How to distribute sketches with libraries to Openframe?

Processing Development
1

I’m currently working on a Processing extension to the Openframe platform

open source platform for artists, curators and art enthusiasts to share, discover and display digital art.

… basically, a smart digital picture frame running on a Raspberry Pi. I would now like to add support for contributed libraries and would like to get some advice on how to best approach it.

In Openframe, all artwork is selected on a Webapp and provided to the device via a public URL. When the user selects artwork, it gets downloaded to the Raspberry Pi. The Processing extension currently accepts a ZIP file, extracts it on the Pi and uses processing-java to compile and run the sketch. This works well as long as there are no contributed libraries used by the sketch. If a library is missing compiling the sketch would obviously fail.

I can imagine two different solutions:

  1. Find and install required libraries automatically
    Ideally, the user wouldn’t have to do anything. I thought this could work similarly to the contribution manager in Processing by using the list of available libraries. Since the manager doesn’t actually do installations automatically, I’m not sure if this is realistic. Is this even a good idea? Libraries and code are changing over time. So maybe it is a good idea to keep everything together. Generally, is there any way to figure out what libraries are used by the sketch?
  2. Add the libraries to sketch package
    I remember, a long time ago, Processing also supported a folder with libraries within the sketch folder. Is this still possible? I can’t find any info on it anymore. To keep a working code base together, it still seems to be a good idea to me. Why has this feature been dropped? If the feature doesn’t exist anymore, the extension could use these libraries and move them to the Processing sketch folder on the Pi. Would adding a libraries folder to the sketch cause any conflicts in Processing?

Another method would be to provide a precompiled sketch to the frame. I want to add support for this too, but don’t want to make it mandatory for sketches with contributed libraries.

What do you guys think?

Thanks.

1 Like
2

What a wonderful project – and so needed by this community. Thank you for working on it.

From what I know about Contributions Manager, it periodically downloads sources.conf

https://raw.githubusercontent.com/processing/processing-docs/master/contrib_generate/sources.conf

…and then gets the relevant metadata by downloading all the listed text files. So, for example, this entry:

https://github.com/atduskgreg/opencv-processing/releases/download/latest/opencv_processing.txt

However, I don’t think it knows what the include statements are or which library they belong to until it actually installs the jars in the Java path. The import statement pattern is not in the metadata. Download url, weirdly, is ALSO not in the metadata file – by convention it is the same as the metadata url, but with “zip” instead of “txt”, so you can derive the location of every download from just the sources.conf file:

https://github.com/atduskgreg/opencv-processing/releases/download/latest/opencv_processing.zip

So if you wanted to script that, you would need a lookup table, something like:

Unfortunately you would need to periodically update your lookup table if the available libraries and their urls changed in sources.conf upstream. But it would create an automagic effect – assuming that the download, unzip, and copy of a library to somewhere on the Java path doesn’t fail, of course!

Yes, sketches can contain their own libraries. In PDE, if you drag and drop a jar onto a sketch window then it is automatically copied into the /code subfolder. I believe that the sketch /code folder is in the Java path for processing-java automatically when it runs on a sketch–it has worked for me in the past.

However, some libraries can run large. quesycam is 77KB, but if you are doing a lot of work with opencv sketches, and installing it in each individual sketch, that’s 285 MB in each individual sketch. Combine that with the video library – which is 645 MB – and you could get into a situation where a user is adding a 1GB download to their Pi every time they add a single sketch. Probably not what you want, especially if these things are often on wifi.

A third option could be to pre-install / bundle a group of popular supported libraries and let anything else be handled by the /code directory. That could be a snapshot of everything in sources.conf (in Contribution Manager) or listed on the Processing libraries page at a given moment – and if you wanted to be even more comprehensive you could browse other third party libraries listed on Archive for Processing (which I run). Let me know if I can help.

1 Like
3

@jeremydouglass thank you so much for elaborating on this. It’s very comprehensive.

  1. The /code subfolder idea seems to be very easy and useful way. I’m going to do some test. But thanks for the heads-up about large files. You definitely have got a point there.

  2. The idea of a bundle or download of all libraries in sources.conf (in Contribution Manager) seems easier but also like brute force to me. This could result in very large data to be downloaded and stored on the Pi. Most of the libraries are probably never going to be used by the user.

  3. The idea of a lookup table sounds very intriguing. I am mostly wondering how and where the script would be run, what language would be best and where the result is going to be stored. I would also like to make sure the information is returned to the community. Creating this might also be useful for the Processing IDE which could then offer auto-install of libraries as well.
    There are a few Python scripts in this repo which seem to be touched manually regularly to update the data: https://github.com/processing/processing-docs/tree/master/contrib_generate
    Do you think writing another Python script for the lookup table and adding it there would be a good idea? In my mind, it would be run manually by whoever is updating the lists there.

If you’ve got the time and like to help, feel free to work on such a lookup table. Otherwise, I might give it a shot at some stage.

Thanks.

4

Yes, although due to the long tail, 3 video sketches with individual /code folders will take up more space than pre-installing the top 30-50 libraries. I’d say you could probably do something curated pretty easily, and for everything else there is /code. Maybe make the install optional – so processing sketches have no library support by default, but there is a convenience script to add support for n common libraries.

Actually, it just occurred to me that video might already be built-in to the PI Processing install? not sure, but if it is that would make a batch of common libraries even smaller / faster to add.

Maybe I misunderstood – I thought when you pushed a sketch to OpenFrame it came with a bundled install script – or that the frame had a generic install script for a sketch of type “Processing” that it ran on any Processing push.

How does it currently work?

So my assumption was that the install was something like:

  1. download zip
  2. unzip and copy sketch to sketch-place
  3. register with frame

So the addition, after unzipping, would be something like:

  • for file in sketch folder ending with .pde or .java:
    • for entry in line in file beginning with “import”:
      • if entry in map of downloads:
        • if download is not on disk:
          • download / install zip to JAVA_PATH

If you go with the default sketch code folder, another option might be to add a “pool libraries” cleanup script – it loops over sketch folders and copies duplicate libraries to shared location found in JAVA_PATH.

5

How do I decide what is important? I’m not using Processing for programming myself much these days.

If the download size is small it sounds like a good idea. But aren’t some libraries conflicting with each other? Why is the Processing IDE not generally doing that? I am going to download all libraries when I come to it and will have a look at the size.

The extension is the install script that handles sketches of type “Processing”. Every time the user selects artwork to be shown the server pushes a URL (of the sketch) which the frame downloads and passes to the extension to deal with it. From there we can do what you described further down in your post. But it would happen on demand for a single sketch.

My question in what you quoted was referring to the lookup table. I thought it would be a good idea to share and maintain it through the community. This could either happen manually by the people maintaining the other lists (on a local machine manually pushing back the results to the repo) or automatically run by a cron job on some server.

6

Ah, perhaps not:

The Video and Sound libraries need to be downloaded through the Library Manager. Libraries / Processing.org

7

Yes – not many, but a few. For example, Geomerative and SVG EXport both use Batik, so they can’t be included in one sketch at the same time. They can both by installed in the libraries folder, however.

I’m not sure I understood the question. Do you mean, why does PDE not pre-download contributed libraries?

It depends on what you are trying to support with OpenFrame – I don’t understand the platform the way you do. Is it primarily meant to be a digital image frame that runs Processing sketches, or is it a base platform for building interactive installations? Should it support sound?

If a frame – starting with the libraries page Libraries / Processing.org it is pretty easy to cull all the things that wouldn’t make sense, like hardware and IO interface. Kinect cameras, motion detectors, thermal tape printers, leap motion, touch interfaces – none of that is relevant to a sketch running on an HDMI monitor. If there is no mouse interaction then you probably don’t need things like PeasyCam or Picking, which are interactive in concept. Without a cam you probably wouldn’t need a QRcode reader. With no interface you probably don’t need GUI packages like controlp5 or g4p. If the point is on-display output, you don’t need stereoscopic renderers like Camera3D, or data export support for OBJ or PDF. Et cetera. All these things could still be used by an OpenFrame sketch somehow in theory – for example, a controlP5 based dashboard that is non-interactive, or a peasycam gimbal that is scripted – but if base support is for a monitor running a no-interaction sketch without attached peripherals, that limits the scope of base library support a lot.

So if you wanted to preinstall (or provide a convenience installation script for) a base set of commonly useful libraries, it might looks something like this:

  • video: GL Video
    125M ./glvideo

  • graphics: Box2D, Geomerative, grafica, HE_Mesh, mesh, PixelFlow, Shapes3D, UnfoldingMaps
    36M ./hemesh
    15M ./PixelFlow
    8.5M ./geomerative
    4.1M ./Shapes3D
    1.2M ./grafica
    656K ./box2d_processing
    72K ./geoMap
    72K ./mesh

  • sound: Sound, minim, themidibus, Beads
    67M ./sound
    23M ./minim
    620K ./themidibus
    14M ./beads

  • frameworks: gicenreUtils, proscene, RiTa, ToxicLibs
    36M ./toxiclibs
    18M ./proscene
    15M ./RiTa
    5.6M ./gicentreUtils

…that’s about 370 MB – in particular because of using the Raspberry Pi video library, not Video which is 620 MB of mostly mac and windows stuff. You could throw a few dozen other math / timing / util libraries in there for another 5MB or so, or it might be best to keep it simple just have people add those to sketches. If you were going to add local network communication I might suggest oscP5 and ArtNet – although I suspect that wouldn’t be easily configurable to work with the generic distribution mechanism that OpenFrame uses.

This is based on my understanding of Java Processing and the most commonly used libraries on these forums in recent years – Raspberry Pi users on that channel of this forum may have more advice about what won’t work or is a must-have for running sketches on a Pi.

Edit

This lists the above libraries alongside their download URLs, sizes, and corresponding contribution manager install locations. The names/urls could be made into JSON / a js script, perhaps.

8

Thanks a lot for creating that list, @jeremydouglass. That’s great.

First of all, I just tested using the code folder with the AdditiveWaves example of Toxiclibs. I copied the jar into the code folder and deleted Toxiclibs in the library folder. It fails to compile. Are you sure this is still working? I remember, there have been some changes at some point. If this is not working, I think we need to download all of the libraries during install of the extension or download them on-demand with each artwork.

If the code folder still works, I think you are right. A curated list of libraries makes sense.

Yes, why does PDE not pre-download contributed libraries?

I think it is primarily meant to be a digital image frame. But that said I think it shouldn’t be limited to a display. That is what makes this project so intriguing to me. It is extensible, becoming a super-smart image frame with more abilities, e.g. to generate real-time images based on data and user input. User input could be anything like a camera, microphone, GPIO sensors or a touchscreen, etc.

For example look at Framed, one of the commercial counterparts of Openframe, which comes with a microphone, camera and speakers.

Also, Openframe has been used in an exhibition where it had a camera attached to it: Openframe I think this could become one of the main use cases think of a Processing class that could exhibit their results on a single screen running different sketches over time.

How do you understand the platform?

I would add a few more things related to touch, hardware or PeasyCam, for example. On the other hand, I don’t see a reason for OscP5 and ArtNet as I see Openframe as a standalone device, and these libraries are for interfacing with other devices. Or at least it wouldn’t be a common use case.

Are there any statistics about the downloads of libraries? Maybe that could help to make decisions.

What do you think about starting a collaborative list/spreadsheet with all libraries and making decisions one by one and if declining a library noting a reason for it? Or is that going to be too much work?

9

Most libraries are only one jar – but toxiclibs is not one of them, it is something like 13 jars plus other resources. You need to copy the full contents of the toxiclibs/library folder into mysketch/code/ folder. Not the folder itself – the contents, like this:

Mac bash shell script:

#!/usr/bin/env bash
SRCLIB=~/Documents/Processing/libraries/toxiclibs
SRCSKETCH=${SRCLIB}/examples/core/math/AdditiveWaves
SKETCH=~/Desktop/AdditiveWaves

cp -r "${SRCSKETCH}" "${SKETCH}"
mkdir "${SKETCH}/code"
cp -r "${SRCLIB}/library/" "${SKETCH}/code/"

The results from source to destination should look like this:

ProcessingInstallLibraryIntoSketchCodeFolder
ProcessingInstallLibraryIntoSketchCodeFolder1262×648 70.3 KB

When you are done, uninstall toxiclibs from PDE, then run the sketch – it works.

You don’t need the examples, reference, or src etc, but you may need anything in the library folder. Note that for most libraries this is just one jar – it is enough to drag-drop its single jar onto the sketch window or manually copy it into the sketch /code folder.

10

This would argue for a greater emphasis on IO / hardware support – in particular, camera support opencv / boofcv would be essential for classes focusing on reactive sketches – Kinect is also popular. As an extensible platform, this starts to look like the question would be what not to included, and why? So now you are looking at 30-50+ libs again rather than 10-15

I think a way to approach this would be to write an openframe-proclibs install script that works with the Processing-Deps repo:

(Edit: its updater script is in a separate repo, processing-deps-updater/update.rb at master · josephtaylor/processing-deps-updater · GitHub)

That repo was created for maven, but it is a big-bunch-of-jars for processing libraries.

  1. The script installs a default list based on processing-deps
  2. users can customize the script to add / remove libs not in the default list
  3. to add a new lib for openframe, contribute it to processing-deps and then add it to the install script.

I think it sounds like a great idea. I have low bandwidth right now, unfortunately, but can chip away at it occasionally. No reason not to get started now, and perhaps others from the forums will join in.

Despite being time-poor, I think this has a lot of potential for digital arts classrooms and public spaces, and really appreciate what you are doing, so I want to help maintain moment.

I’ve invited you to the spreadsheet above – perhaps create another worksheet on it and populate with the processing_deps directory list if that looks useful?

1 Like
11

Thanks for all of your input @jeremydouglass.

I had a look at my toxiclibs installation and it seems I had a really old version installed. The folder structure now looks like you described.
I’ve tested the code folder and realised that there is currently an issue with the extension. The unzipper was not working properly for large zip files and the extension doesn’t yet wait until it’s all unzipped. Once this is solved I can get back to the convenience installation of libraries.

Thanks again for all of your input and enthusiasm also on Github!

12

I would just like to give a status update about the Openframe extension. The most important parts are implemented, which also means it reliably deals with PDE files zip files and libraries in the code folder (at least on my end). I would like to release it asap. At the moment I’m waiting for two things though.

  • I still need access to the openframe-processing NPM package in order to push to it. I’m already in contact with the owner.
  • There are some changes necessary in the Openframe Pi client. I’ve already worked on this and have done a pull request. The owner of the Openframe project is on low bandwidth though, so I have to wait for him to approve and merge the code and do a new release of the Openframe Pi client.

I still think a curated list with preinstalled libraries might be useful in the future. Though first, it might be good to see how the extension behaves in the wild with people using it.

Since the code folder seems to offer an option to ship libraries, I also think this is not a high priority anymore.I see more important things that should be done on Openframe to make it easier to use and grow a community. I would like to dedicate my time to those things first.

That said, I welcome anyone working on preinstalled libraries and/or doing pull requests.

1 Like
13

In the meantime, we’ve worked a lot on the Openframe environment. There is a new forum, revamped docs and a couple of fixes and improvements which should make a start into Openframe a lot smoother.

Now, I would like to announce this extension. What would be the best way to do it though? Just post it here in the forum? What category would be good?

You still need to install Openframe from the master branch to make this work. But it’s fairly easy and all described in the readme the extensions repository:

In general, Openframe is looking for testers. If you are keen to support the project please get in touch in the forum: https://openframe.discourse.group/

Thanks.

1 Like