Hi everyone,
I finally got some time and courage to continue this project.
I started developing the concept on my windows computer using the video library provided by the processing foundation. Sadly, I figure out the hard way that I can’t use that library with a Pi board so I used the glvideo library by gohai instead. I faced many problems to transcript the previous code to the new one and I ended up with big changes in the structure. All that to say that I have created a new repository on github to get a fresh start (the old one is still available):
Now let’s get to the fun part, some screenshots of how it works.
For testing reasons, the tv is replaced by a piece of paper sticked to the wall:
First, you start with a menu giving you all the shortkeys that can be used:
Then you need to calibrate the tool. It consists of 2 steps:
- Display a unique color (different enough than the background) on your tv
- Change the threshold value until only the screen is selected
The tool assists you by displaying a red overlay on top of the pic, showing the selected area.
You can see the difference between a bad threshold and a good one on the 2 followings screenshots:
Once you found a good threshold, the tool bakes some calculations needed at every frame for better performances.
The first task is to find the corners of that area. The second debug mode can show that this operation has been successful:
The second task is to reshape that tetragon to make it a perfect rectangle. In order to save some performance, the accuracy of the process can be changed and only the outer edges (that we are really interested in) is straightened.
To better show the result the background pic is changed:
After, it is just a matter of figuring out the proper color to display. Here are the result with the previous background and the result with a new one to better see the effect:
I’m quite happy with the progress but there is still plenty of work to do:
- Now that I have the result that I wanted I need to figure out a way to send it via bluetooth to an arduino board
- Then I would need to build the LEDs structure and make it works for real!
- Keep improving:
- Change the way the colors of the LEDs are computed. Right now it is just a simple resize function but I’m not sure it gives the best result
- Compensate for brightness difference over the screen. I already have a working exemple but I think it can be improved even more.
- Provide a way to tweak the colors of the LEDs without changing the output of the sketch.
- …
If you have any question or advice please go ahead, and feel free to use it, modify it, improve it… 
I also want to thank @gohai, @msurguy, @GoToLoop and @neilcsmith who help me find my way around using a camera on a Pi.