A universal ambilight


#1

Hi everyone!

This project is still a work in progress but I would like to share with you the progress I have made so far, the direction I want to follow and gather some feedback or advice about it.

I wanted to build an ambilight system for my TV but the problem I ran into was that I was and still am using only an ethernet cable to get my video feed. So I couldn’t use those solutions you can find online using the HDMI, USB or VGA feed.

For those who don’t know what an ambilight system is, here is a link to a corresponding wikipedia page:

To overcome that problem, I got an idea to obtain that feed through a camera and use that as my source to detect the colors of my LEDs.
For this, I will connect a camera (RPI04V2) to a Raspberry PI 3 B+ running a processing sketch. That sketch will have the role to read the feed form the camera and figure out the color of the LEDs for the current image. Then those information will be sent by bluetooth to an arduino connected to the LED.

Now, I am not completely sure it will work because of 3 main factors :

  • the delay that will be introduced between the moment the image is displayed and the moment the LEDs get the information. I have done some tests already and I think it will be ok for normal scenes but maybe a bit slow for action scenes.
  • The quality of the feed : because it is a camera, some darker and brighter zones appear on the image and it can affect the color of the LED. I have already thought about some ways how to tackle this issue and ran some positive tests but it still needs a bit more work
  • The camera parameters : I’m afraid that when the image becomes brighter the camera will adapt its sensitivity to show a darker picture and the other way around for a dark image. I haven’t worked on that one yet.

You can find the current version in this github repository:

What is already done:

  • Display the list of the available cameras and select the one of interest
  • Detect the TV screen position by displaying a unique color on the TV screen
  • Straighten the image of the TV

What needs to be done:

  • Apply the color correction to the image
  • Make it work on the raspberry PI
  • Get the LEDs color
  • Send the information to the arduino via bluetooth
  • Build the LEDs panel
  • Write the arduino code

If you have any idea on how to improve it or if you want me to explain in a bit more in detail how some algorithms work, feel free to post a new message :slight_smile:


Help! processing interface with Json of Philips hue
Multiple Values from processing to arduino through Serial Port
Send 3 variable to arduino?
Annotation processing got disabled, since it requires a 1.6 compliant JVM
Send 3 variable to arduino?
Advice to help Ambilight calculate Arduino
Trouble with bluetooth BLE connection between Pi3 and Blueduino
#2

Hi everyone,

I finally got some time and courage to continue this project.

I started developing the concept on my windows computer using the video library provided by the processing foundation. Sadly, I figure out the hard way that I can’t use that library with a Pi board so I used the glvideo library by gohai instead. I faced many problems to transcript the previous code to the new one and I ended up with big changes in the structure. All that to say that I have created a new repository on github to get a fresh start (the old one is still available):

Now let’s get to the fun part, some screenshots of how it works.

For testing reasons, the tv is replaced by a piece of paper sticked to the wall:

First, you start with a menu giving you all the shortkeys that can be used:

Then you need to calibrate the tool. It consists of 2 steps:

  1. Display a unique color (different enough than the background) on your tv
  2. Change the threshold value until only the screen is selected
    The tool assists you by displaying a red overlay on top of the pic, showing the selected area.
    You can see the difference between a bad threshold and a good one on the 2 followings screenshots:

Once you found a good threshold, the tool bakes some calculations needed at every frame for better performances.
The first task is to find the corners of that area. The second debug mode can show that this operation has been successful:

The second task is to reshape that tetragon to make it a perfect rectangle. In order to save some performance, the accuracy of the process can be changed and only the outer edges (that we are really interested in) is straightened.
To better show the result the background pic is changed:

After, it is just a matter of figuring out the proper color to display. Here are the result with the previous background and the result with a new one to better see the effect:

I’m quite happy with the progress but there is still plenty of work to do:

  1. Now that I have the result that I wanted I need to figure out a way to send it via bluetooth to an arduino board
  2. Then I would need to build the LEDs structure and make it works for real!
  3. Keep improving:
    • Change the way the colors of the LEDs are computed. Right now it is just a simple resize function but I’m not sure it gives the best result
    • Compensate for brightness difference over the screen. I already have a working exemple but I think it can be improved even more.
    • Provide a way to tweak the colors of the LEDs without changing the output of the sketch.

If you have any question or advice please go ahead, and feel free to use it, modify it, improve it… :smiley:

I also want to thank @gohai, @msurguy, @GoToLoop and @neilcsmith who help me find my way around using a camera on a Pi.


#3

Cool project - thanks for sharing the detailed write up @jb4x!


#4

Hi everyone,

Just a quick update, the project is not dead.

I finally found some times to complete the frame:

I also had some problem pairing the blueduino rev3 to my raspberry :crazy_face:

It should be all good now.

The only thing left is the code to send the data to the blueduino and the bueduino code.


#5

Alright guys,

Today’s a really exciting day :smiley::smiley::star_struck:

I finally got it to work!

Here’s a video of how it performed:

There are few points to improve though:

  • The lights are flickering because of the camera feed
  • The colors are not always completely right
  • It is not wireless (I used a usb cable) - Not sure if bluetooth will be fast enough though
  • There is a little delay between the LEDs and the TV

#6

Very cool demo!

In addition to flicker reduction, you might also try out a smoothing / crossfade operation with an adjustable duration – although if I understand right the original aesthetic of the ambilight is to be instantaneous.


#7

Thank you @jeremydouglass for the feedback, really appreciated! :slight_smile:

The problem I have with flicker reduction is that I think it is hardware related. I had huge trouble making it works and currently, for example, I can’t use the camera at more than 20 fps when I should, in theory, be able to go up to 60 fps…

Another issue I have with the camera is the auto-brightness adjustment: I have no control over it what so ever and it is not a simple light sensor outside the camera that I can tape :confused:

And I don’t think I can do something about that flickering effect by code.

Now for the smoothing/crossfade operation, as you said, the goal would be to be as instantaneous as possible to make it feel like it is an extension of the screen. The smoothing would break that effect.

Now if push completely to the limit, the idea can be to not want a perfect match with the screen but to have more like an ambiant lighting around the screen. It would not use just the pixels on the edge but the whole screen, sense the ambiance of the scene, and display something accordingly.

If people have ideas linked to that, I would be happy to read them :slight_smile:


#8

I just linked this recent discussion of the Adafruit Adalight, which seems related to your project –