Sending byte array over processing network (Processing 3)

I’m trying to send an image over a byte array data stream in processing with the code below, but as networking is poorly documented in processing, I think there’s only a way to receive a string and convert it to a byte array, which processing sees as invalid

 PImage fback=Recieve(loadBytes(c.readString()), 640, 480, 3);

the paramaters of the recieve function are as follows:

 PImage Recieve(byte[] data, int w, int h, int ch)

(ch is the color channel, typically 3) Here is my send function:

 s.write(broadcast(video));

Full code for host client: https://pastebin.com/L6Y6FTKp P.S. If you think I’m missing anything, just ask for it.

I think instead of doing

loadBytes(c.readString()

you should do

c.readBytes()

Because reading binary data as a String probably damages the data. If you check the length of the String you get it’s probably shorter than the expected data size.

Here a tiny program to send tiny video over the network:

import processing.net.*;

Server s;
byte bytes[];

void setup() {
  s = new Server(this, 12345);
  bytes = new byte[width*height*3];
}

void draw() {
  float t = frameCount * 0.05;
  background(0);
  ellipse(50 + 50 * cos(t),
    50 + 50 * sin(t), 30, 30);
  loadPixels();
  for(int i=0; i<pixels.length; i++) {
    int c = pixels[i];
    bytes[i*3] = (byte)(c >> 16 & 0xFF);
    bytes[i*3+1] = (byte)(c >> 8 & 0xFF);
    bytes[i*3+2] = (byte)(c & 0xFF);
  }
  s.write(bytes);
}

and the client:

import processing.net.*;

Client c;
byte bytes[];

void setup() {
  c = new Client(this, "127.0.0.1", 12345);
}

void draw() {
  if (c.available() > 0) {
    bytes = c.readBytes();
    if (bytes.length == width*height*3) {
      loadPixels();
      for (int i=0; i<pixels.length; i++) {
        pixels[i] = 
          (255 << 24) + 
          ((int)bytes[i*3] << 16) +
          ((int)bytes[i*3+1] << 8) +
          (int)bytes[i*3+2];
      }      
      updatePixels();
    }
  }
}

Note how I didn’t use the color() function, because doing a few bitwise operations is much faster.

1 Like

(post withdrawn by author, will be automatically deleted in 24 hours unless flagged)

Are the display and the movie the same width and height? if not then it may need some adjusting to make sure there are no out of bound exceptions… In which line you get the error?

the video and the capture are both 640/480 I used to have these lines:

//video.resize(80, 60);
  //image(video, 40, 30);

that were meant to have a smaller window showing yourself, but they were commented out a while back…
the error is typically thrown on this line in the server(nullpointer):

PImage fback=Recieve(c.readBytes(), 640, 480, 3);

and this line on the client(outofbounds):

    outImage.pixels[i] = color(data[i*ch], data[i*ch+1], data[i*ch+2]);

ok, fixed the resolution thing, but now I get a gray screen and these 2 errors:
Client: https://pastebin.com/rmUHrYU5
Server: https://pastebin.com/822Mttsh

For converting byte data to images, you can check how Processing does it here… the beauty of open source.

For the errors for your client and server, you will need to provide your source code. I could guess you are not handling your video data properly but that is as far as I can go. I 'd rather see your whole approach.

Kf

My workplace blocks any port other than 80, 443 and 8080, so I’ll just give you the code to replicate the error:
Server: https:// pastebin .com/Hgybxn1V
Client: https:// pastebin .com/WQDJSwuC
remove the spaces from the links
(Don’t forget to install the libraries that it requires)

I think the biggest problem with your code (and also @hamoid’s code) is that you expect to get each image in one piece. But don’t worry, it can be fixed without much trouble! :slight_smile:

When you send data over TCP like this, it does not arrive grouped in the same way. It is first split into packets (max 64KB) and each packet is sent separately. On the receiving end, you get the bytes as the packets arrive. Sometimes you get only part of the image and the rest arrives later. Sometimes you get end of one image and start of another image together. In theory, you can get two or more images at the same time. The library just gives you bytes which arrived, it does not know how they were grouped when you sent them. That is your job to reconstruct.

You need to have some way of splitting these incoming bytes into individual messages (images). If every image you are sending is the same size, your job is easier, you can keep sending them as you do.

However, on the receiving end, you need to wait until the whole image arrives and then process only that data. You would do it like this:

int imageByteCount = 640 * 480 * 3; // or whatever your image size is
byte[] imageBuffer = new byte[imageByteCount]; // Read buffer, fits exactly one image
Client c; // Client which receives the data

void draw() {

  // Checks if there is at least one whole image in the incoming buffer
  while (c.available() >= imageByteCount) {

    // Loads one full image worth of bytes form the Client
    // The loaded bytes are removed from the Client, the rest stays there
    c.readBytes(imageBuffer);

    // You should process the imageBuffer here, e.g. put the bytes into a PImage
  }
}

The while is important, it makes sure you read all the images which arrived in case there is more of them. If you would read the images slower than they arrived, you would ran out of memory sooner or later. Always make sure you read as much data as you can, but not more! Leave the incomplete images in the Client until the rest of the bytes arrives.

2 Likes

Here’s my altered code, but it throws a nullpointer on the imagebytecount while loop
Here’s the code:

Client: https:// pastebin .com/ve6MQQVM
Server: https:// pastebin .com/nQ9xCmDZ

Oh, my mistake, I read your code again and I didn’t notice you are sending the data as jpg, which will make it a bit more difficult, because each image you send will have a different amount of bytes. I will mention it later.

If I understand it correctly, you have two computers and you want to send video from both of them to the other one.

I see several problems in your code. What I would do is start with a new sketch and try to build it up in several steps. Don’t go to the next step until the previous step works reliably. This way you can learn how each part works and focus on one problem at a time.

1. Making the connection

In the server sketch, you first create a Server and then wait for the client to connect. You would use serverEvent() to get notified when a Client connects. You would save the client in a global variable (like Client c you have there). You can use disconnectEvent() to get notified when the client disconnects and set the reference back to null. In your draw() you write or read bytes only if client c is connected (not null).

In the client sketch, just create Client like you do now. If the server is running, it should connect. If the server is not running, the client will not work and you will need to restart the sketch, or try to create a new client. You can use c.active() to check whether the connection was made or not. If it returns false, you need to create a new client.

2. Sending data of fixed size

Try to send a random int from the client to the server every frame and print them out on the server when they arrive. You would use a similar code as in my previous post, but the size of one message is now 4 bytes (size of int). Try sending ints larger than 255 or less than 0 to check if all four bytes are sent and received correctly.

On the client, you can use

byte[] bytes = java.nio.ByteBuffer.allocate(4).putInt(yourRandomValue).array();
c.write(bytes);

ByteBuffer is like a byte array, but has some useful methods which allow you to put there other data types and get them as a byte array. To optimize, you can create the buffer in setup() and reuse it.

On the server, you would wait until there are at least 4 bytes available, then read them and convert them back to int by using ByteBuffer again:

if (c != null) {
  while (c.available() >= 4) {
    byte[] bytes = c.read(4);
    int i = java.nio.ByteBuffer.wrap(bytes).getInt();
    println("Int arrived:", i);
  }
}

Try sending more ints every frame to check that your code reads all of them.

3. Sending data of variable size

To send messages of variable lengths (like your jpgs), you need to tell the other side how many bytes to expect in each message. Otherwise it won’t be possible to tell where one message ends and the next one begins. You can do this by first sending an int with the number of bytes to expect, then you send the bytes.

[int with size of message A][message A][int with size of message B][message B]...

On the receiving end (server), you wait until there is 4 bytes available (one int), you read the int which tells you the size of the message, then you wait until the whole message arrives, and finally you read it and process it. Then you wait for another int which will tell you the size of the next message. I can help more with this after you make part 2 work.

Give this a try and let us know if you get stuck. Just try to keep it as simple as possible and improve it step by step. You can add video and bidirectional communication once you have this working :wink:

3 Likes

This confuses me… not your fault though, I’m only 16, this is probably way over my head…:confused: