Sending real-time data via TCP

Processing.py
1

Dear fellow Python users,

I would need your help in figuring out how to send a continuous stream of data from an external environment to Processing via TCP. TCP is important because I need the data to remain intact and to arrive in the same order in which it was sent.

The idea is to be able to do some intensive computation task in a separate environment and to send the output (a list of coordinates, indices, …) in real time to Processing for rendering.

For this I am relying on Python socket and pickle modules. I can’t say if they are the most appropriate tools for this specific task so if you can think of a better alternative please let me know.

Tryout

Below a simple test script where I try to send the location of a point around a circle from a Jupyter Notebook. The location changes at each step of the loop and the corresponding coordinates are sent as a serialized tuple.

import socket
import pickle
import math

den = 20 
rad = 100
theta = math.tau / den

HOST = "127.0.0.1" 
PORT = 12000

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
    sock.connect((HOST, PORT)) #connect to server
        
    for i in range(1000):
        i = i%den
        x = math.cos(i*theta) * rad
        y = math.sin(i*theta) * rad
        data = pickle.dumps((x, y), protocol=2) #protocol needs to be between 0 and 2 for Python 2
        sock.send(data) #send data

On the Processing side , I have:

import socket
import pickle

HOST = "127.0.0.1"   
PORT = 12000

def setup():
    size(800, 800)
    
    global s

    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.bind((HOST, PORT))
    s.listen(2)

    
def draw():
    background(255)
    translate(width>>1, height>>1)

    client, address = s.accept()
    
    if client:
        data = client.recv(4096)
        print pickle.loads(data)

In order to get the transmission working the Processsing sketch must be launched prior to the Jupyter script.

Problem

I am only receiving the first coordinate of the loop (100.0, 0.0) and nothing more. I also notice that client, address = s.accept() breaks the draw() function. It seems then to be stopped and only runs once when a TCP message is sent.

I can’t say for sure what is the reason of this behavior but my bet is that I am not correctly setting things on the server side (Processing sketch). I suspect the data to be sent entirely but not processed adequately on reception due to some mistakes that I am probably doing with the socket module.

Would you mind helping me fixing this up ?

1 Like
2

I haven’t looked at socket, but have you already tried using the Processing Net library (import processing.net.*;) ? I believe that it uses TCP (as per https://forum.processing.org/two/discussion/12593/is-the-processing-net-library-use-tcp-or-udp )

1 Like
3

I haven’t ! Thank you for the pointer.

The problem is that I don’t see how I can establish a communication between an external Python library (socket) and a Processing library (Net library for example).

I have tried with oscP5 (that also supports TCP) but couldn’t manage to retrieve the message sent by socket from the Jupyter Notebook.

4

The problem here is that accept() is a blocking method. It waits until a new TCP message has been received. Because of that, usually sockets (and other IO handling) is put into a separate thread to not block your main thread, because this one should be free to draw your scene.

And here comes the processing.net.*; namespace into play (already mentioned by @jeremydouglass). This library sets up a separate thread for you and you only have to deal with the received data:

import processing.net.*;

Server s;

void setup() {
  s = new Server(this, 12000);
}

void draw() {
  
  // check if a client sent something
  Client c = s.available();
  if (c != null) {
    String input = c.readString();
  }
}

This is an example in java, but should be very easy to port to python. And of course, you can also read the bytes with the c.readBytes(int) method.

Regarding OSC, this would work of course, but only if you also send OSC from python. You could use the following library to send the OSC data: https://pypi.org/project/python-osc/ . Even the protocol does not define the transport layer, OSC is usually sent over UDP and not TCP. But you can change that in most libraries.

3 Likes
5

Hi Cancik,

Thank you for the explanation and the sample code. Using the net library seems indeed much more appropriate and easier than the socket module (!).

Unfortunately I still can’t manage to solve my problem.

Not only I just get the first tuple but this time it is followed by an index error:

processing.app.SketchException: IndexError: index out of range: -1

I don’t know if that has to do with the way I am de-serializing the data

add_library('net')
import pickle

def setup():
    size(800, 800)
    
    global s
    
    s = Server(this, 12000)
    
def draw():
    background(255)
        
    c = s.available()
    
    if c:
        data = c.readBytes(4096)
        t = pickle.loads(data)
        print t # <-- throws an error

Client side:

num = 20
rad = 100
theta = math.tau / num

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
    sock.connect((HOST, PORT)) #connect to server
        
    for i in range(1000):
        i = i%den
        x = math.cos(i*theta) * rad
        y = math.sin(i*theta) * rad
        data = pickle.dumps((x, y), protocol=2) 
        sock.sendall(data) #send data

edit: This problem might be related to the pickle module so I just posted a similar question on SO.

6

Upon further testing it seems the TCP network communication only works between 2 “real” Python environments (CPython 3.7 on both client and sever side). I found that the same server side script failed to return the whole message in Processing.py while it succeeded in Thonny IDE.

This leads me think that it is a Jython issue.

I would like to verify the validity of this assumption by running a more comprehensive test but the following throws a TypeError error (line 27) that I am unable to bypass :

– server side (Processing Jython 2.7) –

import pickle
import socket
import struct

HEADER_SIZE = 4
HOST = "127.0.0.1"
PORT = 12000


def receive(nb_bytes, conn):
    # Ensure that exactly the desired amount of bytes is received
    received = bytearray()
    while len(received) < nb_bytes:
        received += conn.recv(nb_bytes - len(received))

    return received

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((HOST, PORT))
s.listen(1)
connection, address = s.accept()

while True:

    # receive header
    header = receive(HEADER_SIZE, connection)
    data_size = struct.unpack(">i", header)[0] # <-- TypeError

    # receive data
    data = receive(data_size, connection)
    print(pickle.loads(data))

processing.app.SketchException: TypeError: unpack(): 2nd arg can’t be coerced to org.python.core.PyArray

How could I possibly get around this error ?

(I must add that this code works in Python 2.7, only Processing.py (Jython 2.7) returns an error)

7

Dunno whether it would still be correct, but it’s worth a try getting a string from the bytearray via its method decode() I guess: :flushed:
header = receive(HEADER_SIZE, connection).decode()

2 Likes
8

Hmm, it solves the error but now I have a new one:

TypeError: StringIO(): 1st arg can’t be coerced to org.python.core.PyArray

9

What is your data rate? I would think you can use UDP… I have use it to stream video and worked fine. This is as an alternative to working with sockets.

Kf

10

Hi @kfrajer, for video I would use UDP but for a mesh or any kind of data structure that defines a shape TCP seems to be necessary since the data has to remain intact and in the correct order.

11

Honestly, this thing is slowly driving me crazy. I have spent the last 3 days trying to figure out a way to send a goddamn serialized tuple to a Python sketch… to no avail.

It is beyond me why the server only returns a single value instead of the whole stream !

I wont get here through all the painful attempts I have made but I can give just a glimpse of the kind of issues I’m currently facing.

Consider the following jython code where I somehow managed to create a minimal (not to say poorly written) server using Java ServerSocket and Scanner modules:

from java.net import ServerSocket
from java.util import Scanner 
import pickle
        
HOST = "127.0.0.1"
PORT = 12000

s = ServerSocket(PORT).accept()
scanner = Scanner(s.getInputStream())

while True:

    if scanner.hasNext():
        data = scanner.next()
        print data
    else:
        break

When running the client (cf: previous post) I should normally receive on the server side a stream of tuples like this:

(100.0, 0.0)
(95.10565162951535, 30.901699437494738)
(80.90169943749474, 58.778525229247315)
(58.778525229247315, 80.90169943749474)
(30.901699437494745, 95.10565162951535)
(6.123233995736766e-15, 100.0)
(-30.901699437494734, 95.10565162951536)
(-58.7785252292473, 80.90169943749474)
(-80.90169943749473, 58.77852522924732)
(-95.10565162951535, 30.901699437494752)
...

However in my case, the code above in Processing (server side) returns this:

(F100.0
F0.0
tp0
.(F95.10565162951535
F30.901699437494738
tp0
.(F80.90169943749474
F58.778525229247315
tp0
.(F58.778525229247315
F80.90169943749474
tp0
.(F30.901699437494745
F95.10565162951535
tp0
.(F6.123233995736766e-15
F100.0
tp0
.(F-30.901699437494734
F95.10565162951536
tp0
.(F-58.7785252292473
F80.90169943749474
tp0
.(F-80.90169943749473
F58.77852522924732
tp0
.(F-95.10565162951535
F30.901699437494752
tp0
.
...

Python type() methods tells me it is unicode but how can it be that I receive unicode when that data has been serialized i.e converted to a byte stream before sending ?

Even stranger, when trying to “unpickle” that data, it disappears instantly. Nothing prints on the console.

data = scanner.next()
print data # <- print strange unusable unicode
print pickle.loads(data) # <- nothing !

I later found that switching the pickling protocol from 0 to 1 somehow allowed to retrieve a tiny portion of that data: the very first tuple and this one only (like in my original example and in the “Net” library example)

#CLIENT SIDE
data = pickle.dumps((x, y), protocol=1) 

#SERVER SIDE
data = scanner.next()
x, y = pickle.loads(data.encode('utf-8')) 
print x, y # <- 100.0 , 0.0 then nothing !

I couldn’t manage to read the data when it was serialized with protocol 3 but I am pretty sure even if I did I would only get either the first tuple or nothing at all.

So the million-dollar question remains, how can i retrieve not only the first but the whole stream of tuples ?

help

12

REFERENCE: https://stackoverflow.com/questions/18743962/python-send-udp-packet

This is my POC using python to send data and Processing listening for any packages. Note that Processing’s draw loop is blocked so you need to send data to see some action. To run this demo, run first your Processing sketch and then run the python script. I have provided some instructions to set up the virtual environment more for my record.

Processing sketch will only store 10 incoming messages in a StringArray object and it gets emptied when it reaches the limit.

Change the sleep time in python to increase the rate

Kf

import java.net.*;
import java.io.*;

import java.nio.charset.StandardCharsets;

final int TEXT_SIZE=16;

// Port we are receiving.
int port = 9100; 
DatagramSocket ds; 
// A byte array to read into (max size of 65536, could be smaller)

StringList lines;

void setup() {
  size(400, 600);
  lines = new StringList();
  textSize(TEXT_SIZE);
  try {
    ds = new DatagramSocket(port);
  } 
  catch (SocketException e) {
    e.printStackTrace();
  }
}

void draw() {
  background(0);
  checkForData();
  printData();
}

void checkForData() {
  byte[] buffer = new byte[65536];
  DatagramPacket p = new DatagramPacket(buffer, buffer.length); 
  try {
    ds.receive(p);
  } 
  catch (IOException e) {
    e.printStackTrace();
  } 
  println("Received datagram with " + p.getLength() + " bytes." );
  
  byte[] data = new byte[p.getLength()];
  System.arraycopy(p.getData(), p.getOffset(), data, 0, p.getLength());
  String mystr = new String(data, StandardCharsets.UTF_8);
  
  if (lines.size()>10) { //Arbitrary value
    lines.clear();
  }
  lines.append(mystr);
}

void printData() {
  int size=lines.size();
  for (int line=0; line<size; line++) {
    text(lines.get(line), 50, line*(2*TEXT_SIZE)+50);
  }
}


# ==================================================
# PYTHON3 INSTRUCTIONS:
#
# python3 -m pip install --upgrade pip
# pip install virtualenv
# which python3
# virtualenv -v venv --python=/usr/bin/python3
# source ./venv/bin/activate
# python -V
# echo "NOW let's send some data"
# python tx-session.py 

import socket
import time

UDP_IP = "127.0.0.1"
UDP_PORT = 9100

def sender(args):
    message = args
    print(f"UDP target IP: {UDP_IP}")
    print(f"UDP target port:  {UDP_PORT}")
    print(f"message: {message}")

    sock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
    sock.sendto(bytes(message, "utf-8"), (UDP_IP, UDP_PORT))

def create_message(base_msg, number_of_stars):
    """ 
    Appends a defined number of stars to the msg parameter and the actual star number
    For example: 'Hello world***[3]'
    """
    msg = base_msg
    for x in range(ctr):
        msg = msg + "*"
    return f"{msg}[{ctr}]" 
        

if __name__ == "__main__":
    ctr = 0
    while 1:
        msg = create_message("Hellow World", ctr)
        sender(msg)
        time.sleep(0.5)  # 500msec
        ctr = ctr + 1
        if ctr > 25:  ## Arbitrary value
            ctr = 0
13

@kfrajer I think this answer belongs to another thread. Again, this is about TCP communication for the sending of list/tuples where order and integrity is preserved. Not about UDP for sending text where delivery can’t be guaranteed. Putting a link to this thread in other UDP related threads is also misleading.

14

Do you have evidence of package drop in your setup or out-of-order? You are reading a generic UDP description and assuming UDP is not to be trusted. If you are running in the same network, this would be reliable comms assuming your data throughput is not high.

I tested some code working with TCP. Run the server first (in Processing Python) and then send the data from the python script.

Kf

Python Script

# ==================================================
# PYTHON3 INSTRUCTIONS:
#
# python3 -m pip install --upgrade pip
# pip install virtualenv
# which python3
# virtualenv -v venv --python=/usr/bin/python3
# source ./venv/bin/activate
# python -V
# python tcp-tx-session.py
#
# REFERENCE: https://stackoverflow.com/questions/34653875/python-how-to-send-data-over-tcp

import socket
import time
import sys

HOST, PORT = "127.0.0.1", 9100

def sender(sock_handler, message):
    print(f"message: {message}")    
    sock_handler.sendall(bytes(message, "utf-8"))
    ##This next closes the session
    #received = str(sock_handler.recv(1024), "utf-8")
    
def create_message(base_msg, number_of_stars):
    """ 
    Appends a defined number of stars to the msg parameter and the actual star number
    For example: 'Hello world***[3]'
    """
    msg = base_msg
    for x in range(ctr):
        msg = msg + "*"
    return f"{msg}[{ctr}]" 
        

if __name__ == "__main__":
    ctr = 0    
    print(f"target IP: {HOST}")
    print(f"target port:  {PORT}")
    sock = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
    sock.connect((HOST, PORT))
    while 1:
        msg = create_message("Hellow World", ctr)
        sender(sock, msg)
        time.sleep(0.5)  # 500msec
        ctr = ctr + 1
        if ctr > 25:
            ctr = 0

In Processing.py

import socket  

host = '127.0.0.1'
port = 9100    # Arbitrary non-privileged port

def setup():
    size(800, 800)
    
    global s    
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind((host, port))

    print(host , port)
    s.listen(1)
    print('Server listening....')

    
def draw():
    background(255)
    translate(width>>1, height>>1)
    conn, addr = s.accept()
    while True:
        try:
            data = conn.recv(1024)
            if not data: break
            print("Client Says: " + bytearray(data, 'utf-8').decode())
            #conn.sendall("Server Says:hi")

        except socket.error:
            print( "Error Occured.")
            conn.close()
            break
15

I don’t have one specific example sketch to share but I did experience data loss in the past when transmitting vertices of a large mesh (several thousands) via UDP from one environment to another *(including Processing) on my computer. That plus the fact that it is generally accepted that TCP is more appropriate for this kind of scenario.

I appreciate all the input but again the question is about sending tuples (not text) and finding a way to retrieve them all. Your example, like mines and like the one suggested by cancik, only returns 1 tuple:

– client side –

import socket
import pickle
import math

HOST = "127.0.0.1"
PORT = 12000

den = 20
rad = 100
theta = math.tau / den

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock:
    sock.connect((HOST, PORT)) #connect to server

    for step in range(10):
        i = step%den
        x = math.cos(i*theta) * rad
        y = math.sin(i*theta) * rad
        data = pickle.dumps((x, y), protocol=2)
        sock.sendall(data)

– server side –

import socket  
import pickle

host = '127.0.0.1'
port = 12000  

def setup():
    size(800, 800)
    
    global s    
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind((host, port))

    print(host , port)
    s.listen(1)
    print('Server listening....')

    
def draw():
    background(255)
    translate(width>>1, height>>1)
    conn, addr = s.accept()
    while True:
        try:
            data = conn.recv(1024)
            if not data: break
            print pickle.loads(data)
            #print("Client Says: " + bytearray(data, 'utf-8').decode())
            #conn.sendall("Server Says:hi")

        except socket.error:
            print( "Error Occured.")
            conn.close()
            break

returns:

('127.0.0.1', 12000)
Server listening....
(100.0, 0.0) # <- first tuple
             # <- rest is missing
16

It seems you need to retrieve all your bytes first in the server side before calling pickle.loads(). Check this approach.

Kf

1 Like