Send a multidimensional numpy array over a socket

Question

Good day,

I\'ve searched for this but haven\'t come up with any responses. I wish to send a multi dimensional numpy array over a socket. Hence, I decided to convert it t

Answer 1

Indeed, .tostring only returns the raw data. This means you additionally need to send the shape and dtype of the array if these aren't known on the other side.

Maybe it's easier to serialize the array using Pickle:

import numpy as np
from cPickle import dumps, loads

x = np.array([[1, 2],[3, 4]], np.uint8)
print loads(dumps(x))
# [[1 2]
#  [3 4]]

Though for very small arrays the size overhead could be significant:

print len(x.tostring()), len(dumps(x))
# 4 171

For more on using Pickle, see here.

Answer 2

Try this example:-

import socket
import numpy as np
from cStringIO import StringIO

class numpysocket():
    def __init__(self):
        pass

    @staticmethod
    def startServer():
        port=7555
        server_socket=socket.socket() 
        server_socket.bind(('',port))
        server_socket.listen(1)
        print 'waiting for a connection...'
        client_connection,client_address=server_socket.accept()
        print 'connected to ',client_address[0]
        ultimate_buffer=''
        while True:
            receiving_buffer = client_connection.recv(1024)
            if not receiving_buffer: break
            ultimate_buffer+= receiving_buffer
            print '-',
        final_image=np.load(StringIO(ultimate_buffer))['frame']
        client_connection.close()
        server_socket.close()
        print '\nframe received'
        return final_image

    @staticmethod
    def startClient(server_address,image):
        if not isinstance(image,np.ndarray):
            print 'not a valid numpy image'
            return
        client_socket=socket.socket()
        port=7555
        try:
            client_socket.connect((server_address, port))
            print 'Connected to %s on port %s' % (server_address, port)
        except socket.error,e:
            print 'Connection to %s on port %s failed: %s' % (server_address, port, e)
            return
        f = StringIO()
        np.savez_compressed(f,frame=image)
        f.seek(0)
        out = f.read()
        client_socket.sendall(out)
        client_socket.shutdown(1)
        client_socket.close()
        print 'image sent'
        pass

In this model client sends multidimensional ndarray to server. There are two functions startServer() and startClient(). startServer takes no arguments but startClient needs server address as well as the ndarray as arguments. First start Server and then start client. Server starts reading from buffer only after receiving the shutdown message from client.

Answer 3

This is a slightly improvised answer for ajsp answer using XML-RPC.

On the server-side when you convert the data, convert the numpy data to a string using the '.tostring()' method. This encodes the numpy ndarray as bytes string. On the client-side when you receive the data decode it using '.fromstring()' method. I wrote two simple functions for this. Hope this is helpful.

1. ndarray2str -- Converts numpy ndarray to bytes string.
2. str2ndarray -- Converts binary str back to numpy ndarray.

    def ndarray2str(a):
        # Convert the numpy array to string 
        a = a.tostring()

        return a

On the receiver side, the data is received as a 'xmlrpc.client.Binary' object. You need to access the data using '.data'.

    def str2ndarray(a):
        # Specify your data type, mine is numpy float64 type, so I am specifying it as np.float64
        a = np.fromstring(a.data, dtype=np.float64)
        a = np.reshape(a, new_shape)

        return a

Note: Only problem with this approach is that XML-RPC is very slow while sending large numpy arrays. It took me around 4 secs to send and receive a (10, 500, 500, 3) size numpy array for me.

I am using python 3.7.4.