How to send and receive webcam stream using tcp sockets in Python?

Question

I am trying to recreate this project. What I have is a server (my computer), and a client (my raspberry pi). What I am doing differently than the original project is that I am t

Answer 1

You can't just display every received buffer of 1-1024 bytes as an image; you have to concatenate them up and only display an image when your buffer is complete.

If you know, out of band, that your images are going to be a fixed number of bytes, you can do something like this:

IMAGE_SIZE = 320*240*3

def handle(self):
    stream_bytes = b''

    try:
        stream_bytes += self.rfile.read(1024)
        while len(stream_bytes) >= IMAGE_SIZE:
            image = np.frombuffer(stream_bytes[:IMAGE_SIZE], dtype="B")
            stream_bytes = stream_bytes[IMAGE_SIZE:]
            print(image.shape)
            cv2.imshow('F', image)
            cv2.waitKey(0)
    finally:
        cv2.destroyAllWindows()
        sys.exit()

If you don't know that, you have to add some kind of framing protocol, like sending the frame size as a uint32 before each frame, so the server can know how many bytes to received for each frame.

---

Next, if you're just sending the raw bytes, without any dtype or shape or order information, you need to embed the dtype and shape information into the server. If you know it's supposed to be, say, bytes in C order in a particular shape, you can do that manually:

image = np.frombuffer(stream_bytes, dtype="B").reshape(320, 240, 3)

… but if not, you have to send that information as part of your framing protocol as well.

Alternatively, you could send a pickle.dumps of the buffer and pickle.loads it on the other side, or np.save to a BytesIO and np.load the result. Either way, that includes the dtype, shape, order, and stride information as well as the raw bytes, so you don't have to worry about it.

---

The next problem is that you're exiting as soon as you display one image. Is that really what you want? If not… just don't do that.

---

But that just raises another problem. Do you really want to block the whole server with that cv.waitKey? Your client is capturing images and sending them as fast as it can; surely you either want to make the server display them as soon as they arrive, or change the design so the client only sends frames on demand. Otherwise, you're just going to get a bunch of near-identical frames, then a many-seconds-long gap while the client is blocked waiting for you to drain the buffer, then repeat.