Pyserial buffer fills faster than I can read

Question

I am reading data from a microcontroller via serial, at a baudrate of 921600. I'm reading a large amount of ASCII csv data, and since it comes in so fast, the buffer get's filled and all the rest of the data gets lost before I can read it. I know I could manually edit the pyserial source code for serialwin32 to increase the buffer size, but I was wondering if there is another way around it?

I can only estimate the amount of data I will receive, but it is somewhere around 200kB of data.

Answer 1

There's a "Receive Buffer" slider that's accessible from the com port's Properties Page in Device Manager. It is found by following the Advanced button on the "Port Settings" tab.

More info:

http://support.microsoft.com/kb/131016 under heading Receive Buffer

http://tldp.org/HOWTO/Serial-HOWTO-4.html under heading Interrupts

Try knocking it down a notch or two.

Answer 2

Have you considered reading from the serial interface in a separate thread that is running prior to sending the command to uC to send the data?

This would remove some of the delay after the write command and starting the read. There are other SO users who have had success with this method, granted they weren't having buffer overruns.

If this isn't clear let me know and I can throw something together to show this.

EDIT

Thinking about it a bit more, if you're trying to read from the buffer and write it out to the file system even the standalone thread might not save you. To minimize the processing time you might consider reading say 100 bytes at a time serial.Read(size=100) and pushing that data into a Queue to process it all after the transfer has completed

Pseudo Code Example

def thread_main_loop(myserialobj, data_queue):
    data_queue.put_no_wait(myserialobj.Read(size=100))

def process_queue_when_done(data_queue):
    while(1):
        if len(data_queue) > 0:
            poped_data = data_queue.get_no_wait()
            # Process the data as needed
        else:
            break;

Answer 3

You do not need to manually change pyserial code.

If you run your code on Windows platform, you simply need to add a line in your code

ser.set_buffer_size(rx_size = 12800, tx_size = 12800)

Where 12800 is an arbitrary number I chose. You can make receiving(rx) and transmitting(tx) buffer as big as 2147483647a

---

See also:

https://docs.python.org/3/library/ctypes.html

https://msdn.microsoft.com/en-us/library/system.io.ports.serialport.readbuffersize(v=vs.110).aspx

You might be able to setup the serial port from the DLL // Setup serial

 mySerialPort.BaudRate = 9600;
 mySerialPort.PortName = comPort;
 mySerialPort.Parity = Parity.None;
 mySerialPort.StopBits = StopBits.One;
 mySerialPort.DataBits = 8;
 mySerialPort.Handshake = Handshake.None;
 mySerialPort.RtsEnable = true;
 mySerialPort.ReadBufferSize = 32768;

Property Value Type: System.Int32 The buffer size, in bytes. The default value is 4096; the maximum value is that of a positive int, or 2147483647

And then open and use it in Python

Answer 4

I am somewhat surprised that nobody has yet mentioned the correct solution to such problems (when available), which is effective flow control through either software (XON/XOFF) or hardware flow control between the microcontroller and its sink. The issue is well described by this web article.

It may be that the source device doesn't honour such protocols, in which case you are stuck with a series of solutions that delegate the problem upwards to where more resources are available (move it from the UART buffer to the driver and upwards towards your application code). If you are losing data, it would certainly seem sensible to try and implement a lower data rate if that's a possibility.