Losing data in received serial string

Question

So part of a larger project needs to receive a long hex character string from a serial port using a raspberry pi. I thought I had it all working but then discovered it was losing a chunk of data in the middle of the string.

def BUTTON_Clicked(self, widget, data= None):

        ser = serial.Serial("/dev/ex_device", 115200, timeout=3)

        RECEIVEDfile = open("RECIEVED.txt", "r+", 0) #unbuffered


        #Commands sent out
        ser.write("*n\r")
        time.sleep(1)
        ser.flush()
        ser.write("*E")
        ser.write("\r")

        #Read back string rx'd
        RECEIVED= ser.read()


        RECEIVED= re.sub(r'[\W_]+', '', RECEIVED) #remove non-alphanumeric characters (caused by noise maybe?)
        RECEIVEDfile.write(re.sub("(.{4})", "\\1\n", RECEIVED, 0, re.DOTALL)) #new line every 4 characters


        RECEIVEDfile.close              
        ser.write("*i\r")
        ser.close

This is the script used to retrieve the data, the baud rate and serial commands are set right and the script is run as "unbuffered" (-u) but yet the full string is not saved. The string is approx 16384 characters long but only approx 9520 characters (it varies) are being saved (can't supply the string for analysis). Anyone know what I'm missing? Cheers for any help you can give me.

Answer 1

Glad my comment helped!

Set timeout to a low number, e.g. 1 second. Then try something like this. It tries to read a large chunk, but times out quickly and doesn't block for a long time. Whatever has been read is put into a list (rx_buf). Then loop forever, as long as you've got pending bytes to read. The real problem is to 'know' when not to expect any more data.

rx_buf = [ser.read(16384)] # Try reading a large chunk of data, blocking for timeout secs.
while True: # Loop to read remaining data, to end of receive buffer.
    pending = ser.inWaiting()
    if pending:
         rx_buf.append(ser.read(pending)) # Append read chunks to the list.
    else:
         break

rx_data = ''.join(rx_buf) # Join the chunks, to get a string of serial data.

The reason I'm putting the chunks in a list is that the join operation is much more efficient than '+=' on strings.

Answer 2

According to this question you need to read the data from the in buffer in chunks (here single byte):

out = ''
# Let's wait one second before reading output (let's give device time to answer).
time.sleep(1)
while ser.inWaiting() > 0:
    out += ser.read(1)

I suspect what is happening in your case is that you're getting an entire 'buffers' full of data, which depending on the state of the buffer may vary.