Line buffered serial input

Question

I have a serial device that I\'m trying to read input from. I sent it a string \"ID\\r\", and it returns \"ID XX\\r\" (where \\r is an ASCII carriage return, hex 0x0d).

Answer 1

Wasted a few hours on this today. It turned out that io.BufferedReader reads until it has filled its buffer and then passes the buffer to io.TextIOWrapper. The default buffer size is 8192, so depending on your device this might take a while.

The correct example code should be:

# buffer size is 1 byte, so directly passed to TextIOWrapper
sio = io.TextIOWrapper(io.BufferedRWPair(ser, ser, 1), encoding='ascii')
print sio.readline()[:-1]

Answer 2

Thanks for the code Keith, but I wanted to keep this code somewhat portable, so I'd like to stick with the default "serial" package.

Plus, since I'm still learning Python, I wanted to try to learn how to use the TextIOWrapper in the way it was intended.

I gave up trying to make serial.readline() work, so for now I'll just use a simple "readLine" function to read a character at a time and look for a carriage return terminator. Though if I run into more serial quirkyness, I may revisit using your code.

Thanks!

def readLine(ser):
    str = ""
    while 1:
        ch = ser.read()
        if(ch == '\r' or ch == ''):  
            break
        str += ch

    #"print "str = " + str

    return str

Answer 3

Caveat: I am using Python 3.4 on a Mac so your mileage may vary, though I believe with TextIOWrapper backported to Python 2.7, the situation in Python 2.7 (and other OSs) will essentially be the same as what I describe below.

The main issue is that io.TextIOWrapper itself uses a buffering mechanism, controlled by the undocumented _CHUNK_SIZE attribute. This is pretty unpleasant. So you have two choices:

1. Use a timeout as you tried out. This is what is hinted on in the documentation of readline on the pyserial documentation page. However, if you use a large value (as you did), when there is not enough data to fill the buffer of TextIOWrapper, your code will block until the timeout is reached. This is what you are experiencing essentially (I did not go after why you have to wait double the timeout value, but I think that this could be sorted out by looking at the implementation of TextIOWrapper and ultimately is irrelevant to your question).

2. The second choice is to change _CHUNK_SIZE to 1. In your case, simply add the line

   sio._CHUNK_SIZE = 1
   

   to your code right after you initialized sio. This has the perhaps unpleasant effect that the buffering within TextIOWrapper itself will be turned off (this is used for the incremental decoding of the input). If performance is not an issue, this is the simplest solution. If performance is an issue, you can set a low value of timeout, not toucing _CHUNK_SIZE. However, in this case be prepared to get an empty string from readline() (if the device sends you an empty line, that will come through as '\n', so it can be distinguished from the empty string that you will get when a read runs out of the alloted time).

There is another problem with your code: When sio will be removed, the close method of ser will be called twice, which will result in an exception when your program will be about to finish (at least this is what happens if I try your code on my computer). You should create (it seems) to instances of serial.Serial and pass those to BufferedRWPair.

I have also created a wrapper class based on TextIOWrapper, which I could also post, if there is interest, just I did not want to litter response with some extra code which, strictly speaking, is not needed.

PS: In the meanwhile, I have experimented with the code on Ubuntu. While on my Mac, I did not see a need for setting the buffer size of io.BufferedRWPair to 1, on Ubuntu I had to do this, too, in addition to setting _CHUNK_SIZE to 1.

Answer 4

This would be hard to debug without actually being there to see it. But see if you can use my tty module.

http://code.google.com/p/pycopia/source/browse/trunk/aid/pycopia/tty.py

Try the SerialPort object in there. I've successfully used this to interact with serial instruments, where "that other serial module" had lots of problems similar to what you describe. This one can also tell you if you have data in the FIFO.

Let me now how it goes.