Eli Bendersky's website - Serial porthttps://eli.thegreenplace.net/2023-06-30T23:16:27-07:00Co-routines as an alternative to state machines2009-08-29T14:32:54-07:002023-06-30T23:16:27-07:00Eli Benderskytag:eli.thegreenplace.net,2009-08-29:/2009/08/29/co-routines-as-an-alternative-to-state-machines
<div class="section" id="jib">
<p>Observation:</p>
<blockquote>
Co-routines are to state machines what recursion is to stacks</blockquote>
<p>When you have to traverse some sort of a nested data structure (say, a binary tree), one approach is to create a stack that remembers where in the tree you are. Another, much more elegant approach, is to write …</p></div>
<div class="section" id="jib">
<p>Observation:</p>
<blockquote>
Co-routines are to state machines what recursion is to stacks</blockquote>
<p>When you have to traverse some sort of a nested data structure (say, a binary tree), one approach is to create a stack that remembers where in the tree you are. Another, much more elegant approach, is to write the function recursively. A recursive function employs the machine stack used to implicitly implement function calls - you get the benefits of the stack without paying the cost of reduced readability.</p>
<p>In this article I'll try to show, using a simple, yet very realistic example why co-routines do the same to state machines.</p>
<div class="section" id="the-problem-serial-framing">
<h4>The problem - serial framing</h4>
<p>I've written a detailed <a class="reference external" href="https://eli.thegreenplace.net/2009/08/12/framing-in-serial-communications/">article</a> about framing earlier this month. The simple summary is: we have an endless incoming stream of bytes, from which we need to deduce structured data frames. That is, we have to find where a frame starts, where it ends and what is the data it carries. For this purpose we use a special header value, footer value and an escape byte (DLE).</p>
<p>A complete Python implementation is described <a class="reference external" href="https://eli.thegreenplace.net/2009/08/20/frames-and-protocols-for-the-serial-port-in-python/">here</a>, but in this article I will present the solution in a simplified manner, keeping all irrelevant details out.</p>
</div>
<div class="section" id="the-state-machine">
<h4>The state machine</h4>
<p>Given a stream and receiving one byte at a time, here is the state machine that describes the framing process:</p>
<img class="align-center" style="width: 285px;" src="https://eli.thegreenplace.net/images/2009/08/statemachine1_framing.png" />
<p>Only inputs and state transitions are shown. The framing process outputs complete frames when moving from the <tt class="docutils literal"><span class="pre">IN_MSG</span></tt> state to the <tt class="docutils literal"><span class="pre">WAIT_HEADER</span></tt> stage (this happens when a footer is received) <a class="footnote-reference" href="#id4" id="id1">[1]</a></p>
</div>
<div class="section" id="implementing-the-state-machine">
<h4>Implementing the state machine</h4>
<p>Here's an implementation of this state machine in Python. The internal state is kept in an object:</p>
<div class="highlight"><pre><span style="color: #00007f; font-weight: bold">class</span> <span style="color: #00007f">ProtocolWrapper</span>(<span style="color: #00007f">object</span>):
<span style="color: #00007f; font-weight: bold">def</span> <span style="color: #00007f">__init__</span>(<span style="color: #00007f">self</span>,
header=<span style="color: #7f007f">'\x61'</span>,
footer=<span style="color: #7f007f">'\x62'</span>,
dle=<span style="color: #7f007f">'\xAB'</span>,
after_dle_func=<span style="color: #00007f; font-weight: bold">lambda</span> x: x):
<span style="color: #00007f">self</span>.header = header
<span style="color: #00007f">self</span>.footer = footer
<span style="color: #00007f">self</span>.dle = dle
<span style="color: #00007f">self</span>.after_dle_func = after_dle_func
<span style="color: #00007f">self</span>.state = <span style="color: #00007f">self</span>.WAIT_HEADER
<span style="color: #00007f">self</span>.frame = <span style="color: #7f007f">''</span>
<span style="color: #007f00"># internal state</span>
(WAIT_HEADER, IN_MSG, AFTER_DLE) = <span style="color: #00007f">range</span>(<span style="color: #007f7f">3</span>)
<span style="color: #00007f; font-weight: bold">def</span> <span style="color: #00007f">input</span>(<span style="color: #00007f">self</span>, byte):
<span style="color: #7f007f">""" Receive a byte.</span>
<span style="color: #7f007f"> If this byte completes a frame, the</span>
<span style="color: #7f007f"> frame is returned. Otherwise, None</span>
<span style="color: #7f007f"> is returned.</span>
<span style="color: #7f007f"> """</span>
<span style="color: #00007f; font-weight: bold">if</span> <span style="color: #00007f">self</span>.state == <span style="color: #00007f">self</span>.WAIT_HEADER:
<span style="color: #00007f; font-weight: bold">if</span> byte == <span style="color: #00007f">self</span>.header:
<span style="color: #00007f">self</span>.state = <span style="color: #00007f">self</span>.IN_MSG
<span style="color: #00007f">self</span>.frame = <span style="color: #7f007f">''</span>
<span style="color: #00007f; font-weight: bold">return</span> <span style="color: #00007f">None</span>
<span style="color: #00007f; font-weight: bold">elif</span> <span style="color: #00007f">self</span>.state == <span style="color: #00007f">self</span>.IN_MSG:
<span style="color: #00007f; font-weight: bold">if</span> byte == <span style="color: #00007f">self</span>.footer:
<span style="color: #00007f">self</span>.state = <span style="color: #00007f">self</span>.WAIT_HEADER
<span style="color: #00007f; font-weight: bold">return</span> <span style="color: #00007f">self</span>.frame
<span style="color: #00007f; font-weight: bold">elif</span> byte == <span style="color: #00007f">self</span>.dle:
<span style="color: #00007f">self</span>.state = <span style="color: #00007f">self</span>.AFTER_DLE
<span style="color: #00007f; font-weight: bold">else</span>:
<span style="color: #00007f">self</span>.frame += byte
<span style="color: #00007f; font-weight: bold">return</span> <span style="color: #00007f">None</span>
<span style="color: #00007f; font-weight: bold">elif</span> <span style="color: #00007f">self</span>.state == <span style="color: #00007f">self</span>.AFTER_DLE:
<span style="color: #00007f">self</span>.frame += <span style="color: #00007f">self</span>.after_dle_func(byte)
<span style="color: #00007f">self</span>.state = <span style="color: #00007f">self</span>.IN_MSG
<span style="color: #00007f; font-weight: bold">return</span> <span style="color: #00007f">None</span>
<span style="color: #00007f; font-weight: bold">else</span>:
<span style="color: #00007f; font-weight: bold">raise</span> AssertionError()
</pre></div>
<p>Note that the code of the <tt class="docutils literal"><span class="pre">input</span></tt> method closely follows the state diagram. This is how implementations of state machines are - it's generally difficult to understand what's going on in the code without having some sort of a state diagram in front of your eyes. In this case the state machine has just 3 states, but it can be easily 20 for more complex needs. Understanding such a state function with 20 states is impossible without a diagram.</p>
<p>Anyhow, here's some test code that simulates a stream of data with a couple of frames and invalid data in between:</p>
<div class="highlight"><pre>bytes = <span style="color: #7f007f">''</span>.join(<span style="color: #00007f">chr</span>(b) <span style="color: #00007f; font-weight: bold">for</span> b <span style="color: #0000aa">in</span>
[<span style="color: #007f7f">0</span>x70, <span style="color: #007f7f">0</span>x24,
<span style="color: #007f7f">0</span>x61, <span style="color: #007f7f">0</span>x99, <span style="color: #007f7f">0</span>xAF, <span style="color: #007f7f">0</span>xD1, <span style="color: #007f7f">0</span>x62,
<span style="color: #007f7f">0</span>x56, <span style="color: #007f7f">0</span>x62,
<span style="color: #007f7f">0</span>x61, <span style="color: #007f7f">0</span>xAB, <span style="color: #007f7f">0</span>xAB, <span style="color: #007f7f">0</span>x14, <span style="color: #007f7f">0</span>x62,
<span style="color: #007f7f">0</span>x7
])
pw = ProtocolWrapper()
<span style="color: #00007f; font-weight: bold">for</span> byte <span style="color: #0000aa">in</span> bytes:
frame = pw.input(byte)
<span style="color: #00007f; font-weight: bold">if</span> frame:
<span style="color: #00007f; font-weight: bold">print</span> <span style="color: #7f007f">'Got frame:'</span>, frame.encode(<span style="color: #7f007f">'hex'</span>)
</pre></div>
<p>This prints:</p>
<div class="highlight"><pre>Got frame: 99afd1
Got frame: ab14
</pre></div>
</div>
<div class="section" id="co-routines">
<h4>Co-routines</h4>
<p>I don't intend to teach the theory behind co-routines here, and I'll assume at least a basic familiarity with the concept. My goal is to show a real-life, relevant example that demonstrates how co-routines relate to state machines.</p>
<p><a class="reference external" href="http://www.chiark.greenend.org.uk/~sgtatham/coroutines.html">This link</a> is a good tutorial on co-routines (in C, of all languages), and there's of course <a class="reference external" href="http://en.wikipedia.org/wiki/Coroutines">Wikipedia</a> and <a class="reference external" href="http://www.c2.com/cgi/wiki?CoRoutine">C2</a>. But the <strong>absolutely best</strong> tutorial, with focus on Python, is David Beazley's presentation from this year's PyCon: <a class="reference external" href="http://www.dabeaz.com/coroutines/index.html">A curious course on coroutines and concurrency</a>. It is while reading this tutorial that the connection finally 'clicked' in my head. It is most highly recommended <a class="footnote-reference" href="#id5" id="id2">[2]</a>.</p>
<p>If there's one description of co-routines you should remember while reading this article and later, it is that co-routines save the control state of a function between calls. Kinda like recursion - you know exactly where are you going to return after a function call.</p>
<p>When you call a co-routine, it doesn't start all over from the beginning. Rather, it starts from right after where it returned (yielded control) the previous time it was called.</p>
<p>This also explains why co-routines can replace state machines. The <tt class="docutils literal"><span class="pre">input</span></tt> method of <tt class="docutils literal"><span class="pre">ProtocolWrapper</span></tt> is invoked multiple times. Since it's a "normal" function, it begins running from its first line for each invocation. This is why it needs to keep a state machine - to know it's current "place in the world" when the next byte is received. With co-routines this isn't necessary - co-routines start exactly where they stopped the previous time they were called - so no state keeping is required!</p>
</div>
<div class="section" id="using-co-routines-for-framing">
<h4>Using co-routines for framing</h4>
<p>Without further ado, here is the co-routine implementation of the framing problem:</p>
<div class="highlight"><pre>@coroutine
<span style="color: #00007f; font-weight: bold">def</span> <span style="color: #00007f">unwrap_protocol</span>(header=<span style="color: #7f007f">'\x61'</span>,
footer=<span style="color: #7f007f">'\x62'</span>,
dle=<span style="color: #7f007f">'\xAB'</span>,
after_dle_func=<span style="color: #00007f; font-weight: bold">lambda</span> x: x,
target=<span style="color: #00007f">None</span>):
<span style="color: #7f007f">""" Simplified framing (protocol unwrapping)</span>
<span style="color: #7f007f"> co-routine.</span>
<span style="color: #7f007f"> """</span>
<span style="color: #007f00"># Outer loop looking for a frame header</span>
<span style="color: #007f00">#</span>
<span style="color: #00007f; font-weight: bold">while</span> <span style="color: #00007f">True</span>:
byte = (<span style="color: #00007f; font-weight: bold">yield</span>)
frame = <span style="color: #7f007f">''</span>
<span style="color: #00007f; font-weight: bold">if</span> byte == header:
<span style="color: #007f00"># Capture the full frame</span>
<span style="color: #007f00">#</span>
<span style="color: #00007f; font-weight: bold">while</span> <span style="color: #00007f">True</span>:
byte = (<span style="color: #00007f; font-weight: bold">yield</span>)
<span style="color: #00007f; font-weight: bold">if</span> byte == footer:
target.send(frame)
<span style="color: #00007f; font-weight: bold">break</span>
<span style="color: #00007f; font-weight: bold">elif</span> byte == dle:
byte = (<span style="color: #00007f; font-weight: bold">yield</span>)
frame += after_dle_func(byte)
<span style="color: #00007f; font-weight: bold">else</span>:
frame += byte
</pre></div>
<p>Look how simple and elegant it is. You can tell immediately what it does just by looking at the source code - no state diagrams are needed.</p>
<p>We loop over frames. A frame starts with a header byte. After a header byte has been received, we accumulate the bytes of the frame until a footer is encountered. The <tt class="docutils literal"><span class="pre">(yield)</span></tt> calls is where the magic is. The function suspends at these points until it is called again <a class="footnote-reference" href="#id6" id="id3">[3]</a>. Then, the value passed in the new call is returned from <tt class="docutils literal"><span class="pre">(yield)</span></tt> and the co-routine proceeds from the same place.</p>
<p>Note how the state machine is <em>implicitly</em> embedded in this code. It's there, but you don't see it - it's hiding in the control structures (the IFs, ELSEs and the WHILEs) of the function.</p>
<p>When a complete frame is received, it is sent to the <tt class="docutils literal"><span class="pre">target</span></tt> of the co-routine, which may process it at will. After executing <tt class="docutils literal"><span class="pre">send</span></tt>, the co-routine breaks out of the inner loop and suspends waiting for a new header in the outer loop.</p>
<p>The <tt class="docutils literal"><span class="pre">@coroutine</span></tt> decorator is a simple utility required for Python co-routines:</p>
<div class="highlight"><pre><span style="color: #00007f; font-weight: bold">def</span> <span style="color: #00007f">coroutine</span>(func):
<span style="color: #00007f; font-weight: bold">def</span> <span style="color: #00007f">start</span>(*args,**kwargs):
cr = func(*args,**kwargs)
cr.next()
<span style="color: #00007f; font-weight: bold">return</span> cr
<span style="color: #00007f; font-weight: bold">return</span> start
</pre></div>
<p>This is needed to bring a co-routine to its first <tt class="docutils literal"><span class="pre">yield</span></tt> and suspend there. You can just use this decorator without worrying about the details, until you become more comfortable with the concept to understand the exact inner workings described in PEP 342.</p>
<p>To test this co-routine implementation we also need a simple "sink" co-routine (using Dave Beazley's terminology from his presentation). This will be the receiver of the <tt class="docutils literal"><span class="pre">send</span></tt> calls made by our co-routine:</p>
<div class="highlight"><pre>@coroutine
<span style="color: #00007f; font-weight: bold">def</span> <span style="color: #00007f">frame_receiver</span>():
<span style="color: #7f007f">""" A simple co-routine "sink" for receiving</span>
<span style="color: #7f007f"> full frames.</span>
<span style="color: #7f007f"> """</span>
<span style="color: #00007f; font-weight: bold">while</span> <span style="color: #00007f">True</span>:
frame = (<span style="color: #00007f; font-weight: bold">yield</span>)
<span style="color: #00007f; font-weight: bold">print</span> <span style="color: #7f007f">'Got frame:'</span>, frame.encode(<span style="color: #7f007f">'hex'</span>)
bytes = <span style="color: #7f007f">''</span>.join(<span style="color: #00007f">chr</span>(b) <span style="color: #00007f; font-weight: bold">for</span> b <span style="color: #0000aa">in</span>
[<span style="color: #007f7f">0</span>x70, <span style="color: #007f7f">0</span>x24,
<span style="color: #007f7f">0</span>x61, <span style="color: #007f7f">0</span>x99, <span style="color: #007f7f">0</span>xAF, <span style="color: #007f7f">0</span>xD1, <span style="color: #007f7f">0</span>x62,
<span style="color: #007f7f">0</span>x56, <span style="color: #007f7f">0</span>x62,
<span style="color: #007f7f">0</span>x61, <span style="color: #007f7f">0</span>xAB, <span style="color: #007f7f">0</span>xAB, <span style="color: #007f7f">0</span>x14, <span style="color: #007f7f">0</span>x62,
<span style="color: #007f7f">0</span>x7
])
unwrapper = unwrap_protocol(
target=frame_receiver())
<span style="color: #00007f; font-weight: bold">for</span> byte <span style="color: #0000aa">in</span> bytes:
unwrapper.send(byte)
</pre></div>
<p>Prints:</p>
<div class="highlight"><pre>Got frame: 99afd1
Got frame: ab14
</pre></div>
</div>
<div class="section" id="conclusion">
<h4>Conclusion</h4>
<p>I'll repeat the quote from the beginning of the article:</p>
<blockquote>
Co-routines are to state machines what recursion is to stacks</blockquote>
<p>Recursion helps process nested data structures without employing explicit stacks.</p>
<p>Similarly, co-routines help solve problems involving state, without using explicit state machines. The resulting code is not centered on the states, but rather on the logic of the tasks, which makes it much simpler to understand.</p>
<p>Co-routines are a useful tool to have in one's toolbox. It is worthwhile to spend some time getting acquainted with them.</p>
<div align="center" class="align-center"><img class="align-center" src="https://eli.thegreenplace.net/images/hline.jpg" style="width: 320px; height: 5px;" /></div>
<table class="docutils footnote" frame="void" id="id4" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Such a state machine is called a <a class="reference external" href="http://en.wikipedia.org/wiki/Mealy">Mealy machine</a> - it generates output based on the current state <em>and</em> input. Most state machines implemented in software are of this type.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id5" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>For Python there's also PEP 342 - but I recommend going over it only after you've read Dave's tutorial.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id6" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>Technically, a co-routine is created once by calling it. Then, we have a "co-routine object" on which we can execute <tt class="docutils literal"><span class="pre">send</span></tt> methods, passing the arguments to <tt class="docutils literal"><span class="pre">yield</span></tt> via <tt class="docutils literal"><span class="pre">send</span></tt>. This is how co-routines are implemented in Python. It might look different in another language, but the concept stays the same.</td></tr>
</tbody>
</table>
</div>
</div>
Frames and protocols for the serial port - in Python2009-08-20T07:01:44-07:002023-02-04T13:41:52-08:00Eli Benderskytag:eli.thegreenplace.net,2009-08-20:/2009/08/20/frames-and-protocols-for-the-serial-port-in-python
<div class="section" id="jab">
<div class="section" id="some-preliminaries">
<h4>Some preliminaries</h4>
<p>If you've been following this blog recently, you must have noticed that many of the posts in these past few weeks are about using Python to communicate via the serial port. I specifically decided to write them as separate posts and not as part of a series, because …</p></div></div>
<div class="section" id="jab">
<div class="section" id="some-preliminaries">
<h4>Some preliminaries</h4>
<p>If you've been following this blog recently, you must have noticed that many of the posts in these past few weeks are about using Python to communicate via the serial port. I specifically decided to write them as separate posts and not as part of a series, because I think that each post is interesting in itself <a class="footnote-reference" href="#id7" id="id1">[1]</a>.</p>
<p>But just in case you got confused, here's the logical order:</p>
<ol class="arabic simple">
<li><a class="reference external" href="https://eli.thegreenplace.net/2009/07/30/setting-up-python-to-work-with-the-serial-port/">Setting up Python to work with the serial port</a></li>
<li><a class="reference external" href="https://eli.thegreenplace.net/2009/08/07/a-live-data-monitor-with-python-pyqt-and-pyserial/">A “live” data monitor with Python, PyQt and PySerial</a></li>
<li><a class="reference external" href="https://eli.thegreenplace.net/2009/08/12/framing-in-serial-communications/">Framing in serial communications</a></li>
</ol>
<p>In this post I want to present some useful Python code to implement the ideas of (3). Additionally, I'll introduce a very useful library for constructing frames from various forms of data.</p>
</div>
<div class="section" id="code">
<h4>Code</h4>
<p>The code for this post is available <a href="https://github.com/eliben/code-for-blog/tree/master/2009/protocol">here</a>.
It contains the modules discussed, the sample code shown and even some unit tests.</p>
</div>
<div class="section" id="arrays-of-data-in-python">
<h4>Arrays of data in Python</h4>
<p>When we think about a sequence of bytes in Python, two approaches come to mind: an array of integers in the range 0-255, or a 'packed' string. Here's some Python terminal action that displays the difference:</p>
<div class="highlight"><pre>>>> arr = [<span style="color: #007f7f">0</span>x45, <span style="color: #007f7f">0</span>xAB, <span style="color: #007f7f">0</span>xC3, <span style="color: #007f7f">0</span>x16]
>>> arr
[<span style="color: #007f7f">69</span>, <span style="color: #007f7f">171</span>, <span style="color: #007f7f">195</span>, <span style="color: #007f7f">22</span>]
>>> <span style="color: #00007f">str</span> = <span style="color: #7f007f">'\x45\xAB\xC3\x16'</span>
>>> <span style="color: #00007f">str</span>
<span style="color: #7f007f">'E\xab\xc3\x16'</span>
>>> <span style="color: #7f007f">''</span>.join(<span style="color: #00007f">chr</span>(b) <span style="color: #00007f; font-weight: bold">for</span> b <span style="color: #0000aa">in</span> arr)
<span style="color: #7f007f">'E\xab\xc3\x16'</span>
>>> [<span style="color: #00007f">ord</span>(b) <span style="color: #00007f; font-weight: bold">for</span> b <span style="color: #0000aa">in</span> <span style="color: #00007f">str</span>]
[<span style="color: #007f7f">69</span>, <span style="color: #007f7f">171</span>, <span style="color: #007f7f">195</span>, <span style="color: #007f7f">22</span>]
</pre></div>
<p>This shows that the two formats are essentially interchangeable, and also that it's very easy to convert between the two.</p>
<p>The format we're going to use is the packed string, because this is what the <tt class="docutils literal"><span class="pre">pyserial</span></tt> module uses to send and receive data.</p>
</div>
<div class="section" id="serializing-data">
<h4>Serializing data</h4>
<p>So, to send data over the serial port we first have to turn it into a packed string - this is called serialization <a class="footnote-reference" href="#id8" id="id2">[2]</a>.</p>
<p>Python has a couple of built-in ways to do that - with the <tt class="docutils literal"><span class="pre">array</span></tt> and <tt class="docutils literal"><span class="pre">struct</span></tt> modules. However, both are suitable for fairly simple and unsophisticated data. To serialize arbitrarily sophisticated data formats, it's much better to use the powerful and flexible <a class="reference external" href="http://construct.wikispaces.com/">construct</a> library <a class="footnote-reference" href="#id9" id="id3">[3]</a>.</p>
<p>Here's a sample message format defined with <tt class="docutils literal"><span class="pre">construct</span></tt> (from <tt class="docutils literal"><span class="pre">sampleformat.py</span></tt> in this article's code archive):</p>
<div class="highlight"><pre><span style="color: #00007f; font-weight: bold">from</span> <span style="color: #00007f">construct</span> <span style="color: #00007f; font-weight: bold">import</span> *
message_crc = Struct(<span style="color: #7f007f">'message_crc'</span>, ULInt32(<span style="color: #7f007f">'crc'</span>))
message_format = Struct(<span style="color: #7f007f">'message_format'</span>,
ULInt16(<span style="color: #7f007f">'msg_id'</span>),
ULInt16(<span style="color: #7f007f">'dest_addr'</span>),
Enum(Byte(<span style="color: #7f007f">'command_type'</span>),
RESTART = <span style="color: #007f7f">0</span>x40,
RESTART_ACK = <span style="color: #007f7f">0</span>x80,
SIGNAL = <span style="color: #007f7f">0</span>x22,
_default_ = Pass
),
BitStruct(<span style="color: #7f007f">'flags'</span>,
Flag(<span style="color: #7f007f">'on'</span>),
BitField(<span style="color: #7f007f">'status'</span>, <span style="color: #007f7f">3</span>),
Flag(<span style="color: #7f007f">'cache'</span>),
Padding(<span style="color: #007f7f">3</span>)
),
Byte(<span style="color: #7f007f">'datalen'</span>),
Array(<span style="color: #00007f; font-weight: bold">lambda</span> ctx: ctx[<span style="color: #7f007f">'datalen'</span>], Byte(<span style="color: #7f007f">'data'</span>)),
Embed(message_crc)
)
</pre></div>
<p>It shows off a few interesting features of <tt class="docutils literal"><span class="pre">construct</span></tt>:</p>
<ul class="simple">
<li>Explicit specification of endianness for multi-byte fields</li>
<li>Enumerations</li>
<li>Support for byte-oriented and bit-oriented fields</li>
<li>Arrays of data with specified length</li>
<li>Embedded structs</li>
</ul>
<p>The message should look roughly familiar for anyone designing and using binary protocols. It's very typical of how real formats look - some ID fields, flags, data, CRC <a class="footnote-reference" href="#id10" id="id4">[4]</a>.</p>
<p>Here's how this message format can be used to pack and unpack a message:</p>
<div class="highlight"><pre>>>> <span style="color: #00007f; font-weight: bold">from</span> <span style="color: #00007f">sampleformat</span> <span style="color: #00007f; font-weight: bold">import</span> message_format
>>> <span style="color: #00007f; font-weight: bold">from</span> <span style="color: #00007f">construct</span> <span style="color: #00007f; font-weight: bold">import</span> *
>>> raw = message_format.build(Container(
... msg_id=<span style="color: #007f7f">0</span>x1234,
... dest_addr=<span style="color: #007f7f">0</span>xacba,
... command_type=<span style="color: #7f007f">'RESTART'</span>,
... flags=Container(on=<span style="color: #007f7f">1</span>, cache=<span style="color: #007f7f">0</span>, status=<span style="color: #007f7f">4</span>),
... datalen=<span style="color: #007f7f">4</span>,
... data=[<span style="color: #007f7f">0</span>x1, <span style="color: #007f7f">0</span>xff, <span style="color: #007f7f">0</span>xff, <span style="color: #007f7f">0</span>xdd],
... crc=<span style="color: #007f7f">0</span>x12345678))
>>> raw.encode(<span style="color: #7f007f">'hex'</span>)
<span style="color: #7f007f">'3412baac40c00401ffffdd78563412'</span>
>>> c = message_format.parse(raw)
>>> <span style="color: #00007f; font-weight: bold">print</span> c
Container:
msg_id = <span style="color: #007f7f">4660</span>
dest_addr = <span style="color: #007f7f">44218</span>
command_type = <span style="color: #7f007f">'RESTART'</span>
flags = Container:
on = <span style="color: #00007f">True</span>
status = <span style="color: #007f7f">4</span>
cache = <span style="color: #00007f">False</span>
datalen = <span style="color: #007f7f">4</span>
data = [
<span style="color: #007f7f">1</span>
<span style="color: #007f7f">255</span>
<span style="color: #007f7f">255</span>
<span style="color: #007f7f">221</span>
]
crc = <span style="color: #007f7f">305419896</span>
</pre></div>
<p>A few things to note here:</p>
<ul class="simple">
<li><tt class="docutils literal"><span class="pre">message_format</span></tt> is an object with two useful methods: <tt class="docutils literal"><span class="pre">build</span></tt> for packing data into a string, and <tt class="docutils literal"><span class="pre">parse</span></tt> for unpacking it back from a string.</li>
<li><tt class="docutils literal"><span class="pre">Container</span></tt> is a class taken from <tt class="docutils literal"><span class="pre">construct</span></tt>. It's just a simple data container holding its data items in attributes. Any compatible object would do here (duck typing!) - for example a <tt class="docutils literal"><span class="pre">namedtuple</span></tt>. I chose <tt class="docutils literal"><span class="pre">Container</span></tt> because it comes with <tt class="docutils literal"><span class="pre">construct</span></tt> anyway and is simple and useful.</li>
<li><tt class="docutils literal"><span class="pre">raw</span></tt> is a packed string. The <tt class="docutils literal"><span class="pre">encode</span></tt> string method is used here to show the hex values of the string's bytes.</li>
</ul>
</div>
<div class="section" id="framing-protocol-wrapping-and-unwrapping">
<h4>Framing (protocol wrapping and unwrapping)</h4>
<p><tt class="docutils literal"><span class="pre">protocolwrapper.py</span></tt> in the code archive</a> is a faithful Python implementation of the <a class="reference external" href="https://eli.thegreenplace.net/2009/08/12/framing-in-serial-communications/">Framing in serial communications article.</a></p>
<p>Not much more to say about it here - the code is commented and should be simple to understand if you're familiar with the theory.</p>
</div>
<div class="section" id="putting-it-all-together">
<h4>Putting it all together</h4>
<p>The process of sending is:</p>
<ol class="arabic simple">
<li>Serialize all the fields into a packed string using the message format object</li>
<li>Compute the CRC and insert it into the frame</li>
<li>Wrap the frame with the protocol</li>
<li>Now we have a string ready to send that represents the complete message</li>
</ol>
<div class="highlight"><pre><span style="color: #00007f; font-weight: bold">from</span> <span style="color: #00007f">zlib</span> <span style="color: #00007f; font-weight: bold">import</span> crc32
<span style="color: #00007f; font-weight: bold">from</span> <span style="color: #00007f">protocolwrapper</span> <span style="color: #00007f; font-weight: bold">import</span> (
ProtocolWrapper, ProtocolStatus)
<span style="color: #00007f; font-weight: bold">from</span> <span style="color: #00007f">sampleformat</span> <span style="color: #00007f; font-weight: bold">import</span> (
message_format, message_crc, Container)
PROTOCOL_HEADER = <span style="color: #7f007f">'\x11'</span>
PROTOCOL_FOOTER = <span style="color: #7f007f">'\x12'</span>
PROTOCOL_DLE = <span style="color: #7f007f">'\x90'</span>
<span style="color: #00007f; font-weight: bold">def</span> <span style="color: #00007f">build_message_to_send</span>(
msg_id, dest_addr, command_type,
flag_on, flag_cache, flag_status, data):
<span style="color: #7f007f">""" Given the data, builds a message for</span>
<span style="color: #7f007f"> transmittion, computing the CRC and packing</span>
<span style="color: #7f007f"> the protocol.</span>
<span style="color: #7f007f"> Returns the packed message ready for</span>
<span style="color: #7f007f"> transmission on the serial port.</span>
<span style="color: #7f007f"> """</span>
datalen = <span style="color: #00007f">len</span>(data)
flags = Container( on=flag_on,
cache=flag_cache,
status=flag_status)
<span style="color: #007f00"># Build the raw message string. CRC is empty</span>
<span style="color: #007f00"># for now</span>
<span style="color: #007f00">#</span>
raw = message_format.build(Container(
msg_id=msg_id,
dest_addr=dest_addr,
command_type=command_type,
flags=flags,
datalen=datalen,
data=data,
crc=<span style="color: #007f7f">0</span>))
<span style="color: #007f00"># Compute the CRC field and append it to the</span>
<span style="color: #007f00"># message instead of the empty CRC specified</span>
<span style="color: #007f00"># initially.</span>
<span style="color: #007f00">#</span>
msg_without_crc = raw[:-<span style="color: #007f7f">4</span>]
msg_crc = message_crc.build(Container(
crc=crc32(msg_without_crc)))
<span style="color: #007f00"># Append the CRC field</span>
<span style="color: #007f00">#</span>
msg = msg_without_crc + msg_crc
pw = ProtocolWrapper(
header=PROTOCOL_HEADER,
footer=PROTOCOL_FOOTER,
dle=PROTOCOL_DLE)
<span style="color: #00007f; font-weight: bold">return</span> pw.wrap(msg)
</pre></div>
<p>The receiving process is:</p>
<ol class="arabic simple">
<li>Unwrap the protocol to receive a frame</li>
<li>Unpack the frame into separate fields using the frame format</li>
<li>Compute the CRC and compare it to the one received</li>
<li>If all is OK, we have received a new valid frame</li>
</ol>
<div class="highlight"><pre><span style="color: #007f00"># Sample: receiving a message</span>
<span style="color: #007f00">#</span>
pw = ProtocolWrapper(
header=PROTOCOL_HEADER,
footer=PROTOCOL_FOOTER,
dle=PROTOCOL_DLE)
<span style="color: #007f00"># Feed all the bytes of 'msg' sequentially</span>
<span style="color: #007f00"># into pw.input</span>
<span style="color: #007f00">#</span>
status = <span style="color: #00007f">map</span>(pw.input, msg)
<span style="color: #00007f; font-weight: bold">if</span> status[-<span style="color: #007f7f">1</span>] == ProtocolStatus.MSG_OK:
rec_msg = pw.last_message
<span style="color: #007f00"># Parse the received CRC into a 32-bit integer</span>
<span style="color: #007f00">#</span>
rec_crc = message_crc.parse(rec_msg[-<span style="color: #007f7f">4</span>:]).crc
<span style="color: #007f00"># Compute the CRC on the message</span>
<span style="color: #007f00">#</span>
calc_crc = crc32(rec_msg[:-<span style="color: #007f7f">4</span>])
<span style="color: #00007f; font-weight: bold">if</span> rec_crc != calc_crc:
<span style="color: #00007f; font-weight: bold">print</span> <span style="color: #7f007f">'Error: CRC mismatch'</span>
<span style="color: #00007f; font-weight: bold">print</span> message_format.parse(rec_msg)
</pre></div>
<p>These are just examples, of course. Your own code will depend on the structure of your frames and how you receive your data. But it can serve as a basic template for implementing arbitrary complex serial protocols in a robust way.</p>
<div align="center" class="align-center"><img class="align-center" src="https://eli.thegreenplace.net/images/hline.jpg" style="width: 320px; height: 5px;" /></div>
<table class="docutils footnote" frame="void" id="id7" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>By the way, all the posts on this topic are collected in the <a class="reference external" href="https://eli.thegreenplace.net/tag/serial-port">Serial Port category</a>.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id8" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>Python already has nice libraries for serialization (<tt class="docutils literal"><span class="pre">pickle</span></tt>, <tt class="docutils literal"><span class="pre">shelve</span></tt>, <tt class="docutils literal"><span class="pre">json</span></tt> and others), but there's a problem! It's usually not Python we have on the other side of the serial link! Two Python programs would find a better, faster method to communicate (like TCP/IP). When we use Python with <tt class="docutils literal"><span class="pre">pyserial</span></tt> it's because we actually want to communicate with some embedded hardware (implemented in C or even as an FPGA/ASIC with VHDL or Verilog) or other physical equipment. So pickling the data won't help here.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id9" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>This is not a tutorial of <tt class="docutils literal"><span class="pre">construct</span></tt> though. There's a pretty good one on <a class="reference external" href="http://construct.wikispaces.com/tutorial">its website</a></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id10" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td><tt class="docutils literal"><span class="pre">construct</span></tt> has sample formats for well-known protocols like TCP and ARP, and binary files like PNG and ELF32.</td></tr>
</tbody>
</table>
</div>
</div>
Framing in serial communications2009-08-12T05:16:47-07:002023-06-30T23:16:27-07:00Eli Benderskytag:eli.thegreenplace.net,2009-08-12:/2009/08/12/framing-in-serial-communications
<div class="section" id="introduction">
<h3>Introduction</h3>
<p>In the <a class="reference external" href="https://eli.thegreenplace.net/2009/08/07/a-live-data-monitor-with-python-pyqt-and-pyserial/">previous post</a> we've seen how to send and receive data on the serial port with Python and plot it live using a pretty GUI.</p>
<p>Notice that the sender script (sender_sim.py) is just sending one byte at a time. The "chunks" of data in the protocol between …</p></div>
<div class="section" id="introduction">
<h3>Introduction</h3>
<p>In the <a class="reference external" href="https://eli.thegreenplace.net/2009/08/07/a-live-data-monitor-with-python-pyqt-and-pyserial/">previous post</a> we've seen how to send and receive data on the serial port with Python and plot it live using a pretty GUI.</p>
<p>Notice that the sender script (sender_sim.py) is just sending one byte at a time. The "chunks" of data in the protocol between the sender and receiver are single bytes. This is simple and convenient, but hardly sufficient in the general sense. We want to be able to send multiple-byte data frames between the communicating parties.</p>
<p>However, there are some challenges that arise immediately:</p>
<ul class="simple">
<li>The receiver is just receiving a stream of bytes from the serial port. How does it know when a message begins or ends? How does it know how long the message is?</li>
<li>Even more seriously, we can not assume a noise-free channel. This is real, physical hardware stuff. Bytes and whole chunks can and will be lost due to electrical noise. Worse, other bytes will be distorted (say, a single bit can be flipped due to noise).</li>
</ul>
<p>To see how this can be done in a safe and tested manner, we first have to learn about the basics of the Data Link Layer in computer networks.</p>
</div>
<div class="section" id="data-link-layer">
<h3>Data Link Layer</h3>
<p>Given a physical layer that can transmit signals between devices, the job of the Data Link Layer <a class="footnote-reference" href="#id9" id="id1">[1]</a> is (roughly stated) to transmit whole frames of data, with some means of assuring the integrity of the data (lack of errors). When we use sockets to communicate over TCP or UDP on the internet, the framing is taken care of deep in the hardware, and we don't even feel it. On the serial port, however, we must take care of the framing and error handling ourselves <a class="footnote-reference" href="#id10" id="id2">[2]</a>.</p>
<div class="section" id="framing">
<h4>Framing</h4>
<p>In chapter 3 of his <a class="reference external" href="https://eli.thegreenplace.net/2009/08/08/book-review-computer-networks-4th-edition-by-andrew-tanenbaum/">"Computer Networks"</a> textbook, Tanenbaum defines the following methods of framing:</p>
<ol class="arabic simple">
<li>Inserting time gaps between frames</li>
<li>Physical layer coding violations</li>
<li>Character count</li>
<li>Flag bytes with byte stuffing</li>
<li>Flag bytes with bit stuffing</li>
</ol>
<p>Methods (1) and (2) are only suitable for a hardware-implemented data link layer <a class="footnote-reference" href="#id11" id="id3">[3]</a>. It is very difficult (read: impossible) to ensure timing when multiple layers of software (running on Windows!) are involved. (2) is an interesting hardware method - but out of the scope of this article.</p>
<p>Method (3) means specifying in the frame header the number of bytes in the frame. The trouble with this is that the count can be garbled by a transmission error. In such a case, it's very difficult to "resynchronize". This method is rarely used.</p>
<p>Methods (4) and (5) are somewhat similar. In this article I'll focus on (4), as (5) is not suitable for serial port communications.</p>
</div>
<div class="section" id="flag-bytes-with-byte-stuffing">
<h4>Flag bytes with byte stuffing</h4>
<p>Let's begin with a simple idea and develop it into a full, robust scheme.</p>
<p><em>Flag bytes</em> are special byte values that denote when a frame begins and ends. Suppose that we want to be able to send frames of arbitrary length. A special start flag byte will denote the beginning of the frame, and an end flag byte will denote its end.</p>
<img src="https://eli.thegreenplace.net/images/2009/08/flags_data.png" />
<p>A question arises, however. Suppose that the value of the end flag is 0x98. What if the value 0x98 appears somewhere in the data? The protocol will get confused and end the message.</p>
<p>There is a simple solution to this problem that will be familiar to all programmers who know about escaping quotes and special characters in strings. It is called <em>byte stuffing</em>, or <em>octet stuffing</em>, or simply <em>escaping</em> <a class="footnote-reference" href="#id12" id="id4">[4]</a>. The scheme goes as follows:</p>
<ul class="simple">
<li>Whenever a flag (start or end) byte appears in the data, we shall insert a special escape byte (ESC) before it. When the receiver sees an ESC, it knows to ignore it and not insert it into the actual data received (de-stuffing).</li>
<li>Whenever ESC itself has to appear in the data, another ESC is prepended to it. The receiver removes the first one but keeps the second one <a class="footnote-reference" href="#id13" id="id5">[5]</a>.</li>
</ul>
<p>Here are a few examples:</p>
<img src="https://eli.thegreenplace.net/images/2009/08/escaping.png" />
<p>Note that we didn't specify what the data is - it's arbitrary and up the the protocol to decide. The only really required part of the data is some kind of error checking - a checksum, or better yet a CRC. This is customarily the last byte (or last word) of the frame, referring to all the bytes in the frame (in its un-stuffed form).</p>
<p>This scheme is quite robust: any lost byte (be it a flag, an escape, a data byte or a checksum byte) will cause the receiver to lose just one frame, after which it will resynchronize onto the start flag byte of the next one.</p>
</div>
<div class="section" id="ppp">
<h4>PPP</h4>
<p>As a matter of fact, this method is a slight simplification of the <a class="reference external" href="http://en.wikipedia.org/wiki/Point-to-Point_Protocol">Point-to-Point Protocol</a> (PPP) which is used by most ISPs for providing ADSL internet to home users, so there's a good chance you're using it now to surf the net and read this article! The framing of PPP is defined in <a class="reference external" href="http://tools.ietf.org/html/rfc1662">RFC 1662</a>.</p>
<p>In particular, PPP does the following:</p>
<ul class="simple">
<li>Both the start and end flag bytes are 0x7E (they shouldn't really be different, if you think about it)</li>
<li>The escape byte is 0x7D</li>
<li>Whenever a flag or escape byte appears in the message, it is escaped by 0x7D and the byte itself is XOR-ed with 0x20. So, for example 0x7E becomes 0x7D 0x5E. Similarly 0x7D becomes 0x7D 0x5D. The receiver unsuffs the escape byte and XORs the next byte with 0x20 again to get the original <a class="footnote-reference" href="#id14" id="id6">[6]</a>.</li>
</ul>
</div>
</div>
<div class="section" id="an-example">
<h3>An example</h3>
<p>Let's now see a completely worked-out example that demonstrates how this works.</p>
<p>Suppose we define the following protocol:</p>
<ul class="simple">
<li>Start flag: 0x12</li>
<li>End flag: 0x13</li>
<li>Escape (DLE): 0x7D</li>
</ul>
<p>And the sender wants to send the following data message (let's ignore its contents for the sake of the example - they're really not that important). The original data is in <strong>(a)</strong>:</p>
<img src="https://eli.thegreenplace.net/images/2009/08/example1.png" />
<p>The data contains two flags that need to be escaped - an end flag at position 2 (counting from 0, of course!), and a DLE at position 4.</p>
<p>The sender's data link layer <a class="footnote-reference" href="#id15" id="id7">[7]</a> turns the data into the frame shown in <strong>(b)</strong> - start and end flags are added, and in-message flags are escaped.</p>
<p>Let's see how the receiver handles such a frame. For demonstration, assume that the first byte the receiver draws from the serial port is not a real part of the message (we want to see how it handles this). In the following diagram, 'Receiver state' is the state of the receiver <em>after</em> the received byte. 'Data buffer' is the currently accumulated message buffer to pass to an upper level:</p>
<img src="https://eli.thegreenplace.net/images/2009/08/example1_rcv.png" />
<p>A few things to note:</p>
<ul class="simple">
<li>The "stray" byte before the header is ignored: according to the protocol each frame has to start with a header, so this isn't part of the frame.</li>
<li>The start and end flags are not inserted into the data buffer</li>
<li>Escapes (DLEs) are correctly handled by a special state</li>
<li>When the frame is finished with an end flag, the receiver has a frame ready to pass to an upper level, and comes back waiting for a header - a new frame.</li>
</ul>
<p>Finally, we see that the message received is exactly the message sent. All the protocol details (flags, escapes and so on) were transparently handled by the data link layer <a class="footnote-reference" href="#id16" id="id8">[8]</a>.</p>
</div>
<div class="section" id="conclusion">
<h3>Conclusion</h3>
<p>There are several methods of handling framing in communications, although most are unsuitable to be used on top of the serial port. Among the ones that are suitable, the most commonly used is <em>byte stuffing</em>. By defining a couple of "magic value" flags and careful rules of escaping, this framing methods is both robust and easy to implement as a software layer. It is also widely used as PPP depends on it.</p>
<p>Finally, it's important to remember that for a high level of robustness, it's required to add some kind of error checking into the protocol - such as computing a CRC on the message and appending it as the last word of the message, which the receiver can verify before deciding that the message is valid.</p>
<div align="center" class="align-center"><img class="align-center" src="https://eli.thegreenplace.net/images/hline.jpg" style="width: 320px; height: 5px;" /></div>
<table class="docutils footnote" frame="void" id="id9" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>The Data Link Layer is layer 2 in the <a class="reference external" href="http://en.wikipedia.org/wiki/OSI_model">OSI model</a>. In the <a class="reference external" href="http://en.wikipedia.org/wiki/TCP/IP_model">TCP/IP model</a> it's simply called the "link layer".</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id10" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>The serial port can be configured to add parity bits to bytes. These days, this option is rarely used, because:</td></tr>
</tbody>
</table>
<ul class="simple">
<li>A single parity bit isn't a very strong means of detecting errors. 2-bit errors fool it.</li>
<li>Error handling is usually done by stronger means at a higher level.</li>
</ul>
<table class="docutils footnote" frame="void" id="id11" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>For example Ethernet (802.3) uses 12 octets of idle characters between frames.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id12" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>You might run into the term DLE - Data Link Escape, which means the same thing. I will use the acronyms DLE and ESC interchangeably.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id13" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id5">[5]</a></td><td>Just like quotes and escape characters in strings! In C: <tt class="docutils literal"><span class="pre">"I</span> <span class="pre">say</span> <span class="pre">\"Hello\""</span></tt>. To escape the escape, repeat it: <tt class="docutils literal"><span class="pre">"Here</span> <span class="pre">comes</span> <span class="pre">the</span> <span class="pre">backslash:</span> <span class="pre">\\</span> <span class="pre">-</span> <span class="pre">seen</span> <span class="pre">it?"</span></tt></td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id14" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id6">[6]</a></td><td>I'd love to hear why this XOR-ing is required. One simple reason I can think of is to prevent the flag and escape bytes appearing "on the line" even after they're escaped. Presumably this improves resynchronization if the escape byte is lost?</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id15" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id7">[7]</a></td><td>Which is just a fancy way to say "a protocol wrapping function", since the layer is implemented in software.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id16" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id8">[8]</a></td><td>Such transparency is one of the greatest ideas of layered network protocols. So when we implement protocols in software, it's a good thing to keep in mind - transparency aids modularity and decoupling, it's a <em>good thing</em>.</td></tr>
</tbody>
</table>
</div>
A "live" data monitor with Python, PyQt and PySerial2009-08-07T14:57:57-07:002023-02-04T15:35:51-08:00Eli Benderskytag:eli.thegreenplace.net,2009-08-07:/2009/08/07/a-live-data-monitor-with-python-pyqt-and-pyserial
<p>The previous <a class="reference external" href="https://eli.thegreenplace.net/2009/07/30/setting-up-python-to-work-with-the-serial-port/">couple</a> of <a class="reference external" href="https://eli.thegreenplace.net/2009/07/31/listing-all-serial-ports-on-windows-with-python/">posts</a> about the PySerial module for serial communications with Python were just a basic introduction. Let's now see something much more useful.</p>
<p>PySerial makes Python a great tool for serial communications from a computer, as it can be easily combined with other powerful Python libraries to …</p>
<p>The previous <a class="reference external" href="https://eli.thegreenplace.net/2009/07/30/setting-up-python-to-work-with-the-serial-port/">couple</a> of <a class="reference external" href="https://eli.thegreenplace.net/2009/07/31/listing-all-serial-ports-on-windows-with-python/">posts</a> about the PySerial module for serial communications with Python were just a basic introduction. Let's now see something much more useful.</p>
<p>PySerial makes Python a great tool for serial communications from a computer, as it can be easily combined with other powerful Python libraries to create full-scale applications. In my case, I'm using PyQt with its plotting supplementary PyQwt to create nice "live" plotting applications, that can be combined with the serial port. Here's a demo:</p>
<p>Download <a href="https://github.com/eliben/code-for-blog/tree/master/2009/plotting_data_monitor">plotting_data_monitor</a>- it's a small demo application written in Python that requires the following modules to be installed on your machine:</p>
<ul class="simple">
<li>PyQt</li>
<li>PyQwt</li>
<li>PySerial</li>
</ul>
<p>What does it do? Well, it basically shows how to combine all these powers of Python into a single application in a simple way. You can choose a serial port and then run the monitor (all via the menu). If another program is sending data to the specified port, you'll see the plot updating "in real time":</p>
<img src="https://eli.thegreenplace.net/images/2009/08/datamonitor_shot.png" />
<p>If you have nothing sending data to your machine, no worries. If you've installed com0com or a similar virtual port emulator like I explained <a class="reference external" href="https://eli.thegreenplace.net/2009/07/30/setting-up-python-to-work-with-the-serial-port/">here</a>, configure it to connect two ports together.</p>
<p>Then, download sender_sim.py which is a very simple data-sending script (the data it provides is a pleasant pseudo-randomized sinusoid). You may want to change the port name hard-coded in it, if your port numbers are different.</p>
<p>When both this sender and the monitor run on the same machine, you'll be able to see the live plotting. Note that I've added a couple of extra features from PyQwt:</p>
<ul class="simple">
<li>A "thermo" bar that shows the average temperature</li>
<li>A knob that sets how often the monitor updates the screen</li>
</ul>
<p>These widgets, and a few others, make PyQwt quite a nice library for emulating Labview-type "lab control" applications in Python. I recommend it highly.</p>
<div class="section" id="how-does-it-work">
<h3>How does it work</h3>
<p>The monitor is a toy demo, but it's based on a few powerful tools I use for real applications. For example, the serial communication itself is performed in a separate thread (in the <tt class="docutils literal"><span class="pre">com_monitor</span></tt> module). The thread issues blocking reads on the serial port in a loop, and communicates the data to the GUI via a <tt class="docutils literal"><span class="pre">Queue</span></tt> (together with an accurate timestamp for every data chunk received). This is a robust and safe implementation that can be used for many kinds of GUI-based monitoring applications.</p>
<p>The GUI itself is updated using a periodic timer (<tt class="docutils literal"><span class="pre">QTimer</span></tt>) that runs whenever the monitor is running. The timer event handler checks whether new data has arrived from the monitor and updates the display widgets accordingly.</p>
<p>The rest of the code is your usual PyQt bureaucracy - creating the menu, the status bar, the GUI widgets, laying everything out nicely and connecting events.</p>
<p>I hope people will find this code useful. If you have trouble running it or understanding how it works, let me know. I recommend using the latest Python 2.6 versions of all the required modules. I checked it only on Windows, but there's no reason whatsoever for it not to run on other OSes out of the box.</p>
</div>
Listing all serial ports on Windows with Python2009-07-31T07:41:00-07:002023-02-04T13:41:52-08:00Eli Benderskytag:eli.thegreenplace.net,2009-07-31:/2009/07/31/listing-all-serial-ports-on-windows-with-python
<div class="section" id="blob">
<p>There are several methods for getting a list of all serial (COM) ports on a Windows machine. Here I want to focus on one that seems to work very well on the computers I tried, and incidentally it's also the simplest to implement.</p>
<div class="section" id="digging-into-the-registry">
<h4>Digging into the registry</h4>
<p>The list of …</p></div></div>
<div class="section" id="blob">
<p>There are several methods for getting a list of all serial (COM) ports on a Windows machine. Here I want to focus on one that seems to work very well on the computers I tried, and incidentally it's also the simplest to implement.</p>
<div class="section" id="digging-into-the-registry">
<h4>Digging into the registry</h4>
<p>The list of serial ports on the machine appears under the <tt class="docutils literal"><span class="pre">HARDWARE\\DEVICEMAP\\SERIALCOMM</span></tt> key in the Windows registry. You can use the <tt class="docutils literal"><span class="pre">regedit</span></tt> tool to reach this key and make sure the serial ports defined on your machine are there (including the <a class="reference external" href="https://eli.thegreenplace.net/2009/07/30/setting-up-python-to-work-with-the-serial-port/">virtual ports</a> created with com0com or another emulator).</p>
<p>Reading the registry in Python is very simple using the <tt class="docutils literal"><span class="pre">_winreg</span></tt> module:</p>
<div class="highlight"><pre><span style="color: #00007f; font-weight: bold">import</span> <span style="color: #00007f">_winreg</span> <span style="color: #00007f; font-weight: bold">as</span> <span style="color: #00007f">winreg</span>
<span style="color: #00007f; font-weight: bold">import</span> <span style="color: #00007f">itertools</span>
<span style="color: #00007f; font-weight: bold">def</span> <span style="color: #00007f">enumerate_serial_ports</span>():
<span style="color: #7f007f">""" Uses the Win32 registry to return an</span>
<span style="color: #7f007f"> iterator of serial (COM) ports</span>
<span style="color: #7f007f"> existing on this computer.</span>
<span style="color: #7f007f"> """</span>
path = <span style="color: #7f007f">'HARDWARE\\DEVICEMAP\\SERIALCOMM'</span>
<span style="color: #00007f; font-weight: bold">try</span>:
key = winreg.OpenKey(winreg.HKEY_LOCAL_MACHINE, path)
<span style="color: #00007f; font-weight: bold">except</span> WindowsError:
<span style="color: #00007f; font-weight: bold">raise</span> IterationError
<span style="color: #00007f; font-weight: bold">for</span> i <span style="color: #0000aa">in</span> itertools.count():
<span style="color: #00007f; font-weight: bold">try</span>:
val = winreg.EnumValue(key, i)
<span style="color: #00007f; font-weight: bold">yield</span> <span style="color: #00007f">str</span>(val[<span style="color: #007f7f">1</span>])
<span style="color: #00007f; font-weight: bold">except</span> EnvironmentError:
<span style="color: #00007f; font-weight: bold">break</span>
</pre></div>
<p>There are other methods for listing the serial ports, but for the time being I'll stick to this one.</p>
</div>
<div class="section" id="port-names">
<h4>Port names</h4>
<p>As I've mentioned in the <a class="reference external" href="https://eli.thegreenplace.net/2009/07/30/setting-up-python-to-work-with-the-serial-port/">last post</a>, Windows requires some port names to appear in a "full" form prepended by several special characters. AFAIK these are all the ports except <tt class="docutils literal"><span class="pre">COM[1..9]</span></tt>. Here's a simple function for generating a "full name" suitable for passing to PySerial:</p>
<div class="highlight"><pre><span style="color: #00007f; font-weight: bold">import</span> <span style="color: #00007f">re</span>
<span style="color: #00007f; font-weight: bold">def</span> <span style="color: #00007f">full_port_name</span>(portname):
<span style="color: #7f007f">""" Given a port-name (of the form COM7,</span>
<span style="color: #7f007f"> COM12, CNCA0, etc.) returns a full</span>
<span style="color: #7f007f"> name suitable for opening with the</span>
<span style="color: #7f007f"> Serial class.</span>
<span style="color: #7f007f"> """</span>
m = re.match(<span style="color: #7f007f">'^COM(\d+)$'</span>, portname)
<span style="color: #00007f; font-weight: bold">if</span> m <span style="color: #0000aa">and</span> <span style="color: #00007f">int</span>(m.group(<span style="color: #007f7f">1</span>)) < <span style="color: #007f7f">10</span>:
<span style="color: #00007f; font-weight: bold">return</span> portname
<span style="color: #00007f; font-weight: bold">return</span> <span style="color: #7f007f">'\\\\.\\'</span> + portname
</pre></div>
<p>While the "simple" name returned by <tt class="docutils literal"><span class="pre">enumerate_serial_ports</span></tt> is readable and suitable for display to the user, only the full name can be passed to <tt class="docutils literal"><span class="pre">serial.Serial</span></tt> for opening the port.</p>
</div>
<div class="section" id="a-simple-gui-for-listing-the-ports">
<h4>A simple GUI for listing the ports</h4>
<p>Finally, I've coded a simple PyQt based GUI for listing the available ports and allowing the user to try and open them (even if the port appears as available on a computer, it may be in use by another program - so opening it will fail).</p>
<img src="https://eli.thegreenplace.net/images/2009/07/listserialports.png" />
<p>Here's the code. It assumes that the <tt class="docutils literal"><span class="pre">enumerate_serial_ports</span></tt> and <tt class="docutils literal"><span class="pre">full_port_name</span></tt> functions were placed in a module called <tt class="docutils literal"><span class="pre">serialutils</span></tt> somewhere on the Python path.</p>
<p>Also, note how the simple port names from <tt class="docutils literal"><span class="pre">enumerate_serial_ports</span></tt> are shown in the list widget, and when trying to actually open the port, the program converts them into full names "under the hood". This is a common practice in serial-port programming.</p>
<div class="highlight"><pre><span style="color: #7f007f">"""</span>
<span style="color: #7f007f">Lists the serial ports available on the</span>
<span style="color: #7f007f">(Windows) computer.</span>
<span style="color: #7f007f">Eli Bendersky (eliben@gmail.com)</span>
<span style="color: #7f007f">License: this code is in the public domain</span>
<span style="color: #7f007f">"""</span>
<span style="color: #00007f; font-weight: bold">import</span> <span style="color: #00007f">sys</span>
<span style="color: #00007f; font-weight: bold">from</span> <span style="color: #00007f">PyQt4.QtCore</span> <span style="color: #00007f; font-weight: bold">import</span> *
<span style="color: #00007f; font-weight: bold">from</span> <span style="color: #00007f">PyQt4.QtGui</span> <span style="color: #00007f; font-weight: bold">import</span> *
<span style="color: #00007f; font-weight: bold">import</span> <span style="color: #00007f">serial</span>
<span style="color: #00007f; font-weight: bold">from</span> <span style="color: #00007f">serial.serialutil</span> <span style="color: #00007f; font-weight: bold">import</span> SerialException
<span style="color: #00007f; font-weight: bold">from</span> <span style="color: #00007f">serialutils</span> <span style="color: #00007f; font-weight: bold">import</span> full_port_name, enumerate_serial_ports
<span style="color: #00007f; font-weight: bold">class</span> <span style="color: #00007f">ListPortsDialog</span>(QDialog):
<span style="color: #00007f; font-weight: bold">def</span> <span style="color: #00007f">__init__</span>(<span style="color: #00007f">self</span>, parent=<span style="color: #00007f">None</span>):
<span style="color: #00007f">super</span>(ListPortsDialog, <span style="color: #00007f">self</span>).__init__(parent)
<span style="color: #00007f">self</span>.setWindowTitle(<span style="color: #7f007f">'List of serial ports'</span>)
<span style="color: #00007f">self</span>.ports_list = QListWidget()
<span style="color: #00007f">self</span>.tryopen_button = QPushButton(<span style="color: #7f007f">'Try to open'</span>)
<span style="color: #00007f">self</span>.connect(<span style="color: #00007f">self</span>.tryopen_button, SIGNAL(<span style="color: #7f007f">'clicked()'</span>),
<span style="color: #00007f">self</span>.on_tryopen)
layout = QVBoxLayout()
layout.addWidget(<span style="color: #00007f">self</span>.ports_list)
layout.addWidget(<span style="color: #00007f">self</span>.tryopen_button)
<span style="color: #00007f">self</span>.setLayout(layout)
<span style="color: #00007f">self</span>.fill_ports_list()
<span style="color: #00007f; font-weight: bold">def</span> <span style="color: #00007f">on_tryopen</span>(<span style="color: #00007f">self</span>):
cur_item = <span style="color: #00007f">self</span>.ports_list.currentItem()
<span style="color: #00007f; font-weight: bold">if</span> cur_item <span style="color: #0000aa">is</span> <span style="color: #0000aa">not</span> <span style="color: #00007f">None</span>:
fullname = full_port_name(<span style="color: #00007f">str</span>(cur_item.text()))
<span style="color: #00007f; font-weight: bold">try</span>:
ser = serial.Serial(fullname, <span style="color: #007f7f">38400</span>)
ser.close()
QMessageBox.information(<span style="color: #00007f">self</span>, <span style="color: #7f007f">'Success'</span>,
<span style="color: #7f007f">'Opened %s successfully'</span> % cur_item.text())
<span style="color: #00007f; font-weight: bold">except</span> SerialException, e:
QMessageBox.critical(<span style="color: #00007f">self</span>, <span style="color: #7f007f">'Failure'</span>,
<span style="color: #7f007f">'Failed to open %s:\n%s'</span> % (
cur_item.text(), e))
<span style="color: #00007f; font-weight: bold">def</span> <span style="color: #00007f">fill_ports_list</span>(<span style="color: #00007f">self</span>):
<span style="color: #00007f; font-weight: bold">for</span> portname <span style="color: #0000aa">in</span> enumerate_serial_ports():
<span style="color: #00007f">self</span>.ports_list.addItem(portname)
<span style="color: #00007f; font-weight: bold">if</span> __name__ == <span style="color: #7f007f">"__main__"</span>:
app = QApplication(sys.argv)
form = ListPortsDialog()
form.show()
app.exec_()
</pre></div>
<p>This runs with ActivePython 2.5.2, PyQt 4.4.3, and the latest PySerial.</p>
<p>Some utility code for this post can be found in <a href="https://github.com/eliben/code-for-blog/tree/master/2009/eblib">this
directory.</a></p>
</div>
</div>
Setting up Python to work with the serial port2009-07-30T07:21:59-07:002022-10-04T14:08:24-07:00Eli Benderskytag:eli.thegreenplace.net,2009-07-30:/2009/07/30/setting-up-python-to-work-with-the-serial-port
<div class="section" id="bla">
<p>The serial port (a.k.a. COM-port, or RS232) is slowly disappearing from desktop and laptop computers, but it's still a very important tool for embedded engineers and people controlling equipment in general. The reason for this is that the serial port is very simple to set-up and use both …</p></div>
<div class="section" id="bla">
<p>The serial port (a.k.a. COM-port, or RS232) is slowly disappearing from desktop and laptop computers, but it's still a very important tool for embedded engineers and people controlling equipment in general. The reason for this is that the serial port is very simple to set-up and use both from embedded software code (the vast majority of MCUs and controllers have UART ports) and from a PC.</p>
<p>A few years ago I've written several posts about working with the serial port using Perl and C++ on a Windows PC.</p>
<p>Since last year, having moved to Python, I've been happily employing the excellent <a class="reference external" href="http://pyserial.sourceforge.net/">PySerial</a> module to do this. I want to write a couple of posts on combining PySerial with GUIs and dynamic charts, but first I'll have to explain how to set it up.</p>
<div class="section" id="installing-a-com-port-emulator">
<h4>Installing a COM-port emulator</h4>
<p>As I've mentioned, it's hard to come by a PC with real COM ports these days. Although there are many <a class="reference external" href="http://www.usbserial.com/">USB to RS-232</a> adapters on the market, the simplest thing to do if all you have is a single computer is installing an emulator. One that I recommend is called <a class="reference external" href="http://com0com.sourceforge.net/">com0com</a>. It is simple to install and works pretty well.</p>
<p>So, download com0com and follow the installation instructions carefully to create two virtual COM ports which are connected to each other. For me com0com created ports named CNCA0 and CNCB0 - in future versions the names may be different, but it really doesn't matter.</p>
<p>Make sure to read the Q&A section in the README file of com0com. For instance, I had turned on buffer overrun emulation because I didn't want the sender to hang when no receiver is connected at the paired port.</p>
<p>At this stage if you're familiar with HyperTerminal, you can check that the ports are indeed working and connected to each other <a class="footnote-reference" href="#id6" id="id1">[1]</a>.</p>
</div>
<div class="section" id="installing-pyserial">
<h4>Installing PySerial</h4>
<p>Installing <a class="reference external" href="http://pyserial.sourceforge.net/">PySerial</a> is very easy either by using <tt class="docutils literal"><span class="pre">easy_install</span></tt> or by downloading the source package and installing it with <tt class="docutils literal"><span class="pre">python</span> <span class="pre">setup.py</span> <span class="pre">install</span></tt>. <tt class="docutils literal"><span class="pre">PySerial</span></tt> doesn't require compilation on Windows.</p>
</div>
<div class="section" id="a-simple-sender">
<h4>A simple sender</h4>
<p>Here's a simple sender script that opens a port, sends the string <tt class="docutils literal"><span class="pre">hello</span></tt>, and closes it:</p>
<div class="highlight"><pre><span style="color: #00007f; font-weight: bold">import</span> <span style="color: #00007f">serial</span>
port = <span style="color: #7f007f">"\\\\.\\CNCB0"</span>
ser = serial.Serial(port, <span style="color: #007f7f">38400</span>)
x = ser.write(<span style="color: #7f007f">'hello'</span>)
ser.close()
</pre></div>
<p>Note the port name. At least on Windows, when the serial port name is not one of <tt class="docutils literal"><span class="pre">COM[1..9]</span></tt>, you'll have to use this funny quoting to make it work <a class="footnote-reference" href="#id7" id="id3">[2]</a>.</p>
<p>This sender can be tested by either opening HyperTerminal on port CNCA0 prior to running it (but make sure to set up the baudrate to 38400, parity to None, stop bits to 1 and hardware control to None) <a class="footnote-reference" href="#id8" id="id4">[3]</a>.</p>
</div>
<div class="section" id="a-non-blocking-receiver">
<h4>A non-blocking receiver</h4>
<p>The beauty of PySerial is that it allows non-blocking reads from the serial port in a very simple way <a class="footnote-reference" href="#id9" id="id5">[4]</a>. The following is a simple non-blocking receiver that listens on a port and prints out received data:</p>
<div class="highlight"><pre><span style="color: #00007f; font-weight: bold">import</span> <span style="color: #00007f">serial</span>
<span style="color: #00007f; font-weight: bold">from</span> <span style="color: #00007f">time</span> <span style="color: #00007f; font-weight: bold">import</span> sleep
port = <span style="color: #7f007f">"\\\\.\\CNCA0"</span>
ser = serial.Serial(port, <span style="color: #007f7f">38400</span>, timeout=<span style="color: #007f7f">0</span>)
<span style="color: #00007f; font-weight: bold">while</span> <span style="color: #00007f">True</span>:
data = ser.read(<span style="color: #007f7f">9999</span>)
<span style="color: #00007f; font-weight: bold">if</span> <span style="color: #00007f">len</span>(data) > <span style="color: #007f7f">0</span>:
<span style="color: #00007f; font-weight: bold">print</span> <span style="color: #7f007f">'Got:'</span>, data
sleep(<span style="color: #007f7f">0.5</span>)
<span style="color: #00007f; font-weight: bold">print</span> <span style="color: #7f007f">'not blocked'</span>
ser.close()
</pre></div>
<p>What this does is basically sleep for half a second, and then check if new data has arrived on the serial port. If it has (the length of received data is not 0), the data is printed out. During the loop the receiver keeps printing <tt class="docutils literal"><span class="pre">not</span> <span class="pre">blocked</span></tt> to demonstrate that the <tt class="docutils literal"><span class="pre">serial.read</span></tt> call isn't blocking.</p>
</div>
<div class="section" id="closing-the-loop">
<h4>Closing the loop</h4>
<p>Make sure HyperTerminal is not running. Run the receiver, and then in a separate window run the sender. You'll get something like the following output:</p>
<div class="highlight"><pre>not blocked
not blocked
not blocked
not blocked
Got: hello
not blocked
not blocked
not blocked
</pre></div>
<p>The amount of <tt class="docutils literal"><span class="pre">not</span> <span class="pre">blocked</span></tt> messages depends on how long the receiver ran before and after you ran the sender.</p>
<img src="https://eli.thegreenplace.net/images/hline.jpg" />
<table class="docutils footnote" frame="void" id="id6" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>It's worthwhile to learn how to use HyperTerminal if you're working with serial ports. HT is a convenient tool for "sanity checking" of your computer's ports and the programs you develop to communicate.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id7" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id3">[2]</a></td><td>Note that half of the backslashes are there to quote other backslashes. Using Python's raw strings it's definitely possible to write the port name as <tt class="docutils literal"><span class="pre">r"\\.\CNCB0"</span></tt>, but I'm keeping the longer syntax for compatibility with standard notation and C code.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id8" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id4">[3]</a></td><td>I'm not just pulling these out of a sleeve. These settings are the defaults of PySerial.</td></tr>
</tbody>
</table>
<table class="docutils footnote" frame="void" id="id9" rules="none">
<colgroup><col class="label" /><col /></colgroup>
<tbody valign="top">
<tr><td class="label"><a class="fn-backref" href="#id5">[4]</a></td><td>Non-blocking reads are very important in interactive applications - you surely don't want to hang your GUI until a new piece of data arrives on the serial port.</td></tr>
</tbody>
</table>
</div>
</div>
null-modem, physical and virtual COM ports2006-11-18T11:41:17-08:002022-10-04T14:08:24-07:00Eli Benderskytag:eli.thegreenplace.net,2006-11-18:/2006/11/18/null-modem-physical-and-virtual-com-ports
When we connect two devices (say a PC and some embedded board, or two PCs) with a serial cable (RS232) we use a female female cross-cable. Apparently this is called a "null modem" connection - and if one doesn't have a cross-cable, one can use a special null modem adapter. <a href="http://en.wikipedia.org/wiki/Null_modem">This …</a>
When we connect two devices (say a PC and some embedded board, or two PCs) with a serial cable (RS232) we use a female female cross-cable. Apparently this is called a "null modem" connection - and if one doesn't have a cross-cable, one can use a special null modem adapter. <a href="http://en.wikipedia.org/wiki/Null_modem">This wikipedia article</a> provides a nice overview of the subject.
If you just want to hack on some serial port code on your computer without actually connecting anything, there are also virtual null modem drivers that emulate serial ports on the PC and can "connect them together". For instance, such a driver can be used to create two new ports, say COM13 and COM14 and "connect them". Then, to applications it seems that ports COM13 and COM14 really exist on the computer, and if one application sends something to COM13 then the application listening at COM14 will receive it, and vice versa.
A couple of free programs to allow this are: <a href="http://developer.berlios.de/projects/n8vbvcomdriver/">N8VB vCOM</a> and <a href="http://sourceforge.net/projects/com0com">com0com</a>.
once again: perl, serial ports and what's between them2006-02-13T22:06:00-08:002022-10-04T14:08:24-07:00Eli Benderskytag:eli.thegreenplace.net,2006-02-13:/2006/02/13/once-again-perl-serial-ports-and-whats-between-them
<p>As I wrote before, I managed to avoid using the Win32::SerialPort module for sending data to the serial port (COM) by employing a C++ slave program that listens to a socket and transmits whatever it gets to the serial port.</p>
<p>Today, I finally perfected this scheme. Now my C …</p>
<p>As I wrote before, I managed to avoid using the Win32::SerialPort module for sending data to the serial port (COM) by employing a C++ slave program that listens to a socket and transmits whatever it gets to the serial port.</p>
<p>Today, I finally perfected this scheme. Now my C++ program (104 Kb !) is a full-duplex serial port <-> socket bridge. It is created by the Perl 'master' script and starts listening to the socket and to the serial port. When it gets something from the socket (sent by the Perl script) it transmits it to the serial port. When it gets something from the serial port, it sends it to the Perl script using the socket.</p>
<p>The Perl script can now either send serial data by shoving it into the socket or query it, in a convenient non-blocking way (thank you ioctl(), see <a href="http://www.perlmonks.org/?node_id=529812" rel="nofollow">this</a>), to see if anything new has arrived.</p>
<p>The C++ bridge works in two threads. One waits (using recv()) on the socket to get data from the master and transmit it to the serial port. The other waits on the serial port and transmits what it gets into the socket.</p>
<p>For now I'm actually doing this using two sockets, one for each way (master -> slave and slave -> master). In theory this is doable with a single bi-directional socket, though I may leave it as it is to allow future flexibility.
</p>
Update (05.03.2008): I've released this publicly.
Here's a <a href="https://github.com/eliben/code-for-blog/tree/master/2005/perl_serial_comm">direct ink</a>.
perl master, C++ slave, bound for serial port programming2005-12-04T22:07:00-08:002022-10-04T14:08:24-07:00Eli Benderskytag:eli.thegreenplace.net,2005-12-04:/2005/12/04/perl-master-c-slave-bound-for-serial-port-programming
<p><b>Introduction</b></p>
<p>I wrote about this topic *SO* much before, but hey, I work with this a lot. Using the PC's serial (COM) port is very useful when working with embedded hardware & software. It's an excellent way to control the hardware from the PC, to gather and plot data, and to …</p>
<p><b>Introduction</b></p>
<p>I wrote about this topic *SO* much before, but hey, I work with this a lot. Using the PC's serial (COM) port is very useful when working with embedded hardware & software. It's an excellent way to control the hardware from the PC, to gather and plot data, and to generally impress your EE co-workers who don't always have an intuitive understanding of how I turn relays on and off using "this Perl program".</p>
<p>One of my best creations in this area is a light-weight but powerful serial port monitor (that can monitor and log time-tagged data from several ports simultaneously), written in C++, following a prototype I wrote in Perl.</p>
<p>But receiving data is not always enough. It's great to be able to plot all my A2D samples from the FPGA in an Excel chart, but the other direction is also important - sending data from the PC to the custom hardware.</p>
<p>For that, on my PC I employ a nice Perl script which uses Win32::SerialPort. But since that module is difficult to install on other PC's, I came up with a solution
to run scripts in a "hosted environment" created with PAR.</p>
<p>However, this method is also not completely satisfactory & flexible enough. Moreover, the PARed executable is a heavy 3 MB which is really inconvenient in some cases.</p>
<p><b>Solution</b></p>
<p>As I mentioned before, the awful Win32::SerialPort module makes it very difficult to work with serial ports in Perl. Until I find the time to write something better, I have real problems to solve at work, so I must pick the best tool for the job. And for this job, the best tool is C++, using the excellent CSerial library which is readily available on the web (LGPL).</p>
<p>On the other hand, general data munging is much nicer with Perl and complex binary data is especially easily managed with pack & unpack. Besides, Perl scripts are easier to change and need no compilation, thus making them generally more convenient to use for "quick tries", which is very important.</p>
<p>So today I finally decided to merge the good from the two worlds - C++'s ability to gracefully handle serial port communications, and Perl's general usability and productivity. The result is a very nice hack, which at its base is very simple.</p>
<p><b>The slave</b></p>
<p>The slave is a small C++ program that starts, receives the serial port's information (name, baudrate, parity, etc.) from the command line, dutifully opens a socket on port 14441 of the localhost (ip 127.0.0.1), using Win32's WinSock library, sends a notification that it's ready to receive data and waits. When it gets a data buffer, it transmits it to the serial port it's tied to.</p>
<p><b>The master</b></p>
<p>The master is a Perl script that runs the slave as a child process (using Win32::Process), and listens on port 14441 (using IO::Socket::INET), waiting for the slave to connect. When the slave connects and says it's ready, the master sends it a buffer of data to transmit to the serial port.</p>
<p><b>Conclusion</b></p>
<p>A happy couple - a 90 Kb executable (slave) + the master script now do anything the more bulky PARed script did, and with much area for future improvement and flexibility (because the C++ CSerial class is so nice to work with). It a good experience of mixing many things together (serial communications, processes, sockets), and I learned once again that sockets are a great IPC technique, even when no more than the local host is needed. But the main conclusion for today is:</p>
<p>Use the right tool for the job !
</p>
<strong>Update:</strong> I've finally packed the code of what I'm describing here for distribution. <a href="https://github.com/eliben/code-for-blog/tree/master/2005/perl_serial_comm">Here it is</a>.