A “live” data monitor with Python, PyQt and PySerial

August 7th, 2009 at 2:57 pm

The previous couple of posts about the PySerial module for serial communications with Python were just a basic introduction. Let’s now see something much more useful.

PySerial makes Python a great tool for serial communications from a computer, as it can be easily combined with other powerful Python libraries to create full-scale applications. In my case, I’m using PyQt with its plotting supplementary PyQwt to create nice "live" plotting applications, that can be combined with the serial port. Here’s a demo:

Download plotting_data_monitor.zip – it’s a small demo application written in Python that requires the following modules to be installed on your machine:

  • PyQt
  • PyQwt
  • PySerial

What does it do? Well, it basically shows how to combine all these powers of Python into a single application in a simple way. You can choose a serial port and then run the monitor (all via the menu). If another program is sending data to the specified port, you’ll see the plot updating "in real time":

http://eli.thegreenplace.net/wp-content/uploads/2009/08/datamonitor_shot.png

If you have nothing sending data to your machine, no worries. If you’ve installed com0com or a similar virtual port emulator like I explained here, configure it to connect two ports together.

Then, download sender_sim.py which is a very simple data-sending script (the data it provides is a pleasant pseudo-randomized sinusoid). You may want to change the port name hard-coded in it, if your port numbers are different.

When both this sender and the monitor run on the same machine, you’ll be able to see the live plotting. Note that I’ve added a couple of extra features from PyQwt:

  • A "thermo" bar that shows the average temperature
  • A knob that sets how often the monitor updates the screen

These widgets, and a few others, make PyQwt quite a nice library for emulating Labview-type "lab control" applications in Python. I recommend it highly.

How does it work

The monitor is a toy demo, but it’s based on a few powerful tools I use for real applications. For example, the serial communication itself is performed in a separate thread (in the com_monitor module). The thread issues blocking reads on the serial port in a loop, and communicates the data to the GUI via a Queue (together with an accurate timestamp for every data chunk received). This is a robust and safe implementation that can be used for many kinds of GUI-based monitoring applications.

The GUI itself is updated using a periodic timer (QTimer) that runs whenever the monitor is running. The timer event handler checks whether new data has arrived from the monitor and updates the display widgets accordingly.

The rest of the code is your usual PyQt bureaucracy – creating the menu, the status bar, the GUI widgets, laying everything out nicely and connecting events.

I hope people will find this code useful. If you have trouble running it or understanding how it works, let me know. I recommend using the latest Python 2.6 versions of all the required modules. I checked it only on Windows, but there’s no reason whatsoever for it not to run on other OSes out of the box.

Related posts:

  1. More PyQt plotting demos
  2. Plotting in Python: matplotlib vs. PyQwt
  3. Setting up Python to work with the serial port
  4. matplotlib with PyQt GUIs
  5. once again: perl, serial ports and what’s between them

55 Responses to “A “live” data monitor with Python, PyQt and PySerial”

  1. BlaineNo Gravatar Says:

    Thanks so much for your helpful posts regarding matplotlib, pyQt, and qwt. This post finally tops the cake. Keep up the good work, and thank you again for your very helpful write-ups.

  2. Dave BehaveNo Gravatar Says:

    feedback: excellent post. valuable information presented very clearly. think seriously about writing a book.
    question:

    using QTimer to update the GUI 10 times/second provides acceptable responsiveness.

    call it OCD, but i would like to think of another way to interface between python Queues and PyQt. a way that is event driven. For example, when the Queue is ‘gettable’ then a Qt signal would be generated.

    It isnt just OCD. there are objective reasons for wanting PyQt to respond to events. For example, if the main thread (which runs the gui) also holds a complex data model, that must react to serial events as the occur (or else the device on the other side will block, or worse, enter an error state)

    I was thinking of using QSocketNotifier. But this has the disadvantage of coupling the GUI with communications medium.

    Can you think of a better way to interface python queues with Qt event driven flow?

    However, if the GUI thread is also managing a complex data model that must act on events (that is

    do you have another method for using python Queues but not

  3. BrunoNo Gravatar Says:

    Looking for the quickest possible way to visualize ADC-data from my Atmega8 I found this very helpfull example. Although a complete idiot with python I’m sometimes able to change existing code to serve my needs. But this time I’m stuck with something I can’t figure out:
    How can I add two simple buttons which send “x” and “y” via the already open Com-port (or one “start”-button which sends “x”, changes from “start” to “stop” and sends “y” the next time it’s pressed)?
    I know it’s a real beginners question, but I have to admitt honestly that I don’t have time to work my way trough thy myriads of tutorials. I need a working solution for a study project and time is running out faster than I can study, so I would be deeply grateful if you could help me out.

  4. elibenNo Gravatar Says:

    @Bruno,

    The current code sample doesn’t support sending data back through the port very easily, and the ComMonitorThread class should be modified for this to happen.

    To add new buttons to the GUI you’ll have to learn some basics of PyQt – adding buttons is very easy. You can find other related code in my blog with relevant code samples, or use the code samples coming with PyQt.

  5. LuisNo Gravatar Says:

    This is my first visit to your blog, and I have to congratulate you as it is very clean and elegantly styled, and the information you post in here can’t possibly be more helpful, relevant and well-presented. I’m a beginner in Python, coming from C++ and Matlab, and have found my introduction to the language very pleasing with these kind of examples. Keep up the good work!

  6. elibenNo Gravatar Says:

    @Luis, thanks for the kind words.

  7. the signal jammerNo Gravatar Says:

    OK thank you that helps quite a bit actually.

  8. Linux_UserNo Gravatar Says:

    Nice app! However, apparently it requires Windows to run – would it be possible to strip the current version from all Windows-related code and publish a second version for Linux? Thanks!

  9. elibenNo Gravatar Says:

    Linux_User,

    Why do you say that it requires Windows to run? As long as you install all its pre-requisites on Linux, I see no reason for it not to work there.

  10. LuisNo Gravatar Says:

    I think it has to do with the implementation of the pyserial convenience wrapper, as it looks in the windows registry for available serial ports, as far as I have read the code. But it might well be a good exercise for Linux_User to port that code to linux, with the help of the examples page of the PySerial documentation.

  11. Data LoggersNo Gravatar Says:

    Great insights,Thanks for the post. I have used pyserial in the past but this is my first experience with
    pyQt. I am using the Python xy package for windows current but might move to Linux. I have a small device that is outputting a basic text string. Maybe your suggestions will help me more to have answers to my questions.

  12. icenovNo Gravatar Says:

    This is very nice code, but I am having difficulty working out how to plot my own data. If I have a string on the serial port, how do I get this into the script? I’m OK to extract the right part of the string, but can’t see how to get it into the script. Any help would be appreciated.

  13. ThijsNo Gravatar Says:

    Thanks a lot for publishing this, I modified this to make a serial datalogging system for some oceanographic equipment and now it’s already been used by some colleagues at another university! This was my first major project with Python and I absolutely fell for it.

    If I may make one suggestion: when plotting much data (I’m displaying 29 lines in real time) that contain a lot of noise, the frame rate of updating the plot slows down a lot (less than 0.5 frames / second). I found that disabling the antialiasing of the line makes the code run faster. It makes the plots look less slick, but when you want speed and simple data-display you don’t really care about this.

  14. Greg HorlerNo Gravatar Says:

    Hi,

    A very interesting article, however I have been unable to load PYQT and get it running on my mac. I have Python 27 installed, but I can’t get the PyQT4 to work. Do you have any advice please? Also is there a gGUI designer for PYQT,PUQWT?

    Also, what level of data sample update rates do you expect using these tools. I need to design a minimalist rs232 data-logger with a fast visual graphing facility. You seem to be able to comment authoritatively on this.

    Any assistance warmly received,Thanks

    Greg

  15. elibenNo Gravatar Says:

    Greg,

    I’m no authority on PyQt installation, certainly on a Mac. You will have better luck asking in PyQt’s mailing list. As for update rates, I’m not sure I understand you question fully. Do you mean RS-232 port speed? Screen update speed?

  16. alNo Gravatar Says:

    Thanks alot for this it is elegant.

    Can you give me some tips on where to start with modifying the ComMonitorThread class so I can incorporatie sending data to the port whilst non blocking reads are happening?

    Background prior to seeing you code I originally created some Python scripts that tells a connected device to start streaming data then parses and plotts that data. Being a beginner at python I couldnt get the threading working so I had one script write the data to a file and a separate script read the file and plot. Messy, but workable.

    I would like to convert this to your method as performance will be better in a separate thread but Im unsure how to send the data I need to setup the device in streaming mode before I start listening.
    Cheers
    Al

  17. elibenNo Gravatar Says:

    al,

    In these cases I usually have another Queue with the thread for sending stuff. The thread then polls this queue and if there’s new stuff in it to send, it sends it. Note that AFAIK this can’t be done while the port is blocking on a read.

  18. alNo Gravatar Says:

    Eli,

    Ive worked it out now.

    Ive modified your code to produce 2 graphs from a medical device that is streaming data at 256 packets a second. Instead of utilising another queue as you suggested for sending extra instructions I found the following method easier.

    Each time I need to send additional instructions to the device I just stop the stream, send the commands, and restart the stream. This makes the comms simplex not duplex but easy for a python beginner like me to implement. Each time the stream is started a folder is created with the name of year_month_day_time and the raw data is saved to a file in that folder.

    Currently your modified code is being used in an FDA trial. Much neater then the usual Labview solution I have had to contend with. Thanks again much appreciated.

    A question I was pondering
    Would it be possible to run your live update script remotely so that it can plot results from a local device? In this way you have complete control when you wish to update the script.

    What I mean is
    -A device is attached to a computer without python installed
    -This computer calls a website that contains a script like your live data monitor script that can access the local com port and produce your temp graph. I realise the com port data would travel to the web server and back to be graphed. Wouldnt matter for low update speeds Im more interested in what would be needed to achieve it.

    Would this be done by CGI and/or qtwebkit??

  19. elibenNo Gravatar Says:

    al,

    I’m happy you figured out your problem.

    Regarding the question on running it remotely. I see a few options:

    1. Package the monitor into a self-contained EXE (using tools like PyInstaller or py2exe) and place it on all monitoring machines (no Python has to be installed)

    2. I don’t think there’s a way to access the local machine’s serial port from the browser (Javascript) directly, you need some kind of bridge application. Luckily it’s not hard to do. I once implemented such a bridge from a serial port to a socket, written in C++. You can download it here – http://eli.thegreenplace.net/files/prog_code/perl_serial_comm.zip – note this is old code I don’t provide support for. You can modify it to send the data to the web (i.e. some sort of REST requests via HTTP, or just connect to another machine on a LAN directly to some known socket, without HTTP)

    Good luck

  20. LuisNo Gravatar Says:

    Hello Eli,

    I was coding a more specific data logging application on top of your ebutils library, when I happened to find an issue: You raise an InterationError exception within the enumerate_serial_ports function in the serialutils module, but this exception does not exist in the Standard Library neither is defined elsewhere in the code. Perhaps did you mean to use StopIteration? This exception was triggered when running the code on a Windows machine that had no serial ports at that moment.

    Thanks!

  21. elibenNo Gravatar Says:

    Luis,

    Thanks, fixed.

  22. LuisNo Gravatar Says:

    Eli, did you change the zip package? I just checked and IterationError is still there… thanks!

  23. LuisNo Gravatar Says:

    My bad, it is updated. Thanks again!

  24. ManniNo Gravatar Says:

    Eli, I must thank You for the great work you have done.
    Being new to the Python language helps a lot to have some good working samples to start off with.

    About the question why the code shouldn’t run on Linux,
    I can confirm there are some issues:

    1. the serial port names are just like file names on the form /dev/ttySx and no Windows registry is available to lookup as in your code

    2. emulation of serial ports is indeed possible, but again by other tools than on Win

    3. the time routines are not fully compatible across the platforms making the code to fail.

    I have just finished a working port and some instructions regardiing the serial ports,
    anyone interested ?

    /Manni

  25. elibenNo Gravatar Says:

    Manni,

    Thanks for the feedback. Sure, could you post your code/notes somewhere? If you don’t have a blog, then maybe some paste-bin? Then link to it from a comment here. Thanks in advance

  26. ManniNo Gravatar Says:

    Ok, Eli, have a look at:
    http://pastebin.com/1R57UbKf

    /Manni

  27. elibenNo Gravatar Says:

    Manni,

    Thanks. I hope folks wanting to run the same code on Linux will find your example useful

  28. ChrisNo Gravatar Says:

    This is a great demonstration. I was wondering if you could give me any tips for expanding it to monitor data from multiple devices attached to multiple com ports.

    Thanks!

  29. elibenNo Gravatar Says:

    Chris,

    Yes, that would be very easy. Just instantiate the port-listening thread several times, once for each port you want to listen to. Since it runs in a separate thread, several such listeners can operate simultaneously without any problem.

  30. ArpiNo Gravatar Says:

    Hi,

    I have tried Manni’s version, but i have failed to make it work.
    My setup was an arduino hooked on /dev/ttyACM0 transmitting with 9600 bps. Sending values from a pot (from 0-255 range), one value/line.
    I can select ttyACM0 from drop down menu. But when I start monitoring that interface I get garbage instead of real data.


    Traceback (most recent call last):
    File “plotting_data_monitor.py”, line 250, in on_timer
    self.read_serial_data()
    File “plotting_data_monitor.py”, line 297, in read_serial_data
    temperature=ord(qdata[-1][0]))
    TypeError: ord() expected a character, but string of length 8 found

    and this is what i have in qdata[-1][0] = ‘\x80x\x00\x80x\x00\x80x’ and qdata[-1][-1]=14.808072090148926.

  31. mikeNo Gravatar Says:

    Hello,
    This article was very helpful, but can u explain how I can use this tool to display sensor data from a Zigbee sensor node connected through a usb-serial cable?

    Thanks

  32. EKNo Gravatar Says:

    Hi
    Im making a very similar application that reads and plots data from an MSP430. The application works great if it is run as a normal python script. Im trying to package the application as a windows exe with py2exe . the compilation process exits normally, but im getting an error when I run the exe : ” Import error : no module named serial “.

    Have you seen this before?

    thanks in advance
    EK

  33. elibenNo Gravatar Says:

    EK,

    Yes, I’ve seen plenty of problems with py2exe, which is why back in the day I ended up using PyInstaller for my Windows packages. However, that’s also far from trivial, so you need to do some googling to set up things correctly.

  34. Don OliverNo Gravatar Says:

    Hi,
    As a newcomer to Py4Qwt, I have downloaded a few examples, which work fine. Using Windows 7
    Your project is just what I have been looking for, as a way to learn how to graph from serial input.
    When I try to run it, however, from the same directory as files that do work, nothing happens, not even an error message.
    I know that this may be not be enough information, but any suggestions would be much appreciated.
    TIA
    Don Oliver

  35. Don OliverNo Gravatar Says:

    Please ignore/delete my above post.
    I realize that it is inappropriate to post such a question here.
    Thanks,
    Don Oliver

  36. waspinatorNo Gravatar Says:

    I’m having trouble getting serial data to read reliably. I made a simple arduino sketch to send data to the monitor as below.

    void setup() {
      Serial.begin(9600);
    }
    
    void loop() {
      Serial.println("thing: 0.234; poo: 123.432; test: 2343");
      delay(1000);
    }

    I then changed some of your code as below:

    full_port_name(str(self.portname.text())),
                9600)
    def read_serial_data(self):
            """ Called periodically by the update timer to read data
                from the serial port.
            """
            qdata = list(get_all_from_queue(self.data_q))
            if len(qdata) > 0:
    
                print "DATA RECIEVED: " + qdata[-1][0]

    but the output is only small fragments of the original message. Any ideas of what I’m doing wrong? Thanks

  37. waspinatorNo Gravatar Says:

    found the problem.

    I had to switch the port_timeout in com_monitor.py from 0.01 to 0.1. Strange, because 0.01 was working fine on an Arduino Uno, but not on an Arduino Nano. Thanks for the demo!

  38. steffanoNo Gravatar Says:

    Hi Eli,

    I need help please I have two sensors outputting count from an Arduino and am using C to code it. have got a 3 digit seven segment display showing the count and am also printing the count on the serial monitor therefore am wondering how I can use my count to plot a graph against time in real time and see the graph updating live on my laptop. Thanks for helping.

    Steffano

  39. waspinatorNo Gravatar Says:

    any ideas on how to modify this to write to serial? Thanks

  40. waspinatorNo Gravatar Says:

    figured out how to write to serial. just add this function to com_monitor.py

    def write_data(self, data):
        self.serial_port.write(data)

    call it from plotting_data_monitor

    self.com_monitor.write_data("Hello\n")
  41. SBNo Gravatar Says:

    Eli, great tutorial, and without having done any Python or Qt, I was able to get to 80% of the destination (seeing live graphing of serial data). Please bear with me now. Python and its framework is seriuosly ****ed up. One version will not work properly with the Qt binding or application of a slightly different version, even when backwards compatibility is assumed and expected. There are dozens of Qwt and Qt packages floating around the internet compiled for very specific python installation, and very specific Qt installation, and it took 5 hours of installing and reinstalling packages, but I still dont understand what is YOUR perfect configuration that worked.

    Here is where I am currently. I installed Python 2.7 and also installed com0com. Pyserial installed correctly, so sending and receiving from the virtual port is working fine. sender_sim.py is working fine. Installed Qt4.8.4 and later PyQt 5 as well sip. I am trying to install PyQwt-5.2, but keep getting the dreadful qmake path messages or python compile errors.

    Here is my request to you: For the working version of live data monitor, please publish the following (with links if you have more energy and time):
    1. Python version
    2. Qt version
    3. PyQt version
    4. SIP version
    5. PyQwt version
    6. Pyserial version

    Thanks!

  42. elibenNo Gravatar Says:

    @SB,

    Yes, the PyQt/PyQwt pair is not trivial to match perfectly. It’s completely not “Python and its framework”, just PyQt. Anyway, I wish I could help but I no longer even have a Windows machine at home :-)

  43. SBNo Gravatar Says:

    @eliben,

    No worries. All I needed was a starting point, and this blog provides one much better than any formal tutorial process. I will figure out the rest over the weekend. Thanks!

  44. ZwilrichNo Gravatar Says:

    @SB,

    The pythonxy distribution at code.google.com/p/pythonxy/ includes PyQt/PyQwt. If you are running Windows XP you need the 2.7.3.1 version which you get from one of the mirrors. The link on the pythonxy download page for 2.7.3.1 is for the update, not the full version.

    I haven’t tried it yet; I’m installing it as I type this.

  45. ZainNo Gravatar Says:

    Hi, I just read your blog and I was surprised to know that something like this existed but I wanted to ask you that is there a way to make it work for plotting sinewaves? I’m working on a project that includes a microcontroller (Beaglebone Black) and I need to make a live plotter like yours for capturing data from the analog inputs of the Bbone. I have been searching for weeks about this and i understand you may not have a windows machine (or if you do, thats great) but can you share some snippets of the code needed for making a sinewave plotter from your existing code or if you can make a new code, that will be fine. I have to present it on Mon and I’m running low on time. Hope to hear from your reply, Thanks

  46. elibenNo Gravatar Says:

    @Zain,

    Obviously the code here can be adapted for any kind of waveform. If the data port will send a sine wave, the application will plot one.

  47. Sebastian NilssonNo Gravatar Says:

    This post inspired me to do something similar but in Processing instead of Python. You can have a look at my project at http://sebastiannilsson.com/en/k/projekt/realtime-plotter/ if you want to.

    Thanks for sharing your code!

  48. HalitNo Gravatar Says:

    Arduino and this app so cool.Thanks!

  49. benNo Gravatar Says:

    This and your post on matplotlib are great!

    Any suggestions for a newbie on developing a serial protocol that can be used on any microcontroller and be used for transferring data to/from a pyQt application?

  50. elibenNo Gravatar Says:

    @ben,

    See: http://eli.thegreenplace.net/2009/08/12/framing-in-serial-communications/

  51. Steven ShamlianNo Gravatar Says:

    Hello,
    Thank you for this post; it’s gotten me bootstrapped on a problem I was trying to solve. I modified your serialutils.py so that it would also work in other OSes (I’m using Linux). Full_port_name needs to not add extra slashes and dots; the simple fix is to check the OS and if it’s not Windows, just return the input. The enumerate_serial_ports function I replaced entirely with something I found here: http://stackoverflow.com/questions/12090503/listing-available-com-ports-with-python . The new serialutils.py looks like this:

    """
    Some serial port utilities for OS compatibility with PySerial (2.6+)
    
    Eli Bendersky (eliben@gmail.com)
     modified by Steven Shamlian (http://www.shamlian.net)
    License: this code is in the public domain
    """
    import re, itertools
    import os, serial
    from serial.tools import list_ports
    
    def full_port_name(portname):
        """ Given a port-name (of the form COM7,
            COM12, CNCA0, etc.) returns a full
            name suitable for opening with the
            Serial class.
        """
        if os.name == 'nt':
            m = re.match('^COM(\d+)$', portname)
            if m and int(m.group(1)) < 10:
                return portname
            return '\\\\.\\' + portname
        else:
            return portname
    
    #replaced with http://stackoverflow.com/questions/12090503/listing-available-com-ports-with-python
    def enumerate_serial_ports():
        """
        Returns a generator for all available serial ports
        """
        if os.name == 'nt':
            # windows
            for i in range(256):
                try:
                    s = serial.Serial(i)
                    s.close()
                    yield 'COM' + str(i + 1)
                except serial.SerialException:
                    pass
        else:
            # unix
            for port in list_ports.comports():
                yield port[0]
    
    if __name__ == "__main__":
        import serial
        for p in enumerate_serial_ports():
            print p, full_port_name(p)
  52. Steven ShamlianNo Gravatar Says:

    Another bug for you:
    In com_monitor, you use time.clock() to time stamp the data coming in off the serial port, but clock() returns processor time, which is related to time the CPU spends on the operation. Because of this, the time stamp can become inaccurate quickly, and will be tied to the update rate (this manifests itself with a lot of data coming into a slow processor). To fix this (and also allow for more than one piece of data coming in during an update, which was also happening with my slow processor), modify the run() function in com_monitor as follows:

    def run(self):
    try:
    if self.serial_port:
    self.serial_port.close()
    self.serial_port = serial.Serial(**self.serial_arg)
    except serial.SerialException, e:
    self.error_q.put(e.message)
    return

    # Save time zero
    startTime = time.time()

    while self.alive.isSet():
    # Reading 1 byte, followed by whatever is left in the
    # read buffer, as suggested by the developer of
    # PySerial.
    #
    data = self.serial_port.read(1)
    data += self.serial_port.read(self.serial_port.inWaiting())

    if len(data) > 0:
    for c in data:
    timestamp = time.time() – startTime
    self.data_q.put((c, timestamp))

    # clean up
    if self.serial_port:
    self.serial_port.close()

  53. Steven ShamlianNo Gravatar Says:

    Let’s try that paste again:

    def run(self):
            try:
                if self.serial_port:
                    self.serial_port.close()
                self.serial_port = serial.Serial(**self.serial_arg)
            except serial.SerialException, e:
                self.error_q.put(e.message)
                return
    
            # Restart the clock
            startTime = time.time()
            print startTime;
    
            while self.alive.isSet():
                # Reading 1 byte, followed by whatever is left in the
                # read buffer, as suggested by the developer of
                # PySerial.
                #
                data = self.serial_port.read(1)
                data += self.serial_port.read(self.serial_port.inWaiting())
    
                if len(data) > 0:
                    for c in data:
                        timestamp = time.time() - startTime
                        print timestamp
                        self.data_q.put((c, timestamp))
    
            # clean up
            if self.serial_port:
                self.serial_port.close()
  54. Steven ShamlianNo Gravatar Says:

    … and you’ll probably want to lose the two debug “print” statements I forgot to remove the second time I pasted in the code. (Sorry. Too much coffee.)

  55. chsNo Gravatar Says:

    how about xonxonff and rtscts setting in com_monitor class…
    it will be required for the real hardware ….?

Leave a Reply

To post code with preserved formatting, enclose it in `backticks` (even multiple lines)