Code sample – socket client based on Twisted with PyQt

May 26th, 2011 at 5:28 am

In an earlier post, I discussed one way of combining blocking socket I/O with a GUI, by means of a separate thread in which the I/O runs. I’ve also mentioned that one alternative is using asynchronous I/O with callbacks integrated into the GUI event loop. Here I want to present some sample code for this alternative.

One of the first things Python programmers will think about then considering asynchronous I/O is Twisted, which is an event-driven networking library written in pure Python. Twisted is a huge library, and it’s quite a job to learn how to use it properly. Here I’ll demonstrate just enough to accomplish the task at hand – re-create the simple socket client from the previous post that works as part of a PyQt GUI.

The full code sample is available for download – socket_client_twisted_pyqt.zip. Here is the socket client class using Twisted:

import struct

from twisted.internet.protocol import Protocol, ClientFactory
from twisted.protocols.basic import IntNStringReceiver


class SocketClientProtocol(IntNStringReceiver):
    """ The protocol is based on twisted.protocols.basic
        IntNStringReceiver, with little-endian 32-bit
        length prefix.
    """
    structFormat = "<L"
    prefixLength = struct.calcsize(structFormat)

    def stringReceived(self, s):
        self.factory.got_msg(s)

    def connectionMade(self):
        self.factory.clientReady(self)


class SocketClientFactory(ClientFactory):
    """ Created with callbacks for connection and receiving.
        send_msg can be used to send messages when connected.
    """
    protocol = SocketClientProtocol

    def __init__(
            self,
            connect_success_callback,
            connect_fail_callback,
            recv_callback):
        self.connect_success_callback = connect_success_callback
        self.connect_fail_callback = connect_fail_callback
        self.recv_callback = recv_callback
        self.client = None

    def clientConnectionFailed(self, connector, reason):
        self.connect_fail_callback(reason)

    def clientReady(self, client):
        self.client = client
        self.connect_success_callback()

    def got_msg(self, msg):
        self.recv_callback(msg)

    def send_msg(self, msg):
        if self.client:
            self.client.sendString(msg)

A couple of notes:

  • Since I want to re-create the same length-prefixed protocol from the previous post, the IntNStringReceiver protocol class from twisted.protocols.basic comes in handy – it’s designed especially for strings prefixed by integer length headers. In our case, the prefix is a 4-byte little-endian unsigned integer, which I specify with the structFormat and prefixLength attributes. In addition I implement a couple of callbacks of the IProtocol interface.
  • SocketClientFactory is a straightforward subclass of Twisted’s ClientFactory, implementing the callbacks we’re interested in here.

This is all quite simple. The bigger problem was finding how to interface Twisted with PyQt. Since Twisted and a typical GUI library are both event-loop based, to make them work together we should use a custom reactor. Unfortunately, due to licensing issues, Twisted doesn’t come with a reactor for PyQt pre-packaged, and it should be obtained separately. Even more unfortunately, the PyQt reactor (qt4reactor.py) doesn’t appear to have a single well-defined address on the web – several slightly different versions of it can be found floating online. In my code sample I’ve included a version of qt4reactor.py which I found to work for my needs.

So, back to the code. This is the implementation of the PyQt client GUI that uses Twisted. Similarly to the thread-based sample, it keeps drawing a nice circle animation to demonstrate it’s never blocked. The full code is in the zip archive, here is only the interesting part:

class SampleGUIClientWindow(QMainWindow):
    def __init__(self, reactor, parent=None):
        super(SampleGUIClientWindow, self).__init__(parent)
        self.reactor = reactor

        self.create_main_frame()
        self.create_client()
        self.create_timer()

    def create_main_frame(self):
        self.circle_widget = CircleWidget()
        self.doit_button = QPushButton('Do it!')
        self.doit_button.clicked.connect(self.on_doit)
        self.log_widget = LogWidget()

        hbox = QHBoxLayout()
        hbox.addWidget(self.circle_widget)
        hbox.addWidget(self.doit_button)
        hbox.addWidget(self.log_widget)

        main_frame = QWidget()
        main_frame.setLayout(hbox)

        self.setCentralWidget(main_frame)

    def create_timer(self):
        self.circle_timer = QTimer(self)
        self.circle_timer.timeout.connect(self.circle_widget.next)
        self.circle_timer.start(25)

    def create_client(self):
        self.client = SocketClientFactory(
                        self.on_client_connect_success,
                        self.on_client_connect_fail,
                        self.on_client_receive)

    def on_doit(self):
        self.log('Connecting...')
        # When the connection is made, self.client calls the on_client_connect
        # callback.
        #
        self.connection = self.reactor.connectTCP(SERVER_HOST, SERVER_PORT, self.client)

    def on_client_connect_success(self):
        self.log('Connected to server. Sending...')
        self.client.send_msg('hello')

    def on_client_connect_fail(self, reason):
        # reason is a twisted.python.failure.Failure  object
        self.log('Connection failed: %s' % reason.getErrorMessage())

    def on_client_receive(self, msg):
        self.log('Client reply: %s' % msg)
        self.log('Disconnecting...')
        self.connection.disconnect()

    def log(self, msg):
        timestamp = '[%010.3f]' % time.clock()
        self.log_widget.append(timestamp + ' ' + str(msg))

    def closeEvent(self, e):
        self.reactor.stop()


#-------------------------------------------------------------------------------
if __name__ == "__main__":
    app = QApplication(sys.argv)

    try:
        import qt4reactor
    except ImportError:
        # Maybe qt4reactor is placed inside twisted.internet in site-packages?
        from twisted.internet import qt4reactor
    qt4reactor.install()

    from twisted.internet import reactor
    mainwindow = SampleGUIClientWindow(reactor)
    mainwindow.show()

    reactor.run()

The most important part of this code is in the last section, where the Twisted reactor and PyQt application are set up. A few steps have to be performed in a careful order:

  • An QApplication is created
  • qt4reactor is imported and installed into Twisted
  • The main window is created
  • Finally, the singleton Twisted reactor (which is actually a qt4reactor, since that’s the one we’ve installed) is run

Note that there’s no app.exec_() here, contrary to what you’d expect from a PyQt program. Since both PyQt and Twisted are based on event loops (in app.exec_() and reactor.run(), respectively), one of them should drive the other. The Twisted way is to let the reactor drive (hence we only call reactor.run() here). Inside its implementation, qt4reactor takes care of running an event loop in a way that dispatches events to both Twisted and PyQt.

Some additional notes:

  • The PyQt main window no longer needs a timer to query results from the client thread. When SocketClientFactory is created in create_client, some methods are passed to it as callbacks. These callbacks will be invoked when interesting events happen.
  • Even though Twisted’s reactor is a global singleton object, it’s good practice to pass it around in the application, instead of importing it from Twisted in multiple places. Here, SampleGUIClientWindow accepts the reactor object in its constructor and uses it later.
  • Twisted’s reactor keeps running until explicitly stopped. The user of a GUI expects the program to exit then the GUI is closed, so I call reactor.stop() in the main window’s closeEvent.

This is it. Once set up, Twisted integrates quite nicely with PyQt. Since both the GUI framework and Twisted are based on the concept of an event loop with callbacks, this is a natural symbiosis.

A final note: I’m very far from being a Twisted expert. In fact, this is the first time I really use it, so may be doing things not in the most idiomatic or optimal way. If you can recommend a better way to implement some parts of this sample, I’ll be very happy to hear about it.

Related posts:

  1. Code sample – socket client thread in Python
  2. Sample using QScintilla with PyQt
  3. New-style signal-slot connection mechanism in PyQt
  4. Boost.Asio with Protocol Buffers code sample
  5. Passing extra arguments to PyQt slots

10 Responses to “Code sample – socket client based on Twisted with PyQt”

  1. GlyphNo Gravatar Says:

    Thanks for posting a Twisted solution to your GUI networking problem! :)

    A few minor notes:

    With the advent of PySide, we may be able to address those licensing issues and have an officially-supported qt reactor again.

    You don’t need to set attributes on your own IntNStringReceiver to get “little-endian 32-bit length prefix” behavior; that’s already in Twisted as Int32StringReceiver.

    Finally, you can save a bit of code by using the ‘endpoints’ connection APIs, which will give you a Deferred that fires with your protocol instance when it’s connected rather than you needing to set up your own clientReady callback.

  2. elibenNo Gravatar Says:

    glyph

    Thank you for the effort invested in Twisted! It’s good to hear about making qt4reactor official.

    Regarding Int32StringReceiver, I definitely saw it, but it’s big-endian (network order) 32-bit integer, while I use a little-endian prefix. Not that I have any particular preference for little-endian, but I just wanted to keep the protocol the same as in the previous sample.

    I will read about endpoints, thanks.

  3. InmakeNo Gravatar Says:

    А мне понравилось!

  4. Jean-YvesNo Gravatar Says:

    Thanks for this solution using twisted and QT

  5. JohannaNo Gravatar Says:

    Hi,
    I am writing a PyQt application which I would like to turn into a client-server app (thiswas not planned at the beginning of the developement). I nerver used Twisted, and in a general way I suck at networking programming. So I guess my question is : is it simple to convert zn existing application to network app using Twisted ? Do you have any good advice apart from your post (which seems to be a good strating point) ?

    Oh, and I am running under Windows 64bits, is it a problem for installing Twisted ?

  6. elibenNo Gravatar Says:

    Johanna,

    Your question is too general. How simple it is to convert an application to Twisted, depends on the application, of course, what would you expect :-) ? I suggest you just get going, and if you run into specific problems, ask questions in the Twisted mailing list or on Stack Overflow. I confess I don’t have much experience with Twisted.

    Re 64-bit Windows, I have no idea. You can always install a 32-bit Python on a Win 64 machine, so Twisted surely can run there. Don’t know about 64-bit Python.

  7. JohannaNo Gravatar Says:

    Thanks for your response.

    About the 64bits issue, I think I’ll just try to run something and ask questions on the mailing list. But it’s not possible to install a 32-Python (not my machine, not my choice..)

    I guess I was just wondering if it was easy to convert something and not start from scratch. But you’re right, I’ll try to get going and see how it turns out !

  8. ScottNo Gravatar Says:

    There is an issue with this application as posted. If you run the application and never hit the “Do It” button but press the window close X the application does not close. I am currently developing an application and I have the same issue.

    If you Cntrl-C the application after pressing the X on the window bar you get the following:
    QObject::killTimers: timers cannot be stopped from another thread
    and application does not exit. Still in a loop somewhere.

    Where is a PyQt4 reactor that works? Meaning you can exit your Qt application. The posted sample above does not allow a close either. I have tried to add a stop method to reactor as well but application will not cleanly exiit.

    I am using the PyQt 4.9.4 which is latest for binary installer. Also using Qt 4.8.1. Some incompatibility with qt4reactor. I have tried launchpad.net version of qt4reactor and can not get Qt QMainWindow to close.

  9. ScottNo Gravatar Says:

    As a followup if you press the ‘Do It’ button then press the window close X button the application does exit to command line. So it appears that is you try or actually connect to backend server the application will exit when pressing the window close X button. It looks like you must do a connect and/or send to server to get everything to close and exit application.

  10. elibenNo Gravatar Says:

    @Scott,

    Thanks for your feedback. Unfortunately, I no longer have the setup to run this sample – so I won’t be able to help here. But yes, qt4reactor interaction is a tricky thing, IIRC, and it wasn’t easy to find a well-functioning implementation.

Leave a Reply

To post code with preserved formatting, enclose it in `backticks` (even multiple lines)