Eli Bendersky's website - EE & Embedded

Convolutions, Polynomials and Flipped Kernels

2025-05-20T20:01:00-07:00

This is a post about multiplying polynomials, convolution sums and the connection between them.

Multiplying polynomials

Suppose we want to multiply one polynomial by another:

\[(3x^3+x^2+2x+1)\cdot(2x^2+6)\]

This is basic middle-school math - we start by cross-multiplying:

\[6x^5+2x^4+4x^3+2x^2+18x^3+6x^2+12x+6\]

And then collect all the terms together by adding up the coefficients:

\[6x^5+2x^4+22x^3+8x^2+12x+6\]

Let's look at a slightly different way to achieve the same result. Instead of cross multiplying all terms and then adding up, we'll focus on what terms appear in the output, and for each such term - what its coefficients are going to be. This is easy to demonstrate with a table, where we lay out one polynomial horizontally and the other vertically. Note that we have to be explicit about the zero coefficient of x in the second polynomial:

The diagonals that add up to each term in the output are highlighted. For example, to get the coefficient of x^3 in the output, we have to add up:

The coefficient of x^3 in the first polynomial, multiplied by the constant (coefficient of x^0) in the second polynomial
The coefficient of x^2 in the first polynomial, multiplied by the coefficient of in the second polynomial
The coefficient of in the first polynomial, multiplied by the coefficients of x^2 in the second polynomial

(if the second polynomial had a x^3 term, there would be another component to add)

For what follows, let's move to a somewhat more abstract representation: a polynomial P can be represented as a sum of coefficients P_i multiplied by corresponding powers of x [1]:

\[P=\sum_{i}P_i x^i\]

Suppose we have two polynomials, P and R. When we multiply them together, the resulting polynomial is S. Based on our insight from the table diagonals above, we can say that for each k:

\[S_k=\sum_{i}P_i\cdot R_{k-i}\]

And then the entire polynomial S is:

\[S=\sum_{k} \left( \sum_{i}P_i\cdot R_{k-i}\right)x^k\]

It's important to understand this formulation, since it's key to this post. Let's use our example polynomials to see how it works. First, we represent the two polynomials as sequences of coefficients (starting with 0, so the coefficient of the constant is first, and the coefficient of the highest power is last):

\[P=[1,2,1,3]\qquad R=[6,0,2]\]

In this notation, P_0 is the first element in the list for P, etc. To calculate the coefficient of x^3 in the product:

\[S_3=\sum_{i}P_i\cdot R_{3-i}\]

Expanding the sum for all the non-zero coefficients of P:

\[S_3=P_0 R_3+P_1 R_2+P_2 R_1+P_3 R_0=0+4+0+18=22\]

Similarly, we'll find that S_4=2, S_2=8 and so on, resulting in the final polynomial as before:

\[6x^5+2x^4+22x^3+8x^2+12x+6\]

There's a nice graphical approach to multiply polynomials using this idea of pairing up sums for each term in the output:

We start with the diagram on the left: one of the polynomials remains in its original form, while the other is flipped around (constant term first, highest power term last). We line up the polynomials vertically, and multiply the corresponding coefficients: the constant coefficient of the output is just the constant coefficient of the first polynomial times the constant coefficient of the second polynomial.

The diagram on the right shows the next step: the second polynomial is shifted left by one term and lined up vertically again. The corresponding coefficients are multiplied, and then the results are added to obtain the coefficient of x in the output polynomial.

We continue the process by shifting the lower polynomial left:

Calculating the coefficients of x^2 and then x^3. A couple more steps:

Now we have all the coefficients of the output. Take a moment to convince yourself that this approach is equivalent to the summation shown just before it, and also to the "diagonals in a table" approach shown further up. They all calculate the same thing [2]!

Hopefully it should be clear why the second polynomial is "flipped" to perform this procedure. This all comes down to which input terms pair up to calculate each output term. As seen above:

\[S_k=\sum_{i}P_i\cdot R_{k-i}\]

While the index i moves in one direction (from the low power terms to the high power terms) in P, the index k-i moves in the opposite direction in R.

If this procedure reminds you of computing a convolution between two arrays, that's because it's exactly that!

Signals, systems and convolutions

The theory of signals and systems is a large topic (typically taught for one or two semesters in undergraduate engineering degrees), but here I want to focus on just one aspect of it which I find really elegant.

Let's define discrete signals and systems first, restricting ourselves to 1D (similar ideas apply in higher dimensions):

Discrete signal: An ordered sequence of numbers x[n] with integer indices n\in \left\{ \dots,-2,-1,0,1,2,\dots \right\}. Can also be thought of as a function x:\mathbb{Z} \rightarrow \mathbb{R}.

Discrete system: A function mapping input signals x[n] to output signals y[n]. For example, y[n]=2x[n] is a system that scales all signals by a factor of two.

Here's an example signal:

This is a finite signal. All values not explicitly shown in the chart are assumed to be 0 (e.g. x[3]=0, x[-1]=0, x[99]=0 and so on).

A very important signal is the discrete impulse:

\[\delta[n]=\begin{cases} 1\quad if \quad n=0\\ 0\quad otherwise \end{cases}\]

In graphical form, here's \delta[n], as well as a couple of time-shifted variants of it. Note how we shift a signal left and right on the horizontal axis by adding to or subtracting from n, correspondingly. Take a moment to double check why this works.

The impulse is useful because we can decompose any discrete signal into a sequence of scaled and shifted impulses. Our example signal has three non-zero values at indices 0, 1 and 2; we can represent it as follows:

\[x[n]=x[0]\delta[n]+x[1]\delta[n-1]+x[2]\delta[n-2]=2\delta[n]+2\delta[n-1]+\delta[n-2]\]

More generally, a signal x[n] can be written as:

\[x[n]=\sum_{k} x[k]\delta[n-k]\]

(for all k where x[k] is nonzero)

In the study of signals and systems, linear and time-invariant (LTI) systems are particularly important.

Linear: suppose y_1[n] is the output of a system for input signal x_1[n], and similarly y_2[n] is the output for x_2[n]. A linear system outputs a\cdot y_1[n]+b\cdot y_2[n] for the input a\cdot x_1[n] + b\cdot x_2[n] where a and b are constants.

Time-invariant: if we delay the input signal by some constant: x_1[n-N], the output is similarly delayed: y_1[n-N]. In other words, the response of the system has a similar shape, no matter when the signal was received (it behaves today similarly to the way it behaved yesterday).

LTI systems are important because of the decomposition of a signal into impulses discussed above. Suppose we want to characterize a system: what it does to an arbitrary signal. For an LTI system, all we need to describe is its response to an impulse!

If the response of our system to \delta[n] is h[n], then:

Its response to c\cdot \delta[n] is c\cdot h[n], for any constant c, because the system is linear.
Its response to \delta[n-k] is h[n-k], for any time shift k, because the system is time-invariant.

We'll combine these and use linearity again (note that in the following x[k] are just constants); the response to a signal decomposed into a sum of shifted and scaled impulses:

\[y[n]=\sum_{k} x[k]\delta[n-k]\]

Is:

\[y[n]=\sum_{k} x[k]h[n-k]\]

This operation is called the convolution between sequences x[n] and h[n], and is denoted with the \ast operator:

\[y[n]=x[n]\ast h[n]\]

Let's work through an example. Suppose we have an LTI system with the following response to an impulse:

The response has h[0]=1, h[1]=3 and zeros everywhere else. Recall our sample signal from the top of this section (the sequence 2, 2, 1). We can decompose it to a sequence of scaled and shifted impulses, and then calculate the system response to each of them. Like this:

The top row shows the input signal decomposed into scaled and shifted impulses; the bottom row is the corresponding system response to each input. If we carefully add up the responses for each n, we'll get the system response y[n] to our input:

Now, let's calculate the same output, but this time using the convolution sum. First, we'll represent the signal x and the impulse response h as sequences (just like we did with polynomials), with x[0] first, then x[1] etc:

\[x=[2,2,1]\qquad h=[1, 3]\]

The convolution sum is:

\[y[n]=\sum_{k} x[k]h[n-k]\]

Recall that k ranges over all the non-zero elements in x. Let's calculate each element of y[n] (noting that h is nonzero only for indices 0 and 1):

\[\begin{align*} y[0]&=x[0]h[0]=2\\ y[1]&=x[0]h[1]+x[1]h[0]=8\\ y[2]&=x[1]h[1]+x[2]h[0]=7\\ y[3]&=x[2]h[1]=3 \end{align*}\]

All subsequent values of y are zero because k only ranges up to 2 and h[4-2]=h[2]=0.

If you look carefully at this calculation, you'll notice that h is "flipped" in relation to x, just like with the polynomials:

We start with x[n] (black) and flipped h[n] (blue), and line up the first non-zero elements. This computes y[0]
In subsequent steps, h[n] is slides right, one element at a time, and the next value of y is computed by adding up the element-wise products of the lined up x and h.

Just like with polynomials [3], the reason why one of the inputs is flipped is clear from the definition of the convolution sum, where one of the the indices increases (k), while the other decreases (n-k).

Properties of convolution

The convolution operation has many useful algebraic properties: linearity, associativity, commutativity, distributivity, etc.

The commutative property means that when computing convolutions graphically, it doesn't matter which of the signals is "flipped", because:

\[y[n]=x[n]\ast h[n]=h[n]\ast x[n]\]

And therefore:

\[y[n]=\sum_{k} x[k]h[n-k]=\sum_{k} x[n-k]h[k]\]

But the most important property of the convolution is how it behaves in the frequency domain. If we denote \mathcal{F}(f) as the Fourier transform of signal f, then the convolution theorem states:

\[\mathcal{F}(f\ast g)=\mathcal{F}(f)\cdot\mathcal{F}(g)\]

The Fourier transform of a convolution is equal to simple multiplication between the Fourier transforms of its operands. This fact - along with advanced algorithms like FFT - make it possible to implement convolutions very efficiently.

This is a deep and fascinating topic, but we'll leave it as a story for another day.

[1]	Theoretically, -\infty<i<\infty, but we only care about the non-zero coefficients. Therefore, we won't be specifying summation bounds; instead, \sum_{i} means "sum over all the non-zero coefficients"

[2]

As a cool exercise, explore how the same technique works for multiplying two numbers together, treating each number as a polynomial of subsequent powers of 10. There's just one slight complication with carries that have to be taken into account in the end, but it works really well! This blog post has more information, if you're interested.

[3]

The similarity in behavior between polynomials and signals here is quite beautiful, and far from accidental. In fact, if we take polynomials with addition and multiplication, we get a ring that's isomorphic to the ring of finite sequences with similar operations. Even operations like "delay" or "shift right" can be emulated with multiplying a polynomial by a power x (which could be negative if we want to "shift left").

This isomorphism is widely employed in mathematics; for example, ordinary generating functions use it to represent sequences as polynomials an then manipulate them algebraically. In signal processing it's also used for the Z transform.

Sum of same-frequency sinusoids

2023-03-11T19:44:00-08:00

I was reviewing an electronics textbook the other day, and it made an offhand comment that "sinusoidal signals of the same frequency always add up to a sinusoid, even if their magnitudes and phases are different". This gave me pause; is that really so? Even with different phases?

Using EE notation, a sinusoidal signal with magnitude A_1, frequency and phase \phi_1 is A_1 sin(wt+\phi_1) [1]. The book's statement amounts to:

\[A_1 sin(wt+\phi_1)+A_2 sin(wt+\phi_2)=A_3 sin(wt+\phi_3)\]

The sum is also a sinusoid with the same frequency, but potentially different magnitude and phase. I couldn't find this equality in any of my reference books, so why is it true?

Empirical probing

Let's start by asking whether this is true at all? It's not at all obvious that this should work. Armed with Python, Numpy and matplotlib, I plotted two sinusoidal signals with the same frequency but different magnitudes and phases:

Now, plotting their sum in green on the same chart:

Well, look at that. It seems to be working. I guess it's time to prove it.

Proof using trig identities

The first proof I want to demonstrate doesn't use any fancy math beyond some basic trigonometric identities. One of best known ones is:

\[sin(a+b)=sin(a)cos(b)+cos(a)sin(b) \hspace{2cm} (id. 1)\]

Taking our sum of sinusoids:

\[A_1 sin(wt+\phi_1)+A_2 sin(wt+\phi_2)\]

Applying (id.1) to each of the terms, and then regrouping, we get:

\[\begin{align*} <sum>&=A_1\left [sin(wt)cos(\phi_1)+cos(wt)sin(\phi_1) \right ]+A_2\left [sin(wt)cos(\phi_2)+cos(wt)sin(\phi_2) \right ]\\ &=\left [A_1 cos(\phi_1) + A_2 cos(\phi_2) \right ]sin(wt)+\left [ A_1 sin(\phi_1) + A_2 sin(\phi_2)\right ]cos(wt)\\ \end{align*}\]

Now, a change of variables trick: we'll assume we can solve the following set of equations for some and [2]:

\[\begin{align*} Bcos(\theta)&=A_1 cos(\phi_1)+A_2 cos(\phi_2) \hspace{2cm} (1)\\ Bsin(\theta)&=A_1 sin(\phi_1)+A_2 sin(\phi_2) \hspace{2cm} (2)\\ \end{align*}\]

To find , we can square each of (1) and (2) and then add the squares together:

\[B^2 cos^2 (\theta)+B^2 sin^2 (\theta)=(A_1 cos(\phi_1)+A_2 cos(\phi_2))^2 + (A_1 sin(\phi_1)+A_2 sin(\phi_2))^2\]

Using the fact that cos^2(a)+sin^2(a)=1, we get:

\[B=\sqrt{(A_1 cos(\phi_1)+A_2 cos(\phi_2))^2 + (A_1 sin(\phi_1)+A_2 sin(\phi_2))^2}\]

To solve for , we can divide equation (2) by (1), getting:

\[\frac{sin(\theta)}{cos(\theta)}=tan(\theta)=\frac{A_1 sin(\phi_1)+A_2 sin(\phi_2)}{A_1 cos(\phi_1)+A_2 cos(\phi_2)}\]

Meaning that:

\[\theta = atan{\frac{A_1 sin(\phi_1)+A_2 sin(\phi_2)}{A_1 cos(\phi_1)+A_2 cos(\phi_2)}}\]

Now that we have the values of and , let's put them aside for a bit and get back to the final line of our sum of sinusoids equation:

\[A_1 sin(wt+\phi_1)+A_2 sin(wt+\phi_2)=\left [A_1 cos(\phi_1) + A_2 cos(\phi_2) \right ]sin(wt)+\left [ A_1 sin(\phi_1) + A_2 sin(\phi_2)\right ]cos(wt)\]

On the right-hand side, we can apply equations (1) and (2) to get:

\[A_1 sin(wt+\phi_1)+A_2 sin(wt+\phi_2)=B cos(\theta) sin(wt)+ B sin(\theta) cos(wt)\]

Applying (id.1) again, we get:

\[A_1 sin(wt+\phi_1)+A_2 sin(wt+\phi_2)=B sin(wt + \theta)\]

We've just shown that the sum of sinusoids with the same frequency is another sinusoid with frequency , and we've calculated and from the other parameters (A_1, A_2, \phi_1 and \phi_2) \blacksquare

Proof using complex numbers

The second proof uses a bit more advanced math, but overall feels more elegant to me. The plan is to use Euler's equation and prove a more general statement on the complex plane.

Instead of looking at the sum of real sinusoids, we'll first look at the sum of two complex exponential functions:

\[A_1 e^{j(wt + \phi_1)} + A_2 e^{j(wt + \phi_2)}\]

Reminder: Euler's equation for a complex exponential is

\[e^{jx}=cosx+jsinx\]

Regrouping our sum of exponentials a bit and then applying this equation:

\[\begin{align*} A_1 e^{j(wt + \phi_1)} + A_2 e^{j(wt + \phi_2)}&=e^{jwt}\left (A_1 e^{j\phi_1} + A_2 e^{j\phi_2}\right )\\ &=e^{jwt}\left ( A_1 cos(\phi_1) + jA_1 sin(\phi_1) + A_2 cos(\phi_2) + jA_2 sin(\phi_2)\right )\\ &=e^{jwt}\left [\left (A_1 cos(\phi_1) + A_2 cos(\phi_2) \right ) + j\left(A_1 sin(\phi_1) + A_2 sin(\phi_2) \right ) \right ] \end{align*}\]

The value inside the square brackets can be viewed as a complex number in its rectangular form: x + jy. We can convert it to its polar form: re^{j\theta}, by calculating:

\[\begin{align*} r&=\sqrt{x^2+y^2}\\ \theta&=atan(\frac{y}{x}) \end{align*}\]

In our case:

\[r=\sqrt{(A_1 cos(\phi_1)+A_2 cos(\phi_2))^2 + (A_1 sin(\phi_1)+A_2 sin(\phi_2))^2}\]

And:

\[\theta = atan{\frac{A_1 sin(\phi_1)+A_2 sin(\phi_2)}{A_1 cos(\phi_1)+A_2 cos(\phi_2)}}\]

Therefore, the sum of complex exponentials is another complex exponential with the same frequency, but a different magnitude and phase:

\[A_1 e^{j(wt + \phi_1)} + A_2 e^{j(wt + \phi_2)}= e^{jwt} r e^{j \theta}=r e^{j(wt + \theta)}\]

From here, we can use Euler's equation again to see the equivalence in terms of sinusoidal functions:

\[\begin{align*} A_1 cos(wt+\phi_1)+jA_1 sin(wt+\phi_1)&+\\ A_2 cos(wt+\phi_2)+jA_2 sin(wt+\phi_2)&=r cos(wt+\theta) + jr sin(wt+\theta) \end{align*}\]

If we only compare the imaginary parts of this equation, we get:

\[A_1 sin(wt+\phi_1)+A_2 sin(wt+\phi_2)=r sin(wt+\theta)\]

With known r and we've calculated earlier from the other constants \blacksquare

Note that by comparing the real parts of the equation, we can trivially prove a similar statement about the sum of cosines (which should surprise no one, since a cosine is just a phase-shifted sine).

[1]	Electrical engineers prefer their signal frequencies in units of radian per second. We also like calling the imaginary unit j instead of i, because the latter is used for electrical current.

[2]	If you're wondering "hold on, why would this work?", recall that any point (x,y) on the Cartesian plane can be represented using polar coordinates with magnitude and angle .

Some notes on Luz - an assembler, linker and CPU simulator

2017-01-05T06:27:00-08:00

A few years ago I wrote about Luz - a self-educational project to implement a CPU simulator and a toolchain for it, consisting of an assembler and a linker. Since then, I received some questions by email that made me realize I could do a better job explaining what the project is and what one can learn from it.

So I went back to the Luz repository and fixed it up to be more modern, in-line with current documentation standards on GitHub. The landing README page should now provide a good overview, but I also wanted to write up some less formal documentation I could point to - a place to show-off some of the more interesting features in Luz; a blog post seemed like the perfect medium for this.

As before, it makes sense to start with the Luz toplevel diagram:

Luz is a collection of related libraries and programs written in Python, implementing all the stages shown in the diagram above.

The CPU simulator

The Luz CPU is inspired by MIPS (for the instruction set), by Altera Nios II (for the way "peripherals" are attached to the CPU), and by MPC 555 (for the memory controller) and is aimed at embedded uses, like Nios II. The Luz user manual lists the complete instruction set explaining what each instructions means.

The simulator itself is functional only - it performs the instructions one after the other, without trying to simulate how long their execution takes. It's not very remarkable and is designed to be simple and readable. The most interesting feature it has, IMHO, is how it maps "peripherals" and even CPU control registers into memory. Rather than providing special instructions or traps for OS system calls, Luz facilitates "bare-metal" programming (by which I mean, without an OS) by mapping "peripherals" into memory, allowing the programmer to access them by reading and writing special memory locations.

My inspiration here was soft-core embeddable CPUs like Nios II, which let you configure what peripherals to connect and how to map them. The CPU can be configured before it's loaded onto real HW, for example to attach as many SPI interfaces as needed. For Luz, to create a new peripheral and attach it to the simulator one implements the Peripheral interface:

class Peripheral(object):
    """ An abstract memory-mapped perhipheral interface.
        Memory-mapped peripherals are accessed through memory
        reads and writes.

        The address given to reads and writes is relative to the
        peripheral's memory map.
        Width is 1, 2, 4 for byte, halfword and word accesses.
    """
    def read_mem(self, addr, width):
        raise NotImplementedError()

    def write_mem(self, addr, width, data):
        raise NotImplementedError()

Luz implements some built-in features as peripherals as well; for example, the core registers (interrupt control, exception control, etc). The idea here is that embedded CPUs can have multiple custom "registers" to control various features, and creating dedicated names for them bloats instruction encoding (you need 5 bits to encode one of 32 registers, etc.); it's better to just map them to memory.

Another example is the debug queue - a peripheral useful for testing and debugging. It's a single word mapped to address 0xF0000 in the simulator. When the peripheral gets a write, it stores it in a special queue and optionally emits the value to stdout. The queue can later be examined. Here is a simple Luz assembly program that makes use of it:

# Counts from 0 to 9 [inclusive], pushing these numbers into the debug queue

    .segment code
    .global asm_main

    .define ADDR_DEBUG_QUEUE, 0xF0000

asm_main:
    li $k0, ADDR_DEBUG_QUEUE

    li $r9, 10                          # r9 is the loop limit
    li $r5, 0                           # r5 is the loop counter

loop:
    sw $r5, 0($k0)                      # store loop counter to debug queue
    addi $r5, $r5, 1                    # increment loop counter
    bltu $r5, $r9, loop                 # loop back if not reached limit

    halt

Using the interactive runner to run this program we get:

$ python run_test_interactive.py loop_simple_debugqueue
DebugQueue: 0x0
DebugQueue: 0x1
DebugQueue: 0x2
DebugQueue: 0x3
DebugQueue: 0x4
DebugQueue: 0x5
DebugQueue: 0x6
DebugQueue: 0x7
DebugQueue: 0x8
DebugQueue: 0x9
Finished successfully...
Debug queue contents:
['0x0', '0x1', '0x2', '0x3', '0x4', '0x5', '0x6', '0x7', '0x8', '0x9']

Assembler

There's a small snippet of Luz assembly shown above. It's your run-of-the-mill RISC assembly, with the familiar set of instructions, fairly simple addressing modes and almost every instruction requiring registers (note how we can't store into the debug queue directly, for example, without dereferencing a register that holds its address).

The Luz user manual contains a complete reference for the instructions, including their encodings. Every instruction is a 32-bit word, with the 6 high bits for the opcode (meaning up to 64 distinct instructions are supported).

The code snippet also shows off some special features of the full Luz toolchain, like the special label asm_main. I'll discuss these later on in the section about linking.

Assembly languages are usually fairly simple to parse, and Luz is no exception. When I started working on Luz, I decided to use the PLY library for the lexer and parser mainly because I wanted to play with it. These days I'd probably just hand-roll a parser.

Luz takes another cool idea from MIPS - register aliases. While the assembler doesn't enforce any specific ABI on the coder, some conventions are very important when writing large assembly programs, and especially when interfacing with routines written by other programmers. To facilitate this, Luz designates register aliases for callee-saved registers and temporary registers.

For example, the general-purpose register number 19 can be referred to in Luz assembly as $r19 but also as $s1 - the callee-saved register 1. When writing standalone Luz programs, one is free to ignore these conventions. To get a taste of how ABI-conformant Luz assembly would look, take a look at this example.

To be honest, ABI was on my mind because I was initially envisioning a full programming environment for Luz, including a C compiler. When you have a compiler, you must have some set of conventions for generated code like procedure parameter passing, saved registers and so on; in other words, the platform ABI.

Linker

In my view, one of the distinguishing features of Luz from other assembler projects out there is the linker. Luz features a full linker that supports creating single "binaries" from multiple assembly files, handling all the dirty work necessary to make that happen. Each assembly file is first "assembled" into a position-independent object file; these are glued together by the linker which applies the necessary relocations to resolve symbols across object files. The prime sieve example shows this in action - the program is divided into three .lasm files: two for subroutines and one for "main".

As we've seen above, the main subroutine in Luz is called asm_main. This is a special name for the linker (not unlike the _start symbol for modern Linux assemblers). The linker collects a set of object files produced by assembly, and makes sure to invoke asm_main from the special location 0x100000. This is where the simulator starts execution.

Luz also has the concept of object files. They are not unlike ELF images in nature: there's a segment table, an export table and a relocation table for each object, serving the expected roles. It is the job of the linker to make sense in this list of objects and correctly connect all call sites to final subroutine addresses.

Luz's standalone assembler can write an assembled image into a file in Intel HEX format, a popular format used in embedded systems to encode binary images or data in ASCII.

The linker was quite a bit of effort to develop. Since all real Luz programs are small I didn't really need to break them up into multiple assembly files; but I really wanted to learn how to write a real linker :) Moreover, as already mentioned my original plans for Luz included a C compiler, and that would make a linker very helpful, since I'd need to link some "system" code into the user's program. Even today, Luz has some "startup code" it links into every image:

# The special segments added by the linker.
# __startup: 3 words
# __heap: 1 word
#
LINKER_STARTUP_CODE = string.Template(r'''
        .segment __startup

    LI      $$sp, ${SP_POINTER}
    CALL    asm_main

        .segment __heap
        .global __heap
    __heap:
        .word 0
''')

This code sets up the stack pointer to the initial address allocated for the stack, and calls the user's asm_main.

Debugger and disassembler

Luz comes with a simple program runner that will execute a Luz program (consisting of multiple assembly files); it also has an interactive mode - a debugger. Here's a sample session with the simple loop example shown above:

$ python run_test_interactive.py -i loop_simple_debugqueue

LUZ simulator started at 0x00100000

[0x00100000] [lui $sp, 0x13] >> set alias 0
[0x00100000] [lui $r29, 0x13] >> s
[0x00100004] [ori $r29, $r29, 0xFFFC] >> s
[0x00100008] [call 0x40003 [0x10000C]] >> s
[0x0010000C] [lui $r26, 0xF] >> s
[0x00100010] [ori $r26, $r26, 0x0] >> s
[0x00100014] [lui $r9, 0x0] >> s
[0x00100018] [ori $r9, $r9, 0xA] >> s
[0x0010001C] [lui $r5, 0x0] >> s
[0x00100020] [ori $r5, $r5, 0x0] >> s
[0x00100024] [sw $r5, 0($r26)] >> s
[0x00100028] [addi $r5, $r5, 0x1] >> s
[0x0010002C] [bltu $r5, $r9, -2] >> s
[0x00100024] [sw $r5, 0($r26)] >> s
[0x00100028] [addi $r5, $r5, 0x1] >> s
[0x0010002C] [bltu $r5, $r9, -2] >> s
[0x00100024] [sw $r5, 0($r26)] >> s
[0x00100028] [addi $r5, $r5, 0x1] >> r
$r0   = 0x00000000   $r1   = 0x00000000   $r2   = 0x00000000   $r3   = 0x00000000
$r4   = 0x00000000   $r5   = 0x00000002   $r6   = 0x00000000   $r7   = 0x00000000
$r8   = 0x00000000   $r9   = 0x0000000A   $r10  = 0x00000000   $r11  = 0x00000000
$r12  = 0x00000000   $r13  = 0x00000000   $r14  = 0x00000000   $r15  = 0x00000000
$r16  = 0x00000000   $r17  = 0x00000000   $r18  = 0x00000000   $r19  = 0x00000000
$r20  = 0x00000000   $r21  = 0x00000000   $r22  = 0x00000000   $r23  = 0x00000000
$r24  = 0x00000000   $r25  = 0x00000000   $r26  = 0x000F0000   $r27  = 0x00000000
$r28  = 0x00000000   $r29  = 0x0013FFFC   $r30  = 0x00000000   $r31  = 0x0010000C

[0x00100028] [addi $r5, $r5, 0x1] >> s 100
[0x00100030] [halt] >> q

There are many interesting things here demonstrating how Luz works:

Note the start up at 0x1000000 - this is where Luz places the start-up segment - three instructions that set up the stack pointer and then call the user's code (asm_main). The user's asm_main starts running at the fourth instruction executed by the simulator.
li is a pseudo-instruction, broken into two real instructions: lui for the upper half of the register, followed by ori for the lower half of the register. The reason for this is li having a 32-bit immediate, which can't fit in a Luz instruction. Therefore, it's broken into two parts which only need 16-bit immediates. This trick is common in RISC ISAs.
Jump labels are resolved to be relative by the assembler: the jump to loop is replaced by -2.
Disassembly! The debugger shows the instruction decoded from every word where execution stops. Note how this exposes pseudo-instructions.

The in-progress RTL implementation

Luz was a hobby project, but an ambitious one :-) Even before I wrote the first line of the assembler or simulator, I started working on an actual CPU implementation in synthesizable VHDL, meaning to get a complete RTL image to run on FPGAs. Unfortunately, I didn't finish this part of the project and what you find in Luz's experimental/luz_uc directory is only 75% complete. The ALU is there, the registers, the hookups to peripherals, even parts of the control path - dealing with instruction fetching, decoding, etc. My original plan was to implement a pipelined CPU (a RISC ISA makes this relatively simple), which perhaps was a bit too much. I should have started simpler.

Conclusion

Luz was an extremely educational project for me. When I started working on it, I mostly had embedded programming experience and was just starting to get interested in systems programming. Luz flung me into the world of assemblers, linkers, binary images, calling conventions, and so on. Besides, Python was a new language for me at the time - Luz started just months after I first got into Python.

Its ~8000 lines of Python code are thus likely not my best Python code, but they should be readable and well commented. I did modernize it a bit over the years, for example to make it run on both Python 2 and 3.

I still hope to get back to the RTL implementation project one day. It's really very close to being able to run realistic assembly programs on real hardware (FPGAs). My dream back then was to fully close the loop by adding a Luz code generation backend to pycparser. Maybe I'll still fulfill it one day :-)

Introducing Luz

2010-05-05T19:43:38-07:00

OK, so the documentation still isn't complete, but I can't wait to introduce my newest concoction - Luz. Luz is a pure-Python implementation of a MIPS-like CPU (as a simulator, of course). This CPU is programmable in an assembly language, a complete assembler for which has been implemented, along with a linker that takes together several object files and creates an executable image to run on the simulator. Oh, and did I mention that it also includes a rudimentary debugger and disassembler? All of this is Luz:

To call Luz new is a bit of a stretch, because I started working on it more than two years ago. It has been a jagged road, with occasional spurts of productivity, but now Luz is finally in a presentable form.

I'll paste from its "getting started guide":

What is Luz useful for? I don't know yet. It's a self-educational project of mine, and I learned a lot by working on it. I suppose that Luz's main value is as an educational tool. Its implementation focuses on simplicity and modularity, and is done in Python, which is a portable and very readable high-level language. Luz can serve as a sample of implementing a complete assembler, a complete linker, a complete CPU simulator. Other such tools exist, but usually not in the clean and self-contained form offered by Luz. In any case, if you've found Luz iseful, I'd love to receive feedback.

This summarizes it, really. Not much more to add, except that Luz is available in source-only form for now, so you'll have to check it out from SVN or just look at the sources in the online browser. Checking the source out is recommended because it allows one to view the documentation in nice HTML format. A few example programs in Luz assembly are available. Luz requires Python 2.6 or higher and the PLY module installed. I tested it on Windows XP and Ubuntu.

I've written an assembler and a CPU simulator before, but that was for a very weird architecture (Knuth's MIX from TAOCP). Luz is a much more useful beast - the CPU is not far from real modern CPUs (the embedded kind, mostly), the assembly language is familiar and best of all, Luz also includes a linker, which will make it much easier to compile C for it in the future.

I'll write more about Luz in sometime later, when I find the time to work on its documentation.

Framing in serial communications

2009-08-12T05:16:47-07:00

Introduction

In the previous post we've seen how to send and receive data on the serial port with Python and plot it live using a pretty GUI.

Notice that the sender script (sender_sim.py) is just sending one byte at a time. The "chunks" of data in the protocol between the sender and receiver are single bytes. This is simple and convenient, but hardly sufficient in the general sense. We want to be able to send multiple-byte data frames between the communicating parties.

However, there are some challenges that arise immediately:

The receiver is just receiving a stream of bytes from the serial port. How does it know when a message begins or ends? How does it know how long the message is?
Even more seriously, we can not assume a noise-free channel. This is real, physical hardware stuff. Bytes and whole chunks can and will be lost due to electrical noise. Worse, other bytes will be distorted (say, a single bit can be flipped due to noise).

To see how this can be done in a safe and tested manner, we first have to learn about the basics of the Data Link Layer in computer networks.

Data Link Layer

Given a physical layer that can transmit signals between devices, the job of the Data Link Layer [1] is (roughly stated) to transmit whole frames of data, with some means of assuring the integrity of the data (lack of errors). When we use sockets to communicate over TCP or UDP on the internet, the framing is taken care of deep in the hardware, and we don't even feel it. On the serial port, however, we must take care of the framing and error handling ourselves [2].

Framing

In chapter 3 of his "Computer Networks" textbook, Tanenbaum defines the following methods of framing:

Inserting time gaps between frames
Physical layer coding violations
Character count
Flag bytes with byte stuffing
Flag bytes with bit stuffing

Methods (1) and (2) are only suitable for a hardware-implemented data link layer [3]. It is very difficult (read: impossible) to ensure timing when multiple layers of software (running on Windows!) are involved. (2) is an interesting hardware method - but out of the scope of this article.

Method (3) means specifying in the frame header the number of bytes in the frame. The trouble with this is that the count can be garbled by a transmission error. In such a case, it's very difficult to "resynchronize". This method is rarely used.

Methods (4) and (5) are somewhat similar. In this article I'll focus on (4), as (5) is not suitable for serial port communications.

Flag bytes with byte stuffing

Let's begin with a simple idea and develop it into a full, robust scheme.

Flag bytes are special byte values that denote when a frame begins and ends. Suppose that we want to be able to send frames of arbitrary length. A special start flag byte will denote the beginning of the frame, and an end flag byte will denote its end.

A question arises, however. Suppose that the value of the end flag is 0x98. What if the value 0x98 appears somewhere in the data? The protocol will get confused and end the message.

There is a simple solution to this problem that will be familiar to all programmers who know about escaping quotes and special characters in strings. It is called byte stuffing, or octet stuffing, or simply escaping [4]. The scheme goes as follows:

Whenever a flag (start or end) byte appears in the data, we shall insert a special escape byte (ESC) before it. When the receiver sees an ESC, it knows to ignore it and not insert it into the actual data received (de-stuffing).
Whenever ESC itself has to appear in the data, another ESC is prepended to it. The receiver removes the first one but keeps the second one [5].

Here are a few examples:

Note that we didn't specify what the data is - it's arbitrary and up the the protocol to decide. The only really required part of the data is some kind of error checking - a checksum, or better yet a CRC. This is customarily the last byte (or last word) of the frame, referring to all the bytes in the frame (in its un-stuffed form).

This scheme is quite robust: any lost byte (be it a flag, an escape, a data byte or a checksum byte) will cause the receiver to lose just one frame, after which it will resynchronize onto the start flag byte of the next one.

PPP

As a matter of fact, this method is a slight simplification of the Point-to-Point Protocol (PPP) which is used by most ISPs for providing ADSL internet to home users, so there's a good chance you're using it now to surf the net and read this article! The framing of PPP is defined in RFC 1662.

In particular, PPP does the following:

Both the start and end flag bytes are 0x7E (they shouldn't really be different, if you think about it)
The escape byte is 0x7D
Whenever a flag or escape byte appears in the message, it is escaped by 0x7D and the byte itself is XOR-ed with 0x20. So, for example 0x7E becomes 0x7D 0x5E. Similarly 0x7D becomes 0x7D 0x5D. The receiver unsuffs the escape byte and XORs the next byte with 0x20 again to get the original [6].

An example

Let's now see a completely worked-out example that demonstrates how this works.

Suppose we define the following protocol:

Start flag: 0x12
End flag: 0x13
Escape (DLE): 0x7D

And the sender wants to send the following data message (let's ignore its contents for the sake of the example - they're really not that important). The original data is in (a):

The data contains two flags that need to be escaped - an end flag at position 2 (counting from 0, of course!), and a DLE at position 4.

The sender's data link layer [7] turns the data into the frame shown in (b) - start and end flags are added, and in-message flags are escaped.

Let's see how the receiver handles such a frame. For demonstration, assume that the first byte the receiver draws from the serial port is not a real part of the message (we want to see how it handles this). In the following diagram, 'Receiver state' is the state of the receiver after the received byte. 'Data buffer' is the currently accumulated message buffer to pass to an upper level:

A few things to note:

The "stray" byte before the header is ignored: according to the protocol each frame has to start with a header, so this isn't part of the frame.
The start and end flags are not inserted into the data buffer
Escapes (DLEs) are correctly handled by a special state
When the frame is finished with an end flag, the receiver has a frame ready to pass to an upper level, and comes back waiting for a header - a new frame.

Finally, we see that the message received is exactly the message sent. All the protocol details (flags, escapes and so on) were transparently handled by the data link layer [8].

Conclusion

There are several methods of handling framing in communications, although most are unsuitable to be used on top of the serial port. Among the ones that are suitable, the most commonly used is byte stuffing. By defining a couple of "magic value" flags and careful rules of escaping, this framing methods is both robust and easy to implement as a software layer. It is also widely used as PPP depends on it.

Finally, it's important to remember that for a high level of robustness, it's required to add some kind of error checking into the protocol - such as computing a CRC on the message and appending it as the last word of the message, which the receiver can verify before deciding that the message is valid.

[1]	The Data Link Layer is layer 2 in the OSI model. In the TCP/IP model it's simply called the "link layer".

[2]	The serial port can be configured to add parity bits to bytes. These days, this option is rarely used, because:

A single parity bit isn't a very strong means of detecting errors. 2-bit errors fool it.
Error handling is usually done by stronger means at a higher level.

[3]	For example Ethernet (802.3) uses 12 octets of idle characters between frames.

[4]	You might run into the term DLE - Data Link Escape, which means the same thing. I will use the acronyms DLE and ESC interchangeably.

[5]	Just like quotes and escape characters in strings! In C: `"I say \"Hello\""`. To escape the escape, repeat it: `"Here comes the backslash: \\ - seen it?"`

[6]	I'd love to hear why this XOR-ing is required. One simple reason I can think of is to prevent the flag and escape bytes appearing "on the line" even after they're escaped. Presumably this improves resynchronization if the escape byte is lost?

[7]	Which is just a fancy way to say "a protocol wrapping function", since the layer is implemented in software.

[8]	Such transparency is one of the greatest ideas of layered network protocols. So when we implement protocols in software, it's a good thing to keep in mind - transparency aids modularity and decoupling, it's a good thing.