Eli Bendersky's website - Math

Notes on Lagrange Interpolating Polynomials

2026-02-28T18:58:00-08:00

Polynomial interpolation is a method of finding a polynomial function that fits a given set of data perfectly. More concretely, suppose we have a set of n+1 distinct points [1]:

\[(x_0,y_0), (x_1, y_1), (x_2, y_2)\cdots(x_n, y_n)\]

And we want to find the polynomial coefficients {a_0\cdots a_n} such that:

\[p(x)=a_0 + a_1 x + a_2 x^2 + \cdots + a_n x^n\]

Fits all our points; that is p(x_0)=y_0, p(x_1)=y_1 etc.

This post discusses a common approach to solving this problem, and also shows why such a polynomial exists and is unique.

Showing existence using linear algebra

When we assign all points (x_i, y_i) into the generic polynomial p(x), we get:

\[\begin{aligned} p(x_0)&=a_0 + a_1 x_0 + a_2 x_0^2 + \cdots a_n x_0^n = y_0\\ p(x_1)&=a_0 + a_1 x_1 + a_2 x_1^2 + \cdots a_n x_1^n = y_1\\ p(x_2)&=a_0 + a_1 x_2 + a_2 x_2^2 + \cdots a_n x_2^n = y_2\\ \cdots \\ p(x_n)&=a_0 + a_1 x_n + a_2 x_n^2 + \cdots a_n x_n^n = y_n\\ \end{aligned}\]

We want to solve for the coefficients a_i. This is a linear system of equations that can be represented by the following matrix equation:

\[{\renewcommand{\arraystretch}{1.5}\begin{bmatrix} 1 & x_0 & x_0^2 & \dots & x_0^n\\ 1 & x_1 & x_1^2 & \dots & x_1^n\\ 1 & x_2 & x_2^2 & \dots & x_2^n\\ \vdots & \vdots & \vdots & \ddots &\vdots \\ 1 & x_n & x_n^2 & \dots & x_n^n \end{bmatrix} \begin{bmatrix} a_0\\ a_1\\ a_2\\ \vdots\\ a_n\\ \end{bmatrix}= \begin{bmatrix} y_0\\ y_1\\ y_2\\ \vdots\\ y_n\\ \end{bmatrix} }\]

The matrix on the left is called the Vandermonde matrix. This matrix is known to be invertible (see Appendix for a proof); therefore, this system of equations has a single solution that can be calculated by inverting the matrix.

In practice, however, the Vandermonde matrix is often numerically ill-conditioned, so inverting it isn’t the best way to calculate exact polynomial coefficients. Several better methods exist.

Lagrange Polynomial

Lagrange interpolation polynomials emerge from a simple, yet powerful idea. Let’s define the Lagrange basis functions l_i(x) (i \in [0, n]) as follows, given our points (x_i, y_i):

\[l_i(x) = \begin{cases} 1 & x = x_i \\ 0 & x = x_j \quad \forall j \neq i \end{cases}\]

In words, l_i(x) is constrained to 1 at and to 0 at all other x_j. We don’t care about its value at any other point.

The linear combination:

\[p(x)=\sum_{i=0}^{n}y_i l_i(x)\]

is then a valid interpolating polynomial for our set of n+1 points, because it’s equal to at each (take a moment to convince yourself this is true).

How do we find l_i(x)? The key insight comes from studying the following function:

\[l'_i(x)=(x-x_0)\cdot (x-x_1)\cdots (x-x_{i-1}) \cdot (x-x_{i+1})\cdots (x-x_n)= \prod_{\substack{0\leq j \leq n \\ j \neq i}}(x-x_j)\]

This function has terms (x-x_j) for all j\neq i. It should be easy to see that l'_i(x) is 0 at all x_j when j\neq i.

What about its value at , though? We can just assign into l'_i(x) to get:

\[l'_i(x_i)=\prod_{\substack{0\leq j \leq n \\ j \neq i}}(x_i-x_j)\]

And then normalize l'_i(x), dividing it by this (constant) value. We get the Lagrange basis function l_i(x):

\[l_i(x)=\frac{l'_i(x)}{l'_i(x_i)}=\prod_{\substack{0\leq j \leq n \\ j \neq i}}\frac{x-x_j}{x_i-x_j}\]

Let’s use a concrete example to visualize this. Suppose we have the following set of points we want to interpolate: (1,4), (2,2), (3,3). We can calculate l'_0(x), l'_1(x) and l'_2(x), and get the following:

Note where each l'_i(x) intersects the axis. These functions have the right values at all x_{j\neq i}. If we normalize them to obtain l_i(x), we get these functions:

Note that each polynomial is 1 at the appropriate and 0 at all the other x_{j\neq i}, as required.

With these l_i(x), we can now plot the interpolating polynomial p(x)=\sum_{i=0}^{n}y_i l_i(x), which fits our set of input points:

Polynomial degree and uniqueness

We’ve just seen that the linear combination of Lagrange basis functions:

\[p(x)=\sum_{i=0}^{n}y_i l_i(x)\]

is a valid interpolating polynomial for a set of n+1 distinct points (x_i, y_i). What is its degree?

Since the degree of each l_i(x) is , then the degree of p(x) is at most . We’ve just derived the first part of the Polynomial interpolation theorem:

Polynomial interpolation theorem: for any n+1 data points (x_0,y_0), (x_1, y_1)\cdots(x_n, y_n) \in \mathbb{R}^2 where no two x_j are the same, there exists a unique polynomial p(x) of degree at most that interpolates these points.

We’ve demonstrated existence and degree, but not yet uniqueness. So let’s turn to that.

We know that p(x) interpolates all n+1 points, and its degree is . Suppose there’s another such polynomial q(x). Let’s construct:

\[r(x)=p(x)-q(x)\]

That do we know about r(x)? First of all, its value is 0 at all our , so it has n+1 roots. Second, we also know that its degree is at most (because it’s the difference of two polynomials of such degree). These two facts are a contradiction. No non-zero polynomial of degree \leq n can have n+1 roots (a basic algebraic fact related to the Fundamental theorem of algebra). So r(x) must be the zero polynomial; in other words, our p(x) is unique \blacksquare.

Note the implication of uniqueness here: given our set of n+1 distinct points, there’s only one polynomial of degree \leq n that interpolates it. We can find its coefficients by inverting the Vandermonde matrix, by using Lagrange basis functions, or any other method [2].

Lagrange polynomials as a basis for P_n(\mathbb{R})

The set P_n(\mathbb{R}) consists of all real polynomials of degree \leq n. This set - along with addition of polynomials and scalar multiplication - forms a vector space.

We called l_i(x) the "Lagrange basis" previously, and they do - in fact - form an actual linear algebra basis for this vector space. To prove this claim, we need to show that Lagrange polynomials are linearly independent and that they span the space.

Linear independence: we have to show that

\[s(x)=\sum_{i=0}^{n}a_i l_i(x)=0\]

implies a_i=0 \quad \forall i. Recall that l_i(x) is 1 at , while all other l_j(x) are 0 at that point. Therefore, evaluating s(x) at , we get:

\[s(x_i)=a_i = 0\]

Similarly, we can show that a_i is 0, for all \blacksquare.

Span: we’ve already demonstrated that the linear combination of l_i(x):

\[p(x)=\sum_{i=0}^{n}y_i l_i(x)\]

is a valid interpolating polynomial for any set of n+1 distinct points. Using the polynomial interpolation theorem, this is the unique polynomial interpolating this set of points. In other words, for every q(x)\in P_n(\mathbb{R}), we can identify any set of n+1 distinct points it passes through, and then use the technique described in this post to find the coefficients of q(x) in the Lagrange basis. Therefore, the set l_i(x) spans the vector space \blacksquare.

Interpolation matrix in the Lagrange basis

Previously we’ve seen how to use the \{1, x, x^2, \dots x^n\} basis to write down a system of linear equations that helps us find the interpolating polynomial. This results in the Vandermonde matrix.

Using the Lagrange basis, we can get a much nicer matrix representation of the interpolation equations.

Recall that our general polynomial using the Lagrange basis is:

\[p(x)=\sum_{i=0}^{n}a_i l_i(x)\]

Let’s build a system of equations for each of the n+1 points (x_i,y_i). For :

\[p(x_0)=\sum_{i=0}^{n}a_i l_i(x_0)\]

By definition of the Lagrange basis functions, all l_i(x_0) where i\neq 0 are 0, while l_0(x_0) is 1. So this simplifies to:

\[p(x_0)=a_0\]

But the value at node is , so we’ve just found that a_0=y_0. We can produce similar equations for the other nodes as well, p(x_1)=a_1, etc. In matrix form:

\[{\renewcommand{\arraystretch}{1.5}\begin{bmatrix} 1 & 0 & 0 & \dots & 0\\ 0 & 1 & 0 & \dots & 0\\ 0 & 0 & 1 & \dots & 0\\ \vdots & \vdots & \vdots & \ddots &\vdots \\ 0 & 0 & 0 & \dots & 1 \end{bmatrix} \begin{bmatrix} a_0\\ a_1\\ a_2\\ \vdots\\ a_n\\ \end{bmatrix}= \begin{bmatrix} y_0\\ y_1\\ y_2\\ \vdots\\ y_n\\ \end{bmatrix} }\]

We get the identity matrix; this is another way to trivially show that a_0=y_0, a_1=y_1 and so on.

Appendix: Vandermonde matrix

Given some numbers \{x_0 \dots x_n\} a matrix of this form:

\[V= {\renewcommand{\arraystretch}{1.5}\begin{bmatrix} 1 & x_0 & x_0^2 & \dots & x_0^n\\ 1 & x_1 & x_1^2 & \dots & x_1^n\\ 1 & x_2 & x_2^2 & \dots & x_2^n\\ \vdots & \vdots & \vdots & \ddots &\vdots \\ 1 & x_n & x_n^2 & \dots & x_n^n \end{bmatrix} }\]

Is called the Vandermonde matrix. What’s special about a Vandermonde matrix is that we know it’s invertible when are distinct. This is because its determinant is known to be non-zero. Moreover, its determinant is [3]:

\[\det(V) = \prod_{0 \le i < j \le n} (x_j - x_i)\]

Here’s why.

To get some intuition, let’s consider some small-rank Vandermonde matrices. Starting with a 2-by-2:

\[\det(V)=\det\begin{bmatrix} 1 & x_0 \\ 1 & x_1 \\ \end{bmatrix}=x_1-x_0\]

Let’s try 3-by-3 now:

\[\det(V)=\det {\renewcommand{\arraystretch}{1.5}\begin{bmatrix} 1 & x_0 & x_0^2 \\ 1 & x_1 & x_1^2 \\ 1 & x_2 & x_2^2 \\ \end{bmatrix} }\]

We can use the standard way of calculating determinants to expand from the first row:

\[\begin{aligned} \det(V)&=1\cdot(x_1 x_2^2 - x_2 x_1^2)-x_0(x_2^2-x_1^2)+x_0^2(x_2 - x_1)\\ &=x_1 x_2^2 - x_2 x_1^2 - x_0 x_2^2+x_0 x_1^2+x_0^2 x_2 - x_0^2 x_1\\ \end{aligned}\]

Using some algebraic manipulation, it’s easy to show this is equivalent to:

\[\det(V)=(x_2-x_1)(x_2-x_0)(x_1-x_0)\]

For the full proof, let’s look at the generalized n+1-by-n+1 matrix again:

Recall that subtracting a multiple of one column from another doesn’t change a matrix’s determinant. For each column k>1, we’ll subtract the value of column k-1 multiplied by from it (this is done on all columns simultaneously). The idea is to make the first row all zeros after the very first element:

\[V= {\renewcommand{\arraystretch}{1.5}\begin{bmatrix} 1 & 0 & 0 & \dots & 0\\ 1 & x_1 - x_0 & x_1^2 - x_1 x_0& \dots & x_1^n - x_1^{n-1} x_0\\ 1 & x_2 - x_0 & x_2^2 - x_2 x_0& \dots & x_2^n - x_2^{n-1} x_0\\ \vdots & \vdots & \vdots & \ddots &\vdots \\ 1 & x_n - x_0 & x_n^2 - x_n x_0& \dots & x_n^n - x_n^{n-1} x_0\\ \end{bmatrix} }\]

Now we factor out x_1-x_0 from the second row (after the first element), x_2-x_0 from the third row and so on, to get:

\[V= {\renewcommand{\arraystretch}{1.5}\begin{bmatrix} 1 & 0 & 0 & \dots & 0\\ 1 & x_1 - x_0 & x_1(x_1 - x_0)& \dots & x_1^{n-1}(x_1 - x_0)\\ 1 & x_2 - x_0 & x_2(x_2 - x_0)& \dots & x_2^{n-1}(x_2 - x_0)\\ \vdots & \vdots & \vdots & \ddots &\vdots \\ 1 & x_n - x_0 & x_n(x_n - x_0)& \dots & x_n^{n-1}(x_n - x_0)\\ \end{bmatrix} }\]

Imagine we erase the first row and first column of . We’ll call the resulting matrix .

\[W= {\renewcommand{\arraystretch}{1.5}\begin{bmatrix} x_1 - x_0 & x_1(x_1 - x_0)& \dots & x_1^{n-1}(x_1 - x_0)\\ x_2 - x_0 & x_2(x_2 - x_0)& \dots & x_2^{n-1}(x_2 - x_0)\\ \vdots & \vdots & \ddots &\vdots \\ x_n - x_0 & x_n(x_n - x_0)& \dots & x_n^{n-1}(x_n - x_0)\\ \end{bmatrix} }\]

Because the first row of is all zeros except the first element, we have:

\[\det(V)=\det(W)\]

Note that the first row of has a common factor of x_1-x_0, so when calculating \det(W), we can move this common factor out. Same for the common factor x_2-x_0 of the second row, and so on. Overall, we can write:

\[\det(W)=(x_1-x_0)(x_2-x_0)\cdots(x_n-x_0)\cdot \det {\renewcommand{\arraystretch}{1.5}\begin{bmatrix} 1 & x_1 & x_1^2 & \dots & x_1^{n-1}\\ 1 & x_2 & x_2^2 & \dots & x_2^{n-1}\\ \vdots & \vdots & \vdots & \ddots &\vdots \\ 1 & x_n & x_n^2 & \dots & x_n^{n-1} \end{bmatrix} }\]

But the smaller matrix is just the Vandermonde matrix for \{x_1 \dots x_{n}\}. If we continue this process by induction, we’ll get:

\[\det(V) = \prod_{0 \le i < j \le n} (x_j - x_i)\]

If you’re interested, the Wikipedia page for the Vandermonde matrix has a couple of additional proofs.

[1]	The -es here are called nodes and the -s are called values.

[2]	Newton polynomials is also an option, and there are many other approaches.

[3]	Note that this means the product of all differences between x_j and where is strictly smaller than j. That is, for n=2, the full product is (x_2-x_1)(x_2-x_0)(x_1-x_0). For an arbitrary , there are \frac{n(n-1)}{2} factors in total.

Notes on Linear Algebra for Polynomials

2026-02-25T18:34:00-08:00

We’ll be working with the set P_n(\mathbb{R}), real polynomials of degree \leq n. Such polynomials can be expressed using n+1 scalar coefficients a_i as follows:

\[p(x)=a_0+a_1 x + a_2 x^2 + \cdots + a_n x^n\]

Vector space

The set P_n(\mathbb{R}), along with addition of polynomials and scalar multiplication form a vector space. As a proof, let’s review how the vector space axioms are satisfied. We’ll use p(x), q(x) and r(x) as arbitrary polynomials from the set P_n(\mathbb{R}) for the demonstration. Similarly, a and b are arbitrary scalars in .

Associativity of vector addition:

\[p(x)+[q(x)+r(x)]=p(x)+q(x)+r(x)=[p(x)+q(x)]+r(x)\]

This is trivial because addition of polynomials is associative [1]. Commutativity is similarly trivial, for the same reason:

Commutativity of vector addition:

\[p(x)+q(x)=q(x)+p(x)\]

Identity element of vector addition:

The zero polynomial 0 serves as an identity element. \forall p(x)\in P_n(\mathbb{R}), we have 0 + p(x) = p(x).

Inverse element of vector addition:

For each p(x), we can use q(x)=-p(x) as the additive inverse, because p(x)+q(x)=0.

Identity element of scalar multiplication

The scalar 1 serves as an identity element for scalar multiplication. For each p(x), it’s true that 1\cdot p(x)=p(x).

Associativity of scalar multiplication:

For any two scalars a and b:

\[a[b\cdot p(x)]=ab\cdot p(x)=[ab]\cdot p(x)\]

Distributivity of scalar multiplication over vector addition:

For any p(x), q(x) and scalar a:

\[a\cdot[p(x)+q(x)]=a\cdot p(x)+a\cdot q(x)\]

Distributivity of scalar multiplication over scalar addition:

For any scalars a and b and polynomial p(x):

\[[a+b]\cdot p(x)=a\cdot p(x) + b\cdot p(x)\]

Linear independence, span and basis

Since we’ve shown that polynomials in P_n(\mathbb{R}) form a vector space, we can now build additional linear algebraic definitions on top of that.

A set of k polynomials p_k(x)\in P_n(\mathbb{R}) is said to be linearly independent if

\[\sum_{i=1}^{k}a_i p_i(x)=0\]

implies a_i=0 \quad \forall i. In words, the only linear combination resulting in the zero vector is when all coefficients are 0.

As an example, let’s discuss the fundamental building blocks of polynomials in P_n(\mathbb{R}): the set \{1, x, x^2, \dots x^n\}. These are linearly independent because:

\[a_0 + a_1 x + a_2 x^2 + \cdots a_n x^n=0\]

is true only for zero polynomial, in which all the coefficients a_i=0. This comes from the very definition of polynomials. Moreover, this set spans the entire P_n(\mathbb{R}) because every polynomial can be (by definition) expressed as a linear combination of \{1, x, x^2, \dots x^n\}.

Since we’ve shown these basic polynomials are linearly independent and span the entire vector space, they are a basis for the space. In fact, this set has a special name: the monomial basis (because a monomial is a polynomial with a single term).

Checking if an arbitrary set of polynomials is a basis

Suppose we have some set polynomials, and we want to know if these form a basis for P_n(\mathbb{R}). How do we go about it?

The idea is using linear algebra the same way we do for any other vector space. Let’s use a concrete example to demonstrate:

\[Q=\{1-x, x, 2x+x^2\}\]

Is the set Q a basis for P_n(\mathbb{R})? We’ll start by checking whether the members of Q are linearly independent. Write:

\[a_0(1-x)+a_1 x + a_2(2x+x^2)=0\]

By regrouping, we can turn this into:

\[a_0 + (a_1-a_0+2a_2)x+a_2 x^2=0\]

For this to be true, the coefficient of each monomial has to be zero; mathematically:

\[\begin{aligned} a_0&=0\\ a_1-a_0+2a_2&=0\\ a2&=0\\ \end{aligned}\]

In matrix form:

\[\begin{bmatrix} 1 & 0 & 0\\ -1 & 1 & 2\\ 0 & 0 & 1\\ \end{bmatrix} \begin{bmatrix}a_0\\ a_1\\ a_2\end{bmatrix} =\begin{bmatrix}0\\ 0\\ 0\end{bmatrix}\]

We know how to solve this, by reducing the matrix into row-echelon form. It’s easy to see that the reduced row-echelon form of this specific matrix is I, the identity matrix. Therefore, this set of equations has a single solution: a_i=0 \quad \forall i [2].

We’ve shown that the set Q is linearly independent. Now let’s show that it spans the space P_n(\mathbb{R}). We want to analyze:

\[a_0(1-x)+a_1 x + a_2(2x+x^2)=\alpha +\beta x + \gamma x^2\]

And find the coefficients a_i that satisfy this for any arbitrary , and \gamma. We proceed just as before, by regrouping on the left side:

\[a_0 + (a_1-a_0+2a_2)x+a_2 x^2=\alpha +\beta x + \gamma x^2\]

and equating the coefficient of each power of separately:

\[\begin{aligned} a_0&=\alpha\\ a_1-a_0+2a_2&=\beta\\ a2&=\gamma\\ \end{aligned}\]

If we turn this into matrix form, the matrix of coefficients is exactly the same as before. So we know there’s a single solution, and by rearranging the matrix into I, the solution will appear on the right hand side. It doesn’t matter for the moment what the actual solution is, as long as it exists and is unique. We’ve shown that Q spans the space!

Since the set Q is linearly independent and spans P_n(\mathbb{R}), it is a basis for the space.

Inner product

I’ve discussed inner products for functions in the post about Hilbert space. Well, polynomials are functions, so we can define an inner product using integrals as follows [3]:

\[\langle p, q \rangle = \int_{a}^{b} p(x) q(x) w(x) \, dx\]

Where the bounds a and b are arbitrary, and could be infinite. Whenever we deal with integrals we worry about convergence; in my post on Hilbert spaces, we only talked about L^2 - the square integrable functions. Most polynomials are not square integrable, however. Therefore, we can restrict this using either:

A special weight function w(x) to make sure the inner product integral converges
Set finite bounds on the integral, and then we can just set w(x)=1.

Let’s use the latter, and restrict the bounds into the range [-1,1], setting w(x)=1. We have the following inner product:

\[\langle p, q \rangle = \int_{-1}^{1} p(x) q(x) \, dx\]

Let’s check that this satisfies the inner product space conditions.

Conjugate symmetry:

Since real multiplication is commutative, we can write:

\[\langle p, q \rangle = \int_{-1}^{1} p(x) q(x) \, dx =\int_{-1}^{1} q(x) p(x) \, dx=\langle q, p \rangle\]

We deal in the reals here, so we can safely ignore complex conjugation.

Linearity in the first argument:

Let p_1,p_2,q\in P_n(\mathbb{R}) and a,b\in \mathbb{R}. We want to show that

\[\langle ap_1+bp_2,q \rangle = a\langle p_1,q\rangle +b\langle p_2,q\rangle\]

Expand the left-hand side using our definition of inner product:

\[\begin{aligned} \langle ap_1+bp_2,q \rangle&=\int_{-1}^{1} (a p_1(x)+b p_2(x)) q(x) \, dx\\ &=a\int_{-1}^{1} p_1(x) q(x) \, dx+b\int_{-1}^{1} p_2(x) q(x) \, dx \end{aligned}\]

The result is equivalent to a\langle p_1,q\rangle +b\langle p_2,q\rangle.

Positive-definiteness:

We want to show that for nonzero p\in P_n(\mathbb{R}), we have \langle p, p\rangle > 0. First of all, since p(x)^2\geq0 for all , it’s true that:

\[\langle p, p\rangle=\int_{-1}^{1}p(x)^2\, dx\geq 0\]

What about the result 0 though? Well, let’s say that

\[\int_{-1}^{1}p(x)^2\, dx=0\]

Since p(x)^2 is a non-negative function, this means that the integral of a non-negative function ends up being 0. But p(x) is a polynomial, so it’s continuous, and so is p(x)^2. If the integral of a continuous non-negative function is 0, it means the function itself is 0. Had it been non-zero in any place, the integral would necessarily have to be positive as well.

We’ve proven that \langle p, p\rangle=0 only when p is the zero polynomial. The positive-definiteness condition is satisfied.

In conclusion, P_n(\mathbb{R}) along with the inner product we’ve defined forms an inner product space.

Orthogonality

Now that we have an inner product, we can define orthogonality on polynomials: two polynomials p,q are orthogonal (w.r.t. our inner product) iff

\[\langle p,q\rangle=\int_{-1}^{1}p(x)q(x)\, dx=0\]

Contrary to expectation [4], the monomial basis polynomials are not orthogonal using our definition of inner product.

For example, calculating the inner product for 1 and x^2:

\[\langle 1,x^2\rangle=\int_{-1}^{1}x^2\, dx=\frac{x^3}{3}\biggr|_{-1}^{1}=\frac{2}{3}\]

There are other sets of polynomials that are orthogonal using our inner product. For example, the Legendre polynomials; but this is a topic for another post.

[1]	There’s a level of basic algebra below which we won’t descend in these notes. We could break this statement further down by saying that something like a_i x^i + a_j x^j can be added to b_i x^i + b_j x^j by adding each power of separately for any and j, but let’s just take it for granted.

[2]

Obviously, this specific set of equations is quite trivial to solve without matrices; I just want to demonstrate the more general approach. Once we have a system of linear equations, the whole toolbox of linear algebra is at our disposal. For example, we could also have checked the determinant and seen it’s non-zero, which means that a square matrix is invertible, and in this case has a single solution of zeroes.

[3]	And actually with this (or any valid) inner product, P_n(\mathbb{R}) indeed forms a Hilbert space, because it’s finite-dimensional, and every finite-dimensional inner product space is complete.

[4]	Because of how naturally this set spans P_n(\mathbb{R}). And indeed, we can define alternative inner products using which monomials are orthogonal.

LaTeX, LLMs and Boring Technology

2025-10-25T06:20:00-07:00

Depending on your particular use case, choosing boring technology is often a good idea. Recently, I've been thinking more and more about how the rise and increase in power of LLMs affects this choice.

By definition, boring technology has been around for a long time. Piles of content have been written and produced about it: tutorials, books, videos, reference manuals, examples, blog posts and so on. All of this is consumed during the LLM training process, making LLMs better and better at reasoning about such technology.

Conversely, "shiny technology" is new, and has much less material available. As a result, LLMs won't be as familiar with it. This applies to many domains, but one specific example for me personally is in the context of LaTeX.

LaTeX certainly fits the "boring technology" bill. It's decades old, and has been the mainstay of academic writing since the 1980s. When I used it for the first time in 2002 (for a project report in my university AI class), it was already very old. But people keep working on it and fixing issues; it's easy to install and its wealth of capabilities and community size are staggering. Moreover, people keep working with it, producing more and more content and examples the LLMs can ingest and learn from.

I keep hearing about the advantages of new and shiny systems like Typst. However, with the help of LLMs, almost none of the advantages seem meaningful to me. LLMs are great at LaTeX and help a lot with learning or remembering the syntax, finding the right packages, deciphering errors and even generating tedious parts like tables and charts, significantly reducing the need for scripting [1].

You can use LLMs either as standalone or fully integrated into your LaTeX environment; Overleaf has a built-in AI helper, and for local editing you can use VSCode plugins or other tools. I'm personally content with TeXstudio and use LLMs as standalone help, but YMMV.

There are many examples where boring technology and LLMs go well together. The main criticism of boring technology is typically that it's "too big, full of cruft, difficult to understand". LLMs really help cutting through the learning curve though, and all that "cruft" is very likely to become useful some time in the future when you graduate from the basic use cases.

To be clear: Typst looks really cool, and kudos to the team behind it! All I'm saying in this post is that for me - personaly - the choice for now is to stick with LaTeX as a "boring technology".

Appendix: some examples of LLMs being helpful with LaTeX

For finding the right math symbols, I rarely need to scan reference materials any longer. LLMs will easily answer questions like "what's that squiggly Greek letter used in math, and its latex symbol?" or "write the latex for Green's theorem, integral form". For the trickiest / largest equations, LLMs are very good at "here's a picture I took of my equation, give me its latex code" these days [2].
"Here's a piece of code and the LaTeX error I'm getting on it; what's wrong?"

This is made more ergonomic by editor integrations, but I personally find that LaTeX's error message problem is hugely overblown. 95% of the errors are reasonably clear, and serious sleuthing is only rarely required in practice. In that minority of cases, pasting some code and the error into a standalone LLM isn't a serious time drain.
Generating TikZ diagrams and plots. For this, the hardest part is getting started and finding the right element names, and so on. It's very useful to just ask an LLM to emit something initial and then tweak it manually later, as needed. You can also ask the LLM to explain each thing it emits in detail - this is a great learning tool for deeper understanding.

Recently I had luck going "meta" with this: when the diagram has repetitive elements, I may ask the LLM to "write a Python program that generates a TikZ diagram ...", and it works well.
Generating and populating tables, and converting them from other data formats or screenshots.
Help with formatting and typesetting (how do I change margins to XXX and spacing to YYY).

[1]

When it comes to scripting, I generally prefer sticking to real programming languages anyway. If there's anything non-trivial to auto-generate I wouldn't use a LaTeX macro, but would write a Python program to generate whatever I need and embed it into the document with something like \input{}.

Typst's scripting system may be marketed as "clean and powerful", but why learn yet another scripting language?

[2]

Ignoring LaTeX's equation notation and doing their own thing is one of the biggest mistakes Typst makes, in my opinion. LaTeX's notation may not be perfect, but it's near universal at this point with support in almost all math-aware tools.

Typst's math mode is a clear sign of the second system effect, and isn't even stable.

Notes on using LaTeX to generate formulae

2025-10-11T08:13:00-07:00

This post collects some notes on using LaTeX to render mathematical documents and formulae, mostly focused on a Linux machine. For background, I typically use LaTeX for one of two (related) purposes:

Render math for my blog posts, which are usually written using reStructuredText. This sometimes includes diagrams generated using TikZ.
Write personal (unpublished) notes on math-y subjects entirely in LaTeX. These are typically short (up to 10-20 pages), single-subject booklets.

I don't currently use LaTeX for either precise typesetting or for authoring very large, book-sized documents.

Editing

For day-to-day authoring, I find TeXstudio to be excellent. It has everything I need for local editing with a convenient preview window. I really like that TeXstudio doesn't hide the fact that it's just a graphical veneer on top of command-line LaTeX tooling, and lets you examine what it's doing through logs.

Note that web-based solutions like Overleaf exist; I can see myself using that, especially if collaborating with others or having to author LaTeX from a diverse set of computers and OSes, but for local editing of git-backed text files, TeXstudio is great.

Converting to other formats with pandoc

pandoc is very capable for converting documents from LaTeX to other formats. Recently I find that it's easier to write math-heavy blog posts in LaTeX, and then convert them to reStructuredText with pandoc.

For example, the recent post on Hilbert spaces was written like this and then converted using this command:

$ pandoc -f latex -s -t rst hilbert.tex

The resulting reStructuredText is very readable and requires very little tweaking before final publishing. pandoc supports many formats, so if you use Markdown or something else, it should work similarly well.

Rendering standalone formulae or diagrams

A useful feature of LaTeX tooling is the ability to render a specific formula in standalone mode to an image. We can write the formula into its own file (call it standaloneformula.tex):

\documentclass[preview]{standalone}
\usepackage{amsmath}
\begin{document}
\(
\int_{-\infty}^\infty e^{-x^2}\,dx=\sqrt{\pi}
\)
\end{document}

In case you were wondering, this is the Gaussian integral:

\[\int_{-\infty}^\infty e^{-x^2}\,dx=\sqrt{\pi}\]

Once we have that standalone .tex file, there's a number of things we can do. First, the texlive package should be installed [1]. Using apt:

$ sudo apt install texlive-full

Now, we can run the tools from texlive, for example pdflatex:

$ pdflatex standaloneformula.tex

This creates a PDF file that's useful for previews. To convert the .tex file to an image in SVG format, we'll use a two-step process:

$ latex standaloneformula.tex

... generates standaloneformula.dvi

$ dvisvgm standaloneformula.dvi

... generates standaloneformula.svg

If we want a PNG file instead of SVG:

$ dvipng -D 300 standaloneformula.dvi

The latexmk tool can build a .tex file into a PDF whenever the input file changes, so running:

$ latexmk -pvc standaloneformula.tex

And opening the PDF in a separate window, we can observe live refreshes of edits without having to recompile explicitly. While useful in some scenarios, I find that TeXstudio already does this well.

The same tooling flow works for TikZ diagrams! A standalone LaTeX document containing a single tikzpicture element can also be rendered to a SVG or PNG using the same exact commands.

Docker versions

If you'd rather not install all these tools directly but use Docker instead, the texlive image can be used to do the same things:

$ docker pull texlive/texlive:latest

And now we can use the same invocations, just through docker:

$ docker run --rm -u $(id -u):$(id -g) \
    -v "$PWD":/workdir -w /workdir texlive/texlive:latest \
    latex standaloneformula.tex

$ docker run --rm -u $(id -u):$(id -g) \
    -v "$PWD":/workdir -w /workdir texlive/texlive:latest \
    dvisvgm standaloneformula.dvi

Positioning of formulae embedded in text

When a formula like \frac{n+1}{n^2-1} is embedded in text, it should be aligned properly to look good with the surrounding text. The information required to do this is emitted by tools like dvisvgm and dvipng; for example:

$ dvisvgm standaloneformula.dvi
pre-processing DVI file (format version 2)
processing page 1
  computing extents based on data set by preview package (version 14.0.6)
  width=81.267395pt, height=9.86894pt, depth=4.388947pt
  graphic size: 81.267395pt x 14.257887pt (28.562223mm x 5.011074mm)
  output written to standaloneformula.svg
1 of 1 page converted in 0.147623 seconds

Note the height=..., depth=... line in the output. The height is the total height of the formula, and depth is its height below the "baseline" (how much down it should stick out from the line). In my blog, these two are translated to attributes on the image element embedding the SVG. Height is translated to style="height: ... and depth to vertical-align: ....

[1]	If the machine already has TeXstudio installed, `texlive` is almost certainly installed as well.

Hilbert space: treating functions as vectors

2025-09-06T06:46:00-07:00

The tools of linear algebra are extremely useful when working in Euclidean space (e.g. \mathbb{R}^3). Wouldn’t it be great if we could apply these tools to additional mathematical constructs, such as functions and sequences? Hilbert space allows us to do exactly this - apply linear algebra to functions.

Intuition - functions as infinite-dimensional vectors

There are several ways to view vectors; a standard interpretation is an ordered list of numbers. Let’s take a vector in \mathbb{R}^3 as an example:

\[v = \begin{bmatrix} 1.4 \\ 4.2 \\ -2.14 \end{bmatrix}\]

This is a list of three numbers, where each number has an index. v[1] is 1.4, v[2] is 4.2 and so on. Another way to think of a vector is a function, in the strict mathematical sense. A vector in \mathbb{R}^3 is a function with the domain {1,2,3} (the indices) and codomain , or:

\[v:\{1,2,3\}\to\mathbb{R}\]

Now imagine that our vector is N-dimensional: \mathbb{R}^N. Using the function notation we can write v:\{1,2,\cdots ,N\}\to\mathbb{R}. This works for any N, and in fact it also works for an infinite N. Our vector then simply becomes a function from the natural numbers to the reals: v:\mathbb{N}\to\mathbb{R}.

But we can take it even further; what if we allow any real number as an index? Our vector is then v:\mathbb{R}\to\mathbb{R}, or we may just change its name to be more familiar: f:\mathbb{R}\to\mathbb{R}. This "vector" is just a function from the reals to the reals.

While we can’t write all the elements down explicitly (there’s an infinite number of them, and most of the indices are irrational which don’t even have a finite representation), we can instead come up with a rule that maps an index to the element. For example: f(x)=x^2 is such a rule. For any given index x, it assigns the value x^2. We’re not used to thinking of functions as vectors, but if we carefully extend some definitions, it’s entirely possible!

So, functions can be seen as vectors with infinite dimensions. The next step is to see how we can define a vector space for functions.

Functions form a vector space

Functions, together with the standard addition and scalar multiplication operations form a vector space.

For full generality, let be any set and \mathbb{F}\in\{\mathbb{R},\mathbb{C}\} (either reals or complex numbers). Let be the set of all functions mapping X\to\mathbb{F}. For f,g\in V and a number a \in \mathbb{F}, we define function addition and scalar multiplication as follows:

\[[f+g](x)=f(x)+g(x)\qquad [a\cdot f](x)=a\cdot f(x)\]

Then along with these operations forms a vector space over \mathbb{F}. For a proof, see Appendix A.

Square integrable functions

A vector space is useful, but to get to Hilbert space and be able to do more interesting operations on functions, we need some additional structure.

From here on, we’ll switch to functions with complex values (functions with real values are just a special case). A function f:\mathbb{R}\to\mathbb{C} is said to be square integrable if:

\[\int_{-\infty}^{\infty}\left | f(x) \right |^2 dx < \infty\]

The set of such functions is commonly denoted L^2, and it forms a subspace of the vector space we discussed in the previous section (for a proof, see Appendix B).

The integral over the square of the function is equivalent to the Euclidean norm for vectors; intuitively, it acts as a measure of length, which is the term used in vectors. For functions, it’s typically referred to as energy [1].

Inner product and norm

To add more tools from the linear algebra toolbox, let’s define an inner product on L^2:

\[\langle f,g \rangle=\int_{-\infty}^{\infty}f^{*}(x)g(x) dx\]

Why is it defined in this way? Here’s the definition of inner product between two N-dimensional vectors with complex values:

\[\langle u,v \rangle=\sum_{i=1}^{N}u_{i}^{*} v_{i}\]

Looks familiar? The function version is just the generalization of this sum over an infinite range (the entire x-axis, if you will), using an integral.

As the next step, we want to show that L^2 is an inner product space, when taken with the inner product operation as defined above. For this to be true, first and foremost we have to show that the inner product is finite for every pair of functions in L^2 (if the integral doesn’t converge, it’s not something we can work with). This can be done using the integral form of the Cauchy-Schwarz inequality:

\[\int_{-\infty}^{\infty}f^{*}(x)g(x) dx \leq \sqrt{\int_{-\infty}^{\infty}|f(x)|^2} dx \sqrt{\int_{-\infty}^{\infty}|g(x)|^2} dx\]

Since f,g\in L^2, the right hand side is finite, and therefore the inner product is finite as well. This is where the square integrability of functions in L^2 comes into play - without being square integrable, the inner product would be impossible to define.

The other properties of inner products can also be demonstrated readily, and there are plenty of resources online that show how [2].

Therefore L^2, coupled with the inner product operation shown here forms an inner product space.

This inner product can be used to define a norm for our space:

\[\|f\| = \sqrt{\langle f,f\rangle} = \sqrt{\int_{-\infty}^{\infty} |f(x)|^2 \, dx}\]

Once again, because our functions in L^2 are square integrable, the norm exists and it’s easy to show it satisfies all the usual requirements for a norm.

Are we Hilbert yet?

We’ve seen that the set of square integrable functions L^2 forms a proper vector space and also an inner product space when coupled with an inner product operation; it also has a norm. So does it have all that’s needed for linear algebra?

Almost. This space should also be complete. The term "complete" is severely overloaded in math, so it’s important to say what it means in this context: put simply, it means the set has no "holes" - no sequence of elements in the set converges to an element outside the set [3]. To put it less simply, a space is complete if all Cauchy sequences of elements of this space converge.

This gets us deep into the large and advanced topic of real analysis. The Riesz-Fischer theorem shows that L^2 is complete.

Once we add completeness to the set of properties of L^2, it becomes a Hilbert space.

You may also hear the term Banach space mentioned in this context. Banach spaces are more general than Hilbert spaces: a complete space with a norm is a Banach space (this norm doesn’t have to come from an inner product). A complete inner-product space is a Hilbert space - the norm of a Hilbert space is defined using its inner product, as we’ve seen above.

Application: generalized Fourier series

The Fourier series is one of the most brilliant and consequential ideas in mathematics. I would really like to dive deeper into this topic, but that would require a post (or a book) of its own.

In short, Fourier series can be defined for functions in L^2 because these form a Hilbert space. Specifically, the inner product for functions lets us define orthogonality and the concept of basis vectors in L^2. These are then used to express any function as a weighted sum of a series of basis functions that span the space. Moreover, the completeness of the space guarantees that Fourier series actually converge to functions within the space.

Interestingly, Fourier put forward his ideas decades before the field of analysis matured and Hilbert space was defined. This is why many mathematicians of the day (most notably Lagrange) objected to Fourier theory as not sufficiently rigorous. The theory worked brilliantly for many useful scenarios, however, and later developments in functional analysis helped put it on a more solid theoretical footing.

Another related example which I find very cool: I’ve mentioned how this theory helps us apply the tools of linear algebra to functions, and generalized Fourier series provides an excellent illustration.

Most people are familiar with the trigonometric Fourier series, but the theory is more general and applies to any set of mutually orthogonal functions that form a basis for the vector space. Is there a polynomial Fourier series? Yes, and it can be derived using one of the classical tools of linear algebra - the Gram-Schmidt process. The result is Legendre polynomials.

Again, all of this is fascinating and I hope to be able to write more on this topic in the future.

Application: Quantum mechanics

In QM, states of particles are described by wavefunctions in a Hilbert space. The inner product can be interpreted as a probability. QM operators can be seen as linear maps on that space. This lets us apply linear algebra in infinite dimensions and unlocks a treasure chest of useful mathematical tools.

Appendix A: proof of vector space axioms for functions

As a reminder, we’re dealing with the set of functions f:X\to\mathbb{F}, where is any set and \mathbb{F} can be either or \mathbb{C}. This set , along with addition between set members and scalar multiplication form a vector space. To prove this, we prove all the vector space axioms:

Associativity of vector addition:

\[\begin{aligned} [f+[g+h]](x)&=f(x)+[g+h](x)\\ &=f(x)+g(x)+h(x)\\ &=[f+g](x)+h(x)\\ &=[[f+g]+h](x) \end{aligned}\]

This proceeds very smoothly because addition on either reals or complex numbers is associative, commutative, etc.

Commutativity of vector addition:

\[\begin{aligned} [f+g](x)&=f(x)+g(x)\\ &=g(x)+f(x)\\ &=[g+f](x) \end{aligned}\]

Identity element of vector addition:

The function z(x)=0 serves as an additive identity element:

\[\begin{aligned} f(x)+z(x)&=f(x)\qquad \forall x \\ [f+x](x)&=f(x) \end{aligned}\]

Inverse elements of vector addition:

We’ll define (-f)(x)=-f(x) as the additive inverse, and z(x) as before:

\[\begin{aligned} f(x)-f(x)&=z(x)\qquad \forall x \\ [f+(-f)](x)&=z(x) \end{aligned}\]

Associativity of scalar multiplication:

For a scalar a\in\mathbb{F}:

\[a(bf(x))=ab(f(x))=(ab)f(x)\]

Identity element of scalar multiplication:

We’ll use the scalar 1 as the identity element of scalar multiplication. Since the result of is a real or complex scalar, it’s trivially true that:

\[1\cdot f(x)=f(x)\]

Distributivity of scalar multiplication over vector addition:

\[\begin{aligned} a\cdot [f+g](x)&=a\cdot (f(x)+g(x))\\ &=a\cdot f(x)+a\cdot g(x)\\ &=[af](x)+[ag](x) \end{aligned}\]

Distributivity of scalar multiplication over scalar addition:

For scalars a and b:

\[(a+b)\cdot f(x)=a\cdot f(x) + b\cdot f(x)\]

Appendix B: proof that square integrable functions form a subspace

To show that L^2 is a subspace of the function vector space, we have to prove the following properties:

Zero element

The zero element z(x)=0 is in L^2:

\[\int_{-\infty}^{\infty}\left | z(x) \right |^2 dx =\int_{-\infty}^{\infty} 0\ dx=0 < \infty\]

Closure under addition

Recall that our functions f,g\in L^2 are complex-valued. For any two complex numbers (see this post):

\[|u+v|^2=|u|^2+|v|^2+2 Re(uv^*)\]

It’s very easy to show that 2Re(uv^*)\leq2|u||v|, and also that 2|u||v|\leq|u|^2+|v|^2. Therefore:

\[|u+v|^2\leq2|u|^2+2|v|^2\]

Armed with this, let’s check if the sum of and g(x) is square integrable:

\[\int_{-\infty}^{\infty}\left | f(x) + g(x)\right |^2 dx\]

Since the values and g(x) are just complex numbers, we’ll use the inequality shown above to write:

\[\int_{-\infty}^{\infty}\left | f(x) + g(x)\right |^2 dx \leq 2\int_{-\infty}^{\infty}\left | f(x) \right |^2 dx+ 2\int_{-\infty}^{\infty}\left | g(x) \right |^2 dx\]

Both integrals on the right hand side are finite, so the one on the left is finite as well.

Closure under scalar multiplication

Given f\in L^2 and a scalar a:

\[\int_{-\infty}^{\infty}a \left | f(x) \right |^2 dx =a^2 \int_{-\infty}^{\infty} \left | f(x) \right |^2\ dx=< \infty\]

[1]	We want to work with functions that have finite total energy. Note that this is a pretty strong restriction! In Fourier analysis, we typically modify the square integrability requirement to be on a finite interval - all the tools still work - and talk about periodic functions.

[2]	I’m not including the proofs here, because some of them are a bit technical and require terminology from real analysis.

[3]	A classical example of a set not satisfying this condition is \mathbb{Q} - the rational numbers; an infinite sum of rational numbers can end up being irrational: \sum_{n=0}^{\infty}\frac{1}{n!}=e. Infinite sums are tricky!