Yesterday I was trying to brush up my skills in probability and came upon this formula on the Wikipedia page about variance:
The article calls this the Bienaymé formula and gives neither proof nor a link to one. Googling this formula proved equally fruitless in terms of proofs.
So, I set out to find why this works. It took me a few hours of digging through books and removing dust from my University-learned probability skills of 8 years ago, but finally I've made it. Here's how.
Note: the Wikipedia article states the Bienaymé formula for uncorrelated variables. Here I'll prove the case of independent variables, which is a more useful and frequently used application of the formula. I'm also proving it for discrete random variables - the continuous case is equivalent.
Expected value and variance
We'll start with a few definitions. Formally, the expected value of a (discrete) random variable X is defined by:
Where is the PMF of X, . For a function :
The variance of X is defined in terms of the expected value as:
From this we can also obtain:
Which is more convenient to use in some calculations.
Linear function of a random variable
From the definitions given above it can be easily shown that given a linear function of a random variable: , the expected value and variance of Y are:
For the expected value, we can make a stronger claim for any g(x):
Multiple random variables
When multiple random variables are involved, things start getting a bit more complicated. I'll focus on two random variables here, but this is easily extensible to N variables. Given two random variables that participate in an experiment, their joint PMF is:
The joint PMF determines the probability of any event that can be specified in terms of the random variables X and Y. For example if A is the set of all pairs that have a certain property, then:
Note that from this PMF we can infer the PMF for a single variable, like this:
The expected value for functions of two variables naturally extends and takes the form:
Sum of random variables
Let's see how the sum of random variables behaves. From the previous formula:
But recall equation (1). The above simply equals to:
We'll also want to prove that . This is only true for independent X and Y, so we'll have to make this assumption (assuming that they're independent means that ).
A very similar proof can show that for independent X and Y:
For any functions g and h (because if X and Y are independent, so are g(X) and h(y)). Now, at last, we're ready to tackle the variance of X + Y. We start by expanding the definition of variance:
Now, note that the random variables and are independent, so:
But using (2) again:
is obviously just , therefore the above reduces to 0.
So, coming back to the long expression for the variance of sums, the last term is 0, and we have:
As I've mentioned before, proving this for the sum of two variables suffices, because the proof for N variables is a simple mathematical extension, and can be intuitively understood by means of a "mental induction". Therefore:
For N independent variables .