Variance and Covariance of Random Variables

Variance and Covariance of Random Variables

Variance and covariance are important concepts in probability theory and statistics that measure the spread (or variability) of random variables and the relationship between two random variables, respectively. Let's go over each concept in detail.

1. Variance of a Random Variable

Definition

The variance of a random variable measures the spread or dispersion of its values around the expected value (mean). It quantifies how much the values of the random variable deviate, on average, from the expected value.

The variance of a random variable $X$, denoted by $\text{Var}(X)$, is defined as the expected value of the squared deviation from the mean $E[X]$:

Alternatively, the variance can be computed using the following formula:

This formula is useful because it allows you to compute the variance by finding the expected value of $X^2$ and subtracting the square of the expected value of $X$.

Interpretation

• A higher variance indicates that the values of $X$ are spread out over a larger range, meaning the data is more dispersed.
• A lower variance indicates that the values of $X$ are clustered more closely around the expected value, meaning the data is less dispersed.

Example

Let’s say you have a random variable $X$ with possible outcomes $1, 2, 3$, and corresponding probabilities $P(X = 1) = 0.2$, $P(X = 2) = 0.5$, and $P(X = 3) = 0.3$.

1. First, calculate the expected value $E[X]$:

   $$E[X] = (1 \cdot 0.2) + (2 \cdot 0.5) + (3 \cdot 0.3) = 0.2 + 1 + 0.9 = 2.1$$

2. Next, calculate $E[X^2]$:

   $$E[X^2] = (1^2 \cdot 0.2) + (2^2 \cdot 0.5) + (3^2 \cdot 0.3) = (1 \cdot 0.2) + (4 \cdot 0.5) + (9 \cdot 0.3) = 0.2 + 2 + 2.7 = 4.9$$

3. Finally, compute the variance:

   $$\text{Var}(X) = E[X^2] - (E[X])^2 = 4.9 - (2.1)^2 = 4.9 - 4.41 = 0.49$$

Thus, the variance of $X$ is 0.49.

2. Standard Deviation

The standard deviation is the square root of the variance and provides a measure of the spread of the random variable in the same units as the random variable itself. It is given by:

Standard deviation is often easier to interpret because it is in the same units as the data.

In our example, since $\text{Var}(X) = 0.49$, the standard deviation is:

3. Covariance of Two Random Variables

Definition

The covariance of two random variables $X$ and $Y$ measures the degree to which the two variables change together. If $X$ and $Y$ tend to increase or decrease together, the covariance will be positive. If one tends to increase while the other decreases, the covariance will be negative. A covariance of zero indicates that the two variables are linearly independent.

The covariance between two random variables $X$ and $Y$, denoted $\text{Cov}(X, Y)$, is defined as:

Alternatively, the covariance can be calculated using the following formula:

Interpretation

• A positive covariance indicates that as $X$ increases, $Y$ tends to increase as well (and vice versa).
• A negative covariance indicates that as $X$ increases, $Y$ tends to decrease.
• A covariance of zero indicates that there is no linear relationship between the two variables.

Example

Let’s say we have two random variables $X$ and $Y$ with the following values and probabilities:

1. First, calculate the expected values $E[X]$ and $E[Y]$:

   $$E[X] = (1 \cdot 0.1) + (1 \cdot 0.4) + (2 \cdot 0.3) + (2 \cdot 0.2) = 0.1 + 0.4 + 0.6 + 0.4 = 1.5$$ $$E[Y] = (2 \cdot 0.1) + (3 \cdot 0.4) + (2 \cdot 0.3) + (4 \cdot 0.2) = 0.2 + 1.2 + 0.6 + 0.8 = 2.8$$

2. Next, calculate $E[XY]$:

   $$E[XY] = (1 \cdot 2 \cdot 0.1) + (1 \cdot 3 \cdot 0.4) + (2 \cdot 2 \cdot 0.3) + (2 \cdot 4 \cdot 0.2)$$ $$E[XY] = (2 \cdot 0.1) + (3 \cdot 0.4) + (4 \cdot 0.3) + (8 \cdot 0.2)$$ $$E[XY] = 0.2 + 1.2 + 1.2 + 1.6 = 4.2$$

3. Finally, compute the covariance:

   $$\text{Cov}(X, Y) = E[XY] - E[X]E[Y] = 4.2 - (1.5 \cdot 2.8) = 4.2 - 4.2 = 0$$

Thus, $\text{Cov}(X, Y) = 0$, meaning that there is no linear relationship between $X$ and $Y$ in this example.

4. Properties of Covariance

Covariance has the following important properties:

• Symmetry:

  $$\text{Cov}(X, Y) = \text{Cov}(Y, X)$$

• Linear Scaling: If $X$ and $Y$ are random variables and $a$ and $b$ are constants, then:

  $$\text{Cov}(aX + b, Y) = a \cdot \text{Cov}(X, Y)$$

• Covariance of a Random Variable with Itself: The covariance of a random variable with itself is simply its variance:

  $$\text{Cov}(X, X) = \text{Var}(X)$$

5. Correlation Coefficient

The correlation coefficient is a normalized measure of the relationship between two random variables. It is defined as:

The correlation coefficient ranges from -1 to 1:

• A value of 1 indicates a perfect positive linear relationship.
• A value of -1 indicates a perfect negative linear relationship.
• A value of 0 indicates no linear relationship.

Summary

1. Variance measures the spread of a random variable around its expected value.
2. Covariance measures the relationship between two random variables—whether they tend to increase or decrease together.
3. Standard Deviation is the square root of the variance and is a measure of the spread in the same units as the data.
4. The correlation coefficient normalizes the covariance to provide a standardized measure of the strength and direction of the linear relationship between two variables.

These concepts are fundamental to understanding how random variables behave, both individually (through variance) and in relation to one another (through covariance).