18 Covariance and Correlation

Quantities like expected value and variance summarize characteristics of the marginal distribution of a single random variable. When there are multiple random variables their joint distribution is of interest. Covariance and correlation summarize a characteristic of the joint distribution of two random variables, namely, the degree to which they “co-deviate from the their respective means”.

Example 18.1 We’ll motivate the calculation of covariance by considering a small example. Say we have data on the sale price \(X\) (measured in millions of dollars) and the living area \(Y\) (measured in thousands of square feet) for a small sample of 10 houses.

repetition	X	Y
1	0.28	1.12
2	0.63	1.05
3	0.76	1.67
4	0.56	1.48
5	0.36	0.43
6	0.89	2.50
7	0.47	1.71
8	0.82	1.88
9	0.71	0.85
10	0.75	1.90
Sum	6.23	14.59
Average	0.62	1.46

Compute the average of the \(X\) values and the average of the \(Y\) values.
Compute the deviation from its mean for each \(X\) value, and the deviation from its mean for each \(Y\) value.
Compute the product of the paired deviations.
Compute the average of the products of the deviations. This is called the covariance. What are the measurement units?

The covariance between two random variables \(X\) and \(Y\) is \[\begin{align*} \text{Cov}(X,Y) & = \text{E}\left[\left(X-\text{E}[X]\right)\left(Y-\text{E}[Y]\right)\right]\\ & = \text{E}(XY) - \text{E}(X)\text{E}(Y) \end{align*}\]
Covariance is the long run average of the products of the paired deviations from the mean
Covariance is the expected value of the product minus the product of expected values.

Example 18.2 Roll a fair four-sided die twice and let \(X\) be the sum of the two rolls, and \(Y\) the larger of the two rolls. Recall Example 17.1 and Table 17.1.

Will \(\text{Cov}(X,Y)\) be positive, negative, or close to 0? Explain without computing.
Compute \(\text{Cov}(X,Y)\) using the definition.
Compute \(\text{Cov}(X,Y)\) using the “shortcut” formula.

\(\text{Cov}(X,Y)>0\) (positive association): above average values of \(X\) tend to be associated with above average values of \(Y\)
\(\text{Cov}(X,Y)<0\) (negative association): above average values of \(X\) tend to be associated with below average values of \(Y\)
\(\text{Cov}(X,Y)=0\) indicates that the random variables are uncorrelated: there is no overall positive or negative association. But be careful: if \(X\) and \(Y\) are uncorrelated there can still be a relationship between \(X\) and \(Y\); there is just no overall positive or negative association.. We will see examples that demonstrate that being uncorrelated does not necessarily imply that random variables are independent.

Example 18.3 We’ll motivate the calculation of correlation by considering a small example. Continuing Example 18.1, say we have data on the sale price \(X\) (measured in millions of dollars) and the living area \(Y\) (measured in thousands of square feet) for a small sample of 10 houses.

repetition	X	Y	Standardized X	Standardized Y	Product
1	0.28	1.12
2	0.63	1.05
3	0.76	1.67
4	0.56	1.48
5	0.36	0.43
6	0.89	2.50
7	0.47	1.71
8	0.82	1.88
9	0.71	0.85
10	0.75	1.90
Sum	6.23	14.59	0	0
Average	0.62	1.46	0	0
SD	0.19	0.57	1	1	NA

Compute the average and SD of the \(X\) values and the average and SD of the \(Y\) values.
Compute the standardized value of each \(X\) value, and the standardized value of each \(Y\) value.
Compute the product of the paired standardized values.
Compute the average of the products of the standardized values. This is called the correlation.
Divide the covariance from Example 18.1 by the product of the two standard deviations. What do you notice?

The correlation (coefficient) between random variables \(X\) and \(Y\) is \[\begin{align*} \text{Corr}(X,Y) & = \text{Cov}\left(\frac{X-\text{E}(X)}{\text{SD}(X)},\frac{Y-\text{E}(Y)}{\text{SD}(Y)}\right)\\ & = \frac{\text{Cov}(X, Y)}{\text{SD}(X)\text{SD}(Y)} \end{align*}\]
The correlation for two random variables is the covariance between the corresponding standardized random variables. Therefore, correlation is a standardized measure of the association between two random variables.
A correlation coefficient has no units and is measured on a universal scale. Regardless of the original measurement units of the random variables \(X\) and \(Y\) \[ -1\le \textrm{Corr}(X,Y)\le 1 \]
\(\textrm{Corr}(X,Y) = 1\) if and only if \(Y=aX+b\) for some \(a>0\)
\(\textrm{Corr}(X,Y) = -1\) if and only if \(Y=aX+b\) for some \(a<0\)
Therefore, correlation is a standardized measure of the strength of the linear association between two random variables.
Covariance is the correlation times the product of the standard deviations. \[ \text{Cov}(X, Y) = \text{Corr}(X, Y)\text{SD}(X)\text{SD}(Y) \]

18.1 Exercises

Exercise 18.1 Continuing Example 18.2. Let \(X\) be the sum of the two rolls, \(Y\) the larger of the two rolls, \(W\) the number of rolls equal to 4, and \(Z\) the number of rolls equal to 1. You should explain your answers without doing any computation.

Is \(\text{Cov}(X, W)\) positive, negative, or zero? Why?
Is \(\text{Cov}(X, Z)\) positive, negative, or zero? Why?
Let \(V=W+Z\); what does this represent? Is \(\text{Cov}(X, V)\) positive, negative, or zero? Why?
Is \(\text{Cov}(W, Z)\) positive, negative, or zero? Why?

Exercise 18.2 What is another name for \(\text{Cov}(X, X)\)?

Exercise 18.3 Sketch a few scatterplots where \(X\) and \(Y\) have a correlation of 0, but they are not independent.

Exercise 18.4 Try the Guessing Correlations applet! Since we’re interested in the long run, set number of points to 1000. There is nothing to submit for this exercise but play around with the applet to get a feel for the difference between a correlation of 0.8-ish versus 0.5-ish versus 0.2-ish.