24  Variance and Mean Square Error of an Estimator

• The bias function of an estimator measures the estimator’s tendency to overestimate or underestimate the parameter, as a function of potential values of the parameter.
• But bias is only one consideration. Bias only considers the average value of the estimator over many samples, which is just one feature of the estimator’s sampling distribution.
• Statistics exhibit sample-to-sample variability: the value of a statistic varies from sample to sample.
• When choosing between two unbiased estimators, the one with smaller variance is generally preferred.
• Remember that the variance of an estimator will be a function of the unknown parameter \(\theta\), so we need to consider the variance function for various potential values of \(\theta\).

Example 24.1 Consider again estimating \(\mu\) for a Poisson(\(\mu\)) distribution based on a random sample \(X_1, \ldots, X_n\) of size \(n\). We have seen that both \(\bar{X}\) and \(S^2\) are unbiased estimators of \(\mu\). If we want to choose between these two estimators, how do we decide?

1. Assume \(n=3\) and \(\mu=2.3\). Describe in full detail how you could conduct a simulation to approximate the sample-to-sample distribution of \(\bar{X}\) and its and its expected value and standard deviation. Then conduct the simulation and record the results. What does the standard deviation measure?
   
   
   
   
   
2. Repeat part 1 for \(S^2\).
   
   
   
   
3. Compare the simulation results for \(\bar{X}\) and \(S^2\) when \(n=3\) and \(\mu = 2.3\). Based on the simulation results, which estimator of \(\mu\) is preferred when \(\mu = 2.3\) — \(\bar{X}\) or \(S^2\)? Why? But then explain why this information isn’t very helpful.
   
   
   
   
   

• For an i.i.d. random sample \(X_1,\ldots, X_n\), the sample mean is the statistic \[ \bar{X}_n = \frac{1}{n}\sum_{i=1}^n X_i = \frac{X_1 + \cdots + X_n}{n} \]
• The sample mean \(\bar{X}\) is an unbiased estimator of the population mean \(\mu\): \[ \textrm{E}(\bar{X}_n) = \mu \]
• Sample-to-sample variability of sample means decreases as the sample size increases \[\begin{align*} \textrm{SD}(\bar{X}_n) & = \frac{\sigma}{\sqrt{n}}\\ \text{sample-to-sample SD of sample means} & = \frac{\text{unit-to-unit SD of values of the variable }}{\sqrt{\text{sample size}}} \end{align*}\]
  • The more variable the individual values of the variable are (i.e. the larger the population SD of the variable, \(\sigma\)), the more variable sample means will be (i.e. the larger than the SE).
  • Sample means are less variable than individual values of the variable
  • Over many random samples, sample means from larger random samples vary less, from sample to sample, than sample means from smaller random samples.
  • The SD of the sample means decreases as the sample size increases, but this reduction in SD follows a “square root rule”. For example, if the sample size is increased by a factor of 4, then the SD is reduced by a factor of 2 (not 4).
    

Example 24.2 Continuing Example 24.1 where the population is Poisson(\(\mu\)).

1. Assuming \(n= 3\) and \(\mu = 2.3\), compute \(\textrm{E}(\bar{X})\) and \(\textrm{SD}(\bar{X})\), and compare to the simulation results.
   
   
   
   
   
2. For a general \(n\) and \(\mu >0\), find expressions for \(\textrm{E}(\bar{X})\) and \(\textrm{SD}(\bar{X})\).
   
   
   
   
   
3. It can be shown that \[ \textrm{Var}(S^2) = \frac{2\mu^2}{n-1} + \frac{\mu}{n} \] Sketch a plot of the variance functions of both \(\bar{X}\) and \(S^2\) as a function of \(\mu\). Regardless of the value of \(\mu\), which estimator has smaller variance? Which of these two unbiased estimators of Poisson \(\mu\), \(\bar{X}\) or \(S^2\), is preferred?
   
   
   
   
   
4. Suppose \(n=3\) and the sample is \((3, 0, 2)\). For this sample \(\bar{x} = 1.67\) and \(s^2 = 2.33\). Which number, 1.67 or 2.33, is a better estimate of \(\mu\)? Explain.
   
   
   
   
   
5. Suppose \(n=3\) and the sample is \((3, 0, 2)\). For this sample \(\bar{x} = 1.67\) and \(s^2 = 2.33\). Which number, 1.67 or 2.33, would you choose as the estimate of \(\mu\) based on this sample? Why?
   
   
   
   
   

• MLE is one procedure for finding an estimator of a parameter \(\theta\). But when several estimators of \(\theta\) are available, how do we decide which is “better”?
• The value of a parameter \(\theta\) is unknown, so it is impossible to determine if any single estimate of \(\theta\) is good or bad.
• An estimator is a random variable that returns different estimates of \(\theta\) for different samples. So even if an estimator produces “good” estimates of \(\theta\) for some samples, it might produce “bad” estimates of \(\theta\) for others.
• Therefore, we can never determine for any particular sample if an estimator produces a good estimate of the true unknown value of \(\theta\).
• Rather, we evaluate the estimation procedure: Over many samples, does an estimator tend to produce reasonable estimates of \(\theta\) for a variety of potential values of \(\theta\)?
• Remember: a statistic/estimator is a random variable whose sampling distribution distribution describes how values of the estimator vary from sample-to-sample over many (hypothetical) samples.
  • We can estimate the degree of this variability by simulating many hypothetical samples from an assumed population distribution and computing the value of the statistic for each sample.
  • However, in practice usually only a single sample is selected and a single value of the statistic is observed, and we don’t know what the population distribution is.

Example 24.3 Continuing the situation of estimating \(\mu\) for a Poisson(\(\mu\)) distribution based on a random sample \(X_1, \ldots, X_n\) of size \(n\). Consider the estimators \[\begin{align*} S^2 & = \frac{1}{n-1}\sum_{i=1}^n \left(X_i - \bar{X}\right)^2\\ \hat{\sigma}^2 & = \frac{1}{n}\sum_{i=1}^n \left(X_i - \bar{X}\right)^2 = \left(\frac{n-1}{n}\right)S^2 \end{align*}\] We have seen that \(S^2\) is an unbiased estimator of \(\mu\) but \(\hat{\sigma}^2\) is not. Does that mean we should prefer \(S^2\) over \(\hat{\sigma}^2\)?

1. Assume \(n=3\) and \(\mu=2.3\). Use simulation to approximate the distribution of \(\hat{\sigma}^2\) and its expected value and standard deviation.
   
   
   
   
   
2. Compare the simulation results to those for \(S^2\) (from earlier). For which estimator is the bias smaller? For which estimator is the variance smaller? Is there a clear preference between the two estimators when \(\mu = 2.3\)?
   
   
   
   
   

• Estimators can be evaluated based on both bias and variability.
• Mean square error is a combined measure of both an estimator’s bias and its variability.
• The mean square error (MSE) of an estimator \(\hat{\theta}\) of a parameter \(\theta\) is \[ \text{MSE}_\theta(\hat{\theta}) = \textrm{E}\left(\left(\hat{\theta}-\theta\right)^2\right), \qquad \text{a function of $\theta$} \]
• Mean square error measures, on average, how far the estimator deviates from the parameter it is estimating.
• The MSE of an estimator is a function of the parameter \(\theta\). It can be shown that¹ \[\begin{align*} \text{MSE}_\theta(\hat{\theta}) & = \textrm{Var}(\hat{\theta}) + \left(\textrm{E}(\hat{\theta})-\theta\right)^2\\ & = \textrm{Var}(\hat{\theta}) +\left(\text{bias}_\theta(\hat{\theta})\right)^2 \end{align*}\]
• There can be many “reasonable” estimators of a parameter. Given two estimators, it’s often the situation that neither one has smaller MSE for all potential values of \(\theta\). Choosing an estimator often involves a tradeoff between bias and variability.

Example 24.4 Continuing Example 24.3.

1. Use the simulation results to compute the MSE of \(S^2\) and \(\hat{\sigma}^2\) when \(\mu = 2.3\). Determine which estimator has smaller MSE when \(\mu=2.3\), but then explain why this information is not very useful.
   
   
   
   
   
2. Plot the MSE functions of \(S^2\) and \(\hat{\sigma}^2\). Which estimator is preferred? Discuss.
   
   
   
   
   

Example 24.5 Continuing estimating \(\mu\) for a Poisson(\(\mu\)) distribution. Consider \(\bar{X}\) and the constant estimator 4.2.

1. Esplain what it means to use 4.2 as an estimation procedure.
   
   
   
   
   
2. Find the MSE of the constant estimator 4.2.
   
   
   
   
   
3. Find the MSE of \(\bar{X}\).
   
   
   
   
   
4. Suppose \(n=3\). Plot the MSE functions of the two estimators. Does either estimator have a better MSE? Explain.
   
   
   
   
   
5. Donny Don’t says: “neither estimator has smaller MSE for all values of \(\mu\). So we’ll use the constant estimator 4.2 when \(\mu\) is near 4.2 (between 3.17 and 5.56 if \(n=3\)) and we’ll use \(\bar{X}\) for other values of \(\mu\)”. Do you agree with Donny’s strategy? Do you see any problems with it?
   
   
   
   
   
6. What happens to the MSEs as \(n\) increases? In particular, what happens to the range of values of \(\mu\) for which the constant estimator 4.2 has smaller MSE than \(\bar{X}\)?
   
   
   
   
   

• Given a parameter \(\theta\), it is never possible to find a single estimator that has the smallest MSE for all potential values of \(\theta\).
• Choosing an estimator often involves a tradeoff between bias and variability.
• MLEs generally have good properties. For many situations, roughly,
  • The MLE is asymptotically unbiased; that is, the bias is small when the sample size is large
  • The variance of an MLE is about as small as it can be for an unbiased estimator
• But there are many reasonable estimation procedures besides MLE (e.g., Bayes estimators).
• Bias is estimation is not necessarily bad. Being willing to accept a little bit of bias can often lead to a beneficial reduction in variability.
  • On the other hand, try to minimize bias in data collection: how the sample is selected, how the variables are measured, etc
  • A lot of the theory assumes we have a perfect random sample, which is never true in practice

24.1 Exercises

Exercise 24.1 Consider estimating the rate parameter \(\lambda\) of an Exponential population based on data \(X_1, \ldots, X_n\) from a random sample of size \(n\). (For example, you want to estimate the rate at which earthquakes occur based on a sample of times between earthquakes.)

We have seen that the MLE of \(\lambda\) is \(1/\bar{X}\).

1. Explain in full detail how you could conduct a simulation to approximate the MSE of \(1/\bar{X}\) when \(\lambda = 2\) for a given \(n\).
   
   
   
   
   
2. Explain in full detail how you could conduct a simulation to approximate the MSE function of \(1/\bar{X}\) for a given \(n\).
   
   
   
   
   

Exercise 24.2 Recall Darth’s estimator of \(\mu\) in the Poisson(\(\mu\)) car dealership problem of Example 21.3

\[ \hat{\mu} = \frac{n}{n+100}\bar{X}+ \frac{100}{n+100}(2.3), \]

1. Compute the bias, variance, and MSE of \(\hat{\mu}\) when \(\mu=2\) for a sample of size \(n=3\).
   
   
   
   
   
2. Compute the bias, variance, and MSE functions of \(\hat{\mu}\) for a sample of size \(n=3\).
   
   
   
   
   

---

1. Recall: By definition \(\textrm{Var}(Y)=\textrm{E}[(Y-\textrm{E}(Y))^2]\) but remember the useful computational formula
   \(\textrm{Var}(Y) = \textrm{E}(Y^2) - (\textrm{E}(Y))^2\). Apply this to \(Y=\hat{\theta}-\theta\), and remember that \(\hat{\theta}\) is random but \(\theta\) is not.