22  Maximum Likelihood Estimation

Example 22.1 Suppose we are willing to assume a Poisson(\(\mu\)) model for some variable \(X\). But we don’t know the value of \(\mu\), so we’ll collect data and use it to estimate \(\mu\). Suppose first that we have just a single observation of \(X\). (We’ll look at a sample of size \(n\) soon.)

For context, think of \(\mu\) as the long run average number of cars sold per day over many days at a new car dealership, and we’re trying to estimate \(\mu\) based on just the number of cars sold in a single day. But what we really want is an estimation procedure that we can use any time we have data from a Poisson distribution, regardless of the context.

1. Does \(X\) take values on a discrete or continuous scale? What about \(\mu\)?
   
   
   
   
   
2. Write the formula for the Poisson(\(\mu\)) pmf.
   
   
   
   
   
3. Now remember that \(\mu\) is unknown. Suppose that a single observation yields \(x=3\) (e.g. 3 cars are sold in a single day.) Now write the pmf plugging in everything we know; what is this a function of?
   
   
   
   
   
4. Compute and intepret \(\text{P}(X = 3)\) if \(\mu = 2.3\).
   
   
   
   
   
5. Compute and intepret \(\text{P}(X = 3)\) if \(\mu = 1\).
   
   
   
   
   
6. Compute and intepret \(\text{P}(X = 3)\) if \(\mu = 4.8\).
   
   
   
   
   
7. Suppose that \(x=3\). Based just on this observation, which value — 1, 2.3, or 4.8 — would you choose as your estimate for \(\mu\)? Why?
   
   
   
   
   
8. We obviously have more choices for our estimate of \(\mu\) than just 1, 2.3, or 4.8. Describe in principle the process you would follow to find the estimate of \(\mu\) based on a single observation \(x=3\).
   
   
   
   
   

• Consider a single observation \(X\) from a population \(p_\theta\) with parameter \(\theta\): Let \(X\sim p_\theta\). The likelihood function given \(X=x\) is defined as \[L(\theta) \equiv L_x(\theta) = p_\theta(x)\]
• That is, the likelihood function is the probability (or density for continuous data \(X\)) of observing the given sample data \(x\) viewed as a function of the unknown parameter(s) \(\theta\). The observed value of \(x\) (data) is treated as a fixed constant.
• The maximum likelihood estimate of \(\theta\) is the value of \(\theta\) for which the observed data has the highest likelihood of occurring.
• The maximum likelihood estimator (MLE) of \(\theta\) is the RV \(\hat{\theta}\) which maximizes the likelihood, i.e. the value \(\hat{\theta}\) which satisfies \[L(\hat{\theta}) \ge L(\theta) \text{ for all } \theta.\]
• Remember that an estimator \(\hat{\theta}\) is a function of \(X\), the observed data.
• It is often convenient to consider the log-likelihood¹ function of \(\theta\) \[\ell(\theta) \equiv \ell_x(\theta) = \log L_x(\theta)\\ \]
• Because \(\log\) is an increasing function it follows that \(\hat{\theta}\) is the MLE of \(\theta\) if and only if \[\ell(\hat{\theta}) \ge \ell(\theta) \text{ for all } \theta.\]

Example 22.2 Suppose we are willing to assume a Poisson(\(\mu\)) model for some variable \(X\). But we don’t know the value of \(\mu\), so we’ll collect data and use it to estimate \(\mu\). Let \(X_1, \ldots, X_n\) be the data in a random sample of size \(n\). For example, suppose \(n=4\) and we observe \(x_1 = 3\), \(x_2 = 0\), \(x_3=2\), \(x_4=0\).

For context, think of \(\mu\) as the long run average number of cars sold per day over many days at a new car dealership, and we’re trying to estimate \(\mu\) based on just the number of cars sold in \(n\) days (e.g., 3 cars sold on the first day, 0 on the second, 2 on the third, and 0 on the fourth). But what we really want is an estimation procedure that we can use any time we have data from a Poisson distribution, regardless of the context.

1. Write an expression for the likelihood function. What is this a function of?
   
   
   
   
   
2. Compute and intepret the likelihood if \(\mu = 2.3\).
   
   
   
   
   
3. Compute and intepret the likelihood if \(\mu = 1\).
   
   
   
   
   
4. Compute and intepret the likelihood if \(\mu = 4.8\).
   
   
   
   
   
5. Suppose we observe the sample (3, 0, 2, 0). Based just on this sample, which value — 1, 2.3, or 4.8 — would you choose as your estimate for \(\mu\)? Why?
   
   
   
   
   
6. We obviously have more choices for our estimate of \(\mu\) than just 1, 2.3, or 4.8. Describe in principle the process you would follow to find the estimate of \(\mu\) based on the (3, 0, 2, 0) sample.
   
   
   
   
   

• Consider a random sample of \(n\) values from a population \(p_\theta\) with parameter \(\theta\): Let \(X_1,\ldots,X_n\) be i.i.d. \(\sim p_\theta\).
• The likelihood function of \(\theta\), given \(X_1=x_1, \ldots, X_n=x_n\), is \[L(\theta)\equiv L_{x_1, \ldots, x_n}(\theta) = \prod_{i=1}^n p_\theta(x_i) = p_\theta(x_1)\times\cdots\times p_\theta(x_n)\]
• That is, the likelihood function is the joint density² of the sample \(X_1,\ldots,X_n\), evaluated at the observed data values \(x_1,\ldots,x_n\), viewed as a function of the (unknown) parameter(s) \(\theta\).
• Note that \(x_1, \ldots, x_n\) represent distinct values (like \(x_1=3, x_2=0, x_3=2\) in the example).
• The MLE of \(\theta\) is the value of \(\theta\) for which the observed sample data has the highest likelihood of occurring.
• The maximum likelihood estimator (MLE) of \(\theta\) is the RV \(\hat{\theta}\) which maximizes the likelihood, i.e. the value \(\hat{\theta}\) which satisfies \[L(\hat{\theta}) \ge L(\theta) \text{ for all } \theta.\]
• Remember that an estimator \(\hat{\theta}\) is a function of \(X_1, \ldots, X_n\).
• The log-likelihood function of \(\theta\) is \[\begin{aligned} \ell(\theta) \equiv \ell_{x_1, \ldots, x_n}(\theta)& = \log L(\theta)\\ & = \sum_{i=1}^n\log p_\theta(x_i) = \log p_\theta(x_1) + \cdots + \log p_\theta(x_n) \end{aligned}\]
• Because \(\log\) is an increasing function it follows that \(\hat{\theta}\) is the MLE of \(\theta\) if and only if \[\ell(\hat{\theta}) \ge \ell(\theta) \text{ for all } \theta.\]

Example 22.3 We have seen that \(\bar{X}\) is the MLE of \(\mu\) in a Poisson(\(\mu\)) model. (Example 22.4 proves this fact using calculus.) Now we’ll consider estimating some other parameters in the Poisson situation. Again, our goal is to determine procedures for Poisson models regardless of the context, but you can think in terms of the car dealership context.

1. Consider \(\sqrt{\mu}\). Is \(\sqrt{\mu}\) a parameter or a statistic? How would you interpret \(\sqrt{\mu}\) in this context?
   
   
   
   
   
2. What do you think the MLE of \(\sqrt{\mu}\) is, in general and for the observed (3, 0, 2, 0) sample?
   
   
   
   
   
3. Consider \(e^{-\mu}\). Is \(e^{-\mu}\) a parameter or a statistic? How would you interpret \(e^{-\mu}\) in this context?
   
   
   
   
   
4. What do you think the MLE of \(e^{-\mu}\) is, in general and for the observed (3, 0, 2, 0) sample
   
   
   
   
   
5. Consider \(\mu e^{-\mu}\). Is \(\mu e^{-\mu}\) a parameter or a statistic? How would you interpret \(\mu e^{-\mu}\) in this context?
   
   
   
   
   
6. What do you think the MLE of \(\mu e^{-\mu}\) is, in general and for the observed (3, 0, 2, 0) sample?
   
   
   
   
   
7. We started by assuming a Poisson model, but how do we know that is a reasonable assumption? Suppose we observe data for a random sample of size \(n\). Suggest you might use the sample data to investigate whether a Poisson model is appropriate.
   
   
   
   

• A parameter is any characteristic of a population distribution
• For many populations, there are a few “main” parameters (like \(\mu\) for Poisson(\(\mu\)), or \((\mu\), \(\sigma\)) for Normal(\(\mu\), \(\sigma\)), but any characteristic of a population is parameter and can be estimated
• Invariance property of MLEs. If \(\hat{\theta}\) is the MLE of \(\theta\), then \(g(\hat{\theta})\) is the MLE of \(g(\theta)\).
• That is, the “MLE of a function is the function of the MLE”
• The invariance principle says that if you find the MLE of the “main parameters” then you automatically get the MLE of all the parameters that are based on it

Example 22.4 We wish to estimate the parameter \(\mu\) for a Poisson(\(\mu\)) distribution based on a random sample of \(n\) values \(X_1\ldots, X_n\).

1. Suppose that \(n=4\) and the sample is (3, 0, 2, 0). Carefully write the likelihood function.
   
   
   
   
   
2. Suppose that \(n=4\) and the sample is (3, 0, 2, 0). Carefully write the log-likelihood function.
   
   
   
   
   
3. Use calculus to find the MLE of \(\mu\) given \(n=4\) and the sample (3, 0, 2, 0). Compare to what we found previously using graphical methods.
   
   
   
   
   
4. Now consider a general sample of size \(n\). Carefully write the likelihood function.
   
   
   
   
   
5. Now consider a general sample of size \(n\). Carefully write the log-likelihood function.
   
   
   
   
   
6. Use calculus to find the MLE of \(\mu\) given a sample of size \(n\). Compare to what we found previously using graphical methods.
   
   
   
   
   

• In many situations, the full sample data is not necessary to find the MLE; rather, a few descriptive statistics are often sufficient to evaluate the likelihood and determine the MLE

• Poisson(\(\mu\)): the MLE of \(\mu\) is the sample mean \(\bar{X}\)

• Binomial(\(n\), \(p\)), with \(n\) known: the MLE of \(p\) is the sample proportion \(\hat{p} = X/n\), where \(X\) is the number of “successes” in the sample

• Exponential(\(\lambda\)): the MLE of \(\lambda\) is the sample rate \(1/\bar{X}\); the usual situation is where the \(X\)’s measure times between “events” and so \(\bar{X}\) is the sample mean time between “events”.

• Normal(\(\mu\), \(\sigma\)), with both \(\mu\) and \(\sigma\) unknown: the MLEs of \(\mu\), \(\sigma^2\), and \(\sigma\), are, respectively \[\begin{align*} \hat{\mu} & = \bar{X} = \frac{1}{n} \sum_{i=1}^n X_i\\ \hat{\sigma}^2 & = \frac{1}{n} \sum_{i=1}^n \left(X_i - \bar{X}\right)^2\\ \hat{\sigma} & = \sqrt{\frac{1}{n} \sum_{i=1}^n \left(X_i - \bar{X}\right)^2} \end{align*}\]

• Maximum likelihood estimation procedures are widely applicable and maximum likelihood estimators have many nice theoretical properties.

• However, there are some drawbacks:

  • May require numerical maximization, which can be computationally intensive (especially if many parameters)
  • May not be reliable estimators in small samples.

22.1 Exercises

Exercise 22.1 Suppose that times (minutes) between earthquakes (of any magnitude in a certain region) follow an Exponential distribution with rate parameter \(\lambda\). Let \(X_1, \ldots, X_n\) be a random sample of times (minutes) between earthquakes for \(n\) earthquakes. Suppose the data for a sample of size \(n=5\) is: 20, 37, 13, 10, 25 minutes.

1. Carefully write the likelihood function for the observed sample. (Hint: For a random sample from a continuous distribution, to find the likelihood function you plug each observed value into the pdf and then form the product.)
   
   
   
   
   
2. Use any software you want to carefully plot the likelihood function from the previous part. Then use your plot—you can just zoom in as needed—to find the MLE of \(\lambda\) based on this sample. (Hint: you should find that the MLE is 0.0476, which is equal to \(1/\bar{x} = 5/105 = 0.0476\).)
   
   
   
   
   
3. Write a clearly worded sentence reporting in context your estimate of \(\lambda\) from the previous part.
   
   
   
   
   

Exercise 22.2 Suppose that within a certain population, credit scores follow a Normal(\(\mu\), \(\sigma\)) distribution. In a random sample of 5 individuals from this population, the credit scores are 600, 630, 680, 700, 770.

1. Compute the MLE of \(\mu\).
   
   
   
   
   
2. Compute the MLE of \(\sigma\).
   
   
   
   
   
3. Explain in words what it means for these parameters to be the MLE.
   
   
   
   
   
4. Compute the MLE of the 10th percentile of credit scores for this population
   
   
   
   
   
5. Compute the MLE of the population proportion of individuals with a credit score above 750.
   
   
   
   
   

---

1. “log” always means natural log.

2. We usually have i.i.d. observations and so the joint density is just the product of the marginal densities as we have defined it. However, for more general situations the likelihood can be defined directly as the joint density.