21  Statistical Estimation: From Probability to Statistics

Example 21.1 Consider the following two scenarios. Which is a probability scenario and which is a statistics scenario? Discuss some features of each scenario

• Assume that the number of cars sold daily at a certain used car dealership follows a Poisson distribution with parameter 2.3, independently from day to day. How likely is it for the dealership to sell more than 100 cars in a month?
• The car dealership observes how many cars they sell each day for the next month. Given their data, is it reasonable to assume daily car sales follow a Poisson distribution? If so, how could they use the observed data to estimate the Poisson parameter?
  
  
  
  
  

• Probability: Assume a model for a random (uncertain) process and evaluate probabilities of potential outcomes or events — what might the data look like?

  • In probability, a model is assumed and parameters are assumed to be known, but the data is unknown.

  • In probability, we typically call the model a “distribution” and the data “random variables”.

• Statistical inference: Observe sample data and make conclusions about the process that generated the data

  • In statistics, the data is observed but parameters are unknown.

  • In statistics, we typically call the model the “population” and the data the “sample”.

• Statistical inference involves using sample data to make conclusions about the population.

  • Point estimation: Estimate an unknown population parameter with a single number based on sample data

  • Interval estimation: Estimate an unknown population parameter with a plausible range of values based on sample data

  • Hypothesis testing: Assess the evidence that the sample data provides regarding a particular claim about unknown population parameters.

• The population distribution is the distribution of individual values of the variable of interest. A parameter, generically denoted \(\theta\), is a number that summarizes the population distribution.

• The population distribution, denoted \(p_\theta\) or \(p_\theta(x)\), is a probability model for the distribution of values of a variable¹ \(X\) over all individuals in the population.

  • If the variable \(X\) is discrete, then \(p_\theta\) represents a probability mass function (pmf): \(p_\theta(x) = \textrm{P}(X=x)\)

  • If the variable \(X\) is continuous, then \(p_\theta\) represents a probability density function (pdf): \(p_\theta(x) \approx \frac{1}{\epsilon}\textrm{P}(x - \epsilon/2 < X < x + \epsilon/2)\)

• A parameter, generically denoted \(\theta\), is a characteristic of the population distribution, often computed as an expected value of some function of \(X\).

  • In some cases \(\theta\) represents a vector of multiple parameters.

  • For example, if \(p_\theta\) represents a Normal distribution, then \(\theta\) might represent the pair \(\theta \equiv (\mu, \sigma)\) consisting of the population mean and the population standard deviation.

• A parameter is a fixed number but its value is unknown².

• Many statistical studies involve random sampling from a population.

• A (simple) random sample of size \(n\) is a collection of random variables \(X_1,\ldots,X_n\) that are independent and identically distributed (i.i.d.)

  • Identically distributed — each individual \(X_i\) has the same marginal distribution — the population distribution — since they are sampled from the same population: \[X_i \sim p_\theta \quad \text{for all $i=1, \ldots, n$}\]

  • Independent — the values are sampled independently, so the joint distribution³ is the product of marginal distributions: \[p_\theta(x_1, \ldots, x_n) = p_\theta(x_1)p_\theta(x_2)\cdots p_\theta(x_n)\quad \text{for all $x_1,\ldots, x_n$}\]

• A statistic is a characteristic of the sample which can be computed from the data. More precisely, a statistic is a function of \(X_1,\ldots,X_n\), but not of \(\theta\).

  • That is, a statistic is itself a random variable, and therefore has its own probability distribution, which describes how values of the statistic would vary from sample-to-sample over many (hypothetical) samples.

  • The probability distribution of a statistic is called a sampling distribution.

  • The distribution of a statistic will depend on \(\theta\), but the value of a statistic will not.

• A statistic \(\hat{\theta}\equiv T(X_1,\ldots,X_n)\) used to estimate a parameter \(\theta\) is called a (point) estimator.

  • Naturally, we hope that there is some correspondence between our estimator \(\hat{\theta}\) and the parameter it is estimating \(\theta\) — for example, estimating the population mean with the sample mean — but the definition does not require this.

  • When the numerical value of a statistic is calculated from observed sample data, its value is called a (point) estimate of the parameter.

  • The estimator \(\hat{\theta}\) is a RV whose value changes from sample to sample; an estimate is the observed value computed based on the results of a single sample.

Example 21.2 Jerry has just opened a car dealership in a new location. He assumes that the number of cars he’ll sell in a day has a Poisson(\(\mu\)) distribution. But he doesn’t know the value of \(\mu\), so he plans to use data from his first few days of business to estimate \(\mu\). Let \(X_1, \ldots, X_n\) represent the number of cars sold for a sample of \(n\) days.

1. What plays the role of the population?
   
   
   
   
   
2. What is the variable of interest?
   
   
   
   
   
3. What is the population distribution?
   
   
   
   
   
4. What is the main parameter of interest? How do you interpret it?
   
   
   
   
   
5. Suggest one estimator of \(\mu\).
   
   
   
   
   
6. Suppose Jerry sells 3 cars on the first day, 0 cars on the second day, and 2 on the third day (that is, \(x_1=3, x_2 = 0, x_3 = 2\)). What would your estimate of \(\mu\) be based on the (3, 0, 2) sample?
   
   
   
   
   
7. Suggest another estimator of \(\mu\). What would your estimate of \(\mu\) be based on the (3, 0, 2) sample?
   
   
   
   
   

Example 21.3 Continuing Example 21.2, we’ll consider a few different estimators of \(\mu\), which yield a few different estimates of \(\mu\) based on the (3, 0, 2) sample.

1. Ahsoka suggests that since \(\mu\) is the expected value or population mean of a Poisson distribution, our estimator should be the sample mean \[ \bar{X} = \frac{1}{n}\sum_{i=1}^n X_i \] Compute Ahsoka’s estimate of \(\mu\) based on the (3, 0, 2) sample.
   
   
   
   
   
2. Boba suggests that since \(\mu\) is the population variane of a Poisson distribution, our estimator should be the sample variance \[ \hat{\sigma}^2 = \frac{1}{n}\sum_{i=1}^n (X_i-\bar{X})^2 \] Compute Boba’s estimate of \(\mu\) based on the (3, 0, 2) sample.
   
   
   
   
   
3. Cassian likes Boba’s idea, but he heard that you should divide by \(n-1\) instead of \(n\) when computing the sample variance, so Cassian says our estimator should be this sample variance \[ S^2 = \frac{1}{n-1}\sum_{i=1}^n (X_i-\bar{X})^2 \] Compute Cassian’s estimate of \(\mu\) based on the (3, 0, 2) sample.
   
   
   
   
   
4. Darth has data from 100 days of sales at another car dealership where the sample mean was 2.3, and he thinks that information should be able to help us. He decides to take a weighted average of the sample mean we observe at the new dealership (\(\bar{X}\)) and 2.3, the average from the other dealership. Darth’s estimator is \[ \frac{n}{n+100}\bar{X}+ \frac{100}{n+100}(2.3), \] reasoning that the more data we collect from the new dealership (represented by \(n\)), the less weight the 2.3 from the other dealership should get. Compute Darth’s estimate of \(\mu\) based on the (3, 0, 2) sample.
   
   
   
   
   
5. Ezra remembers the Poisson pattern. In particular, he remembers that the probability of 0 cars sold in a day is \(p_0=e^{-\mu}\), and rearranging shows that \(\mu = -\log p_0\). Ezra suggests that we estimate \(p_0\) with the sample proportion of days with 0 cars sold, \(\hat{p}_0\), so that \[ -\log \hat{p}_0 \] is Ezra’s estimator of \(\mu\). Compute Ezra’s estimate of \(\mu\) based on the (3, 0, 2) sample.
   
   
   
   
   
6. We have seen a few different estimators of \(\mu\), each with a seemingly reasonable rationale behind it. Each estimator produces a different estimate of \(\mu\) based on the same (3, 0, 2) sample data. Which of these 5 numbers is the best estimate of \(\mu\)? That is, which of 5 values is closest to the true value of \(\mu\)?
   
   
   
   
   
7. How might you investigate which of the estimators corresponds to the best estimation procedure of estimaing \(\mu\) for a Poisson distribution?
   
   
   
   
   

• Because the true value of the parameter \(\theta\) is unknown, we never know if the sample data provides a reasonable estimate of \(\theta\)
• Rather, we evaluate the estimation procedure for potential values of \(\theta\): How does the estimation procedure perform over many possible samples given potential values of \(\theta\)?
• Some questions of interest when estimating a parameter \(\theta\)
  • How do we find an estimator of \(\theta\)?
  • What are properties of “good” estimators?
  • How can we tell if one estimator is “better” than another?
  • Can we find a “best” estimator of \(\theta\)? (Short answer: no)
  • How do we determine the margin of error for an estimator?

21.1 Exercises

Exercise 21.1 In each of the following scenarios, one of (i) and (ii) is a probability and one is a statistics question. Classify each question as “probability” or “statistics”. Then discuss some general features of the probability questions, and some general features of the statistics questions

1. U.S. Satisfaction

   • Assume that 30% of Americans are satisfied with the way things are going in the U.S. If a random sample of 1000 Americans is selected, what are the chances that more than 330 Americans in the sample are satisfied with the way things are going in the U.S?

   • In a random sample of 1000 Americans, 330 Americans are satisfied with the way things are going in the U.S. What percent of all Americans are satisfied with the way things are going in the U.S?

2. Great white shark length

   • Assume that lengths of female great white sharks follow a Normal distribution with mean 16 ft and standard deviation 3 ft. If a random sample of 40 female great white sharks is selected, what are the chances that the average length in the sample is between 16.5 and 17.5 ft?

   • In a sample of 40 female great white sharks, the sample mean length is 17.0 ft. What is the average length of female great white sharks?

3. In a clinical trial, 200 subjects are randomly assigned to receive either an experimental vaccine or no treatment (e.g., a placebo), and then each subject is tested to see if they developed immunity.

   • If the vaccine is in reality not effective, what are the chances that the proportion of subjects who develop immunity is twice as large for those who received the vaccine than for those who did not?

   • Among the 100 subjects who received the vaccine, 20 developed immunity; among the 100 subjects who did not receive the vaccine, 10 developed immunity. Is there evidence that the vaccine is effective?

4. Snap streaks

   • Assume that lengths of streaks on Snapchat follow an Exponential distribution, with mean 75 for teen users and mean 40 for adult users. A random sample of 200 Snapchat users is selected, 100 teens and 100 adults. What are the chances that the sample mean streak length for teens is with within 20 of the sample mean streak length for adults?

   • In a random sample of 100 teen Snapchat users, the sample mean streak length is 80. In a random sample of 100 adult Snapchat users, the sample mean streak length is 35. In general, how much longer do streak lengths for teen Snapchat users tend to be than for adult Snapchat users?
     
     
     
     
     
     
     
     
     

Exercise 21.2 Continuing Exercise 21.2. In each scenario, identify

1. What plays the role of the population(s)?
   
   
   
   
   
2. What is the variable(s) of interest?
   
   
   
   
   
3. What is the population distribution(s)?
   
   
   
   
   
4. What is the main parameter(s) of interest?
   
   
   
   
   

---

1. Many problems of interest involve multiple variables (e.g., height, weight, income), in which case the population distribution is a joint distribution. But it is simpler — both conceptually and notationally — to consider just a single variable of interest.

2. There are two main philosophical approaches to statistical inference that differ mainly in whether they emphasize “fixed number” or “unknown”. The frequentist approach treats a parameter as a fixed number. The Bayesian approach treats an unknown parameter— in other words, an uncertain parameter—as a random variable with a probability distribution that describes the degree of uncertainty. Both approaches are valid, each with advantages and disadvantages. The frequentist approach has been historically more prevalent, but the Bayesian approach has gained in popularity in the last 30 or so years.

3. Abuse of notation in the expression: \(p_\theta\) on the left represents the joint distribution of \((X_1, \ldots, X_n)\), a function with \(n\) inputs; \(p_\theta\) on the right represents that marginal distribution of each \(X_i\) (since i.d.), a function with a single input.