2 Statistical estimation

When we are dealing with large populations (the production of items such as LEDs, light bulbs, piston rings etc.) it is extremely unlikely that we will be able to calculate population parameters such as the mean and variance directly from the full population.

We have to use processes which enable us to estimate these quantities. There are two basic methods used called point estimation and interval estimation. The essential difference is that point estimation gives single numbers which, in the sense defined below, are Ôbest estimatesÕ of population parameters, while interval estimates give a range of values together with a figure called the confidence that the true value of a parameter lies within the calculated range. Such ranges are usually called confidence intervals .

Statistically, the word ‘estimate’ implies a defined procedure for finding population parameters. In statistics, the word ‘estimate’ does not mean a guess, something which is rough-and-ready. What the word does mean is that an agreed precise process has been (or will be) used to find required values and that these values are ‘best values’ in some sense. Often this means that the procedure used, which is called the ‘estimator’, is:

1. consistent in the sense that the difference between the true value and the estimate approaches zero as the sample size used to do the calculation increases;
2. unbiased in the sense that the expected value of the estimator is equal to the true value;
3. efficient in the sense that the variance of the estimator is small.

Expectation is covered in Workbooks 37 and 38. You should note that it is not always possible to find a ‘best’ estimator. You might have to decide (for example) between one which is

consistent, biased and efficient

and one which is

consistent, unbiased and inefficient

when what you really want is one which is

consistent, unbiased and efficient.

2.1 Point estimation

We will look at the point estimation of the mean and variance of a population and use the following notation.

Notation

Estimating the mean

This is straightforward.

$\stackrel{̂}{\mu }=\bar{x}$

is a sensible estimate since the difference between the population mean and the sample mean disappears with increasing sample size. We can show that this estimator is unbiased. Symbolically we have:

$\stackrel{̂}{\mu }=\frac{x_1+x_2+\⋯x_n}{n}$

so that

Note that the expected value of $x_1$ is $\text{E}\left(X\right)$ , i.e. $\text{E}\left(x_1\right)=\text{E}\left(X\right)$ . Similarly for $x_1,x_2,\⋯,x_n$ .

Estimating the variance

This is a little more difficult. The true variance of the population is ${\sigma }^2=\frac{\sum {\left(x-\mu \right)}^2}{N}$ which suggests the estimator, calculated from a sample, should be ${\stackrel{̂}{\sigma }}^2=\frac{\sum {\left(x-\mu \right)}^2}{n}$ .

However, we do not know the true value of $\mu$ , but we do have the estimator $\stackrel{̂}{\mu }=\bar{x}$ .

Replacing $\mu$ by the estimator $\stackrel{̂}{\mu }=\bar{x}$ gives

${\stackrel{̂}{\sigma }}^2=\frac{\sum {\left(x-\bar{x}\right)}^2}{n}$

This can be written in the form

${\stackrel{̂}{\sigma }}^2=\frac{\sum {\left(x-\bar{x}\right)}^2}{n}=\frac{\sum x^2}{n}-{\left(\bar{x}\right)}^2$

Hence

$\text{E}\left({\stackrel{̂}{\sigma }}^2\right)=\frac{\text{E}\left(\sum x^2\right)}{n}-\text{E}\left\{{\left(\bar{X}\right)}^2\right\}=\text{E}\left(X^2\right)-\text{E}\left\{{\left(\bar{X}\right)}^2\right\}$

We already have the important result

$\text{E}\left(x\right)=\text{E}\left(\bar{x}\right)$   and   $\text{V}\left(\bar{x}\right)=\frac{\text{V}\left(x\right)}{n}$

Using the result $\text{E}\left(x\right)=\text{E}\left(\bar{x}\right)$ gives us

This result is biased , for an unbiased estimator the result should be ${\sigma }^2$ not $\frac{n-1}{n}{\sigma }^2$ .

Fortunately, the remedy is simple, we just multiply by the so-called Bessel’s correction, namely $\frac{n}{n-1}$ and obtain the result

${\stackrel{̂}{\sigma }}^2=\frac{n}{n-1}\frac{\sum {\left(x-\bar{x}\right)}^2}{n}=\frac{\sum {\left(x-\bar{x}\right)}^2}{n-1}$

There are two points to note here. Firstly (and rather obviously) you should not take samples of size 1 since the variance cannot be estimated from such samples. Secondly, you should check the operation of any hand calculators (and spreadsheets!) that you use to find out exactly what you are calculating when you press the button for standard deviation. You might find that you are calculating either

${\sigma }^2=\frac{\sum {\left(x-\mu \right)}^2}{N}$ or ${\stackrel{̂}{\sigma }}^2=\frac{\sum {\left(x-\bar{x}\right)}^2}{n-1}$

It is just as well to know which, as the first formula assumes that you are calculating the variance of a population while the second assumes that you are estimating the variance of a population from a random sample of size $n$ taken from that population.

From now on we will assume that we divide by $n-1$ in the sample variance and we will simply write $s^2$ for $s_{n-1}^2.$

2.2 Interval estimation

We will look at the process of finding an interval estimation of the mean and variance of a population and use the notation used above.

Interval estimation for the mean

This interval is commonly called the Confidence Interval for the Mean.

Firstly, we know that while the sample mean $\bar{x}=\frac{x_1+x_2+\⋯+x_n}{n}$ is a good estimator of the population mean $\mu$ . We also know that the calculated mean $\bar{x}$ of a sample of size $n$ is unlikely to be exactly equal to $\mu$ . We will now construct an interval around $\bar{x}$ in such a way that we can quantify the confidence that the interval actually contains the population mean $\mu$ .

Secondly, we know that for sufficiently large samples taken from a large population, $\bar{x}$ follows a normal distribution with mean $\mu$ and standard deviation $\frac{\sigma }{\sqrt{n}}$ . Thirdly, looking at the following extract from the normal probability tables,

we can see that $2\times 47.5\%=95\%$ of the values in the standard normal distribution lie between $\pm 1.96$ standard deviation either side of the mean.

So before we see the data we may say that

$\text{P}\left(\mu -1.96\frac{\sigma }{\sqrt{n}}\le \bar{x}\le \mu +1.96\frac{\sigma }{\sqrt{n}}\right)=0.95$

After we see the data we say with $95\%$ confidence that

$\mu -1.96\frac{\sigma }{\sqrt{n}}\le \bar{x}\le \mu +1.96\frac{\sigma }{\sqrt{n}}$

which leads to

$\bar{x}-1.96\frac{\sigma }{\sqrt{n}}\le \mu \le \bar{x}+1.96\frac{\sigma }{\sqrt{n}}$

This interval is called a 95% confidence interval for the mean $\mu$ .

Note that while the $95\%$ level is very commonly used, there is nothing sacrosanct about this level. If we go through the same argument but demand that we need to be $99\%$ certain that $\mu$ lies within the confidence interval developed, we obtain the interval

$\bar{x}-2.58\frac{\sigma }{\sqrt{n}}\le \mu \le \bar{x}+2.58\frac{\sigma }{\sqrt{n}}$

since an inspection of the standard normal tables reveals that $99\%$ of the values in a standard normal distribution lie within 2.58 standard deviations of the mean.

The above argument assumes that we know the population variance. In practice this is often not the case and we have to estimate the population variance from a sample. From the work we have seen above, we know that the best estimate of the population variance from a sample of size n is given by the formula

${\stackrel{̂}{\sigma }}^2=\frac{\sum {\left(x-\bar{x}\right)}^2}{n-1}$

It follows that if we do not know the population variance, we must use the estimate $\stackrel{̂}{\sigma }$ in place of $\sigma$ . Our 95% and 99% confidence intervals (for large samples) become

$\bar{x}-1.96\frac{\stackrel{̂}{\sigma }}{\sqrt{n}}\le \mu \le \bar{x}+1.96\frac{\stackrel{̂}{\sigma }}{\sqrt{n}}$ and $\bar{x}-2.58\frac{\stackrel{̂}{\sigma }}{\sqrt{n}}\le \mu \le \bar{x}+2.58\frac{\stackrel{̂}{\sigma }}{\sqrt{n}}$

where

${\stackrel{̂}{\sigma }}^2=\frac{\sum {\left(x-\bar{x}\right)}^2}{n-1}$

When we do not know the population variance, we need to estimate it. Hence we need to gauge the confidence we can have in the estimate.

In small samples, when we need to estimate the variance, the values 1.96 and 2.58 need to be replaced by values from the Student’s $t$ -distribution. See HELM booklet  41.

Example 1

After 1000 hours of use the weight loss, in gm, due to wear in certain rollers in machines, is normally distributed with mean $\mu$ and variance ${\sigma }^2.$ Fifty independent observations are taken. (This may be regarded as a “large” sample.) If observation $i$ is $y_i,$ then ${\sum }_{i=1}^{50}{y}_i=497.2$ and ${\sum }_{i=1}^{50}{y}_i^2=5473.58.$

Estimate $\mu$ and ${\sigma }^2$ and give a 95% confidence interval for $\mu .$

Solution

We estimate $\mu$ using the sample mean: $ȳ=\frac{\sum y_i}{n}=\frac{497.2}{50}=9.944\text{gm}$

We estimate ${\sigma }^2$ using the sample variance:

The estimated standard error of the mean is $\sqrt{\frac{s^2}{n}}=\sqrt{\frac{10.8046}{50}}=0.4649\text{gm}$

The 95% confidence interval for $\mu$ is $ȳ\pm 1.96\sqrt{\frac{s^2}{n}}.$ That is $9.479<\mu <10.409$

Exercises

1. The voltages of sixty nominally 10 volt cells are measured. Assuming these to be independent observations from a normal distribution with mean $\mu$ and variance ${\sigma }^2,$ estimate $\mu$ and ${\sigma }^2.$ Regarding this as a “large”sample, find a 99% confidence interval for $\mu .$ The data are:

   10.3	10.5	9.6	9.7	10.6	9.9	10.1	10.1	9.9	10.5
   10.1	10.1	9.9	9.8	10.6	10.0	9.9	10.0	10.3	10.1
   10.1	10.3	10.5	9.7	10.1	9.7	9.8	10.3	10.2	10.2
   10.1	10.5	10.0	10.0	10.6	10.9	10.1	10.1	9.8	10.7
   10.3	10.4	10.4	10.3	10.4	9.9	9.9	10.5	10.0	10.7
   10.1	10.6	10.0	10.7	9.8	10.4	10.3	10.0	10.5	10.1

2. The natural logarithms of the times in minutes taken to complete a certain task are normally distributed with mean $\mu$ and variance ${\sigma }^2.$ Seventy-five independent observations are taken. (This may be regarded as a “large” sample.) If the natural logarithm of the time for observation $i$ is $y_i,$ then $\sum y_i=147.75$ and $\sum y_i^2=292.8175.$

   Estimate $\mu$ and ${\sigma }^2$ and give a 95% confidence interval for $\mu .$

   Use your confidence interval to find a 95% confidence interval for the median time to complete the task.

Answer