1 Regression

As we have already noted, relationship(s) between variables are of interest to engineers who may wish to determine the degree of association existing between independent and dependent variables. Knowing this often helps engineers to make predictions and, on this basis, to forecast and plan. Essentially, regression analysis provides a sound knowledge base from which accurate estimates of the values of a dependent variable may be made once the values of related independent variables are known.

It is worth noting that in practice the choice of independent variable(s) may be made by the engineer on the basis of experience and/or prior knowledge since this may indicate to the engineer which independent variables are likely to have a substantial influence on the dependent variable. In summary, we may state that the principle objectives of regression analysis are:

1. to enable accurate estimates of the values of a dependent variable to be made from known values of a set of independent variables;
2. to enable estimates of errors resulting from the use of a regression line as a basis of prediction.

Note that if a regression line is represented as $y=f\left(x\right)$ where $x$ is the independent variable, then the actual function used (linear, quadratic, higher degree polynomial etc.) may be obtained via the use of a theoretical analysis or perhaps a scatter diagram (see below) of some real data. Note that a regression line represented as $y=f\left(x\right)$ is called a regression line of $y$ on $x$ .

1.1 Scatter diagrams

A useful first step in establishing the degree of association between two variables is the plotting of a scatter diagram . Examples of pairs of measurements which an engineer might plot are:

1. volume and pressure;
2. acceleration and tyre wear;
3. current and magnetic field;
4. torsion strength of an alloy and purity.

If there exists a relationship between measured variables, it can take many forms. Even though an outline introduction to non-linear regression is given at the end of this Workbook, we shall focus on the linear relationship only.

In order to produce a good scatter diagram you should follow the steps given below:

1. Give the diagram a clear title and indicate exactly what information is being displayed;
2. Choose and clearly mark the axes;
3. Choose carefully and clearly mark the scales on the axes;
4. Indicate the source of the data.

Examples of scatter diagrams are shown below.

• Figure 1 shows an association which follows a curve, possibly exponential, quadratic or cubic;
• Figure 2 shows a reasonable degree of linear association where the points of the scatter diagram lie in an area surrounding a straight line;
• Figure 3 represents a randomly placed set of points and no linear association is present between the variables.

Note that in Figure 2, the word ‘reasonable’ is not defined and that while points ‘close’ to the indicated straight line may be explained by random variation, those ‘far away’ may be due to assignable variation.

The rest of this Section will deal with linear association only although it is worth noting that techniques do exist for transforming many non-linear relationships into linear ones. We shall investigate linear association in two ways, firstly by using educated guess work to obtain a regression line ‘by eye’ and secondly by using the well-known technique called the method of least squares.

1.2 Regression lines by eye

Note that at a very simple level, we may look at the data and, using an ‘educated guess’, draw a line of regression ‘by eye’ through a set of points. However, finding a regression line by eye is unsatisfactory as a general statistical method since it involves guess-work in drawing the line with the associated errors in any results obtained. The guess-work can be removed by the method of least squares in which the equation of a regression line is calculated using data. Essentially, we calculate the equation of the regression line by minimising the sum of the squared vertical distances between the data points and the line.

1.3 The method of least squares - an elementary view

We assume that an experiment has been performed which has resulted in $n$ pairs of values, say $\left(x_1,y_1\right),\left(x_2,y_2\right),\cdots ,\left(x_n,y_n\right)$ and that these results have been checked for approximate linearity on the scatter diagram given below.

Figure 4

The vertical distances of each point from the line $y=a+bx$ are easily calculated as

$y_1-a-b{x}_1,y_2-a-b{x}_2,y_3-a-b{x}_3\cdots y_n-a-b{x}_4$

These distances are squared to guarantee that they are positive and calculus is used to minimise the sum of the squared distances. Effectively we are minimizing the sum of a two-variable expression and need to use partial differentiation. If you wish to follow this up and look in more detail at the technique, any good book (engineering or mathematics) containing sections on multi-variable calculus should suffice. We will not look at the details of the calculations here but simply note that the process results in two equations in the two unknowns $m$ and $c$ being formed. These equations are:

$\sum xy-a\sum x-b\sum x^2=0$ (i)

and

$\sum y-na-b\sum x=0$ (ii)

The second of these equations (ii) immediately gives a useful result. Rearranging the equation we get

$\frac{\sum y}{n}-a-b\frac{\sum x}{n}=0$ or, put more simply $ȳ=a+b\bar{x}$

where $\left(\bar{x},ȳ\right)$ is the mean of the array of data points $\left(x_1,y_1\right),\left(x_2,y_2\right),\cdots ,\left(x_n,y_n\right)$ .

This shows that the mean of the array always lies on the regression line. Since the mean is easily calculated, the result forms a useful check for a plotted regression line. Ensure that any regression line you draw passes through the mean of the array of data points.

Eliminating $a$ from the equations gives a formula for the gradient $b$ of the regression line, this is:

$b=\frac{\frac{\sum xy}{n}-\frac{\sum x}{n}\frac{\sum y}{n}}{\frac{\sum x^2}{n}-{\left(\frac{\sum x}{n}\right)}^2}$ often written as $b=\frac{S_{xy}}{S_x^2}$

The quantity $S_x^2$ is, of course, the variance of the $x$ -values. The quantity $S_{xy}$ is known as the covariance (of $x$ and $y$ ) and will appear again later in this Workbook when we measure the degree of linear association between two variables.

Knowing the value of $b$ enables us to obtain the value of $a$ from the equation $ȳ=a+b\bar{x}$

It should be noted that the coefficients $b$ and $a$ obtained here will give us the regression line of $y$ on $x$ . This line is used to predict $y$ values given $x$ values. If we need to predict the values of $x$ from given values of $y$ we need the regression line of $x$ on $y$ . The two lines are not the same except in the (very) special case where all of the points lie exactly on a straight line. It is worth noting however, that the two lines cross at the point $\left(\bar{x},ȳ\right)$ . It can be shown that the regression line of $x$ on $y$ is given by Key Point 2:

Example 1

A warehouse manager of a company dealing in large quantities of steel cable needs to be able to estimate how much cable is left on his partially used drums. A random sample of twelve partially used drums is taken and each drum is weighed and the corresponding length of cable measured. The results are given in the table below:

Find the least squares regression line in the form $y=mx+c$ and use it to predict the lengths of cable left on drums whose weights are:

(i) 35 kg (ii) 85 kg (iii) 100 kg

In the latter case state any assumptions which you make in order to find the length of cable left on the drum.

Solution

Excel calculations give   $\sum x=700,\sum x^2=44200,\sum y=1860\sum xy=118600$   so that the formulae

$b=\frac{\frac{\sum xy}{n}-\frac{\sum x}{n}\frac{\sum y}{n}}{\frac{\sum x^2}{n}-{\left(\frac{\sum x}{n}\right)}^2}$ and $a=ȳ-b\bar{x}$

give $a=-20$ and $b=3$ . Our regression line is $y=-20+3x,$ so $y=3x-20$ .

Hence, the required predicted values are:

$y_{35}=3\times 35-20=85y_{85}=3\times 85-20=235y_{100}=3\times 100-20=280$

all results being in metres.

To obtain the last result we have assumed that the linearity of the relationship continues beyond the range of values actually taken.

Task!

An article in the Journal of Sound and Vibration 1991 (151) explored a possible relationship between hypertension (defined as blood pressure rise in mm of mercury) and exposure to noise levels (measured in decibels). Some data given is as follows:

1. Draw a scatter diagram of the data.
2. Comment on whether a linear model is appropriate for the data.
3. Calculate a line of best fit of $y$ on $x$ for the data given.
4. Use your regression line predict the expected rise in blood pressure for a exposure to a noise level of 97 decibels.

Answer

1.4 The method of least squares - a modelling view

We take the dependent variable $Y$ to be a random variable whose value, for a fixed value of $x$ depends on the value of $x$ and a random error component say $e$ and we write

$Y=\alpha +\beta x+e$

Adopting the notation of conditional probability, we are looking for the expected value of $Y$ for a given value of $x$ . The expected value of $Y$ for a given value of $x$ is denoted by

$\text{E}\left(Y|x\right)=\text{E}\left(\alpha +\beta x+e\right)=\text{E}\left(\alpha +\beta x\right)+\text{E}\left(e\right)$

The variance of $Y$ for a given value of $x$ is given by the relationship

$\text{V}\left(Y|x\right)=\text{V}\left(\alpha +\beta x+e\right)=\text{V}\left(\alpha +\beta x\right)+\text{V}\left(e\right)$ , assuming independence.

If ${\mu }_{Y|x}$ represents the true mean value of $Y$ for a given value of $x$ then

${\mu }_{Y|x}=\alpha +\beta x$ , assuming a linear relationship holds,

is a straight line of mean values. If we now assume that the errors $e$ are distributed with mean 0 and variance ${\sigma }^2$ we may write

$\text{E}\left(Y|x\right)=\text{E}\left(\alpha +\beta x\right)+\text{E}\left(e\right)=\alpha +\beta x$ since $\text{E}\left(e\right)=0$ .

and

$\text{V}\left(Y|x\right)=\text{V}\left(\alpha +\beta x\right)+\text{V}\left(e\right)={\sigma }^2$ since $\text{V}\left(\alpha +\beta x\right)=0$ .

This implies that for each value of $x$ , $Y$ is distributed with mean $\alpha +\beta x$ and variance ${\sigma }^2$ . Hence when the variance is small the observed values of $Y$ will be close to the regression line and when the variance is large, at least some of the observed values of $Y$ may not be close to the line. Note that the assumption that the errors $e$ are distributed with mean 0 may be made without loss of generality. If the errors had any other mean, we could subtract it and then add the mean to the value of $c$ . The ideas are illustrated in the following diagram.

Figure 5

The regression line is shown passing through the means of the distributions for the individual values of $x$ . The value of $y$ corresponding to the $x$ -value $x_i$ can be represented by the equation

$y_i=\alpha +\beta x_i+e_i$

where $e_i$ is the error of the observed value of $y,$ that is the difference from its expected value, namely

$\text{E}\left(Y|x_i\right)={\mu }_{y|x_i}=\alpha +\beta x_i$

Now, if we estimate $\alpha$ and $\beta$ with $a$ and $b$ , the residual , or estimated error, becomes

${ê}_i=y_i-a-b{x}_i$

so that the sum of the squares of the residuals is given by

$S=\sum {ê}_i^2=\sum {\left(y_i-a-b{x}_i\right)}^2$

and we may minimize the quantity $S$ by using the method of least squares as before. The mathematical details are omitted as before and the equations obtained for $b$ and $a$ are as before, namely

$b=\frac{\frac{\sum xy}{n}-\frac{\sum x}{n}\frac{\sum y}{n}}{\frac{\sum x^2}{n}-{\left(\frac{\sum x}{n}\right)}^2}$   and   $a=ȳ-b\bar{x}$ .

Note that since the error $e_i$ in the $i$ th observation essentially describes the error in the fit of the model to the $i$ th observation, the sum of the squares of the errors $\sum e_i^2$ will now be used to allow us to comment on the adequacy of fit of a linear model to a given data set.

1.5 Adequacy of fit

We now know that the variance $\text{V}\left(Y|x\right)={\sigma }^2$ is the key to describing the adequacy of fit of our simple linear model. In general, the smaller the variance, the better the fit although you should note that it is wise to distinguish between ‘poor fit’ and a large error variance. Poor fit may suggest, for example, that the relationship is not in fact linear and that a fundamental assumption made has been violated. A large value of ${\sigma }^2$ does not necessarily mean that a linear model is a poor fit.

It can be shown that the sum of the squares of the errors say $S{S}_E$ can be used to give an unbiased estimator ${\stackrel{̂}{\sigma }}^2$ of ${\sigma }^2$ via the formula

${\stackrel{̂}{\sigma }}^2=\frac{S{S}_E}{n-p}$

where $p$ is the number of independent variables used in the regression equation. In the case of simple linear regression $p=2$ since we are using just $x$ and $c$ and the estimator becomes:

${\stackrel{̂}{\sigma }}^2=\frac{S{S}_E}{n-2}$

The quantity $S{S}_E$ is usually used explicitly in formulae whose purpose is to determine the adequacy of a linear model to explain the variability found in data. Two ways in which the adequacy of a regression model may be judged are given by the so-called Coefficient of Determination and the Adjusted Coefficient of Determination .

1.6 The coefficient of determination

Denoted by $R^2$ , the Coefficient of Determination is defined by the formula

$R^2=1-\frac{S{S}_E}{S{S}_T}$

where $S{S}_E$ is the sum of the squares of the errors and $S{S}_T$ is the sum of the squares of the totals given by $\sum {\left(y_i-{ŷ}_i\right)}^2=\sum y_i^2-n{ȳ}^2$ . The value of $R^2$ is sometimes described as representing the amount of variability explained or accounted for by a regression model. For example, if after a particular calculation it was found that $R^2=0.884$ , we could say that the model accounts for about 88% of the variability found in the data. However, deductions made on the basis of the value of $R^2$ should be treated cautiously, the reasons for this are embedded in the following properties of the statistic. It can be shown that:

1. $0\le R^2\le 1$
2. a large value of $R^2$ does not necessarily imply that a model is a good fit;
3. adding a regressor variable (simple regression becomes multiple regression) always increases the value of $R^2$ . This is one reason why a large value of $R^2$ does not necessarily imply a good model;
4. models giving large values of $R^2$ can be poor predictors of new values if the fitted model does not apply at the appropriate $x$ -value.

Finally, it is worth noting that to check the fit of a linear model properly, one should look at plots of residual values. In some cases, tests of goodness-of-fit are available although this topic is not covered in this Workbook.

1.7 The adjusted coefficient of determination

Denoted (often) by $R_{adj}^2$ , the Adjusted Coefficient of Determination is defined as

$R_{adj}^2=1-\frac{S{S}_E∕\left(n-p\right)}{S{S}_T∕\left(n-1\right)}$

where $p$ is the number of variables in the regression equation. For the simple linear model, $p=2$ since we have two unknown parameters in the regression equation, the intercept $c$ and the coefficient $m$ of $x$ . It can be shown that:

1. $R_{adj}^2$ is a better indicator of the adequacy of predictive power than $R^2$ since it takes into account the number of regressor variables used in the model;
2. $R_{adj}^2$ does not necessarily increase when a new regressor variable is added.

Both coefficients claim to measure the adequacy of the predictive power of a regression model and their values indicate the proportion of variability explained by the model. For example a value of

$R^2\text{or}{R}_{adj}^2=0.9751$

may be interpreted as indicating that a model explains 97.51% of the variability it describes. For example, the drum and cable example considered previously gives the results outlined below with

$R^2=96.2\text{and}{R}_{adj}^2=0.958$

In general, $R_{adj}^2$ is (perhaps) more useful than $R^2$ for comparing alternative models. In the context of a simple linear model, $R^2$ is easier to interpret. In the drum and cable example we would claim that the linear model explains some 96.2% of the variation it describes.

Task!

Use the drum and cable data given in Example 1 (page 7) and set up a spreadsheet to verify the values of the Coefficient of Determination and the Adjusted Coefficient of Determination calculated on page 12.

Answer

1.8 Significance testing for regression

Note that the results in this Section apply to the simple linear model only. Some additions are necessary before the results can be generalized.

The discussions so far pre-suppose that a linear model adequately describes the relationship between the variables. We can use a significance test involving the distribution to decide whether or not $y$ is linearly dependent on $x$ . We set up the following hypotheses:

$H_0:\beta =0\text{and}{H}_1:\beta \ne 0$

Note that we have one degree of freedom since we are testing only one parameter ( $m$ ) and that $n$ denotes the number of pairs of $\left(x,y\right)$ values. A set of tables giving the 5% values of the $F$ -distribution is given at the end of this Workbook (Table 1).

Example 2

Test to determine whether a simple linear model is appropriate for the data previously given in the drum and cable example above.

Solution

We know that

$S{S}_T=S{S}_R+S{S}_E$

where  $S{S}_T=\sum y^2-\frac{{\left(\sum y\right)}^2}{n}$ is the total sum of squares (of $y$ ) so that (from the spreadsheet above) we have:

$S{S}_R=31500-1200=30300$

Hence

$F_{test}=\frac{S{S}_R}{S{S}_E∕\left(n-2\right)}=\frac{30300}{1200∕\left(12-2\right)}=252.5$

From Table 1, the critical value is $f_{0.05,1,10}=241.9$ .

Hence, since $F_{test}>f_{0.05,1,10}$ , we reject the null hypothesis and conclude that $\beta \ne 0$ .

1.9 Regression curves

The Section should be regarded as introductory only. The reason for including non-linear regression is to demonstrate how the method of least squares can be extended to deal with cases where the relationship between variables is, for example, quadratic or exponential.

A regression curve is defined to be the curve passing through the expected value of $Y$ for a set of given values of $x$ . The idea is illustrated by the following diagram.

Figure 6

We will look at the quadratic and exponential cases in a little detail.

1.10 The quadratic case

We are looking for a functional relation of the form

$y=\alpha +\beta x+\gamma x^2$

and so, using the method of least squares, we require the values of $a,b$ and $c$ which minimize the expression

$f\left(a,b,c\right)={\sum }_{r=1}^n{\left(y_r-a-b{x}_r-c{x}_r^2\right)}^2$

Note here that the regression described by the form

$y=\alpha +\beta x+\gamma x^2$

is actually a linear regression since the expression is linear in $\alpha ,\beta$ and $\gamma$ .

Omitting the subscripts and using partial differentiation gives

At a minimum we require

$\frac{\partial f}{\partial a}=\frac{\partial f}{\partial b}=\frac{\partial f}{\partial c}=0$

which results in the three linear equations

which can be solved to give the values of $a,b$ and $c$ .

1.11 The exponential case

We use the same technique to look for a functional relation of the form

$y=\alpha e^{\beta x}$

As before, using the method of least squares, we require the values of $a$ and $b$ which minimize the expression

$f\left(a,b\right)={\sum }_{r=1}^n{\left(y_r-a{\text{e}}^{b{x}_r}\right)}^2$

Again omitting the subscripts and using partial differentiation gives

At a minimum we require

$\frac{\partial f}{\partial a}=\frac{\partial f}{\partial b}=0$

which results in the two non-linear equations

which can be solved by iterative methods to give the values of $a$ and $b$ .

Note that it is possible to combine (for example) linear and exponential regression to obtain a regression equation of the form

$y=\left(\alpha +\beta x\right){\text{e}}^{\gamma x}$

The method of least squares may then be used to find estimates $a,b,c$ of $\alpha ,\beta ,\gamma$ .