The easiest way to observe the relation of two variables is to draw a scatter plot with one variable as X axis and the other as Y axis. If two variables are related, data will gather together with a certain pattern, and if not related, data will be scattered around. The correlation analysis is a method of analyzing the degree of linear relationship between two variables. It is to investigate how linearly the other variable increases or decreases as one variable increases.

The relation between two variables can be roughly investigated using a scatter plot like this. However, a measure of the extent of the relation can be used together to provide a more accurate and objective view of the relation between two variables. As a measure of the relation between two variables, there is a covariance. The population covariance of the two variables \(X\) and \(Y\) is denoted as \(Cov(X,Y)\). When the random samples of two variables are given as \( (X_1 , Y_1 ) , (X_2 , Y_2 ), ... , (X_n , Y_n ) \), the estimate of the population covariance using samples, which is called the sample covariance, \(S_{XY}\), is defined as follows: $$\small \begin{align} S_{XY} &= \frac{1}{n-1} \sum_{i=1}^{n} ( X_i - \overline X )( Y_i - \overline Y ) \\ &= \frac{1}{n-1} ( \sum_{i=1}^{n} X_i Y_i - n {\overline X}{\overline Y} ) \end{align} $$ In the above equation, \(\small \overline X\) and \(\small \overline Y\) represent the sample means of \(X\) and \(Y\) respectively.

In order to understand the meaning of covariance, consider a case that \(Y\) increases if \(X\) increases. If the value of \(X\) is larger than \(\small \overline X\) and the value of \(Y\) is larger than \(\small \overline Y\), then \(\small (X - \overline X)(Y- \overline Y) \) always has a positive value. Also, if the value of \(X\) is smaller than \(\small \overline X\) and the value of \(Y\) is smaller than \(\small \overline Y\), then \(\small (X - \overline X)(Y- \overline Y) \) has a positive value. Therefore, their mean value which is the covariance tends to be positive. Conversely, if the value of the covariance is negative, the value of the other variable decreases as the value of one variable increases. Hence, by calculating covariance, we can see the relation between two variables: positive correlation (i.e., increasing the value of one variable will increase the value of the other) or negative correlation (i.e., decreasing the value of the other).

Covariance itself is a good measure, but, since the covariance depends on the unit of \(X\) and \(Y\), it makes difficult to interpret the covariance according to the size of the value and inconvenient to compare with other data. Standardized covariance which divides the covariance by the standard deviation of \(X\) and \(Y\), \(\sigma_{X}\) and \(\sigma_{Y}\), to obtain a measurement unrelated to the type of variable or specific unit, is called the population correlation coefficient and denoted as \(\rho\).

$$ \text{Population Correlation Coefficient: } \rho = \frac{Cov (X, Y)} { \sigma_X \sigma_Y } $$

<Figure 12.1.3> shows different scatter plots and its values of the correlation coefficient.

The correlation coefficient \(\rho\) is interpreted as follows:

『eStatU』 provides a simulation of scatter plot shapes for different correlations as in <Figure 12.1.4>.

[Correlation Simulation]

An estimate of the population correlation coefficient using samples of two variables is called the sample correlation coefficient and denoted as \(r\). The formula for the sample correlation coefficient \(r\) can be obtained by replacing each parameter with the estimates in the formula for the population correlation coefficient. $$ r = \frac {S_{XY}} { S_X S_Y } $$ where \(S_{XY}\) is the sample covariance and \(S_{X}\), \(S_{Y}\) are the sample standard deviations of \(X\) and \(Y\) as follows: $$\small \begin{align} S_{XY} &= \frac{1}{n-1} \sum_{i=1}^{n} (X_i - \overline X )(Y_i - \overline Y ) \\ S_X^2 &= \frac{1}{n-1} \sum_{i=1}^{n} (X_i - \overline X )^{2} \\ S_Y^2 &= \frac{1}{n-1} \sum_{i=1}^{n} (Y_i - \overline Y )^{2} \\ \end{align} $$ Therefore, the formula \(r\) can be written as follows $$\small \begin{align} r &= \frac {\sum_{i=1}^{n} (X_i - \overline X )(Y_i - \overline Y )} { \sqrt{\sum_{i=1}^{n} (X_i - \overline X )^{2} \sum_{i=1}^{n} (Y_i - \overline Y )^{2} } } \\ &= \frac {\sum_{i=1}^{n} X_i Y_i - n \overline X \overline Y } { \sqrt{\left (\sum_{i=1}^{n} X_{i}^{2} - n {\overline X}^2 \right) \left( \sum_{i=1}^{n} Y_{i}^{2} - n {\overline Y}^{2} \right) } } \end{align} $$

Sample correlation coefficient \(r\) can be used for testing hypothesis of the population correlation coefficient. The main interest in testing hypothesis of \(\rho\) is \(H_0 : \rho = 0\) which tests the existence of linear correlation. This test can be done using \(t\) distribution as follows:

If there are more than three variables in the analysis, the relationship can be viewed using the scatter plots for each combination of two variables and the sample correlation coefficients can be obtained. However, to make it easier to see the relationship between the variables, the correlations between the variables can be arranged in a matrix format which is called a correlation matrix. 『eStat』 shows the result of a correlation matrix and the significance test for those values. The result of the test shows the t value and p-value.

Regression analysis is a statistical method that first establishes a reasonable mathematical model of relationships between variables, estimates the model using measured values of the variables, and then uses the estimated model to describe the relationship between the variables, or to apply it to the analysis such as forecasting. For example, a mathematical model of the relationship between sales (\(Y\)) and advertising costs (\(X\)) would not only explain the relationship between sales and advertising costs, but would also be able to predict the amount of sales for a given investment.

As such, the regression analysis is intended to investigate and predict the degree of relation between variables and the shape of the relation. In regression analysis, a mathematical model of the relation between variables is called a regression equation, and the variable affected by other related variables is called a dependent variable. The dependent variable is the variable we would like to describe which is usually observed in response to other variables, so it is also called a response variable. In addition, variables that affect the dependent variable are called independent variables. The independent variable is also referred to as the explanatory variable, because it is used to describe the dependent variable. In the previous example, if the objective is to analyse the change in sales amounts resulting from increases and decreases in advertising costs, the sales amount is a dependent variable and the advertising cost is an independent variable.

If the number of independent variables included in the regression equation is one, it is called a simple linear regression. If the number of independent variables are two or more, it is called a multiple linear regression.

Simple linear regression analysis has only one independent variable and the regression equation is shown as follows: $$ Y = f(X,\alpha,\beta) = \alpha + \beta X $$ In other words, the regression equation is represented by the linear equation of the independent variable, and \(\alpha\) and \(\beta\) are unknown parameters which represent the intercept and slope respectively. The \(\alpha\) and \(\beta\) are called the regression coefficients. The above equation represents an unknown linear relationship between \(Y\) and \(X\) in population and is therefore, referred to as the population regression equation.

In order to estimate the regression coefficients \(\alpha\) and \(\beta\), observations of the dependent and independent variable are required, i.e., samples. In general, all of these observations are not located in a line. This is because, even if the \(Y\) and \(X\) have an exact linear relation, there may be a measurement error in the observations, or there may not be an exact linear relationship between \(Y\) and \(X\). Therefore, the regression formula can be written by considering these errors together as follows: $$ Y_i = \alpha + \beta X_i + \epsilon_{i}, \quad i=1,2,...,n $$ where \(i\) is the subscript representing the \(i^{th}\) observation, and \(\epsilon_i\) is the random variable indicating an error with a mean of zero and a variance \(\sigma^2\) which is independent of each other. The error \(\epsilon_i\) indicates that the observation \(Y_i\) is how far away from the population regression equation. The above equation includes unknown population parameters \(\alpha\), \(\beta\) and \(\sigma^2\), and is therefore, referred to as a population regression model.

If \(a\) and \(b\) are the estimated regression coefficients using samples, the fitted regression equation can be written as follows: It is referred to as the sample regression equation. $$ {\hat Y}_i = a + b X_i $$ In this expression, \({\hat Y}_i\) represents the estimated value of \(Y\) at \(X=X_i\) as predicted by the appropriate regression equation. These predicted values can not match the actual observed values of \(Y\), and differences between these two values are called residuals and denoted as \(e_i\). $$ \text{Residuals} \qquad e_i = Y_i - {\hat Y}_i , \quad i=1,2,...,n $$ The regression analysis makes some assumptions about the unobservable error \(\epsilon_i\). Since the residuals \(e_i\) calculated using the sample values have similar characteristics as \(\epsilon_i\), they are used to investigate the validity of these assumptions. (Refer to Section 12.2.6 for residual analysis.)

When sample data, \((X_1 , Y_1 ) , (X_2 , Y_2 ) , ... , (X_n , Y_n ) \), are given, a straight line representing it can be drawn in many ways. Since one of the main objectives of regression analysis is prediction, we would like to use the estimated regression line that would make the residuals smallest that the error occurs when predicting the value of Y. However, it is not possible to minimize the value of the residuals at all points, and it should be chosen to make the residuals 'totally' smaller. The most widely used of these methods is the method which minimizes the total sum of squared residuals, that is called the method of least squares regression.

To obtain the values of \(\alpha\) and \(\beta\) by the least squares method, the sum of squares above should be differentiated partially with respect to \(\alpha\) and \(\beta\), and equate them zero respectively. If the solution of \(\alpha\) and \(\beta\) of these equations is \(a\) and \(b\), the equations can be written as follows: $$ \begin{align} a \cdot n + b \sum_{i=1}^{n} X_i &= \sum_{i=1}^{n} Y_i \\ a \sum_{i=1}^{n} X_i + b \sum_{i=1}^{n} X_i^2 &= \sum_{i=1}^{n} X_i Y_i \\ \end{align} $$

The above expression is called a normal equation. The solution \(a\) and \(b\) of this normal equation is called the least squares estimator of \(\alpha\) and \(\beta\) and is given as follows:

If we divide both the numerator and the denominator of \(b\) by \(n-1\), \(b\) can be written as \(b = \frac{S_{XY}}{S_{X}^2}\). Since the correlation coefficient is \(r = \frac{S_{XY}}{S_X S_Y}\) and therefore, the slope \(b\) can also be calculated by using the correlation coefficient as follows: $$ b = \frac{S_{XY}}{S_X ^2} = \frac{ r S_X S_Y } {S_X ^2 } = r \frac{S_Y}{S_X} $$

After estimating the regression line, it should be investigated how valid the regression line is. Since the objective of a regression analysis is to describe a dependent variable as a function of an independent variable, it is necessary to find out how much the explanation is. A residual standard error and a coefficient of determination are used for such validation studies.

Residual standard error \(s\) is a measure of the extent to which observations are scattered around the estimated line. First, you can define the sample variance of residuals as follows: $$ s^2 = \frac{1}{n-2} \sum_{i=1}^{n} ( Y_i - {\hat Y}_i )^2 $$ The residual standard error \(s\) is defined as the square root of \(s^2\). The \(s^2\) is an estimate of \(\sigma^2\) which is the extent that the observations \(Y\) are spread around the population regression line. A small value of \(s\) or \(s^2\) indicates that the observations are close to the estimated regression line, which in turn implies that the regression line represents well the relationship between the two variables.

However, it is not clear how small the residual standard error \(s\) is, although the smaller value is the better. In addition, the size of the value of \(s\) depends on the unit of \(Y\). To eliminate this shortcoming, a relative measure called the coefficient of determination is defined. The coefficient of determination is the ratio of the variation described by the regression line over the total variation of observation \(Y_i\), so that it is a relative measure that can be used regardless of the type and unit of the variable.

As in the analysis of variance in Chapter 9, the following partitions of the sum of squares and degrees of freedom are formed in the regression analysis:

Description of the above three sums of squares is as follows:

Total Sum of Squares : \( \small SST = \sum_{i=1}^{n} ( Y_i - {\overline Y} )^2\)
The total sum of squares indicating the total variation in observed values of \(Y\) is called the total sum of squares (\(SST\)). This \(SST\) has the degree of freedom, \(n-1\), and if \(SST\) is divided by the the degree of freedom, it becomes the sample variance of \(Y_i\).

Error Sum of Squares : \( \small SSE = \sum_{i=1}^{n} ( Y_i - {\hat Y}_i )^2\)
The error sum of squares (\(SSE\)) of the residuals represents the unexplained variation of the total variation of the \(Y\). Since the calculation of this sum of squares requires the estimation of two parameters \(\alpha\) and \(\beta\), \(SSE\) has the degree of freedom \(n-2\). This is the reason why, in the calculation of the sample variance of residuals \(s^2\), it was divided by \(n-2\).

Regression Sum of Squares : \( \small SSR = \sum_{i=1}^{n} ( {\hat Y}_i - {\overline Y} )^2 \)
The regression sum of squares (\(SSR\)) indicates the variation explained by the regression line among the total variation of \(Y\). This sum of squares has the degree of freedom of 1.

If the estimated regression equation fully explains the variation in all samples (i.e., if all observations are on the sample regression line), the unexplained variation \(SSE\) will be zero. Thus, if the portion of \(SSE\) is small among the total sum of squares \(SST\), or if the portion of \(SSR\) is large, the estimated regression model is more suitable. Therefore, the ratio of \(SSR\) to the total variation \(SST\), called the coefficient of determination, is defined as a measure of the suitability of the regression line as follows: $$ R^2 = \frac{Explained \;\; Variation}{Total \;\; Variation} = \frac{SSR}{SST} $$ The value of the coefficient of determination is always between 0 and 1 and the closer the value is to 1, the more concentrated the samples are around the regression line, which means that the estimated regression line explains the observations well.

If we divide three sums of squares obtained in the above example by its degrees of freedom, each one becomes a kind of variance. For example, if you divide the \(SST\) by \(n-1\) degrees of freedom, then it becomes the sample variance of the observed values \(Y_1 , Y_2 , ... , Y_n\). If you divide the \(SSE\) by \(n-2\) degrees of freedom, it becomes \(s^2\) which is an estimate of the variance of error \(\sigma^2\). For this reason, addressing the problems associated with the regression using the partition of the sum of squares is called the ANOVA of regression. Information required for ANOVA, such as calculated sum of squares and degrees of freedom, can be compiled in the ANOVA table as shown in Table 12.2.2.

The sum of squares divided by its degrees of freedom is referred to as mean squares, and Table 12.2.2 defines the regression mean squares (\(MSR\)) and error mean squares (\(MSE\)) respectively. As the expression indicates, \(MSE\) is the same statistic as \(s^2\) which is the estimate of \(\sigma^2\).

The \(F\) value given in the last column are used for testing hypothesis \(H_0: \beta = 0 ,\; H_1 : \beta \ne 0 \). If \(\beta\) is not 0, the \(F\) value can be expected to be large, because the assumed regression line is valid and the variation of \(Y\) is explained in large part by the regression line. Therefore, we can reversely decide that \(\beta\) is not zero if the calculated \(F\) ratio is large enough. If the assumptions about the error terms mentioned in the population regression model are valid and if the error terms follows a normal distribution, the distribution of \(F\) value, when the null hypothesis is true, follows distribution with 1 and \(n-2\) degrees of freedom. Therefore, if \(F_0 > F_{1,n-2; α}\), then we can reject \(H_0 : \beta = 0\) .

One assumption of the error term \(\epsilon\) in the population regression model is that it follows a normal distribution with the mean of zero and variance of \(\sigma^2\). Under this assumption the regression coefficients and other parameters can be estimated and tested. Note that, under the assumption above, the regression model \(Y = \alpha + \beta X + \epsilon \) follows a normal distribution with the mean \(\alpha + \beta X \) and variance \(\sigma^2\).

The inference for each regression parameter in the previous section is all based on some assumptions about the error term \(\epsilon\) included in the population regression model. Therefore, the satisfaction of these assumptions is an important precondition for making a valid inference. However, because the error term is unobservable, the residuals as estimate of the error term are used to investigate the validity of these assumptions which are referred to as a residual analysis.

First, let's look at the assumptions in the regression model.

\(\quad \) Assumptions in regression model
\(\quad \;\; A_1\): The assumed model \(Y = \alpha + \beta X + \epsilon\) is correct.
\(\quad \;\; A_2\): The expectation of error terms \(\epsilon_i\) is 0.
\(\quad \;\; A_3\): (Homoscedasticity) The variance of \(\epsilon_i\) is \(\sigma^2\) which is the same for all \(X\).
\(\quad \;\; A_4\): (Independence) Error terms \(\epsilon_i\) are independent.
\(\quad \;\; A_5\): (Normality) Error terms \(\epsilon_i\)’s are normally distributed.

Review the references for the meaning of these assumptions. The validity of these assumptions is generally investigated using scatter plots of the residuals. The following scatter plots used primarily for each assumption:

\(\quad \)1) Residuals versus predicted values (i.e., \(e_i\) vs \(\hat Y_i\)) : \(\quad A_3\)
\(\quad \)2) Residuals versus independent variables (i.e., \(e_i\) vs \(X_i\)) : \(\quad A_1\)
\(\quad \)3) Residuals versus observations (i.e., \(e_i\) vs \(i\)) : \(\quad A_2 , A4\)

In the above scatter plots, if the residuals show no particular trend around zero, and appear randomly, then each assumption is valid.

The assumption that the error term \(\epsilon\) follows a normal distribution can be investigated by drawing a histogram of the residuals in case of a large amount of data to see if the distribution is similar to the shape of the normal distribution. Another method is to use the quantile–quantile (Q-Q) scatter plot of the residuals. In general, if the Q-Q scatter plot of the residuals forms a straight line, it can be considered as a normal distribution.

Since residuals are also dependent on the unit of the dependent variable, standardized values of the residuals are used for consistent analysis of the residuals, which are called standardized residuals. Both the scatter plots of the residuals described above and the Q-Q scatter plot are created using the standardized residuals. In particular, if the value of the standardized residuals is outside the \(\pm\)2, an anomaly value or an outlier value can be suspected.

In 『eStatU』 it is possible to do experiments on how much a regression line is affected by an extreme point (<Figure 12.2.8>). A point can be created by clicking the mouse on the screen in the link below. If you create multiple dots, you can see how much the regression line changes each time. You can observe how sensitive the correlation coefficient and the coefficient of determination are as you move a point along with the mouse.

[Regression Experiment ]

For actual applications of the regression analysis, the multiple regression models with two or more independent variables are more frequently used than the simple linear regression with one independent variable. This is because it is rare for a dependent variable to be sufficiently explained by a single independent variable, and in most cases, a dependent variable has a relationship with several independent variables. For example, it may be expected that sales will be significantly affected by advertising costs, which are examples of simple linear regression, but will also be affected by product quality ratings, the number and size of stores sold. The statistical model used to identify the relationship between one dependent variable and several independent variables is called a multiple linear regression analysis. However, the simple linear regression and multiple linear regression analysis differ only in the number of independent variables involved, and there is no difference in the method of analysis.

In the multiple linear regression model, it is assumed that the dependent variable \(Y\) and \(k\) number of independent variables have the following relational formulas: $$ Y_i = \beta_0 + \beta_1 X_{i1} + \cdots + \beta_k X_{ik} + \epsilon_i $$ This means that the dependent variable is represented by the linear function of independent variables and a random variable that represents the error term as in the simple linear regression model. The assumption of the error terms is the same as the assumption in the simple linear regression. In the above equation, \(\beta_0\) is the intercept of \(Y\) axis and \(\beta_i\) is the slope of the Y axis and \(X_i\) which indicates the effect of \(X_i\) to \(Y\) when other independent variables are fixed.

In general, matrix and vectors are used to facilitate expression of formula and calculation of expressions. For example, if there are \(k\) number of independent variables, the population multiple regression model at the observation point \(i=1,2,...,n\) is presented in a simple manner as follows: $$ \mathbf {Y = X} \boldsymbol{\beta + \epsilon} $$

Here \(\mathbf {Y, X}, \boldsymbol{\beta , \epsilon}\) are defined as follows: $$ {\bf Y} = \left[ \matrix{ Y_1 \\ Y_2 \\ \cdot \\ \cdot \\ Y_n } \right], \quad {\bf X} = \left[ \matrix{ 1 & X_{11} & X_{12} & \cdots & X_{1k} \\ 1 & X_{21} & X_{22} & \cdots & X_{2k} \\ & & \cdots \\ & & \cdots \\ 1 & X_{n1} & X_{n2} & \cdots & X_{nk} } \right], \quad {\boldsymbol \beta} = \left[ \matrix{ \beta_0 \\ \beta_1 \\ \cdot \\ \cdot \\ \beta_k } \right], \quad {\boldsymbol \epsilon} = \left[ \matrix{ \epsilon_1 \\ \epsilon_2 \\ \cdot \\ \cdot \\ \epsilon_n } \right] $$

In a multiple regression analysis, it is necessary to estimate the \(k+1\) number of regression coefficients \(\beta_0 , \beta_1 , ... , \beta_k\) using samples. In this case, the least squares method which minimizes the sum of the squared errors is also used. We find \(\boldsymbol \beta\) which minimizes the following sum of the error squares as follows: $$ S = \sum_{i=1}^{n} \epsilon_{i}^2 = {\boldsymbol \epsilon ' \boldsymbol \epsilon} = ( \bf Y - \bf X' \boldsymbol \beta )'( \bf Y - \bf X' \boldsymbol \beta ) $$ As in the simple linear regression, the above error sum of squares is differentiated with respect to \(\boldsymbol \beta\) and then equate to zero which is called a normal equation. The solution of the equation, denoted as \(\bf b\) which is called the least squares estimate of \(\boldsymbol \beta\), should satisfy the following normal equation. $$ \bf {(X'X) b = X'y} $$ Therefore, if there exists an inverse matrix of \(\bf {X'X}\), the least squares estimator of \(\boldsymbol \beta\), \(\bf b\), is as follows: $$ \bf {b = (X'X)^{-1} X'y} $$ (Note: Statistical packages uses a different formula, because the above formula causes large amount of computing error)

If the estimated regression coefficients are \({\bf b} = (b_0 , b_1 , ... , b_k )\), the estimate of the response variable \(Y\) is as follows: $$ {\hat Y}_i = b_0 + b_1 X_{i1} + \cdots + b_k X_{ik} $$ The residuals are as follows: $$ \begin{align} e_i &= Y_i - {\hat Y}_i \\ &= Y_i - (b_0 + b_1 X_{i1} + \cdots + b_k X_{ik} ) \end{align} $$ By using a vector notation, the residual vector \(\bf e\) can be defined as follows: $$ \bf {e = Y - X b} $$

In order to investigate the validity of the estimated regression line in the multiple regression analysis, the standardized residual error and coefficient of determination are also used. In the simple linear regression analysis, the computational formula for these measures was given as a function of the residuals, i.e., observed value of \(Y\) and its predicted value, there is nothing to do with the number of independent variables. Therefore, the same formula can be used in the multiple linear regression and there is only a difference in the value of the degrees of freedom that each sum of squares has.

In the multiple linear regression analysis, the standard error of residuals is defined as follows: $$ s = \sqrt { \frac{1}{n-k-1} \sum_{i=1}^{n} (Y_i - {\hat Y}_i )^2} $$ The difference from the simple linear regression is that the degrees of freedom for residuals is \(n-k-1\), because the \(k\) number of regression coefficients must be estimated in order to calculate residuals. As in simple linear regression, \(s^2\) is a statistic such as the residual mean squares (\(MSE\)). The coefficient of determination is given in \(R^2 = \frac{SSR}{SST}\) and its interpretation is as shown in the simple linear regression.

The sum of squares is defined by the same formula as in the simple linear regression, and can be divided with corresponding degrees of freedom as follows and the table of the analysis of variance is shown in Table 12.3.2.

\(\quad\) Sum of squares \(\quad \quad \;\;SST = SSE + SSR\)
\(\quad\) Degrees of freedom \(\quad (n-1) = (n-k-1) + k\)

The \(F\) value in the above ANOVA table is used to test the significance of the regression equation, where the null hypothesis is that all independent variables are not linearly related to the dependent variables. $$ \begin{align} H_0 &: \beta_1 = \beta_2 = \cdots = \beta_k = 0 \\ H_1 &: \text{At least one of } k \text { number of } \beta_i \text{s is not equal to 0} \end{align} $$ Since \(F_0\) follows \(F\) distribution with \(k\) and \((n-k-1)\) degrees of freedom under the null hypothesis, we can reject \(H_0\) at the significance level \(\alpha\) if \(F_0 \gt F_{k,n-k-1 ; α}\). Each \(\beta_i\) can also be tested which is described in the following sections. (Also, 『eStat』 calculates the \(p\)-value for this test, so use this \(p\)-value to test. That is, if the \(p\)-value is less than the significance level, the null hypothesis is rejected.)

Parameters that are of interest in multiple linear regression, as in the simple linear regression, are the expected value of Y and each regression coefficients \(\beta_0 = \beta_1 = \cdots = \beta_k\). The inference of these parameters \(\beta_0 = \beta_1 = \cdots = \beta_k\) is made possible by obtaining a probability distribution of the point estimates \(b_i\). Under the assumption that the error terms \(\epsilon_i\) are independent and all have a distribution of \(N(0, \sigma^2 )\), it can be shown that the distribution of \(b_i\) is as follows: $$ b_i \sim N( \beta_i , c_{ii} \cdot \sigma^2 ), \quad i=0,1,2,...,k $$ The above \(c_{ii}\) is the \(i^{th}\) diagonal element of the \((k+1)\times (k+1)\) matrix \(\bf {(X'X)^{-1}}\). In addition, using an estimate \(s^2\) instead of a parameter \(\sigma^2\), you can make inferences about each regression coefficient using the \(t\) distribution.

Residual analysis of the multiple linear regression is the same as in the simple linear regression.