Estimation of Multiple Regression Models

Multiple regression is a predictive analysis technique that investigates the relationship between a dependent variable and several independent
variables.

Y is the dependent variable or explained variable, while X₁,X₂ ,…,X_k are the independent variables or explanatory variables, and u represents the error terms.

Given that there are k independent variables along with an intercept, equation encompasses k+1 unknown parameters. Here, β₀ is the intercept, while β_i denote the slope parameters. Consequently, the equation contains k+1 unknown parameters, which comprise β₀, the intercept, and k slope parameters, β_i. The presence of k independent variables and an intercept account for these k+1 unknown parameters.

This assumption means that the variance of the residuals should be constant across all levels of the independent variables. In other words, the spread of the residuals should be similar for all values of the independent variables, without any discernible pattern. This assumption is known as homoscedasticity.

The Ordinary Least Squares (OLS) method is a foundational statistical technique used to estimate the parameters of a linear regression model by minimizing the sum of the squared differences between observed and predicted values.

The estimated coefficients are derived by calculating the partial derivatives of the objective function and setting them equal to zero. First we arrive at the normal equations. The OLS estimators of β^0 , β^1,..., β^k are obtained by solving this system of equations.

β^0 (Intercept) = 129,939.06: The estimated average price for a house with 0 square footage and 0 years old is 129,936 Dollars. This interpretation might not be meaningful in a real-world context. But the intercept term gives us the average value of the dependent variable (here it is price of the house) when the explanatory variables are equal to zero.

β^2 (Age of House) = -3,140.93: This implies that, on average, an increase of 1 year in the age of the house is associated with a decrease of $3,140 in the house price, holding the square footage fixed.

It is similar to ceteris paribus assumption in economics. In multiple regression, we are usually interested in understanding how a change in a particular independent variable affects the dependent variable, assuming no change in the other independent variables.

The residuals for observation, which show the difference between the actual values of the dependent variable and the fitted values.

1. The sum of the residuals is zero,
2. The mean of the fitted values is equal to the mean of the observed values,
3. The residuals are uncorrelated with X_1i and X_2i,
4. The residuals are uncorrelated with fitted values.

One common measure of goodness of fit in multiple regression is the coefficient of determination, also known as R-squared.

R² shows the proportion of variation in Y explained collectively by multiple independent variables, such as X₂ and X₃.

Where ESS is the explained sum of squares, TSS is the total sum of squares and RSS is the residual sum of squares.

Adjusted R² (denoted as R ̅ ² ) is used. It adjusts or reduces the R² value, providing a more accurate estimation of the model’s fit to the data

The adjusted R² value of 0.87 indicates that approximately 87% of the variation in the house price can be explained by the independent variables (size and age of house) included in the model.

The standard error of regression is a statistical tool used to measure the average distance between the observed data points and the fitted regression line in a dataset. It helps evaluate the accuracy of the regression fit and confidence level of estimated values

An estimator is unbiased if its expected value equals the parameter of the population. Mathematically,

where ‘E’ is the expectation operator. The unbiasedness property of OLS estimators is the basic minimum requirement to be satisfied by any estimator, but it is not sufficient.

Linearity, Unbiasedness, Efficiency and Consistency are the key properties of OLS estimators in a multiple regression model.

The efficiency of an estimator refers to the concept that an estimator has the smallest variance among all unbiased estimators of a parameter. Essentially, an efficient estimator minimizes the likelihood of deviating substantially from the actual value of a parameter.