Estimation of Simple Regression Models

The aim of econometrics is to analyze the economic relations determined by the economic theory within the framework of quantitative models, to express the reasons for the change of the variables with numerical measures, to predict the future values of these variables and to test the accuracy of the economic theories.

Economic theory often postulates hypotheses. While economic theory focuses on the direction of the relationship between two or more variables, it is far from giving a numerical measurement of the relationship. Econometrics, on the other
hand, adds an empirical content to economic theory. The task of mathematical economics is to put economic theory into mathematical patterns. The conversion and verification of mathematical equations into econometric equations is the subject of econometrics. This field focuses on collectingand processing economic data. Confirmation of
economic theories by using these collected data and making numerical estimations is the subject of econometrics.

Regression is a statistical method that investigates the possibility of estimating the value of the parameters that establish a relationship between dependent and independent variables.

A simple linear regression model has precisely one dependent variable (Y) and exactly one independent variable (X). If the number of variables changes, this regression model will no longer be referred to as a simple regression model.

Data are sources of information that contain numerical values for all the variables, including one dependent variable and one or more independent variables. Data are needed to make predictions using the econometric models and the performance of the model depends on appropriate data.

In statistics, a population refers to the entire set of individuals, objects, or events that you are interested in studying or making inferences about. It represents the complete group that you want to draw conclusions about.

A sample, on the other hand, is a subset or smaller representation of the population. It is a smaller group selected from the population for the purpose of analysis. Sampling is done when it is impractical or not feasible to study the entire population. By studying a representative sample, you can make inferences and draw conclusions about the population as a whole

Statistical inference is the process of drawing conclusions about a population based on sample data. It involves using probability theory and statistical techniques to make inferences about a population parameter based on sample data.

One observation contains one numeric value of each variable. All of the observations are called the data.

Using regression analysis, it becomes possible to address questions such as whether a relationship exists between the height of parents and the height of their children. You can then compare your prior assumptions on these matters.

The aim in OLS is to find β estimators that minimize the errors; more specifically the sum of squared errors.

BLUE is the best linear unbiased estimator.

For a regression model that satisfy the following 10 assumptions, the OLS estimators, in the class of unbiased linear estimators, have minimum variance; that is, they are the best linear unbiased estimators (BLUE).

1. The econometric model has to be a linear regression model. The parameters (β0, β1) have to be a polynomial of order 1, but linearity has not to be confused with data representation. Linearity in terms of variables is not required, the model can have terms like X2 or any power. If the model is not linear in parameters, then the model may be transformed to make it linear and calculate the BLUE estimators for
the transformed model.

2. X values are fixed (nonstochastic) in repeated sampling. More technically, if a survey is made, the X values used in this survey must
also be used in all the following surveys.

3. Zero average (mean) value of error term ui.

4. Homoscedasticity or equal variance of ui.

5. Homoscedasticity or equal variance of ui.

6. Zero covariance between ui and Xi. There should not be a relation between these two terms.

7. The number of observations n must be greater than the number of parameters to be estimated.

8. Variability in X values. The X values in a given sample must not all be the same and preferably, there should be a very large difference between them.

9. The regression model is correctly specified. The relation in the real world and the model have to be exactly the same.

10. There is no perfect multicollinearity. The independent variables should not be written in deterministic linear function of each other; otherwise, OLS estimators cannot be calculated. 

Unbiasedness is a very important property of OLS estimator because it promises to calculate the population parameters.

Each random variable has its probabilistic density function (PDF). This function has a center of the distribution measured with the statistics of mean (μ) or the weighted average of random variables. In addition, this function shows the density of observations with the statistics of variance σ2.

OLS estimator formulas are linear because β’s can be written as a linear polynomialfunction of Yi’s.

Coefficient of determination measures how well the sample regression line fits the data.

TSS (Total Sum of Squares) is the total variation in data, ESS (Explained Sum of Squares) is the variation that the estimates capture to explain, and RSS (Residual Sum of Squares) is the unexplained value by the model.

Correlation coefficient r, which measures the strength and direction of linear relation between Y and X.