|San José State University|
& the Gateway
to the Rockies
A Real Derivation|
of Regression Analysis
in Matrix Form
This is a real derivation of regression analysis in matrix form. An internet search for such a derivation brings up purported derivations which are not really derivations. They set up the material for a derivation and then simply give the final result.
Let Y be an n-dimensional row vector, a 1 by n matrix. This is the data for the dependent variable. The data for the m explanatory variables is given as an m by n matrix. Let B be an m dimensional row vector, a 1 by m matrix.
The deviations D between the dependent variables and the estimates based upon the coefficients B are given by
The sum S of the squared deviations can be expressed as
The best B is the one that minimizes S. They are called the Least Squares estimates. Here is how they are derived.
Now consider S(B+ΔB) which is
In the expression for ΔS=[S(B+ΔB) − S(B)] all of the terms not involving ΔB are eliminated. That leaves
Now let ΔB go to an infinitesimal dB. This eliminates the term ΔBXXTΔBT which would involve the products of infinitesimals. Thus
The term dB(XXTBT− XYT) is just the transpose of (BXXT−YXT)dBT and both are 1x1 matrices; i.e., scalars.
This means that
For an extremum (∂S/∂B) must be the zero row vector 0 and hence
The second order conditions for an unconstained minimum are that the matrix of the second derivatives of the parameters is positive definite. Let the second derivatives of the regression coefficients B be denoted as
The matrix (XXT) by the nature of its construction is positive definite. Therefore the regression estimates
minimize the sum of the squared deviations between the actual dependent variables and their regression estimates.
The Least Squares estimates of the regression coefficients for the depemdent variables in terms of the independent explanatory variables X are given by
HOME PAGE OF Thayer Watkins