Multiple Linear Regression: How to Calculate the Coefficient

#2
Unfortunately there's no easy way to walk through the steps quite so simply as in the univariate case, since it gets more and more complicated the larger the number of variables is. However, if you're ok with matrix algebra, it's actually quite simple.

Let your model be defined as y=Xb+e, where X is the matrix of data (m columns, one for each variable, and n rows, one for each observation), y is the vector of dependent variable observations, b is the vector of true coefficients, and e is a vector of error terms.

The vector of estimated coefficients in a multivariate OLS regression, b*, is given by:

b*=(X'X)^-1X'y, where X' is the transpose of X, and By ^-1 I mean the inverse of the matrix (X'X).

Notice that this doesn't depend on the actual values of m or n--it's the same equation no matter the size of the matrices. You can check to confirm that when m=1, this is the same procedure as the univariate case.

So to code multiple regression you'd just need to to input, transpose, and invert matrices.

Prediction: Once you have your vector of betas, you can just take an observation [x1(i), x2(i), ..., xm(i)] and multiply it by the b* vector. The resulting scalar is your prediction for y(i).
 

Draze

New Member
#3
Unfortunately there's no easy way to walk through the steps quite so simply as in the univariate case, since it gets more and more complicated the larger the number of variables is. However, if you're ok with matrix algebra, it's actually quite simple.

Let your model be defined as y=Xb+e, where X is the matrix of data (m columns, one for each variable, and n rows, one for each observation), y is the vector of dependent variable observations, b is the vector of true coefficients, and e is a vector of error terms.

The vector of estimated coefficients in a multivariate OLS regression, b*, is given by:

b*=(X'X)^-1X'y, where X' is the transpose of X, and By ^-1 I mean the inverse of the matrix (X'X).
Matrix algebra is something I've never delved into, but for this purpose, I've tried to teach myself in order to better understand your solution. Complexity isn't an issue since a computer will be doing the work, all I need is the concept.

So I'm going to take the following data, and apply this formula b*=(X'X)^-1X'y... I am inexperienced with matrix algebra so please correct me where I am wrong.

Y (DV): 1,2,3
X1 (IV): 1,2,3
X2 (IV): 3,2,1

Code:
	   [B][U]X'X[/U][/B]
[B]   X        X'[/B]
[1 2 3] * [1 3] = [1*1+2*2+3*3  1*3+2*2+3*1] = [14 10]
[3 2 1] * [2 2]   [3*1+2*2+1*3  3*3+2*2*1*1]   [10 14] 
	  [3 1]                                correct?

	  [B][U]X'X^-1[/U][/B]
= [-14 -10]  or positional inverse? [10 14]
  [-10 -14]                         [14 10]

	   [B][U]X'Y[/U][/B]
[B]  X'     Y[/B]        
[1 3] * [1] = [1*1+3*1] = [4 ]
[2 2] * [2]   [2*2+2*2]   [8 ]
[3 1] * [3]   [3*3+1*3]   [12]

       [B][U](X'X^-1) * (X'y)[/U][/B]
[10 14] * [4 ] = ?
[14 10]   [8 ]
          [12]
Thank you for your reply and time.