摘要: 本文是吴恩达 (Andrew Ng)老师《机器学习》课程，第五章《多变量线性回归》中第33课时《正规方程》的视频原文字幕。为本人在视频学习过程中记录下来并加以修正，使其更加简洁，方便阅读，以便日后查阅使用。现分享给大家。如有错误，欢迎大家批评指正，在此表示诚挚地感谢！同时希望对大家的学习能有所帮助。
In this video (article), we'll talk about the normal equation, which for some linear regression problems, will give us a much better way to solve for the optimal value of the parameters theta.
Concretely, so far the algorithm that we've been using for linear regression is gradient descent where in order to minimize the cost function 
, we would take this iterative algorithm that takes many steps, multiple iterations of gradient descent to converge to the global minimum. In contrast, the normal equation would give us a method to solve for 
analytically, so that rather than needing to run this iterative algorithm, we can instead just solve for the optimal value for theta all at one go, so that in basically one step you get to the optimal value right there. It turns out that the normal equation method has some advantages and some disadvantages, but before we get to that and talk about when you should use it, let's get some intuition about what this method does.
For this explanatory example, let's take a very simplified cost function , that's just the function of a real number . So, for now, imaging that is just a scalar value or that is just a real value. It's just a number, rather than a vector. Imagine that we have a cost function 
that's a quadratic function of this real value parameter , so looks like that. Well, how do you minimize a quadratic function? For those of you that know a little bit of calculus, you may know that the way to minimize a function is to take derivatives and to set derivatives equal to zero. So we take the derivative of with respect to the parameter of . You get some formula which I am not going to derive, you set that derivative to zero, and this allows you to solve for the value of that minimizes . That was a simpler case of when was just a real number. In the problem that we are interested in, is no longer just a real number, but instead is this n+1-dimentional parameter vector, and, a cost function is a function of this vector value or 
through 
. And the cost function looks like this, some square cost function on the right. How do you minimize this cost function ? Calculus actually tells us that one way to do so, is to take the partial derivative of , with respect to very parameter of 
in turn, and then, to set all of these to 0. If you do that, you solve the values of , 
up to . Then this will give you that value of the to minimize the cost function . Well if you actually work through the calculus and work through the solution to the parameters , up to , the derivation ends up being somewhat involved. And, what I'm going to do in this video (article), is actually to not go through the derivation, which is kind of long and kind of involved, but what I want to do is just tell you what you need to know in order to implement this process so you can solve for the values of s that corresponds to where the partial derivatives is equal to zero. Or alternatively, or equivalently, the values of s that minimize the cost function . I realize that some of the comments I made may have made more sense only to those of you that are little more familiar with calculus. So, but if you don't know, if you're less familiar with calculus, don't worry about it. I'm just going to tell you what you need to know in order to implement this algorithm and get it to work.
For the example that I want to use as a running example. Let's say that I have 
training examples. In order to implement this normal equation method, what I'm going to do is the following. I'm going to take my dataset, so here are my four training examples. In this case let's assume that, these four examples is all the data I have. What I am going to do is take my dataset and add an extra column that corresponds to my extra feature 
, that is always takes on this value of 1. What I'm going to do is I'm then going to construct a matrix called 
that's a matrix that basically contains all of the features from my training data. So, concretely, here are my features and we're going to take all those numbers and put them into this matrix "", okay? So just copy the data over one column at a time and then I am going to do something similar for 
s. I am going to take the values that I'm trying to predict and construct now vector, like so, and call that a vector . So is going to be a 
-dimensional matrix, and is going to be a 
-dimension vector. Where is the training examples and 
is the number of features, 
, because of this extra feature that I had. Finally if you take your matrix and if you just compute this, and set 
, this would give you the value of that minimizes your cost function. There was a lot that happened on the slides and I work through it using one specific example of one dataset. Let me just write this out in a slightly more general form. And then later on in this video (article), let me explain this equation a little bit more. In case it is not yet entirely clear how to do this.
In a general case, let us say we have training examples, so 
up to 
and features. So, each of the training example 
may looks like a vector like this, that is a dimensional feature vector. The way I'm going to construct the matrix "", this is also called the design matrix is as follows. Each training example gives me a feature vector like this. Let's say, sort of n+1 dimensional vector. The way I'm going to construct my design matrix is only construct the matrix like this. And what I'm going to do is take the first training example, so that is a vector, take its transpose so it ends up being this long flat thing and make 
the first row of my design matrix . Then I am going to take my second training example 
, take the transpose of that and put that as the second row of and so on, down until my last training example. Take the transpose of that, and that's my last row of my matrix . And, so this makes my matrix , an dimensional matrix. As a concrete example, let's say I have only one feature, really, only one feature other than , which is always equal to 1. So if my feature vector is , which are equal to this 1, which is , then some real feature, like maybe the size of the house, then my design matrix , would be equal to this. For the first row, I'm going to basically take this and take its transpose, So I'm going to end up with 1 and then 
. For the second row we are going to end up with 1 and then 
and so on down to 1, 
. And this will be a 
dimensional matrix. So, that's how to construct the matrix . And the vector maybe sometimes I might write an arrow on top to denote that it is a vector, but very often I'll just write this as , either way. The vector is obtained by taking all the labels, all the correct prices of houses in my training set, and just stacking them up into an m-dimensional vector, and that's . Finally, having constructed the matrix and the vector , we then just compute .
I just want to make sure that this equation makes sense to you and you know how to implement it. So, concretely, what is this 
? Well, it is the inverse of the matrix 
. Concretely, if you were to set 
, so 
is a matrix, gives you another matrix, and we call that matrix 
. Then, you know, is just you take this matrix and you invert it, right! This gives, let's say 
. And so that's how you computes this thing. You compute and then you compute this inverse. We haven't yet talked about Octave. We'll do so in the later set of videos, but in the octave programming language, or a similar view, and also the matlab programming language is very similar. The command to compute this quantity, 
, is as follows. In Octave
(X prime) is the notation that you use to denote . And so, this expression that's boxed in red (
), that's computing . pinv is a function for computing the inverse of a matrix, so this (
) computes , and then you multiply that by , and you multiply that by y. So you end up computing that formula which I didn't prove, but it is possible to show mathematically even though I'm not going to do so here. This formula gives you the optimal value of in the sense that if you set equal to this, that's the value of that minimizes the cost function for linear regression. One last detail, in an earlier video, I talked about feature scaling and the idea of getting features to be on similar ranges of values of each other. If you are using this normal equation method then feature scaling isn't actually necessary and is actually okay if, say, any feature 
is between 0 and 1, and some feature 
is between ranges from 0 to 1000 and some feature 
ranges from 0 to 
. And if you are using the normal equation method this is okay and there is no need to do feature scaling. Although of course if you're using gradient descent, then, feature scaling is still important.
Finally, when should you use the gradient descent and when should you use the normal equation method. Here are some of their advantages and disadvantages. Let's say you have training examples and features. One disadvantage of gradient descent is that, you need to choose the learning rate 
. And, often, this means running it few times, with different learning rate and seeing what works best. And so that is sort of extra work and extra hassle. Another disadvantage with gradient descent is it needs many more iterations. So, depending on the details, that could make it slower, although there's more to the story as we'll see in a second. As for the normal equation, you don't need to choose any learning rate , so that, you know, makes it really convenient, makes it simple to implement. You just run it and it usually just works. And you don't need to iterate, so you don't need to plot , check the convergence or take all those extra steps. So far, the balance seems to favor the normal equation. Here are some disadvantages of the normal equation, and some advantages of gradient descent. Gradient descent works pretty well, even when you have a very large number of features. So, even if you have millions of features, you can run gradient descent and it will be reasonably efficient. It will do something reasonable. In contrast, the normal equation, in order to solve for the parameters , we need to solve for this term. We need to compute this term, . This matrix . That's an 
matrix, if you have n features. Because, if you look at the dimensions of and dimension of , you figure out what the dimension of the product is, the matrix is an matrix where n is the number of features, and for almost computed implementations, the cost of inverting the matrix, grows roughly as the cube of the dimension of the matrix. So, computing this inverse costs roughly order 
time. Sometimes, it's slightly faster than but it's close enough for our purposes. So if n, the number of features is very large, then computing this quantity can be slow and the normal equation method can actually be much slower. So if n is large then I might usually use gradient descent because we don't want to pay this order time. But, if n is relatively small, then the normal equation might give you a better way to solve the parameters. What does small and large mean? Well, if n is on the order of a hundred, then inverting a hundred-by-hundred matrix is no problem by modern computing standards. If n is a thousand, I would still use the normal equation method. Inverting a thousand-by-thousand matrix is actually really fast on a modern computer. If n is ten thousand, then I might start to wonder. Inverting a ten thousand-by-ten thousand matrix starts to get kind of slow, and I might then start to maybe lean in the direction of gradient descent, but maybe not quite. n equals ten thousand, you can invert a ten-thousand-by-ten-thousand matrix. But if it gets much bigger than that, then, I would probably use gradient descent. So, if n equals to 
with a million features, then inverting a million-by-million matrix is going to be very expensive, and I would definitely favor gradient descent if you have that many features. So exactly how large set of features has to be before you convert to gradient descent, it's hard to give a strict number. But, for me, it is usually around ten thousand but I might start to consider switching over to gradient descent or maybe some other algorithms that we'll talk about later in this class. To summarize, so long as the number of features is not too large, the normal equation gives us a great alternative method to solve for the parameter . Concretely, so long as the number of features is less than 1000, you know, I would use, I would usually just use the normal equation method rather than gradient descent.
To preview some ideas that we'll talk about later in this course, as we get to the more complex learning algorithm, for example, when we talk about classification algorithm, like a logistic regression algorithm, we'll see that those algorithm... The normal equation method actually don't work for those more sophisticated learning algorithms, and we will have to resort to gradient descent for those algorithm. So, gradient descent is a very useful algorithm to know. Both for linear regression when we have a large number of features and for some of the other algorithms that we'll see in this course, because for them the normal equation method just doesn't apply and doesn't work. But for this specific model of linear regression, the normal equation can give you an alternative that can be much faster than gradient descent. So, depending on the detail of your algorithm, depending on the detail of the problems and how many features you have, both of the algorithms are well worth knowing about.
<end>
