Sign in to continue learning
Mathematics for Machine Learning and Data Science
Beginner
Topics
Deep Learning
Mathematical Foundations
Supervised Learning
In the example from last video, you had a 3x3 matrix which had 3 different eigenvalues and 3 different eigenvectors, one for each of the eigenvalues. Could it be that all 3x3 matrices always have 3 eigenvectors? Well, it turns out that that's not always the case. Let's look into some examples. Consider this 3x3 matrix A with the values shown here. Now find the characteristic polynomial as the determinant of A minus lambda I. Using the formula for the determinant for 3x3 matrices, we get 2 minus lambda squared times minus 4 lambda. All the remaining terms are 0 due to all the zeros in the original matrix. Now to find the eigenvalues, you need to find the zeros of this polynomial, which in this case is super simple. This gives the eigenvalues 4, 2, and 2. Notice that the number 2 is repeated twice. Now watch what happens when we find the eigenvectors associated with them. Let's start with the eigenvalue 4. The eigenvector needs to satisfy that A times the vector x1, x2, x3 equals the vector 4 times x1, 4 times x2, 4 times x3. The product between the matrix and the vector can be expanded as the vector with entries 2x1 minus x1 plus 4x2 minus 0.5x3 and 2x3. And we want this to be equal to the vector 4x1, 4x2, 4x3, which leads to this system of equations. This one can be written as negative 2x1 equals 0, negative x1 minus 0.5x3 equals 0, and negative 2x3 is equal to 0. Notice that all I did here was moving everything to the left of the equal sign. Now we can simplify a little bit and get that the first one is equivalent to x1 equals 0, the third one is equivalent to x3 equals 0, and now since x2 doesn't appear in any of these equations at all, then the second equation is 0 for free, and x2 can be any number in order to satisfy these equations. So to keep it simple, we can set x2 to be equal to 1, and you get the eigenvector 0, 1, 0 associated to the eigenvalue 4. Now let's repeat for the eigenvalue 2. In this case, a dot product with the vector x1, x2, x3 should be equal to the vector 2x1, 2x2, 2x3, and the dot product can be expanded with the same expression as before. We again get a system of equations, 2x1 equals 2x1, negative x1 plus 4x2 minus 0.5x3 is 2x2, and 2x3 equals 2x3. When we manipulate this by taking everything to the left of the equal sign, we get that the first equation is quite trivial, it's 0 equals 0. The second one is negative 1 plus 2x2 minus 0.5x3 equals 0, and the third one is again trivial, it's 0 equals 0. You can rewrite the second equation as x1 equals 2 times x2 minus 0.5 times x3. This equation has infinitely many solutions depending on what values of x2 and x3 you choose. So let's say that you choose x2 equals 1 and x3 equals 0. That means x1 has to be 2 for it to be an eigenvector, given the vector 2, 1, 0. But you can also choose x2 to be 1 and x3 to be 2, which gives the eigenvector 1, 1, 2. These two vectors point in different directions, so they are actually two different eigenvectors. Remember that you only care about the direction of the eigenvector, since any scaled version of it will still be an eigenvector to the same eigenvalue. It is worth mentioning that you could have found different pairs of eigenvectors depending on the values of x2 and x3, but the important thing is that you can always find two distinct directions for the eigenvalue 2. To sum up, this matrix A has the following pairs of eigenvalues and eigenvectors. The first eigenvalue lambda 1, which is 4, has as an eigenvector 0, 1, 0. The second eigenvalue lambda 2, which is 2, has as an eigenvector the vector 2, 1, 0. And finally, the third eigenvalue, lambda 3, also equals 2 and has as an eigenvector 1, 1, 2. In this case, you were able to find three distinct eigenvectors, even though you had a repeated eigenvalue. Let's look into one more example where this doesn't happen. Let's change just one value in the matrix and see what happens. So the characteristic polynomial stays the same as before. And once again, we're going to have the values 4, 2, and again 2 as eigenvalues. Let's repeat the process from before to find the eigenvectors and start with the eigenvalue 4. The only equation that changes here is the last one, where now you have 4 times x2 plus 2 times x3. And that gives us the system of equations, where the first two remain the same, being negative 2 times x1 equals 0 and negative 1 minus 0.5 x3 equals 0. And again, let's name them equations 1, 2, and 3, respectively. From equation 1, you get that x1 has to be 0. And combining equations 3 and 1, you get that x3 has to be 0. This leaves again x2 to take on any value. Just as with the previous matrix, you can choose the vector 0, 1, 0 as the eigenvector. And it's the same one for the previous matrix. For the eigenvalue 2, the same thing happens. When you solve this equation, you get that 2x1 is equal to 2x1, that negative x1 plus 4x2 minus 0.5x3 is 2x2, and that 4x1 plus 2x3 is 2x3. Which gives rise to the following system of equations, that can be simplified as 0 equals 0, as negative x1 plus 2x2 minus 0.5x3 equals 0, and finally, 4 times x1 equals 0. This imposes the restriction that x1 can only be 0. Now from equations 2 and 3, you get that x3 must equal 4 times x2. Now you only have one degree of freedom, since x1 is always 0, and once you fix x2, the value of x3 gets fixed as well. For example, you can consider x1 equals 0 and x2 equals 1, which makes x3 equal 4, given the eigenvector 0, 1, 4. What would happen if, for example, you choose x2 to be 1 half? Then x3 must be 2, given the eigenvector 0, 1 half, 2. However, these two vectors lie on the same line. One is simply a scaling from the other, which means that they are actually the same eigenvector. You can test it. You can't find any vector which satisfies equations 1, 2, and 3 that spans a different line. This means that the number 2 is 2 times an eigenvalue, but you can only find one eigenvector associated with it. Then the eigenvectors are of the form 0, k, 4, k, but there's only one direction which can be paired with eigenvalue 2, even though the number 2 appears twice as an eigenvalue. So, recap. For the matrix entries 2, 0, 0, negative 1, 4, negative 0.5, 4, 0, 2, you have an eigenvalue of 4 associated to the eigenvector 0, 1, 0, then an eigenvalue of 2 associated to the eigenvector 0, 1, 4, and again an eigenvalue of 2 which doesn't have an associated eigenvector. That means that you can't create an eigenbasis for the three-dimensional space because you're lacking one vector to span the whole space. As a summary, if you have a 2 by 2 matrix with eigenvalues lambda 1 and lambda 2, if the two eigenvalues are different, then you always get two distinct eigenvectors. However, if the eigenvalues are the same, then you can have either one or two eigenvectors. If you have a 3 by 3 matrix with eigenvalues lambda 1, lambda 2, and lambda 3, then you get some more options to explore. If all the three eigenvalues are different, then you can always find three distinct eigenvectors. If you have one eigenvalue repeated twice with the other being different, then you can have either two or three eigenvectors. These are the two examples you just explored. However, if you have the same eigenvalue repeated three times, you can have any number of eigenvectors between 1 and 3. In this week's programming assignments, you will get to explore many more interesting examples of matrices with different properties and number of eigenvectors.
