Sign in to continue learning
Advanced Learning Algorithms
33 hours 27 mins
So, without further ado, let's jump into the vectorized implementation of a neural network. We'll look at the code that you have seen in an earlier video, and hopefully matmul, that is, that matrix multiplication calculation, will make more sense. Let's jump in. So you saw previously how you can take the matrix A and compute A transpose times W, resulting in this matrix here, Z. In code, if this is the matrix A, this is a NumPy array with the elements corresponding to what I wrote on top, then A transpose, which I'm going to write as AT, is going to be this matrix here, with, again, the columns of A now laid out in rows instead. And, by the way, instead of setting up AT this way, another way to compute AT in NumPy would be to write AT equals A dot T. That's the transpose function that takes the columns of the matrix and lays them on the side. In code, here's how you would initialize the matrix W as another 2D NumPy array, and then to compute Z equals A transpose times W, you would write Z equals NP dot matmul AT comma W. And that will compute this matrix Z over here, giving you this result down here. And, by the way, if you read others' code, sometimes you see Z equals AT and then the at symbol W. This is an alternative way of calling the matmul function, although I find using NP dot matmul to be clearer, and so the code you see in this class will just use the matmul function like this, rather than this at symbol. So, let's look at what a vectorized implementation of forward prop looks like. I'm going to set A transpose to be equal to the input feature values 217. So these are just the usual input feature values, 200 degrees, roasting coffee for 17 minutes. So this is a 1 by 2 matrix, and I'm going to take the parameters W1, W2, and W3, and stack them in columns like this to form this matrix capital W. And the values B1, B2, B3, I'm going to put into a 1 by 3 matrix that is this matrix B as follows. Then it turns out that if you were to compute Z equals A transpose W plus B, that will result in these three numbers, and that's computed by taking the input feature values and multiplying that by the first column and then adding B to get 165. Taking these feature values, dot producting with the second column, that is the weights W2, and adding B2 to get negative 531, and these feature values dot product with the weights W3 plus B3 to get 900. Feel free to pause the video if you wish to double check these calculations, but this gives you the values of Z11, Z12, and Z13. And then finally, if the function G applies the sigmoid function to these three numbers element-wise, that is applies the sigmoid function to 165, to negative 531, and to 900, then you end up with A equals G of this matrix Z ends up being 101. And it's 101 because sigmoid of 165 is so close to 1 that up to numerical round off, it's basically 1, and these are basically 0 and 1. Let's look at how you implement this in code. A transpose is equal to this, is this 1 by 2 array of 217. The matrix W is this 2 by 3 matrix, and B is this 1 by 3 matrix. And so the way you can implement forward prop in the layer is dense input A transpose WB is equal to Z equals matmul A transpose times W plus B. So that just implements this line of code. And then A out, that is the output of this layer, is equal to G, the activation function applied element-wise to this matrix Z. And you return A out, and that gives you this value. In case you're comparing this slide with the slide a few videos back, there was just one little difference, which was by convention, the way this is implemented in TensorFlow, and rather than calling this variable AT, we were calling it AIN, which is why this too is a correct implementation of the code. And there is a convention in TensorFlow that individual examples are actually laid out in rows in the matrix X, rather than in the matrix X transpose, which is why the code implementation actually looks like this in TensorFlow. But this explains why, with just a few lines of code, you can implement forward prop in the neural network, and moreover, get a huge bonus because modern computers are very good at implementing matrix multiplication such as matmul efficiently. That's the last video of this week. Thanks for sticking with me all the way through the end of these optional videos. For the rest of this week, I hope you also take a look at the quizzes and the practice labs, and also the optional labs to exercise this material even more deeply. You now know how to do inference and forward prop in the neural network, which I think is really cool, so congratulations. After you have gone through the quizzes and the labs, please also come back, and in the next week, we'll look at how to actually train the neural network. So I look forward to seeing you next week.