Sign in to continue learning
Advanced Learning Algorithms
34h14m
In order to build a neural network that can carry out multiclass classification, we're going to take the softmax regression model and put it into essentially the output layer of a neural network. Let's take a look at how to do that. Previously, when we were doing handwritten digit recognition with just two classes, we used a neural network with this architecture. If you now want to do handwritten digit classification with 10 classes, all the digits from 0 to 9, then we're going to change this neural network to have 10 output units, like so. And this new output layer will be a softmax output layer. So sometimes we'll say this neural network has a softmax output or that this output layer is a softmax layer. And the way forward propagation works in this neural network is, given an input x, a1 gets computed exactly the same as before, and then a2, deactivations for the second hidden layer, also get computed exactly the same as before. And we now have to compute deactivations for this output layer, that is, a3. This is how it works. If you have 10 output classes, we will compute z1, z2, through z10 using these expressions. So this is actually very similar to what we had previously for the formula you used to compute z. z1 is w1 dot product with a2, deactivations from the previous layer, plus v1, and so on for z1 through z10. Then a1 is equal to e to the z1 divided by e to the z1 plus dot dot dot plus up to e to the z10, and that's our estimate of the chance that y is equal to 1. And similarly for a2, and similarly all the way up to a10. And so this gives you your estimates of the chance of y being equal to 1, 2, and so on up through the 10th possible label for y. And just for completeness, if you want to indicate that these are the quantities associated with layer 3, technically I should add these superscript 3s there. It does make the notation a little bit more cluttered, but this makes explicit that this is, for example, the z31 value, and this is the parameters associated with the first unit of layer 3 of this neural network. And with this, your softmax output layer now gives you estimates of the chance of y being any of these 10 possible output labels. I do want to mention that the softmax layer, or sometimes also called the softmax activation function, it is a little bit unusual in one respect compared to the other activation functions we've seen so far, like sigmoid, ReLU, and linear, which is that when we're looking at sigmoid or ReLU or linear activation functions, a1 was a function of z1, and a2 was a function of z2 and only z2. In other words, to obtain the activation values, we could apply the activation function g, be it sigmoid or ReLU or something else, element-wise, to z1 and z2 and so on to get a1 and a2 and a3 and a4. But with the softmax activation function, notice that a1 is a function of z1 and z2 and z3 all the way up to z10. So each of these activation values depends on all of the values of z, and this is a property that's a bit unique to the softmax output or the softmax activation function. Or stated differently, if you want to compute a1 through a10, that is a function of z1 all the way up to z10 simultaneously, and this is unlike the other activation functions we've seen so far. Finally, let's look at how you would implement this in TensorFlow. If you want to implement the neural network that I've shown here on this slide, this is the code to do so. Similar as before, there are three steps to specifying and training a model. The first step is to tell TensorFlow to sequentially string together three layers. First layer is this 25 units of a ReLU activation function. Second layer, 15 units of a ReLU activation function. And then the third layer, because there are now 10 output units, you want to output a1 through a10, so there are now 10 output units, and we'll tell TensorFlow to use the softmax activation function. And the cost function that you saw in the last video, TensorFlow calls that the sparse categorical cross-entropy function. So I know this name is a bit of a mouthful, whereas for logistic regression, we had the binary cross-entropy function. Here, we're using the sparse categorical cross-entropy function. And what sparse categorical refers to is that you're still classifying y into categories, so it's categorical, it takes on values from 1 to 10, and sparse refers to that y can only take on one of these 10 values. So each image is either 0 or 1 or 2 or so on up to 9, you're not going to see a picture that is simultaneously the number 2 and the number 7. So sparse refers to that each digit is only one of these categories. So that's why the loss function that you saw in the last video is called in TensorFlow, the sparse categorical cross-entropy loss function. And then the code for training the model is just the same as before. And if you use this code, you can train a neural network on a multiclass classification problem. Just one important note. If you use this code exactly as I've written here, it will work, but don't actually use this code because it turns out that in TensorFlow, there's a better version of the code that makes TensorFlow work better. So even though the code shown in this slide works, don't use this code the way I've written it here, because in a later video this week, you'll see a different version. There's actually a recommended version of implementing this that will work better. But we'll take a look at that in a later video. So now you know how to train a neural network with a softmax output layer with one caveat. There's a different version of the code that will make TensorFlow able to compute these probabilities much more accurately. Let's take a look at that in the next video, which will also show you the actual code that I recommend you use if you're training a softmax neural network. Let's go on to the next video.