Sign in to continue learning
Unsupervised Learning, Recommenders, Reinforcement Learning
27h16m
The Lunar Lander lets you land a simulated vehicle on the moon. It's like a fun little video game that's been used by a lot of reinforcement learning researchers. Let's take a look at what it is. In this application, you're in command of a lunar lander that is rapidly approaching the surface of the moon, and your job is to fire thrusters at the appropriate times to land it safely on a landing pad. To give you a sense of what it looks like, this is the lunar lander landing successfully, and it's firing thrusters downward and to the left and right to position itself to land between these two yellow flags. Or if the reinforcement learning algorithms policy does not do well, then this is what it might look like where the lander unfortunately has crashed on the surface of the moon. In this application, you have four possible actions. On every time step, you could either do nothing, in which case the forces of inertia and gravity pull you toward the surface of the moon, or you can fire a left thruster. When you see a little red dot come out on the left, that's firing the left thruster, they'll tend to push the lunar lander to the right, or you can fire the main engine that's thrusting down the bottom here, or you can fire the right thruster, and that's firing the right thruster which will push you to the left. And your job is to keep on picking actions over time so as to land the lunar lander safely between these two flags here on the landing pad. In order to give the actions a shorter name, I'm sometimes going to call the actions nothing, meaning do nothing, or left, meaning fire the left thruster, or main, meaning fire the main engine downward or right. So I'm going to call the actions nothing, left, main, and right for short later in this video. How about the state space of this MTP? The states are its position, x and y, so how far to the left or right and how high up is it, as well as velocity, x dot, y dot. How fast is it moving in the horizontal and vertical directions? And then also its angle, so how far is the lunar lander tilted to the left or tilted to the right? Its angular velocity, theta dot. And then finally, because a small difference in positioning makes a big difference in whether or not it's landed, we're going to have two other variables in the state vector, which we'll call L and R, which corresponds to whether the left leg is grounded, meaning whether or not the left leg is sitting on the ground, as well as R, which corresponds to whether or not the right leg is sitting on the ground. So whereas x, y, x dot, y dot, theta, theta dot are numbers, L and R will be binary valued and can take on only values 0 or 1, depending on whether the left and right legs are touching the ground. Finally, here's the reward function for the lunar lander. If it manages to get to the landing pad, then it receives a reward between 100 and 140, depending on how well it's flown and gotten to the center of the landing pad. We also give it an additional reward for moving toward or away from the pad. So if it moves closer to the pad, it receives a positive reward. If it moves away and drifts away, it receives a negative reward. If it crashes, it gets a large negative 100 reward. If it achieves a soft landing, that is a landing that's not a crash, it gets a plus 100 reward. For each leg, the left leg or the right leg that it gets grounded, it receives a plus 10 reward. And finally, to encourage it not to waste too much fuel and fire thrusters unnecessarily, every time it fires the main engine, we give it a negative 0.3 reward. And every time it fires the left or the right side thrusters, we give it a negative 0.03 reward. Notice that this is a moderately complex reward function. The designers of the lunar lander application actually put some thought into exactly what behavior you want and codified it in the reward function to incentivize more of the behaviors you want and fewer of the behaviors, like crashing, that you don't want. You find when you're building your own reinforcement learning application, it usually takes some thought to specify exactly what you want or don't want and to codify that in the reward function. But specifying the reward function should still turn out to be much easier than specifying the exact right action to take from every single state, which is much harder for this and many other reinforcement learning applications. So, the lunar lander problem is as follows. Our goal is to learn a policy pi that, when given a state s, as written here, takes an action a equals pi of s so as to maximize the return, the sum of discounted rewards. And usually for the lunar lander, we'll use a fairly large value for gamma. We'll use a value of gamma that's equal to 0.985, so pretty close to 1. And if you can learn a policy pi that does this, then you successfully land this lunar lander. Exciting application, and we're now finally ready to develop a learning algorithm which will turn out to use deep learning on neural networks to come up with a policy to land the lunar lander. Let's go on to the next video where we'll start to learn about deep reinforcement learning.