I know, this article is quite lengthy but I hopefully think that it could help some of you! On the other hand, since we are selecting the action $a_i$, $N(s, a_i)$ will be incremented by $1$. Since we have run $1600$ simulations, it means that:$N(s, a_1) = 200 \quad\quad N(s, a_2) = 750 \quad\quad N(s, a_3) = 650$Now, we can finally understand the $\sim$ symbol. In March 2016, Deepmind’s AlphaGo beat 18 times world champion Go player Lee Sedol 4–1 in a series watched by over 200 million people. actually incorporates all the information about how good our action $a$ is. So, we want to know when $U_{bad}(s, a_i)$ starts to be smaller than $U_{best}(s, a_i)$. To do that we will suppose that we have finished to run the $1600$ Then: $W(s, a_i)/N(s, a_i) = (-0.2 + 0.3 - 0.2)/3 \approx -0.033$Since an action that has been selected $1200$ times will add approximately $0.033$ to the Now, We’re going to use absurd reasoning. The second simulation is depicted on Since the action that maximizes $U(s, a)$ is the action $13$, we will expand the nodes from this action. The first part of the That’s it. We pass the current board game as well as the $15$
I Deep Learning. The AlphaGo Zero AI relies on 2 main components. So all the bad actions $a_i$ will be selected as long as $U_{bad}(s, a_i) > U_{best}(s, a_i)$. Before I try to convince you that their criterion is good, I’d like to make you aware that, in real life you would The penalizing term can be broken down into 2 parts:Let’s understand this formula. For this, we will assume that something is Now, if we assume that, at the end of the $1600$ simulations, $U(s, a_2) \gg U(s, a_1)$, then, because the The only option for $U(s, a_2) \not{\gg} U(s, a_1)$ is that $P(s, a_2) \ll P(s, a_1)$. This implementation is largely inspired from the non official minigo implementation. Well, the best you can do is to add all the numbers and you’ll get $5 + 8 + 12 + 25 + 3 = 53$ which is very far from return the best next move to choose at each step of the game. When we will run the $1600$ $v(s, a_i)=1$ if we win and $v(s, a_i) = -1$ otherwise.Now, the previous argument doesn’t hold anymore since $v(s, a_i)$ is not any random variable output by a untrained Neural Network. According to the previous argument we can just solve:Hence, once we have selected the bad actions $19$ times each, we are Hence, we will associate a higher probability to the actions that have been selected the most during the To train our Neural Network we will use the data generated during the self-play games.
Shortly after they have released their research paper explaining how work their algorithm, they have released another paper called That’s it! Figure I.3: The input given to the Neural Network consists of the $15$ previous states of the game + the current state of the game and the color which encodes which player is to play. I see that you’re asking yourself the right questions!
There is nothing fancy. To do that, the Neural Network will browse its own architecture in the opposite direction.