Yihua Zhang PRO

@zinchse

You were not misunderstanding anything. The post was misleading on exactly that point, and I have rewritten it.

@zinchse is right, and the heart of it is that nothing about the opponent has to be known. Write out the log of the trajectory probability:

log P(tau|theta) = log rho_0(s_0) + sum_i [ log P(s_{i+1}|s_i, a_i) + log pi_theta(a_i|s_i) ]

The environment term log P(s_{i+1}|s_i, a_i) is there, and you are right that we do not know it, but it contains no theta. So when we differentiate with respect to theta it goes to zero and drops out, leaving only sum_i grad log pi_theta(a_i|s_i). We never need its value, only the fact that theta does not appear inside it. Whether the opponent replies by a fixed rule, by a coin flip, or by any policy at all changes nothing in that argument. This is what model-free means, and it is why policy gradient methods work in environments that nobody has a model of.

What sent you down this path was my own wording. The note under that derivation said the opponent's moves "are determined by fixed rules", which suggests both that the opponent plays deterministically and that we would have to know the
rules. Neither is required. It now says that the transition absorbs whatever the opponent does, carries no dependence on theta, and therefore vanishes under the gradient, whether the reply is by rule or at random.

The P.S. about an adapting opponent deserves underlining as well. If the opponent changes how it plays in response to us, then P(s_{i+1}|s_i, a_i) does depend on theta, the term no longer differentiates away, and the derivation above stops holding. That is the multi-agent, non-stationary setting, and it is a genuinely different problem rather than a footnote.

Thank you for asking, and thanks to @zinchse for the answer.

Yes, you are right, and so are @sienna223 and @winvswon78 . The formula was wrong and it has been fixed.

It should read

grad J ~= (1/N) sum_i sum_t [ grad log pi(a_it|s_it) * ( sum_{t'=t}^T r(s_it', a_it') ) ]

rather than having the two sums sitting side by side as separate factors. As written it was not even well formed, since t appears as the lower limit of the reward sum while t is bound by the gradient sum, so outside that sum it has no value at all.

@zinchse has the important half of the answer and I want to underline it, because "its reward" can be read two ways. Each action's gradient is multiplied by every reward from that step to the end of the episode, not by the single reward r_t collected at that step. The reward-to-go form is derived by starting from grad log P(tau) * R(tau) for the whole trajectory and then dropping, for each action, the rewards that came before it, since an action cannot influence what already happened. What remains in front of each gradient term is the full remaining return.

The same wrong factoring had propagated to the two formulas that follow, the one in section 2.6 that subtracts the baseline and the one in section 2.7 that replaces the reward-to-go with the advantage. In 2.7 the advantage had also lost its index and read A^pi(s, a) instead of A^pi(s_it, a_it). All three are corrected now.

Thanks for flagging it, and thanks to @zinchse for the explanation in the thread.

Yes, you are right, and thank you for the careful read. Your screenshot is what the derivation actually gives.

Starting from

pi*(y|x) * Z(x) / pi_ref(y|x) = exp( r*(x,y) / beta )

take the log of both sides and multiply by beta:

r*(x,y) = beta * log( pi*(y|x) / pi_ref(y|x) ) + beta * log Z(x)

The Z(x) comes out of the log and becomes the second term. Leaving it inside the first log as well counts it twice, which is exactly what the post did.

It had spread further than that one line. The same wrong expression appears in the section 5.1 summary, in the Bradley-Terry substitution, in the Plackett-Luce substitution, and inside both Plackett-Luce products, including the final multi-response objective. All eight places are fixed now.

One thing worth adding for anyone else reading. The error never reaches the conclusion. In the pairwise case the two beta * log Z(x) terms cancel as soon as you take r*(x, y_w) - r*(x, y_l), and in the ranking case the Z factor appears in the numerator and in every term of the denominator, so it cancels out of each normalized ratio. The DPO loss at the end is therefore the same with or without the extra Z. That is almost certainly why the mistake survived this long, and it is also why it was worth catching, since a reader following the algebra line by line would have been stuck trying to reproduce a step that does not work.

Thanks also to @zinchse for confirming it in this thread.

Thank you for taking the time to write these up. All four are fair, and three of them were outright errors on my part. The post has been corrected. I am sorry for this late reply.

1/ You are right, and my phrasing was simply wrong. The clip applies to the ratio alone, and the interval [1-eps, 1+eps] bounds the ratio, not the ratio times the advantage. I have rewritten that passage, and I also added the point you make
about the min, which the post skipped entirely. The clipping only bites once the policy has already moved in the direction the advantage points, so we stop pushing further on evidence a single trajectory cannot support, while a move in the wrong direction is still corrected in full. That asymmetry is the whole design, and saying "whichever is smaller" without explaining it misses it.

2/ Agreed. I have softened those statements and added an explicit note that the variance reductions are empirical regularities rather than formal guarantees.

3/ This is the one I have left alone, and I would rather explain why than quietly disagree. The readers I had in mind are people who have used PPO or GRPO without ever deriving GAE, and for them the fact that the arithmetic is easy is exactly what makes it worth seeing in full. You are right, though, that it interrupts the flow, since GAE has already been defined and used by that point. If I restructure the post again, moving the expansion to an appendix is what I will do.

4/ Correct, the sign was wrong. It now reads

L_PPO = L_POLICY - c1 * L_VF + c2 * L_ENTROPY

with a note that L_POLICY and the entropy are maximized while the squared error is minimized, and that an implementation calling a minimizer negates the whole expression.

While I was in there I also fixed two things other readers caught. The extra Z(x) in the optimal reward expression that you confirmed for @mabobbi is gone, in all eight places it had propagated to. And @woolpeeker was right that the reward-to-go weight sat outside the sum over timesteps, which had spread to the baseline and advantage formulas as well.

Thanks again. Comments like this are the reason it is worth posting these things in public.

The issues have been fixed. Thanks for letting me know. This is because the math engine in HuggingFace is different from the one we usually use. It uses "\(" and "\)" to wrap in-line equations, rather than "(" and ")". Obviously, I was not aware of that before.