Implementation of advantage function

[gauss-clb]

actor_output = self.actor(sequences, attention_mask)
action_log_probs = calc_action_log_probs(actor_output, sequences, num_actions)
base_model_output = self.initial_model(sequences, attention_mask)
base_action_log_probs = calc_action_log_probs(base_model_output, sequences, num_actions)
value = self.critic(sequences, action_mask, attention_mask)
r = self.reward_model(sequences, attention_mask)
reward = compute_reward(r, self.kl_coef, action_log_probs, base_action_log_probs, action_mask=action_mask)

advantage = reward - value

https://github.com/hpcaitech/ColossalAI/blob/main/applications/Chat/coati/experience_maker/naive.py#L52

Why value only uses prompt part, https://github.com/hpcaitech/ColossalAI/blob/main/applications/Chat/coati/models/base/critic.py#L49, but r uses prompt+response?
Why reward=r-self.kl_coef*kl_divergence(action_log_probs, base_action_log_probs), is there any theory to support it?