Fix misc. expiring logit/sparam bias bugs

[dungquixote42]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

Previous PRs: #1731, #1770

This PR addresses an issue where context rewinds (speculative decoding rollbacks or string bans) would desynchronize the Expiring Logit Bias (ELB) and Expiring Sampling Parameter Bias (EPB) state machines.

Also fixes various miscellaneous bugs + code QC.

Tests

Below are representative tests from the previous PRs, retested to see no regression.

\n\n                      # paragraph 1 end
\n\n                      # paragraph 2 end
("In conclusion," : 999)  # paragraph 3 begins with "In conclusion,"
.                         # paragraph 3 sentence 1 end
("<channel|>" : 999)      # paragraph 3 sentence 2 begins with "<channel|>" (end-of-thinking tag)

(-1: temp ~ 0.4, top_p ~ 0.05, xtc_probability ~ -0.2)  # paragraph 1
\n\n                                                    # paragraph 1 end
(-1: temp ~ -0.4, top_p ~ -0.05, min_p ~ -0.05)         # paragraph 2
\n\n                                                    # paragraph 2 end

\n\n
(-1: temp ~ 0.2)
\n\n
(-1: temp ~ 0.1)
\n\n
(-1: temp ~ -0.1)
\n\n
(-1: temp ~ -0.3)
("In conclusion," : 999)
\n\n
(-1: temp ~ -0.1)
\n\n
(-1: temp ~ 0.1)
\n\n
(-1: temp ~ 0.3)
\n\n
(-1: temp ~ 0.1)
"<channel|>" >>  # jump here if <channel|> is generated

(("`" : temp ~ -0.4))  # opening backtick decreases temp     ... (1)
(("\"" : temp ~ 0.3))  # opening double quote increases temp ... (2)
\n\n                   # paragraph 1 end
\n\n                   # paragraph 2 end, (1) and (2) was added
(())                   # paragraph 3 + clear persistent entries
\n\n                   # paragraph 3 end, (1) and (2) was not added

New Test

This is a loopback test for testing compatibility with the rewind mechanism.
With the string "The quick brown fox jumps over the lazy dog" is banned, the server is stuck repeatedly injecting the string and rewinding it.
The rewinding part does not print the sparam bias, but the reapplication shows it was correctly restored.

(("brown" : temp ~ 0.3))
(("lazy" : xtc_probability ~ -0.2))

\n\n
("The quick brown fox jumps over the lazy dog" : 999)

[dungquixote42]

@Ph0rk0z did you get around to test without bans? Also, do you see the bugs in the first response, or in the second and later?

[Ph0rk0z]

It still happens kind of randomly. Sometimes the first reply will be correct and reasoning ends, other times even that will eat the whole 2048 token budget.

I also thought that maybe XTC is interacting with it too, due to the sporadic nature? I mean if we can't use string bans, xtc or any of that this will end up kind of pointless. Ideally it should control reasoning where the string bans don't really fire and I'm loading it from a file to bypass API bugs.

[dungquixote42]

It looks like the rewind mechanism does not fully clear speculated tokens. ELB seems fine otherwise? Maybe.

[dungquixote42]

@Ph0rk0z I have been testing with XTC and string ban enabled. Simple cases, but things look good on my end.

[Ph0rk0z]

I will have to give it a go.

[dungquixote42]

@Ph0rk0z just fixed a major bug.