【FR】Bug: Multiple FlagGems operators degrade accuracy on nvidia (upstage/SOLAR-10.7B-Instruct-v1.0)

[buzhengjing]

Bug Report: 【FR】Multiple FlagGems operators degrade accuracy, target not reachable on nvidia (upstage/SOLAR-10.7B-Instruct-v1.0)

Description

On model upstage/SOLAR-10.7B-Instruct-v1.0 under FlagOS, accuracy cannot reach the target.
Key observation: disabling more FlagOS operators (routing them to reference) increases the
GPQA score (30.81% -> 32.32%), indicating these FlagOS operator implementations hurt accuracy.
However, even after fixing 17 operators to reference, V3 = 31.82%, still below the NV baseline
34.0% (relative drop 6.41% > 5%), so accuracy_ok=false.

Note: operator control in plugin mode is via vllm-plugin-FL nvidia.yaml op_backends, not gems.txt.

Environment

Item	Value
Hardware	NVIDIA H20-3e
PyTorch	2.11.0+cu130
vLLM	0.20.2
FlagGems	5.0.2
vllm-plugin-FL	0.2.0rc2.post1

Accuracy Data (per round)

Round	Newly disabled ops (cumulative)	GPQA
R1	full, ones, zeros	30.81%
R2	+ add, cos, div, reciprocal, sin, sub	31.31%
R3	+ argmax, softmax	31.82%
R4	+ arange, exponential, lt, rand_like, randn, where	32.32%

• NV baseline: 34.0%, threshold 5%
• V3 final: 31.82% (rel_drop 6.41%), NOT passed

Affected Operators (17 fixed-disabled)

add, arange, argmax, cos, div, exponential, full, lt, ones, rand_like, randn,
reciprocal, sin, softmax, sub, where, zeros
(disabling them raises the score -> all negatively affect accuracy)

Possible Directions

• Prioritize precision review of high-frequency ops (add / cos / div / softmax / reciprocal)
• Investigate why more disabling yields higher scores: suspected cumulative numerical error
  across multiple FlagOS operators