[qgallouedec]

@SunMarc suggested in huggingface/accelerate#4177 that this check belongs here rather than in accelerate

It reads layer_types and sliding_window, which are transformers concepts, and there is already _supports_sdpa / _supports_flex_attn for this kind of capability flag.

CP can only express full causal attention. A layer with a stricter mask (sliding-window, chunked) gets that mask dropped and silently trains as full causal, and a layer carrying a recurrent state along the sequence (linear attention) never has that state exchanged between ranks.

huggingface/accelerate#4177 currently pattern-matches the config to detect this. With this property it becomes if not model.supports_context_parallel: raise.

_supports_context_parallel = False also lets a model rule it out whatever its config, which the config check cannot express (gpt-oss, where attention sinks need a custom kernel).

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[ArthurZucker]

LGTM, but this belongs in the DistributedMixin? (or not? TrainingMixing would be a better name?)

[qgallouedec]

Kept it in PreTrainedModel on purpose: it's a capability flag, and those all live there today (_supports_sdpa, _supports_flash_attn, supports_tp_plan, etc), next to which _supports_context_parallel sits.
DistributedMixin holds the plans and the orchestration and has no supports_* at all. If you'd rather the whole supports_* set move into the mixin (or a renamed TrainingMixin), happy to do that as its own PR ☺️

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 33431074964:1
Result: failure | Jobs: 16 | Tests: 184,774 | Failures: 0 | Duration: 14h 44m