[generate] torch.distributed-compatible DynamicCache

[gante]

What does this PR do?

DynamicCache was not compatible with torch.distributed before, and #36212 exposed the issue. See the discussion starting here for more details. The added code contains comments explaining what's going on.

Fixes Trainer + DP when use_cache=True, such as in prefix tuning training runs.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[gante]

@S1ro1 it should be okay now 👌 torch.distributed is clever and builds the tensors themselves correctly, all we need is to place them in the right variables

[S1ro1]

@gante Comments look good to me, maybe small nit would be writing down the shapes? i.e. (num_layers, kv, (...)), else looks good to me, test as well.

[BenjaminBossan]

Thanks for fixing this, LGTM.

[gante]

(merging to unblock PEFT)