Add Deepseek AI's Janus model

[ArthurZucker]

Model description

🚀 Janus-Series: Unified Multimodal Understanding and Generation Models

• https://github.com/deepseek-ai/Janus
• https://huggingface.co/deepseek-ai/Janus-Pro-1B

Open source status

• The model implementation is available
• The model weights are available

Provide useful links for the implementation

No response

[yaswanth19]

@ArthurZucker I will work on integrating this Model 🤩🤗

[ArthurZucker]

Awesome! 🚀

[hsilva664]

@yaswanth19 I am also interested in working on this. If you have something already it would be nice if you could open a PR. I can help code the model or review/test it.

[geetu040]

Hi @ArthurZucker and @zucchini-nlp, do you plan to add "janus-flow" in transformers as well?

"janus" and "janus-pro" use the same architecture, but "janus-flow" is quite different and cannot be used under the same JanusModel that is being worked on.

Below, I've added some links and info for the 3 models in "janus-series".

Janus

• Weights: deepseek-ai/Janus-1.3B - Downloads 63,883
• Code: deepseek-ai/Janus
• Research Paper: Janus - Published: 17 Oct 2024 - Citations: 21

Janus-Flow

• Weights: deepseek-ai/JanusFlow-1.3B - Downloads: 9,874
• Code: deepseek-ai/Janus/janusflow
• Research Paper: JanusFlow - Published: 12 Nov 2024 - Citations: 2

Janus-Pro

• Weights: deepseek-ai/Janus-Pro-1B - Downloads: 64,859
• Weights: deepseek-ai/Janus-Pro-7B - Downloads: 223,437
• Code: deepseek-ai/Janus (same as Janus)
• Research Paper: Janus-Pro - Published: 29 Jan 2025 - Citations: 0

[zucchini-nlp]

Janus-flow is a flow based model and we don't have any flow based models in transformers currently. I don't think it is worth adding it in the moment

[geetu040]

Janus-flow is a flow based model and we don't have any flow based models in transformers currently. I don't think it is worth adding it in the moment

@zucchini-nlp And what about the Deepseek-VL series? they are different from Janus-series

• Deepseek-VL: paper - code - weights
• Deepseek-VL2: paper - code - weights

[zucchini-nlp]

@geetu040 yep, a very much needed model. I am not 100% sure if anyone is working on it, we have a PR for deepseek-v3 (#35926) and for the v2 no-one is working currently as per last comment (#35317).

CMIIW, Deepseek-VL2 uses v2 as backbone, so we'll have to first add the LM v2 model. If there isn't much difference between deepseek-v2 and v3, I think we can add support for v2 easily with modular after the above PR is merged. So feel free to comment under the above issue and work on v2 support :)

[geetu040]

Thanks, I realised the constraints. I will start work on these.

[ArthurZucker]

Created #36110 and #36111 for tracking