SD_api_pictures: using the loaded model to translate the prompt in english.

[cmhamiche]

Getting SD_api_pictures to do a translation from user's language to a prompt in english using the currently loaded model before the call to Stable Diffusion, so everything's running locally.

I'm using Open Assistant llama's model (oasst-llama13b-4bit-128g) and it's very capable at languages like french or german and pretty decent at translations.

I'd love to be able to use the Stable Diffusion API pictures for TextGen extension but unfortunately if I'm having a conversation in french it will send a prompt to Stable Diffusion... in french. Results are off to say the least.

An option for SD_api_pictures to translate the prompt in the background before calling Stable Diffusion would make everything simpler and would also make Stable Diffusion multilingual, sort of.

Here's an other extensions I'm aware of:

For Stable-Diffusion-webui:

Stable-Diffusion-Webui-Prompt-Translator:
It's either using an online service (or an AI translation engine that wou'd have to setup called MarianM which apparently isn't that good anyway). Clicking the "Translate" button in stable diffusion's interface is required, not ideal.
https://github.com/butaixianran/Stable-Diffusion-Webui-Prompt-Translator

For text-generation-webui:
google_translate extension: It's also using an online service.
https://github.com/oobabooga/text-generation-webui/blob/main/extensions/google_translate/script.py

[janvarev]

I'm waiting for my PR to be merged: #1321
PR allow to use in google_translate extension offline server LibreTranslate, so it'll be easier to integrate it with sd_api_extension.

[cmhamiche]

Thank you, I'd probably use it if there's no alternatives but I'd really appreciate a simple offline solution like using the model if it is good enough for translation.

Open Assistant llama's model is actually great for translation but I think that even llama 7B 4bit could give decent results so why not using the same model, just translate the last inference before sending it to SD api.

Edit: I didn't realize that it could be used offline, I'll give it a try.

[janvarev]

Edit: I didn't realize that it could be used offline, I'll give it a try.

Also, it significantly faster than GPT model, and can be succesfully run on CPU. In my installation it requires near 0.5-1 seconds to translate.

[cmhamiche]

I did some comparisons. I've tried it locally and while LibreTranslate is extremely fast and great at translating, there's a lot of context lost in translation (when it's not completely wrong) and the responses are slower on my system than just infering when I'm using the extension. It really hinders the model's capabilities at understanding the request and when it understands, it tend to give short anwsers (that's with a 13B, 4bit model).

There's also an issue where the pic comes back with a very verbose prompt that has not been translated at all. I'll try to get the details of that issue tomorrow and come up with exemples if necessary.

Still, that extension is a great addition for people who can't get finetuned models for their languages and it's really trivial to set up. LibreTranslate is fairly impressive.

[janvarev]

Thanks for reply!
As I understand, you make some requests to llama 7B 4bit to make translation of text. Could you provide some good examples of them? I wanna try such a way too.

[janvarev]

@cmhamiche Please, try my updated google_translation ext. I've added option to translate using local already loaded Alpaca model - as you say, it works better in some cases.
#1321 (comment)

[cmhamiche]

Man, that is incredible.

I'm currently using oasst-llama13b-4bit-128g which is pretty good at translating but requires 10 gb ram if I remember correctly.
There's some language stats here, it is multilingual. https://open-assistant.io/fr/stats
But I used to "struggle" with alpaca 7B 4bits just a few days ago on a 1060 6gb and it was doing translation fine in french, so I'll also test that for the fellow low specs users. Thank you.

Let me switch back to ubuntu, I'll try real quick and report back.

[cmhamiche]

Okay it works great with alpaca:

python server.py --model 4bit_alpaca-7b-native-4bit --wbits 4 --groupsize 128 --sdp-attention --auto-launch --chat --listen --no-stream --listen-port 8008 --extension google_translate sd_api_pictures
Gradio HTTP request redirected to localhost :)
bin /home/linux/miniconda3/envs/textgen/lib/python3.10/site-packages/bitsandbytes/libbitsandbytes_cuda117.so
Loading settings from settings.json...
Loading 4bit_alpaca-7b-native-4bit...
Found the following quantized model: models/4bit_alpaca-7b-native-4bit/alpaca7b-4bit.pt
Loading model ...
Done.
Replaced attention with sdp_attention
Loaded the model in 12.92 seconds.
Loading the extension "google_translate"... Ok.
Loading the extension "sd_api_pictures"... Ok.
Loading the extension "gallery"... Ok.
Running on local URL:  http://0.0.0.0:8008

To create a public link, set `share=True` in `launch()`.
LocalTranslation prompt: Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Translate phrase from French to English
### Input:
Quel est la recette de la pizza ?
### Response:
Output generated in 3.07 seconds (3.26 tokens/s, 10 tokens, context 57, seed 1845646554)
LocalTranslation answer: What is the recipe for pizza?
LocalTranslation prompt: Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Translate phrase from English to French
### Input:
The classic recipe for pizza includes making a thin crust by mixing flour, yeast and salt in warm water; then adding olive oil, water and sugar to the dough before rolling it out and transferring it to a baking sheet. Toppings such as mozzarella cheese, tomatoes, onions, pepperoni or sausage can be added before baking at 375 degrees Fahrenheit for about 20 minutes.
### Response:
Output generated in 9.06 seconds (12.58 tokens/s, 114 tokens, context 145, seed 615984167)
LocalTranslation answer: La recette classique pour une pizza comprend de mixer du farine, du yeast et un sel dans l'eau; puis ajouter de l'huile, de l'eau et du sucre à la duret avant de roller et transférer sur une plaque; des toppings tels que mozzarella cheese, tomates, échalotes, pepperoni ou saucisson peuvent être ajoutés avant cuisson à 375°F pendant environ 20 minutes.
Output generated in 17.37 seconds (5.76 tokens/s, 100 tokens, context 43, seed 165764203)

I know what the issue was about the slow output:
You need to pass --no-stream, otherwise you get an inference mid sentence then a translation then an inference then a translation. Happens with both LibreTranslate and LocalAlpaca and it starts looking like it's beam search (while it's sphynx-moth). Check this:

LocalTranslation answer: Les ingrédients fondamentaux pour faire une.
LocalTranslation prompt: Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Translate phrase from English to French
### Input:
The basic ingredients for making a p
### Response:
Output generated in 2.29 seconds (5.25 tokens/s, 12 tokens, context 55, seed 145336152)
LocalTranslation answer: Les ingrédients fondamentaux pour faire une p
LocalTranslation prompt: Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Translate phrase from English to French
### Input:
The basic ingredients for making a pizza
### Response:
Output generated in 2.00 seconds (6.51 tokens/s, 13 tokens, context 56, seed 741753326)
LocalTranslation answer: Les ingrédients fondamentaux pour faire une pizza
LocalTranslation prompt: Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction:
Translate phrase from English to French
### Input:
The basic ingredients for making a pizza usually
### Response:
Output generated in 2.44 seconds (7.78 tokens/s, 19 tokens, context 57, seed 214619914)
LocalTranslation answer: Les ingrédients fondamentaux pour faire une pizza sont habituellement.

Also I might need to adapt the extension because I don't need it for every inference. My goal isn't to translate every inferences, the model I use is actually great at french.
I just need to translate the ones that will to go to sd_api_pictures so Stable Diffusion get an english prompt.

I also uninstalled LibreTranslator as it seems that there's a conflict with the latest text-generation-webui if you do pip install -r requirements.txt --upgrade.

It's getting late. I'll try with oasst-llama13b-4bit-128g tomorrow. I appreciate the responsiveness on your part, thank you very much.

edit: translation for french with alpaca-7B-4bit isn't as good as LibreTranslator for sure but it might be on par with a finetuned model (like Vigogne for french); it needs to be tested. I can't really compare the speed now tho, I wish I knew but what I though was slow was the lack of --no-stream unfortunately.

[janvarev]

Oh, thanks!
Yes, it works ok with --no-stream stuff due to reason you mentioned, I forget to say about it.

About your request to translate only specific stuff in sd_api... I don't work with sd_api now, but can recommend this:

• track your needed phrases in sd_api code
• then check how to call another plugin in it
• use in my version of google_translator plugin function like local_translator(params['language string'],'en',string,tpl_alpaca) to translate phrase to your lang with LocalAlpaca. Thats all you need.

[cmhamiche]

Thanks for the hints.

[Brawlence]

@cmhamiche
IMO this shouldn't be a part of SD_api_pictures logic itself. But I can always make the prompts for generation which go into the model be customizable; then one could provide it with custom request phrase (like for example "describe 'the thing' it but in English")

[cmhamiche]

Yeah, I glanced at the script real quick, what I want is fairly trivial so it's up to you.
It could be a temporary fork in the meantime also.

If implementing the changes you suggested for non english and multilingual models can wait I'm fine with it, it will probably be confusing for newcomers. It will happen when everything gets localized eventually.

I've never coded but I'll definitely be able to edit the script to my liking, thanks to the comments. Thanks guys.

edit: confusing for newcomers that the script is able to use their trigger expressions in their native language which is great but the picture from Stable Diffusion is borked since the prompt is still in their native language instead of english, I just wanted to clarify.

edit2: also I think that, right before sending the prompt, instead of a translation I will try to regenerate the prompt in english instead, I think it might get better results (maybe not, maybe only if the preset is deterministic or at least not too verbose, idk yet).