Add examples of large language models

[yuguolong]

Will examples of large language models be added?

[FilipFan]

I have the same question—I’d like to see an LLM example based on LiteRT, but I can’t seem to find one.

Additionally, the available samples appear scattered and inconsistent:

• This sample repo referenced at https://ai.google.dev/edge/litert/android doesn’t include an LLM example.
• https://github.com/tensorflow/examples/tree/master/lite/examples is linked in Use LiteRT on Android, but it only contains TensorFlow Lite examples and hasn’t been updated in two years.
• https://github.com/google-ai-edge/gallery, referenced in the LLM Inference guide for Android, depends on higher-level MediaPipe APIs.
• https://github.com/android/ai-samples/tree/main/gemini-nano, linked in Android AI code samples, relies on the AICore API instead.

[protobird-git]

@yuguolong, @FilipFan Sorry for the long delayed reply.

LLMs are supported by LiteRT-LM project, https://github.com/google-ai-edge/LiteRT-LM/ which runs LLMs over LiteRT. It has GPU and NPU accelerations thanks to LiteRT. Give a try and let us know how it works for you.

[shivaylamba]

@protobird-git can you add sample on how to use LiteRT-LM with NPU support in the Kotlin API - it is not currently supported.

[protobird-git]

@ztenghui, could you share the status of LiteRM-LM Korlin API?