Define + document core API surface for language models, provide language bindings

[GregoryComer]

🚀 The feature, motivation and pitch

In order to make language models easy to use on ExecuTorch, we likely need to define the "core" components for running LLMs on ET. This is an evolving area, of course, but there is a lot of interest now on running decoder-only auto-regressive generation, which we want to provide a clean solution for. The most important use, in my opinion, is providing a streamlined path for runtime integration of HF models, though the more general, the better.

In a nutshell, I hope to be able to give users "the" way to run LLMs on ET. Don't make them make technical decisions when we can avoid it. We can have lower-level composable components and power-user APIs, but I hope to handle the majority of use cases by being able to tell users, "get your model from HF, export with Optimum, and here's the runner APIs in C++, Java, and Obj-C/Swift". Minimal decision making or lower-level understanding required.

The Android + iOS bindings are likely most critical here, but I want to ensure that we standardize on the components and API design before building these.

Requirements

1. Define "core" runtime components needed to run text generation decoder models, particularly with a focus on models from HF through Optimum.
2. Move all reusable components out of the examples directory (may already be done).
3. Clearly document these components.
4. Provide API bindings for these components from Java and Objective-C / Swift. Ideally, the API surface should match as closely as possible while respecting the appropriate language conventions, such that there is parity in usage and capability between Java, Obj-C/Swift, and C++ runners.

CC @guangy10 @larryliu0820 for core APIs, @kirklandsign @shoumikhin for language bindings, @byjlw @mergennachin for usability workstream

Alternatives

No response

Additional context

No response

RFC (Optional)

No response

cc @mergennachin @cccclai @helunwencser @jackzhxng @byjlw

[guangy10]

When users are validating HF models via Optimum today, they are actually using the llama runtime via pybind. Ideally runner APIs in C++, Java, and Obj-C/Swift and via pybind should be identical. If they can get (a) running a model e2e in optimum easily, why would it be difficult to do (b) use exact same runtime on-device w/o pybind.

[GregoryComer]

When users are validating HF models via Optimum today, they are actually using the llama runtime via pybind. Ideally runner APIs in C++, Java, and Obj-C/Swift and via pybind should be identical. If they can get (a) running a model e2e in optimum easily, why would it be difficult to do (b) use exact same runtime on-device w/o pybind.

Makes sense, but is the pybind version relying on the HuggingFace generator loop / pipeline / tokenizer components? I'm wondering about the steps for handling tokenization and generator loop in ET, matching the model export and structure conventions used by HF / ET LLMs.

[guangy10]

is the pybind version relying on the HuggingFace generator loop / pipeline / tokenizer components?

@GregoryComer Yes, that's where we are today. I talked to @larryliu0820 about this a while ago and proposed to hide the generation loop and tokenizer inside the runtime. Here is how it's implemented in optimum in python today and with the notes I put during the initial integration: https://github.com/huggingface/optimum-executorch/blob/main/optimum/executorch/modeling.py#L284-L286

[GregoryComer]

is the pybind version relying on the HuggingFace generator loop / pipeline / tokenizer components?

@GregoryComer Yes, that's where we are today. I talked to @larryliu0820 about this a while ago and proposed to hide the generation loop and tokenizer inside the runtime. Here is how it's implemented in optimum in python today and with the notes I put during the initial integration: https://github.com/huggingface/optimum-executorch/blob/main/optimum/executorch/modeling.py#L284-L286

Make sense. Thanks. I do want to make sure that we have this runtime solution in place before we start seriously advertising HF compliance, since right now, users can export their model, but can't actually run it. Or at least we don't have a clear path and instructions for them to run with Java, Obj-C/Swift, etc.

[guangy10]

since right now, users can export their model, but can't actually run it

@GregoryComer I want to clarity that the pain is when running the model on Android/iOS, not running/validating the model on host via optimum, if we just look at the usability, though the perf is not optimal.