C/C++ binding interface

[roman-kruglov]

Do you guys plan to officially support such a binding? It seems pretty logical, after all Rust produces native code. We have a product in C++ and need to implement a RoBERTa \ GPT2 \ BPE tokenizer. The options are either reimplement it from the python version from transformers or use this rust implementation.

[julien-c]

We'd welcome an external contribution to add this support, @roman-kruglov

[glennkroegel]

@julien-c Does this mean if we want to use pytorch's jit scripting we are kind of stuck? Not much in the way of using a pytorch module in a Rust app with this tokenizer.

[Narsil]

You can use tch-rs to load models and use tokenizers from Rust if that's a viable option for you.

[glennkroegel]

You can use tch-rs to load models and use tokenizers from Rust if that's a viable option for you.

If people think it's stable / reliable then it should be fine.

[alexeyr]

@julien-c I am planning to work on this (probably using https://docs.rs/cxx/0.5.6/cxx/).

[alexeyr]

I expect to make a draft PR this week (or early next week). It will only have enough classes for BERT tokenizer, to get feedback on the API and design. However, I need string_view (and possibly optional). This means one of 3 options, in order of my own preference:

1. Copy https://github.com/martinmoene/string-view-lite/blob/master/include/nonstd/string_view.hpp and the license.
2. Include the repository as a submodule. Most of it isn't actually needed, but I remember there is a way to fetch only the necessary files.
3. Require C++17.

I would strongly prefer to avoid 3. If this repository already had submodules, I'd definitely go with option 2, but adding the first one is a bigger burden IMO. Which would you prefer?

[n1t0]

That's great news @alexeyr! I'm looking forward to seeing this!

I think you can go with copying it for now, that seems totally fine.

[alexeyr]

One more question then (I was planning to ask in the draft PR, but can as well do it now). Report errors using exceptions or expected<T, E> (which is like the Rust Result and again will require a single-header dependency)? Unfortunately,

1. idiomatic C++ is quite split on this, with many libraries defining their own Result-like types;
2. currently the second option still requires exceptions to be enabled, but some future cxx version is likely to change this;
3. going with exceptions lets users pick their preferred Result-like type.

[n1t0]

What would you be your take on this? Does being opinionated on this (by picking our preferred Result-like types) makes it harder for somebody to use it?

[alexeyr]

A bit, yes (especially if they prefer a more "advanced" option like https://boostorg.github.io/leaf/).

Actually, I thought of a way to make it configurable, so it will throw exceptions by default and let users define macros before including our header files to use a Result-like. Let me just try it out and see if it works...

[alexeyr]

Looks like it does work (at least compiles):

https://github.com/alexeyr/tokenizers/blob/c35c14833d3b2506616dd859dc73d550255201e2/bindings/cpp/src/tokenizers_util.h#L27-L48

https://github.com/alexeyr/tokenizers/blob/613e208ab5944e8f9d1488ff66947f8580f2d14d/bindings/cpp/src/redefine_result_tests.cpp#L5-L27

And if this approach runs into problems later, simply picking between exceptions and a single specific Result-like should work.

[alexeyr]

So it took much longer than expected, but here it is. The basic API is here:

tokenizers/bindings/cpp/tokenizers-cpp/tests.h

Lines 77 to 114 in 6358c24

	TEST_SUITE("Tokenizers") {
	TEST_CASE("Bert") {
	Tokenizer tokenizer(
	WordPieceBuilder()
	.files(data_file("/bert-base-uncased-vocab.txt"))
	.unk_token("[UNK]")
	.max_input_chars_per_word(100)
	.build());
	tokenizer.with_normalizer(BertNormalizerOptions().build())
	.with_pre_tokenizer(PreTokenizer::bert())
	.with_post_processor(
	PostProcessor::bert("[SEP]", 101, "[CLS]", 102));
	Encoding encoding = tokenizer.encode("My name is John", true);
	std::vector<rust::String> expected_tokens{"[CLS]", "my", "name",
	"is", "john", "[SEP]"};
	COMPARE_CONTAINERS(expected_tokens, encoding.get_tokens());
	std::vector<uint32_t> expected_ids{102, 2026, 2171, 2003, 2198, 101};
	COMPARE_CONTAINERS(expected_ids, encoding.get_ids());
	}
	TEST_CASE("GPT2") {
	Tokenizer tokenizer(BpeBuilder()
	.files(data_file("/gpt2-vocab.json"),
	data_file("/gpt2-merges.txt"))
	.build());
	tokenizer.with_pre_tokenizer(PreTokenizer::byte_level(false, true));
	Encoding encoding = tokenizer.encode("My name is John", true);
	std::vector<rust::String> expected_tokens{"My", "Ġname", "Ġis",
	"ĠJohn"};
	COMPARE_CONTAINERS(expected_tokens, encoding.get_tokens());
	std::vector<uint32_t> expected_ids{3666, 1438, 318, 1757};
	COMPARE_CONTAINERS(expected_ids, encoding.get_ids());
	}
	}

[alexeyr]

Normalizers, pre-tokenizers, and models are done. I will try to finish the remaining parts on Friday, but don't expect any more significant API changes. So if anybody wants to review what is there, now is a good time.

[mtszkw]

Hi, how is it going? Are you still working on this one, @alexeyr?