Repository navigation
Fix most of the tokenizer tests. - #41
Conversation
|
The documentation is not available anymore as the PR was closed or merged. |
|
Thanks a lot for all this work @lockon-n ! I can have a deeper look on Thursday, is that ok? |
|
Thanks @NielsRogge and @SaulLu! @lockon-n will work with you on this PR. |
SaulLu
left a comment
SaulLu
left a comment
There was a problem hiding this comment.
Thank you so much for working on adding MarkupLM 🤗 🙌 ! Integrating the tokenizer and understanding the tests is not a simple thing.
I left a lot of questions in the comments to make sure I understood the motivation behind your changes.
| ], | ||
| ) | ||
|
|
||
| class MarkupLMConverter(Converter): |
There was a problem hiding this comment.
Do you remember why Roberta's converter didn't work?
There was a problem hiding this comment.
I compare roberta's to bert's, seems it does not add the unknown token to BPE model.
There was a problem hiding this comment.
Thanks! Yes, that's possible as Roberta's is a byte-level BPE (we can't have unknown tokens). MarkupLM's tokenizer can produce unknown tokens?
There was a problem hiding this comment.
Well. We assume MarkupLM's tokenizer is exactly the same as Roberta's, so unknown tokens shouldn't exist. So the only problem is this test is originally designed for LayoutLM, which is based on Bert's tokenizer. Maybe we can ignore it.
There was a problem hiding this comment.
It's test_rust_and_python_full_tokenizers.
You can see in the setUp() of MarkupLMTokenizationTest, we have an unknown token for the temporary slow tokenizer, however if using the default RoBERTa converter, the unknown token is removed when building fast tokenizer. Then ids and rust_ids in line 1122/1123 (test_tokenization_markuplm.py) are not the same (the first token of ids is '<unk>', while rust_ids does not have it).
| overflowing_xpath_tags_seq = pair_xpath_tags_seq[-window_len:] | ||
| overflowing_xpath_subs_seq = pair_xpath_subs_seq[-window_len:] |
| text = [['this', 'is', 'the'], ["how", "are", "you"]] | ||
| xpaths = [["html/body"] * 4, ["html/body"] * 3] |
There was a problem hiding this comment.
I think I'm missing something here: I see 3 nodes in the ['this', 'is', 'the'] list. Why do we have 4 values in the corresponding xpaths list?
There was a problem hiding this comment.
This should be a typo, sorry about that.
There was a problem hiding this comment.
No problem! If it is not possible to have a case where the number of text nodes and xpath nodes is not the same, shouldn't an error be raised (btw I wonder how the tokenizer handles this difference currently)?
There was a problem hiding this comment.
I think raising an error is enough, or something like assume len(text)==len(xpaths). In fact we do not consider such difference when designing relevant module, but xpaths longer than text may be ignored I guess.
| inputs = new_tokenizer(text, xpaths=xpaths) | ||
| self.assertEqual(len(inputs["input_ids"]), 2) | ||
| decoded_input = new_tokenizer.decode(inputs["input_ids"][0], skip_special_tokens=True) | ||
| expected_result = "this is the" | ||
| expected_result = "thisisthe" # original expected result "this is the" seems contradicts to roberta-based tokenizer |
There was a problem hiding this comment.
This behaviour led me at the time to investigate how tokenization was done in the original repository.
For context, in a Roberta tokenizer spaces are transformed into Ġ characters and not removed from the tokenization. With Roberta's tokenizer, the tokenization of the individual words - 'this', 'is', and 'the'- would be:
- with
add_prefix_space=False:['this'],['is'], and['the'] - with
add_prefix_space=True:['Ġthis'],['Ġis'], and['Ġthe']
The spaces being lost and the decoder not artificially adding spaces explains the current behaviour of the decode method. Nevertheless, this led me to 2 questions:
- in the original code did the text in each node start with a space (even if this space was artificily added with a
add_prefix_spaceargument)? In other words, was the tokenization of the nodes[['this'], ['is'], ['the']]or was it[['Ġthis'], ['Ġis'], ['Ġthe']]. - in the case where the text in each node does not start with a space, do we need a custom decoder to reintroduce spaces (is this possible?)? To answer this question I think the most important thing is to know if MarkupLM can solve generative tasks.
These questions led me to propose a new integration test in this PR #37 which is unfortunately based on a very old version of the library and therefore does not make the PR readable now. To create this integration test I investigate the tokenization produced by the authors of MarkUpLM on the downstream task of WebSRC. It seems to me that the integration test I proposed to add was this one:
transformers/tests/test_tokenization_markuplm.py
Lines 2196 to 2251 in 36dcacc
In addition to the problems I had identified in the PR message, the targets defined in the test allowed me to conclude that there was no space added at the beginning of each node (which would answer my first question above). But you would surely know better than I what you did in your original code and what should be our ground truth!
There was a problem hiding this comment.
For question 1, the tokenization should be [['this'], ['is'], ['the']]. For question 2, now MarkupLM can not handle generative questions and I think a custom decoder is not necessary.
Frankly speaking, we do not consider too much about the "prefix space". For each node, we will first do strip() to remove the possible prefix/suffix spaces, and all things left is to use a default Roberta Tokenizer to tokenize the span and generate tokens.
There was a problem hiding this comment.
Very clear to me!
As MarkupLM can't do generative tasks, I think we should put a warning message when the decode method is called saying that the decoding is experimental and subject to change.
There was a problem hiding this comment.
Sure, that's exactly what we want :)
| # Ensure consistency | ||
| output_txt = tokenizer.decode(toks_ids, clean_up_tokenization_spaces=False) | ||
| # an extra blank will cause inconsistency: ["a","b",] & "a b" | ||
| ''' | ||
| if " " not in output_txt and len(toks_ids) > 1: | ||
| output_txt = ( | ||
| tokenizer.decode([toks_ids[0]], clean_up_tokenization_spaces=False) | ||
| + " " | ||
| + tokenizer.decode(toks_ids[1:], clean_up_tokenization_spaces=False) | ||
| ) | ||
| ''' | ||
| if with_prefix_space: | ||
| output_txt = " " + output_txt | ||
| nodes = output_txt.split(" ") | ||
| xpaths = ["html/body" for i in range(len(nodes))] | ||
| output_ids = tokenizer.encode(nodes, xpaths=xpaths, add_special_tokens=False) | ||
|
|
||
| return nodes, xpaths, output_ids |
There was a problem hiding this comment.
For this automatic script, I wonder if what is currently generated is the most useful. Currently nodes and xpaths are both composed of a single element, ['lowerstidner'] and ['html/body'] respectively. Wouldn't it be more useful to have an example here with multiple nodes and xpaths?
There was a problem hiding this comment.
I'm quite confused about this question, would you like to give more details?
There was a problem hiding this comment.
Since in most use cases the input will have several nodes, I think we should also have several nodes in the example returned here. Indeed, by not having several nodes one might not be able to spot potential problems.
For example: ['lowerstidner', 'lower', 'lowersti'] and ['html/body', 'html/body/div', 'html/body/div/table']
There was a problem hiding this comment.
Yes, having multiple nodes is more common and it's better to have an example like that. But I think there's no difference if we only consider the xpath converter part.
I remember that the corresponding test in LayoutLM can generate multiple nodes, which may also be a 'bert2roberta' problem.
There was a problem hiding this comment.
It is possible, I think in this case it is ok to override this method to propose a sufficiently adversarial example. 😊
| @@ -2216,7 +2215,7 @@ def test_markuplm_integration_test(self): | |||
| nodes, xpaths = self.get_nodes_and_xpaths() | |||
|
|
|||
| # fmt: off | |||
| expected_results = {'input_ids': [101, 1037, 6881, 2135, 3231, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'bbox': [[0, 0, 0, 0], [423, 237, 440, 251], [427, 272, 441, 287], [427, 272, 441, 287], [419, 115, 437, 129], [1000, 1000, 1000, 1000], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]} # noqa: E231 | |||
| expected_results = {'input_ids': [0, 42891, 8331, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'xpath_tags_seq': [[216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [109, 25, 50, 120, 50, 178, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [109, 25, 50, 120, 50, 178, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216], [216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216, 216]], 'xpath_subs_seq': [[1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [109, 25, 50, 120, 50, 178, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [109, 25, 50, 120, 50, 178, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001], [1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001, 1001]], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]} | |||
There was a problem hiding this comment.
Thanks for generating the right targets! Out of curiosity, what was your workflow for generating these targets (so that I know how to do it if we want to redo it again for other examples in the futur)?
There was a problem hiding this comment.
As you can see the original expected results are for LayoutLM (e.g., bbox) but not MarkupLM.
To generate it, just use a roberta tokenizer to tokenize it, and do some padding is ok. The xpath tags/subscripts part are from the get_xpath_seq method in MarkupLMTokenizer.
There was a problem hiding this comment.
Thank you very much for your feedback. I wonder if there might be a way to get the targets out of the original code.
The reason I'm wondering this is that while investigating the WebSRC downstream task I noticed that the template (=adding tokens at the beginning of a sentence, at the separation of sentences and at the end of a sentence) the pair of sentences (questions and textual nodes) was not identical to Roberta's. From what I understand:
- Roberta's template:
'<s>', tokens from sentence 1 '</s>', '</s>', tokens from sentence 2 '</s>' - WebSRC MarkupLM's template:
'<s>', tokens from sentence 1 '</s>', tokens from sentence 2 '</s>'
But maybe in the original code several templates were also used depending on the task, or I didn't understand something...
So, in this example I wonder if the target for get_question_nodes_and_xpaths and get_question_nodes_and_xpaths_batch is indeed a target representative of the modeling proposed in MarkupLM.
There was a problem hiding this comment.
Oh I get your point.
The current targets are wrong, and what we want is WebSRC's template, which means the element related to the second </s> should be removed and a padding element should be added in 'input_ids', 'xpath_tags_seq', 'xpath_subs_seq' and 'attention_mask'. We leave 'token_type_ids' alone as it is always all-zero. Sorry for this mistake.
The source code for WebSRC is BERT's style, and the early version for MarkupLM is also BERT's style. In other words, there is only one sep token in between. This feature is kept after we switch to RoBERTa's tokenizer. I have to admit this is our ignorance, because we do not know standard RoBERTa tokenizer inserts two consecutive sep tokens at that time. As a result, our pre-training is also conducted in this manner (one sep), so the template in fine-tuning should also be that kind.
To fix this issue, simply removing the second </s> and its pad in xpath tag/subs is OK in MarkupLMTokenizer.
There was a problem hiding this comment.
Thank you very much for the confirmation! There are really no worries!
Unfortunately this means that for the tokenizer fast we have to wait for a new feature that should arrive soon in the tokenizers library to be able to combine two post-processors: the one for the byte-level and the one for the template that is not the one from roberta.
|
@SaulLu Hi there. I have tried my best to answer the questions. Feel free to raise more lol. |
|
@SaulLu More feedback updated. Looking for your reply ( ,and possibly, more questions). 😄 |
|
Hi @NielsRogge , I think MarkupLM is almost done except that MarkupLMTokenizer will insert two between the question and the page context, so we just need to delete the second one. But the tokenizerfast may wait for some new features as @SaulLu said. |
|
Hi @SaulLu and @NielsRogge, how about the progress of the tokenizer fast? |
|
Hi @NielsRogge and @lockon-n and @wolfshow , As the feature for tokenizers fast is not ready yet, what we can do is to add the fast version of Markup LM with partial availability: the only case that will not work is if I have opened a PR showing this case here: lockon-n#1 I think this is a good compromise to add MarkUp LM to transformers. If we want to use this workaround, the |
proposal of a fix for the MarkupLM fast tokenizer
|
Hi @NielsRogge , I have merged the PR proposed by @SaulLu . I think MakrupLM is ready for transformers now. |
|
Great! 🙌 I just think we need to add an integration test for the fast tokenizer too and change the template for the slow tokenizer with the question 😄 (and add the warning discussed in this message: #41 (comment)) |
…late in slow tokenizer
|
@SaulLu Issues have been fixed in latest update. The integration test for fast tokenzier fails (double problem) when text-pair encoding, is that expected? I have set trim_offset as False in default. |
| def test_markuplm_integration_test(self): | ||
|
|
||
| tokenizer_p = MarkupLMTokenizer.from_pretrained("microsoft/markuplm-base") | ||
| # no fast tokenizer for now | ||
| # tokenizer_r = MarkupLMTokenizerFast.from_pretrained("microsoft/markuplm-base") | ||
| tokenizer_r = MarkupLMTokenizerFast.from_pretrained("microsoft/markuplm-base") |
There was a problem hiding this comment.
@lockon-n , can you re-try with this change? 🙏
| tokenizer_r = MarkupLMTokenizerFast.from_pretrained("microsoft/markuplm-base") | |
| tokenizer_r = MarkupLMTokenizerFast.from_pretrained("microsoft/markuplm-base", from_slow=True) |
If it solves the issue it's just because the ŧokenizer.json file at https://huggingface.co/microsoft/markuplm-base needs to be updated
There was a problem hiding this comment.
Oh 😞
Could this be coming from the xpath_tags_seq list?
I have trouble remembering how the nodes and xpaths need to be tied. I particular I don't remember why line 102 is xpaths = ["/html/body/div/li[1]/div/span", "/html/body/div/li[1]/div/span"] and not xpaths = ["/html/body/div/li[1]/div/span"]
transformers/tests/models/markuplm/test_tokenization_markuplm.py
Lines 99 to 112 in 485b2ca
There was a problem hiding this comment.
This seems to be a typo, but does not leads to mistakes. The second xpath is discarded when processing I guess.
|
|
||
| tokenizer.post_processor = processors.TemplateProcessing( | ||
| single="<s> $A </s>", | ||
| pair="<s> $A </s> $B </s>", | ||
| special_tokens=[ | ||
| ("<s>", ot.convert_tokens_to_ids("<s>")), | ||
| ("</s>", ot.convert_tokens_to_ids("</s>")), | ||
| ], | ||
| ) |
There was a problem hiding this comment.
oh I'm sorry the change on this file slipped through the cracks when I commited the files.... but indeed this is the fix!
There was a problem hiding this comment.
@SaulLu I rewrite the converter, this solve the double problem. i.e., the tokenizer.post_processor(). However, the tokenizerfast seem to tokenize "what's his name?" as ["what","'s","his","name","?"] while the slow-tokenizer and groundtruth is ["what","'s","Ġhis","Ġname","?"]. Seems the space problem caused by trim_offsets?
There was a problem hiding this comment.
@SaulLu I rewrite the converter, this solve the double problem. i.e., the tokenizer.post_processor().
🙌
However, the tokenizerfast seem to tokenize "what's his name?" as ["what","'s","his","name","?"] while as ["what","'s","Ġhis","Ġname","?"]. Seems the space problem caused by trim_offsets?
Can you share the snippet you used to see that please?
There was a problem hiding this comment.
run test_markuplm_integration_test(), and you can compare the difference between encoding_p['input_ids] and encoding_r['input_ids'] in Line 2291&2292. I convert the ids into tokens, and gets the results above.
Sorry for not using an easier way like a snippet of codes.
There was a problem hiding this comment.
So... we reach a release-ready version?
There was a problem hiding this comment.
Thanks so much for your effort !!!! 👏 👏 really nice experience to collaborate with such nice staffs, also excited to see our model appearing in this famous repo (a common dream for NLPers I guess 😆)
| tokenizer.post_processor = processors.TemplateProcessing( | ||
| single="<s> $A </s>", | ||
| pair="<s> $A </s> $B </s>", | ||
| special_tokens=[ | ||
| ("<s>", ot.convert_tokens_to_ids("<s>")), | ||
| ("</s>", ot.convert_tokens_to_ids("</s>")), | ||
| ], | ||
| ) |
There was a problem hiding this comment.
| tokenizer.post_processor = processors.TemplateProcessing( | |
| single="<s> $A </s>", | |
| pair="<s> $A </s> $B </s>", | |
| special_tokens=[ | |
| ("<s>", ot.convert_tokens_to_ids("<s>")), | |
| ("</s>", ot.convert_tokens_to_ids("</s>")), | |
| ], | |
| ) | |
| cls = str(self.original_tokenizer.cls_token) | |
| sep = str(self.original_tokenizer.sep_token) | |
| cls_token_id = self.original_tokenizer.cls_token_id | |
| sep_token_id = self.original_tokenizer.sep_token_id | |
| tokenizer.post_processor = processors.TemplateProcessing( | |
| single="cls $A sep", | |
| pair="cls $A sep $B sep", | |
| special_tokens=[ | |
| (cls, cls_token_id), | |
| (sep, sep_token_id), | |
| ], | |
| ) |
|
Ok, awesome work :) can you confirm I can merge this now? |
|
I think yes @NielsRogge |
yes! 🙌 |
|
So I've merged + rebased with the main branch of Transformers, seems like there are 4 tests to be fixed. Running returns: |
|
@NielsRogge Hi, I have modify the codes to tackle these tests. Should another PR be raised? |
|
Yes, you can do that. Note that I've rebased the |
* remove one of the last deps * update fast image processor after refactor * styling * more quality of life improvements * nit * update * cleanups * some cleanups * vllm updates * update fake image token * [convert] Fix typo * [convert] Strip extraneous bytes from shards * [convert] Minor fixes * [convert] Use num_experts * multi-image fixes in modeling + processor * fixup size * 128 experts * Use default rope * Unfuse mlp * simplify a lot inputs embeds merging * remove .item() 👀 * fix from review * Address feedback * Use None "default" for rope_scaling. Add eot. * set seed * return aspect ratios and bug fixes * Moe 128 rebased (#8) * 128 experts * Use default rope * Unfuse mlp * Address feedback * Use None "default" for rope_scaling. Add eot. * Meta/llama quant compat (#7) * add quant compatible model & conversion code for llama4 * fix a few issues * fix a few issues * minor type mapping fix --------- Co-authored-by: Lu Fang <fanglu@fb.com> * use a new config parameter to determine which model definition to use for MoE --------- Co-authored-by: Pedro Cuenca <pedro@huggingface.co> Co-authored-by: Lu Fang <fanglu@fb.com> * un-comment write_tokenizer from converting script * remove un-used imports * [llama4] Pop aspect_ratios from image processor output in Llama4Processor Signed-off-by: Jon Swenson <jmswen@gmail.com> * Fix parameter_count name * Update src/transformers/models/llama4/configuration_llama4.py * nit * Add changes for no_rope, moe_layers, chunked attention. Just need to test all * Update src/transformers/models/llama4/image_processing_llama4_fast.py * nit * fix post merge with main * support flex attention * fixes * fix * add layer * small updates * rebase and delete llm_compressor * nit * [llama4/mm] Add back <|image|> token that delimits global tile * [llama4/mm] Fix Llama 4 image processing unit tests * add explicit dtype Signed-off-by: Jon Swenson <jmswen@gmail.com> * sdpa works * comment todo small * fix model loading Signed-off-by: Zijing Liu <liuzijing2014@gmail.com> * revert * nits * small fix for TP on 1 node * Read new params from config * Add <|eom|> * lol don't know how this got here * adding fp8 * Save processor, fix chat template * style * Add boi/eoi tokens We don't use them. * fixes for now flex seems to work :) * updates * nits * updates * missking keys * add context parallel * update * update * fix * nits * add worldsize and make eager attn work for vision * Ignore new key present in base models * add tp_plan * fix nope Signed-off-by: Zijing Liu <liuzijing2014@gmail.com> * minor fix Signed-off-by: Zijing Liu <liuzijing2014@gmail.com> * Clean up Llama4 vision model * current updates * add support for `attn_temperature_tuning` * add floor scale * add missing attn scales * push what works, dirty trick for the device synch * oups * Fix pad_token_id See https://huggingface.co/ll-re/Llama-4-Scout-17B-16E/discussions/2/files Confirmed in the original codebase. * fix causallml loading * rm * fix tied-weights * fix sdpa * push current version * should work with both short and long * add compressed_tensos & fix fbgemm tp * Fix flex impl * style * chunking * try to revert the potentially breaking change * fix auto factory * fix shapes in general * rm processing * commit cache utils cleanup * Fix context length * fix * allocate * update tp_plan * fix SDPA! * Add support for sparse `Llama4TextMoe` layer from the kernel hub * cleanup * better merge * update * still broken fixing now * nits * revert print * Write max_position_embeddings and max_model_length * Update modeling_llama4.py * Save attention_chunk_size * Sync eos terminators * Read initializer_range * style * remove `dict` * fix * eager should use `chunked_attention_mask` * revert * fixup * fix config * Revert "Merge pull request #36 from huggingface/sparse-llama4-moe" This reverts commit ccda19f, reversing changes made to a515579. * Fix typo and remove warning with compiled flex and chunked prefill * Fix MoE vs FF (#41) * fix * Use correct no_rope_layers if provided one is empty list * update tests * fix * skipping some tests * fix fp8 loading Signed-off-by: Zijing Liu <liuzijing2014@gmail.com> * fix text geneartion pipeline Signed-off-by: Zijing Liu <liuzijing2014@gmail.com> * eager needs 4D mask * fix * Some cleanup * fix * update * fix * replace correctly module * patch * modulelist * update * update * clean up * Don't move to `cuda:0` in distributed mode * restrict to compressed tensors for now * rm print * Docs! * Fixes * Update docs/source/en/model_doc/llama4.md Co-authored-by: Pedro Cuenca <pedro@huggingface.co> * Fixes * cuda graph fix * revert some stuff * fixup * styling * Update src/transformers/models/llama4/modeling_llama4.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * fixup * commit licence, cleanup here and there and style * more styling changes * fix dummies * fix and clean docstrings * remove comment * remove warning * Only fast image processor is supported * nit * trigger CI * fix issue with flex encoder * fix dynamic cache * Code quality * Code quality * fix more tests for now * Code quality * Code quality * Nuke bunch of failing stuff * Code quality * Code quality * cleanup removal of slow image processor * ruff fix fast image processor * fix * fix styling * Docs * Repo consistency * Repo consistency * fix sliding window issue * separate llama cache * styling * Repo consistency * Repo consistency * push waht works * L4 Repo consistency * Docs * fix last last alst alst alst alstsaltlsltlaslt --------- Signed-off-by: Jon Swenson <jmswen@gmail.com> Signed-off-by: Zijing Liu <liuzijing2014@gmail.com> Co-authored-by: yonigozlan <yoni.gozlan10@gmail.com> Co-authored-by: Pedro Cuenca <pedro@huggingface.co> Co-authored-by: Pablo Montalvo <pablo.montalvo.leroux@gmail.com> Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com> Co-authored-by: Keyun Tong <tongkeyun@gmail.com> Co-authored-by: Zijing Liu <liuzijing2014@users.noreply.github.com> Co-authored-by: Lu Fang <fanglu@fb.com> Co-authored-by: Zijing Liu <liuzijing2014@gmail.com> Co-authored-by: Jon Swenson <jmswen@gmail.com> Co-authored-by: jmswen <jmswen@users.noreply.github.com> Co-authored-by: MekkCyber <mekk.cyber@gmail.com> Co-authored-by: Mohamed Mekkouri <93391238+MekkCyber@users.noreply.github.com> Co-authored-by: Mohit Sharma <mohit21sharma.ms@gmail.com> Co-authored-by: Yong Hoon Shin <yhshin@meta.com> Co-authored-by: Marc Sun <marc@huggingface.co> Co-authored-by: drisspg <drisspguessous@gmail.com> Co-authored-by: Cyril Vallez <cyril.vallez@gmail.com> Co-authored-by: Daniël de Kok <me@danieldk.eu> Co-authored-by: Lysandre <hi@lysand.re> Co-authored-by: Ye (Charlotte) Qi <ye.charlotte.qi@gmail.com> Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
What does this PR do?
I have fixed most of the tokenizer tests except
test_maximum_encoding_length_pair_input.Seems these tests are copied from LayoutLMv2, but LayoutLMv2 is based on BERT tokenizer, so some tests may not suitable for MarkupLM which is based on RoBERTa tokenizer. I tried my best to fix both the test part and the tokenizer part, with necessary annotations left. However,
test_maximum_encoding_length_pair_inputis beyond my ability, I only fixed the slow tokenizer part in this test.I will be very pleased if this commit is useful!
Fixes # (issue)
Before submitting
Pull Request section?
to it if that's the case.
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
@NielsRogge Feel free to ask questions or give suggestions for changes.