delete_documents fails when using a hypen or an understore as the value

[normanb]

I am using a uuid for my id field and when I delete by id the delete does not happen (same with underscores). I don't think a hypen or an underscore needs to be escaped?

Removing the hypen or underscore and the delete works fine. I can work around this by removing the hyphen before writing

I was thinking that the - is being interpreted as a negation but it is underscores as well.

The Python code to recreate is below;

import json
from tantivy import Index, SchemaBuilder

schema_builder = SchemaBuilder()
schema_builder.add_text_field("id", fast=True)
schema = schema_builder.build()

index = Index(schema)

writer = index.writer()

id_str = "test-1"

source_doc = {
    "id": id_str,
}

query = index.parse_query(f"id:{id_str}")
writer.add_json(json.dumps(source_doc))

writer.commit()
writer.wait_merging_threads()
index.reload()
result = index.searcher().search(query)
print(f"Insert: {result}")

writer = index.writer()
writer.delete_documents("id", id_str)
writer.commit()
index.reload()

result = index.searcher().search(query)
print(f"Deleted: {result}")

And I get the result

Insert: SearchResult(hits: [(0.57536423, DocAddress { segment_ord: 0, doc: 0 })], count: 1)
Deleted: SearchResult(hits: [(0.57536423, DocAddress { segment_ord: 0, doc: 0 })], count: 1)

Whereas without the hyphen I get as expected;

Insert: SearchResult(hits: [(0.28768212, DocAddress { segment_ord: 0, doc: 0 })], count: 1)
Deleted: SearchResult(hits: [], count: 0)

[cjrh]

This is the same issue as #297

The root cause is that for a tokenized field, the tokenizer is being used when adding data (which modifies the stored value) but the the .delete_documents method does not apply the tokenizer to the parameters.

If you use the "raw" tokenizer, you will find that it works:

schema_builder.add_text_field("id", fast=True, tokenizer_name="raw")

Output:

$ python main.py 
Insert: SearchResult(hits: [(0.28768212, DocAddress { segment_ord: 0, doc: 0 })], count: 1)
Deleted: SearchResult(hits: [], count: 0)

We acknowledge this is somewhat of a footgun at the moment. The delete_documents method should be modified to use whatever tokenizer is defined on the given field.

[cjrh]

I started working on this last night. We already have delete_query implemented under the hood so that's what we'll use for text fields I think. That will solve the problem.

[cjrh]

@normanb For your use-case, are you happy to use the raw tokenizer to solve your UUID usecase? (it's odd to have a tokenizer on a uuid field I reckon). If so we can close this. I am considering more general options for deleting documents via query in the associated PR, but I think for UUIDs specifically, and other "non natural language" cases, the correct solution is to use the raw tokenizer.

[normanb]

Yes, this can be closed. The raw tokenizer will solve the issue. I also just wrapped the id before putting them as a fast field but this will be easier. Thanks!