Add a way to surface/detect indexing failures

[shannonantony]

@bsbodden
Thank you again for your continued support in helping the Redis community.

I’m encountering an issue where there is a mismatch between the key count in my document repository and the number of documents reported by the Redis index. For example, I have a repository with around 1 million records. Occasionally, I notice that while the key count remains at 1 million, the FT.INFO command for the index returns a num_docs value that differs from the repository count.

Have you come across a similar issue? What might be causing this discrepancy?

I appreciate any insights or advice you may have.

Thanks again,
Jay

[bsbodden]

@shannonantony Jay, there could be indexing issues with certain records. If you launch the cli or Redis Insight, you can try an FT.INFO index_name and scrolldown the output, there should be a hash_indexing_failures, if that number is not zero then some records contained some data that could not be indexed.

[shannonantony]

@bsbodden .
Thank you for your response.

I’ve observed some indexing failures, including the following:

hash_indexing_failures
94763
My document repository primarily consists of a collection of IDs, and I’m uncertain about what might be causing these failures. Could you please advise on how I can investigate the issue further and prevent such errors in the future?

Thank you again for your help.

Best regards,
Jay

[bsbodden]

If you can identify one of the records with the indexing failures, post the data here (is it JSON or Hashes?) and also post one of the records that is in the index. Typically it is something simple with the data that causes the error. For example, a typical error we see is geo field with Lon-Lat reversed (is Lon-Lat in Redis for some reason)