How to deal with potentially dangerous or broken HTML, some options:

[berkes]

My content contains potentially dangerous HTML and often broken HTML: I don't control the source.
On output this breaks the page and might cause XSS-injections.

Describe the solution you'd like
Either one, or more, of the following:

• Documented best practices WRT indexing HTML.
• Documented best practices WRT rendering indexed (potentially broken or dirty) HTML. Highlighting included.
• Have the _formatted fields strip all HTML after cropping, and before highlighting.
• Have the _formatted fields encode all HTML after cropping, and after highlighting, but before inserting the highlight em-tags.

Describe alternatives you've considered
I see several opportunities and like to discuss those, before writing a PR:

1. Leave it outside the responsibility of Meilisearch. In this case maybe a warning in documentation about potential XSS is prudent. In this case user would need to:
   1. Sanitize, encode or strip HTML before indexing.
   2. Sanitize, encode or strip HTMLclient-side, before returning to the user. As is done in e.g. html sanitize #539
   3. Sanitize, encode or strip HTML in a proxy or backend. The (HTML/JS) client would request from a custom backend which searches in Meilisearch and then sanitizes the results.
2. Always encode or strip HTML before placing the fields in _formatted. Meilisearch would be opinionated, choose one and enforce that for all users. This is how e.g. elasticsearch does its result_fields.
3. Have an option to encode or strip HTML. A user would need to add additional options, e.g. attributesToStrip=*,overview alongside the crop and highlight feature, which would either encode or strip the configured fields.

Please check some tests I added in 6b0f4eb which demonstrate the problem caused when cropping or highlighting. The XSS is not covered as that is implied in those tests, but not directly related to cropping and highlighting.
Alternatives have some down- and upsides:

• 1.i: Will loose information that the tokenizer might (in future?) employ to improve ranking. E.g. rank a word in <strong> higher than one in <small>.
• 1.ii: Makes the highlighter complex and often not work. Will show garbled content to users, in many cases.
• 1.iii: Requires serverside app. Useful when already needed for e.g. access control, but adds complexity and performance overhead.
• 1. Any of those do allow great control over the output for several use-cases. E.g. an implementor might want to keep tags like p, br and to have some line-support, or em and strong to have some markup, but may want to remove headers, a-tags etc.
• 2. Is simple and straightforward for the implmentor but gives little control. Also requires some thinking on how to deal with cropping offsets and highlighting: should those apply to the sanitized, cropped, or stripped version or to the orig? Since both the _formatted and original fields are returned, users still have the ability to use the raw, original value and e.g. sanitize that themselves.
• 3. is more complex to implement. In the simplest form, it may cover some use-cases, but will always exclude some (like stripping only attributes or only certain tags, etc). Or it would quickly turn into a full-blown HTML-sanitizer feature that is controlled through search-parameters, which sounds like a bad plan to me. :D.

[gmourier]

Hello Berkes! 👋

Thanks for the great explanation, this is a great analysis and the pros and cons are well explained.

Indeed, it is a difficult choice to make if MeiliSearch handles sanitization. However, there are a few things that strike me.

In my opinion, the client code should do its own sanitization before sending data or before displaying data.

This is a complex subject, as MeiliSearch is not a sanitizer but a pure search engine, how can we guarantee that our choices on the engine side are the right ones for most of the use-cases knowing that there are libraries on the client side allowing to do custom sanitization rules according to the user's need?

If we decide to leave sanitization on the client-side, I totally agree with your idea to document this and to warn users to use client-side sanitization against XSS vulnerabilities. In fact, this is exactly what Algolia does in a dedicated page.

On a related topic, we may decide to not make it possible to search on HTML tags to increase the relevance of search results. We could do this in the default tokenizer. In case of someone needs to be able to search specifically for html tags, we could provide a tokenizer that allows this or find a way to load a custom tokenizer made by the user.

Can you elaborate on the fact that you can't control the source? I'm curious about your use case!

Thanks for this feedback. I am always willing to discuss and debate this idea!

[berkes]

Can you elaborate on the fact that you can't control the source? I'm curious about your use case!

Certainly!

I'm building a bot that indexes job postings and job-candidates from the so-called Fediverse (mastodon and the likes). Source here: https://github.com/Flockingbird/hunter2/

The HTML that this bot receives can be dirty, broken, or full of XSS. So we really need to sanitize it somewhere along the path.
This issue here was mostly to find out if my ideas match with those of MeiliSearch.

When I look at other storage backends, here is how I would do it there:

• Postgres (or any RDBMS): sanitize on output on the server-side app that presents the PG data to the clients.
• ElasticSearch: use the built-in sanitization that comes with the highlighter so that clients can interact with the database directly.

I presumed MeiliSearch fits more in the ES line, esp since it advises the following:

MeiliSearch queries should be sent directly from the front-end. The more proxy there is between MeiliSearch and the end-user, the less fast queries and thus search-experience will be.
from Phylosophy » Anti-pattern.

But this only works if the data in Meilisearch is reasonably clean, or the client has tooling to sanitize client-side.

From this, I presumed there may be plans to offer some in-database(index) sanitzation. In which case it would make more sense for me to spend the effort helping with that, than to implement sanitisation in the bot or in the client-side.

[gmourier]

Thank you @berkes for explaining your use case in detail! Really appreciated.

Indeed, for the moment, it would be necessary to do this at the bot level or at the client side. Concerning the antipattern, the best is to do it on the bot side because it will not directly impact the users doing the searches since the data will already be clean and ready to be displayed.

It is possible that one day we will also decide to add a search pipeline that would allow us to handle the sanitization case upstream of the engine.

The engine is always evolving and the ideas/feedback from our users are important to us. We have a tool that allows us to collect the ideas of our users and I invite you to add yours so that other users can vote later and put it forward. Features ideas board 🙂

[berkes]

I've added it to the ideas board. I was not aware of that board, sorry.

I you prefer, I can close this issue, but if you want to close it, by all means: feel free.

[gmourier]

Thanks @berkes, I'm closing this issue!