Tell us more about your dump feature usage

[gmourier]

Hello, community!

To improve the upgrade of a Meilisearch instance, we would like to know how important dumps are for you and your frustrations with using them.

If you don't use them, what are the reasons?

[shivaylamba]

@gmourier thanks for starting this discussion!
Usecase:
I recently got a dump from a v0.24 Meilisearch instance, and then indexed it to a v0.28 instance.

Issue:
I am not entirely sure how to actually change the contents from the dump. If we are required to do that, we could add documentation on how we can use the dump file and probably convert it into a JSON file type.
The reason behind this is that if we are transitioning from a really old version, certain data might require transformation( for example: nested field support addition in v0.27).

Also currently I used the Meilisearch CLI to actually index data to another instance using the dump file.

Verdict:
Dumps are easy to use but just sharing documentation on how to easily edit the content in the dump would be a good addition.
+
CLI tool to index data from a dump could potentially be made better

• adding support for being able to index data through dump via the MeilisearchJS API.

[irevoire]

Hey, @shivaylamba, it’s cool to see you around!

For this discussion, it’s a bit out of scope, but it would be a pretty good addition to meilisearch! And we have all the data available in jsonlines. Thus, it’s doable.

[gmourier]

This message that occurs when a version cannot start with a data.ms from an older version could easily be improved to unblock users and help them.

Error: Expected Meilisearch engine version: 0.28.0, current engine version: 0.29.0. To update Meilisearch use a dump.

Some ideas:

• Rewrite it to make it more informative about what is happening.
• Link to a documentation page explaining how to use the dump feature.

[MihaiNueleanu]

We don't use the dump feature at all.

The reason: meilisearch is not a "primary" database, so we treat it as ephemeral (in a sense). To bootstrap meilisearch after a version upgrade, we use the same scripts that we wrote on our first data migration from postgres. Which are the same scripts that we run periodically to ensure consistency against the primary database (postgres).

We never got into the dumps feature because of that. It seems like an easier workflow to use our own, already existent sync scripts.

[Fronix]

We do pretty much the same, our meilisearch indexes are just a small copy of a larger database. So whenever we have an update we just reindex everything, since we use code to reindex it will always be the same.

[MihaiNueleanu]

We have a decent-sized database (~5 mil records total in meilisearch). And we're doing the restore rather "inefficiently". But the 30 minutes runtime for a full sync + cleanup is acceptable for us at this point.

[gmourier]

Hi @Fronix and @MihaiNueleanu 👋

FYI, if you don't delete the data, v0.29 should speed up the syncing; we introduced some optimization for that particular case.

[MihaiNueleanu]

@gmourier that's great, thanks for the heads up! You guys are delivering really nice performance improvements with every new release, which is really really great!

[juchom]

Hi guys,

For my use case I'm using Meilisearch with docker image : using dumps is not friendly at all and I suggested to update the CLI a few months ago here : #396

I gave up on using dumps for now and I have a script that rebuild all indexes from postgre like @MihaiNueleanu, it's ok for now beceause I have low volumes of documents, but I won't keep this solution in the future.

[curquiza]

Hello @juchom thanks for this feedback and the previous one!

So far, our CLI is something we want to improve more generally than just for the dumps. Unfortunately not the priority for 2022 since we want to stabilize Meilisearch first, but definitely a good suggestion for a new CLI!

Thanks again!