Improve export speed

[tokee]

The current experimental fields-exporter in solrWayback issues standard cursorMark requests against the underlying Solr.

Assume we have 100 Solr nodes and page 1000 documents at a time (a reasonable approximation of the index at the Royal Danish Library which has 143 nodes).

In a classic Solr Cloud setup each cursorMark request for 1000 documents will cause processing at each Solr node, where each node will return 1000 document-IDs and sort values, but only 10 will be used on average (1000 documents / 100 nodes = 10 documents/node). This means there is a lot of wasted processing for each request. A local test gave an export with high variance in speed, from 12-130 records/second and high load across the Solr Cloud.

If the Solr nodes were queried sequentially, each cursorMark request would involve only a single Solr node and all 1000 documents returned would be used. This was tested against the setup at Netarkivet and the export speed here was ~1200 documents/second with load localized to a single Solr node. It would be quite feasible to query 1/4 of all nodes on parallel, which would increase export speed to ~30,000 documents/second in our not-so-hypothetical setup.

There is clearly much to gain by rewiring the SolrWayback export code. Two possibilities comes to mind:

External shard requests

Introduce an endpoint listing the shards in the Solr setup and add a shardID parameter to the export endpoint.

• Pro: Very easy to implement in SolrWayback
• Con: Puts the burden of merging (and grouping if that is required) the result on the caller

Internal shard requests

Add parallel explicit shard requests to the existing code, taking care to support grouping. Basically a mimic of Solr streaming with the crucial difference that export of stored fields is possible.

• Pro: Makes field export performant without any extra work for the end user
• Con: Not trivial to implement, needs consideration of Solr Cloud topology

[tokee]

While there is some bookkeeping when implementing the internal shard requests, the overall logistics is reasonably straight forward:

1. Use CLUSTERSTATUS to get the full topology
2. Get the true collection IDs by checking the alias section
3. For each true collection, get all shard IDs
4. For all shard IDs, create a streaming document provider, either by specifying the shard ID in the request or directly connecting to the base_url stated in the topology map
5. Create a merger for the shard-specific streams that uses the same order/grouping logic as the basic document provider

Note: Yes, this is basically a re-implementation of Solr streaming export. It might be easier to add support for stored fields to Solr streaming export :-)

Note 2: There should probably be a sanity check so that the current simpler exporter is used for "small" result sets.

[tokee]

I am working on this on branch 329_export_speedup. Most of the code has been written and is in the "write new unit test, fix error, repeat"-phase.

[tokee]

The current status on 329_export_speedup is that the code has been written and needs performance testing to see if it makes a positive difference.

[tokee]

Preliminary results are in: 10 times the performance for the many-shard index at the Royal Danish Library, less improvement for "not a lot of shards"-setups. Report at https://sbdevel.wordpress.com/2023/10/24/beware-the-cursormark-my-son/

[thomasegense]

*10 is massive!
We probably have to add a boolean to the solrwayback.property , value is default false (not enabled).
Large collections can then enable it. See you did that.

You have done large scale import and compare result? The ordering will be different I guess?
But just exporting ID's , sort and compare two files.

[tokee]

I did make a few small scale tests that showed that the same documents are exported but for larger scale there is a difference which I need to investigate before making a pull request.