Provide streaming mode

[alexey-milovidov]

Describe the bug

I cannot successfully run scc on 20 TB dump of GitHub (231 219 569 files) without OOM on a server with 128 GiB RAM,
even if I run scc with csv output format.

[alexey-milovidov]

This is needed for ClickHouse/ClickHouse#15323

[boyter]

Oh that's an interesting one... not one I ever considered, since I have never tried running it over that much data in one go.

I guess a stream option for CSV and SQL works in this case well enough.

[boyter]

@alexey-milovidov if you are able to build from source you can get from master and try the new stream mode for csv

scc --format csv-stream

It should dump the results as they come in to standard output avoiding any memory issues (in theory).

[boyter]

Closing down as done and ready for release.