Async Api

[cillierburger]

Are there any plans to support an async version of the API, or some kind of multi-threaded access ?

[auxten]

Hi @cillierburger
chDB could run in multi-threaded env, but you need to reuse the Session.
Could describe more details of your needs?

[cillierburger]

Hi @auxten ,

Sure, we have a tool that builds and runs ETL pipelines on top of clickhouse. This tool potentially runs multiple different queries "concurrently".

I'm exploring changing the adapter to use chDb, at first for running test cases. After that we would like to considering slicing out parts of our pipeline and running them using chDb.

So the idea would be that something like this:

tasks.append(session.query('SELECT sleep(2)'))
tasks.append(session.query('SELECT sleep(2)'))
tasks.append(session.query('SELECT sleep(2)'))

await asyncio.gather(*tasks)

Would complete in ~ 2s

I found the "parralel" test in the repository, but this just seems to prove that the lock is being used.

Happy to provide a more complete example if useful.

Thanks! 🙂

https://github.com/chdb-io/chdb/blob/main/tests/test_parallel.py

[cillierburger]

Hi, just checking in on this again.

Is what we are trying to do possible with ChDB at this stage ? Is there maybe a different way to get the queries to run "concurrently" ?

Thanks again. Any input appreciated

[auxten]

@kafka1991 is working on the related feature, maybe he could answer this question

[kafka1991]

Hi @cillierburger
The PR #405 that adds support for concurrent session access has been completed. We plan to have it reviewed and merged before the next release. We may also publish an experimental version in advance to gradually roll out this feature.

[cillierburger]

@kafka1991 @auxten this is awesome,thank you ! Looking forward to testing it out