Dask engine can only have 4 partitions - Pull Requests #2492

[zhfanrui]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): CentOS release 6.5 (Final)
• Modin version (modin.__version__): 0.8.2 installed from conda-forge channel
• Python version: 3.7.3
• Code we can use to reproduce:

from distributed import Client
client = Client(n_workers=10)
import modin.pandas as pd
df = pd.read_csv(filename)

Describe the problem

Hi there, I recently got a really huge CSV file to read and found that no matter how I declare the number of the worker by using

from distributed import Client
client = Client(n_workers=10)
import modin.pandas as pd

The partition of the Dataframe object is still 4 causing the efficiency really low.

So I reviewed some of the code briefly (actually it takes me a whole afternoon to track the problem) and found pd.DEFAULT_NPARTITIONS is 4 defaulted (in file modin/modin/pandas/__init__.py) and the init function (_update_engine) didn't change the value of it when using Dask engine.

Hence, I change pd.DEFAULT_NPARTITIONS after importing pd by

pd.DEFAULT_NPARTITIONS = 10

and it works well!

So I decided to add a line of code to solve this problem (See Pull Requests #2492 ). However, it failed in test_read_csv_datetime. Really confused about that.

Source code / logs

[devin-petersohn]

Hi @zhfanrui, thanks for the report!

I can reproduce the issue, and your PR does fix the issue. I am not sure why the tests failed in that way because I do not see the issue when locally running the tests with that change.

Relevant Test Traceback:

  File "/modin/modin/pandas/__init__.py", line 138, in _update_engine
    num_cpus = len(dask_client.nthreads())
TypeError: object of type 'coroutine' has no len()

Any ideas here?

[YarShev]

@devin-petersohn , when running locally I got the same error. Maybe that is related to how we execute _update_engine via Engine.subcribe. @vnlitvinov , are there any thoughts on this?
The log from the PR can be found here (link).

[devin-petersohn]

@YarShev which tests did you run to reproduce? I ran the I/O tests and did not have the error.

[YarShev]

@devin-petersohn , I ran pytest modin/pandas/test/test_io.py::TestReadCSV::test_read_csv_datetime. After some digging it was found out we fall down into this place. That's why we can see the error.