to_pandas in PandasQueryCompiler loses meta information about indices and columns

[dchigarev]

System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Windows 10
• Modin version (modin.__version__): 0.7.3+198.ge4e3fecb
• Python version: 3.7.5
• Code we can use to reproduce:

if __name__ == "__main__":
    import modin.pandas as pd
    import numpy as np

    data = {"A": np.arange(256)}

    index = pd.MultiIndex.from_tuples((i, i*2) for i in np.arange(257)).drop(0)

    md_df = pd.DataFrame(data, index=index)

    print("md_df.index:", len(md_df.index.levels[0]))  # md_df.index: 257 
    print("pd_df.index:", len(md_df._to_pandas().index.levels[0]))  # pd_df.index: 256

Describe the problem

to_pandas loses information about index levels which is useful for operations that operates with indices levels internally (like unstack)

[dchigarev]

I figured out that to_pandas does not consider indices and columns from modin_frame at all, only from partitions. For example if modin frame has columns = ['A', 'B'] but in partitions columns = [1, 2, 3] (different columns lengths could appear because of incorrect function implementation) then to_pandas will return us the DataFrame with columns = [1, 2, 3].

More explicit example why it is important:
Many tests in modin uses df_equals to compare modin_result with pandas_result. Under the hood df_equals translates modin_df to pandas and after that compares its like two pandas dataframes. Since to_pandas does not consider modin frame indices, we will never know if modin frame itself has correct indices/columns in such comparison.

For example, let's take nlargest function, its tests pass ok, but when we would like to use its result without converting it to pandas, we will see that it's incorrect (#1734).

Many tests started to fail after I fixed that to_pandas does not use modin frame indices in the conversion to pandas (#1729 #1731 #1732 #1733 #1734), I've fixed most of them in #1727.

@devin-petersohn please take a look, I guess that all the PRs with new pandas API functions implementation should be blocked for now, because it can't be properly tested