Add a hidden column __natural_order__.

[ueshin]

Adding a hidden column __natural_order__ for ordering the rows as it is, especially for window-like operations like cumxxx.

[codecov-io]

Codecov Report

Merging #1146 into master will decrease coverage by <.01%.
The diff coverage is 98.07%.

@@            Coverage Diff             @@
##           master    #1146      +/-   ##
==========================================
- Coverage   95.22%   95.22%   -0.01%     
==========================================
  Files          35       35              
  Lines        7062     7055       -7     
==========================================
- Hits         6725     6718       -7     
  Misses        337      337

Impacted Files	Coverage Δ
databricks/koalas/series.py	96.46% <100%> (-0.03%)	⬇️
databricks/koalas/indexing.py	94.22% <100%> (ø)	⬆️
databricks/koalas/generic.py	96.18% <100%> (ø)	⬆️
databricks/koalas/groupby.py	91.6% <100%> (-0.02%)	⬇️
databricks/koalas/frame.py	97.02% <100%> (-0.02%)	⬇️
databricks/koalas/internal.py	96.5% <85.71%> (+0.07%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0865228...3d48413. Read the comment docs.

[softagram-bot]

Softagram Impact Report for pull/1146 (head commit: 3d48413)

⚠️ Copy paste found

ℹ️ frame.py: Copy paste fragment on line 5792 shared with ../namespace.py:

              on: Union[str, List[str], Tuple[str, ...], List[Tuple[str, ...]]] = None,
              left_on: Union[str, List[str], Tuple[s...(truncated 273 chars)

ℹ️ frame.py: Copy paste fragment inside the same file on lines 7294, 7377:

        # TODO: there is a similar logic to transpose in, for instance,
        #  DataFrame.any, Series.quantile. Maybe ...(truncated 1065 chars)

ℹ️ frame.py: Copy paste fragment inside the same file on lines 4913, 4935:

            sdf = self._sdf.select(
                self._internal.index_scols +
                [self._internal.scol_for(idx...(truncated 505 chars)

ℹ️ groupby.py: Copy paste fragment inside the same file on lines 1153, 1224:

            window = Window.partitionBy(groupkey_cols) \
                .orderBy(order_column, NATURAL_ORDER_COLUMN_NAME)
    ...(truncated 1017 chars)

ℹ️ groupby.py: Copy paste fragment inside the same file on lines 2058, 2104:

        sdf = sdf.withColumn('rank', F.row_number().over(window)).filter(F.col('rank') <= n)
        internal = _...(truncated 497 chars)

Now that you are on the file, it would be easier to pay back some tech. debt.

⭐ Change Overview

(Open in Softagram Desktop for full details)

⭐ Details of Dependency Changes

(Open in Softagram Desktop for full details)

📄 Full report

• Permalink: Full report for pull/1146

Impact Report explained. Give feedback on this report to [email protected]

[HyukjinKwon]

My concern is that it harms readability in the codes ... Now hidden columns are going around in the Spark DataFrame. But I guess we have no choice to deal with the natural order ..

I am merging this for now but I really think we should improve the readability .. at least we can have, for instance, sdf_without_hidden_columns property or another layer or util to control such things ..

[HyukjinKwon]

no big deal but we don't we just use a list to keep the order? I don't think it's likely to have a duplicated columns if that was the concern.

[HyukjinKwon]

It think StructType has names property for this line.

[HyukjinKwon]

Do we need to select the hidden columns here explicitly? I suspect it was selected since they are map-like operations (?). Maybe we should just don't do this in all code base if I am not mistaken ...

[ueshin]

Actually in most cases we can remove, but in some cases we can't, e.g., right after _cum, and window-like functions after #1151 is merged.

[HyukjinKwon]

I am merging this for now. I will think about how to clean up too ... we can address the comments together later.