BUG: inconsistent concat casting EA vs non-EA

[jbrockmendel]

• closes #xxxx
• tests added / passed
• passes black pandas
• passes git diff upstream/master -u -- "*.py" | flake8 --diff
• whatsnew entry

Because we dont have 2D EAs, we end up passing axis=0 to concat_compat from internals.concat. As a result we drop empty arrays only when EAs are present. This leads to dtypes being preserved in EA cases while being cast to object in non-EA cases.

One solution is just to support 2D EAs. But since I like the dropping-empties behavior better anyway, this is good too.

[jreback]

can you add a whatsnew note

[jorisvandenbossche]

BTW, originally this is not directly related to having 2D EAs or not. The inconsistency already existed between DataFrame vs Series (because when concatting Series/SingleBlockManager, axis=0 is used for any dtype). So when concatting Series, empties are ignored, and when concatting DataFrames, empties are not ignored.
Only, now with extension dtypes (and maybe before with non-consolidatable block as well), also for DataFrame columns involving EAs, empties are dropped, creating an additional inconsistency within concatting DataFrames.

But so we should also tackle the general inconsistency between DataFrame and Series (which this PR might be doing already, but that should probably be made more explicit).

For the actual behaviour we want, it's also an option to go for the DataFrame behaviour of not dropping empties.
I think there are also good arguments for this behaviour: the resulting dtype of a concat-operation only depends on the input dtypes, and not on the exact content (the exact values, how many values (shape)). In general we want to get rid of value-dependent behaviour.

[jbrockmendel]

this is not directly related to having 2D EAs or not

Yes it is. We pass axis=0 when we have EAs as a kludge bc they aren't 2D.

For the actual behaviour we want, it's also an option to for the DataFrame behaviour of not dropping empties [...] In general we want to get rid of value-dependent behaviour.

That's not unreasonable.

[jreback]

hmm I though we did drop empties on DataFrame, but i guess not. yeah would be +1 on making that change for consistency.

[jorisvandenbossche]

I think this was merged a bit too quickly. As noted above (#38843 (comment)), IMO there is much to say for solving it the other way around: changing Series behaviour to match the DataFrame behaviour (of not dropping empties).

---

this is not directly related to having 2D EAs or not

Yes it is. We pass axis=0 when we have EAs as a kludge bc they aren't 2D.

I only wanted to mention that, even without EAs at all, we have an inconsistency in behaviour caused by the axis=0 check. And this inconsistency already existed before EAs, and it's something that needs to be solved regardless. But yes, for EA dtypes, this causes an additional inconsistency, as I acknowledged.

BTW, as general advice, I would recommend to stop constantly calling 1D EAs a "kludge", it's not helping the discussion in any way.

[jbrockmendel]

BTW, as general advice, I would recommend to stop constantly calling 1D EAs a "kludge", it's not helping the discussion in any way.

The 1D EAs themselves are not the kludge, it is the workarounds that become necessary as a result of 1D EAs. In this case passing axis=0 to concat_compat.

[jorisvandenbossche]

(Understood, but in general comments, it's not always clear to the reader which of both you mean, so I keep my advice ;))

Anyway, more importantly, can I get some response on the actual issue that I raised (the API change for concat with empties for DataFrame vs Series) ?

[jbrockmendel]

the API change for concat with empties for DataFrame vs Series

I'm open to it.