ENH: Add new implementation of DataFrame.stack

[rhshadrach]

• closes #xxxx (Replace xxxx with the GitHub issue number)
• Tests added and passed if fixing a bug or adding a new feature
• All code checks passed.
• Added type annotations to new arguments/methods/functions.
• Added an entry in the latest doc/source/whatsnew/vX.X.X.rst file if fixing a bug or adding a new feature.

We can either deprecate the current implementation in 2.1, or wait until 2.2. It's slightly more gradual to wait until 2.2; users in 2.1 could opt into the new behavior without any warnings.

[jorisvandenbossche]

How do you see the deprecation/update process? If we don't yet warn in 2.1, I doubt that many users will already try out v3=True. And for warning, do you think it would be possible to detect when to warn? Or were you planning to warn for all cases to say that people should specify v3=True to get rid of the warning? (ideally we would only warn if v3 would actually give a different result, otherwise it will be a noisy annoying warning for those (the majority?) use cases that don't need it)

[jbrockmendel]

Instead of a v3 keyword could use the "future" option from #53025

[rhshadrach]

@jorisvandenbossche

How do you see the deprecation/update process? If we don't yet warn in 2.1, I doubt that many users will already try out v3=True. And for warning, do you think it would be possible to detect when to warn?

I was thinking we add the deprecation in 2.2; I agree not many will use v3=True in 2.1, but I don't see any negatives of waiting until 2.2 to deprecate.

For doing the deprecation, we could get really complicated. I think behavior is the same when:

• stacking where columns are a non-MultiIndex;
• stacking where columns are a MultiIndex and every combination between stacked and non-stacked levels occurs and dropna=False;
• stacking where columns are a MultiIndex and every combination between stacked and non-stacked levels occurs and dropna=True and the input does not have any NAs; or
• stacking where columns are a MultiIndex, dtypes are all float, dropna=True, and the input does not have NaN values

However, since we'd be deprecating dropna we'd still need to warn in the 2nd bullet above.

@jbrockmendel

Instead of a v3 keyword could use the "future" option from #53025

I'd definitely be on board with this.

[rhshadrach]

@jbrockmendel @jorisvandenbossche @mroeschke assuming the CI agrees I think this is ready for review.

@jbrockmendel - regarding using a future.* option here, I think I'd prefer the additional argument since this is just on a single method. But if we're wanting something like "all opting into future behaviors is done via future.* options", I'd be on board with that and happy to rework that here. Thoughts?

[jbrockmendel]

But if we're wanting something like "all opting into future behaviors is done via future.* options"

Definitely not

[rhshadrach]

linter didn't like that this was named factorize now that we import factorize as well, and rightfully so.

[rhshadrach]

Thanks @mroeschke - I think this is good for another look.

[mroeschke]

Looks pretty good. Just a code check failure

[mroeschke]

Awesome, thanks @rhshadrach