[EPIC] Automatically generate all function documentation from code

[alamb]

Is your feature request related to a problem or challenge?

Currently, when we add a new function to DataFusion library we have to remember to document that function in the documentation, but currently we have to remember this during code review.

This has a few downsides:

1. we have forgotten to document some functions
2. the documentation has drifted over time,
3. help text for various functions can only be found on the DataFusion website, and not, for example within the function itself.

@Omega359 has created the basic framework for automatically generating content in #12668

This ticket tracks the work required to port the rest of the documentation to programatic form

Tasks

• Automatically generate function documentation from comments / code #12432
• Improve description of function migration #12743
• Remove redundant aggregate/window/scalar function documentation #12745
• feat: Add alternative_syntax function for docs #13139
• Remove old static documentation function pages (Aggregate/Scalar) #12741
• Remove old static Window Functions page #13171
• Add support for DESCRIBE xyz; where xyz is a function. Ideally this would support not just the core functions in DF but also any added externally (for example those in https://github.com/datafusion-contrib/datafusion-functions-extra) and verify it works in the CLI
• Look into generating the function sections in expressions.md (it's a bit out of date)
• Make generate_function_docs.sh error if there is a function without documentation

Aggregate Function Documentation Migration

• Migrate documentation for aggregate functions from aggregate_functions.md to code #12827
• Port / Add Documentation for BitwiseOperation
• Port / Add Documentation for VarianceSample Port / Add Documentation for VarianceSample and VariancePopulation #12742
• Port / Add Documentation for VariancePopulation Port / Add Documentation for VarianceSample and VariancePopulation #12742

Scalar Function Documentation Migration

• [EPIC] Improve examples to make them easier to navigate #11172
• Migrate documentation for all string functions from scalar_functions.md to code  #12775
• Migrate documentation for all array functions from scalar_functions.md to code #12927
• Migrate documentation for all crypto functions from scalar_functions.md to code #12828
• Migrate documentation for all datetime functions from scalar_functions.md to code #12859
• Migrate documentation for all encoding functions from scalar_functions.md to code #12858
• Migrate documentation for all math functions from scalar_functions.md to code #12867
• Migrate documentation for all regex functions from scalar_functions.md to code #12868
• Migrate documentation for all string functions from scalar_functions.md to code #12934
• Migrate documentation for all unicode functions from scalar_functions.md to code #12935

Window Function Documentation Migration

• Port / Add Documentation for Lead
• Port / Add Documentation for RowNumber
• Migrate documentation for remaining window functions to window_functions.md #12936

[alamb]

Update is I have a few follow on PRs

• Remove old static documentation function pages (Aggregate/Scalar) #12741
• Automatically generate function documentation from comments / code #12432

Then I figure I will file one or two other tickets to see how easy it is to port other functions

And if all goes well I'll make a ticket storm to port the docs

[Omega359]

FYI I've finished up the string expression migration locally - just waiting on the updated code from #12742 to land before pushing a PR.

[alamb]

FYI I've finished up the string expression migration locally - just waiting on the updated code from #12742 to land before pushing a PR.

Thanks, I'll merge those as soon as they get a review (I can't merge them until they get an approval from another committer)

[Omega359]

No rush I just wanted to make sure no one was duplicating work needlessly