tracking issue: array API standard document

[rgommers]

Timeline (high-level)

• Aug 31: array RFC draft complete enough for final internal review
• Sep 15: array RFC published + array-api repo public

Array API standard document intermediate steps:

Aug 10:

• Broadcasting (status: merged)
• Mutability (status: pending discussion, see Copy-view behaviour and mutating arrays #24)
• Dtypes and casting (status: merged)
• API signatures: statistical reductions (status: merged)

Aug 17:

• Purpose and scope (status: scope part (Add Scope and Stakeholders sections #22) merged, "how to adopt" (Add a description of how to adopt the API #33) under review)
• Type signatures (status: merged)
• Static typing (status: Add section on static typing #62 for review)
• Indexing (status: under review)
• API signatures: linear algebra (status: first set of functions merged, others like svd and qr TODO)
• API signatures: sorting (status: under review)
• API signatures: searching (status: merged)
• Accuracy (status: under review)
• Methodology: data on existing API design & usage (status: under review)

Aug 24:

• Array duck typing
• API signatures: array creation (status: Add use cases section #35 merged, a few functions (mainly meshgrid) to be added)
• API signatures: array manipulation (status: under review)
• API signatures: set functions (status: under review)
• API signatures: logical functions (status: merged)
• C API (status: WIP content in issue WIP: strategies for C/Cython API usage #51)
• Use cases (status: merged)
• Assumptions (status: merged)
• API signatures: math, non element-wise

Aug 31:

• Array object (status: Add array object specification #53 for review)
• Future API standard evolution (status: under review, see Add section on future API standard evolution #37; for the actual process see https://github.com/data-apis/workgroup/issues/22)
• Parallelism (status: Add content to the Parallelism section. #61 for review)
• Data interchange mechanisms (see Add data interchange section, using DLPack #34)
• Verification - test suite (status: merged)
• API: constants (status: merged)
• versioning (status: under review, see Add section on future API standard evolution #37)

Added later:

• Python operators, and equivalent dunder methods and library functions/methods (see Proposal to standardize element-wise arithmetic operations #9)
• Device support (see device support #39)

[rgommers]

Current status:

There are 11 open PRs that are either (close to) ready for merging or that need input on specific choices to be made. I'll comment only on the ones that need action:

• Add specifications for set functions #25 (set functions): needs a decision on sort order for unique. In particular input from TensorFlow needed (@alextp). update: merged, opened Specifying a sort order when returning unique values #40 for the TODO
• Add specifications for sorting functions #19 (sort functions): needs update from @kgryte on stable sorting decision.
• Add Scope and Stakeholders sections #22 (scope section): needs an update from @rgommers for the TBD about device support.
• Add a description of how to adopt the API #33 ("how to adopt the API" section): needs review
• Add data interchange section, using DLPack #34 (data interchange / DLPack): needs review
• Add accuracy document #29 (accuracy): needs completing, and input on how to specify accuracy for what's outside of the IEEE-754 spec.
• Add section on tooling #18 (tooling / methodology): needs a bit more detail on array-api-comparison and how the data produced by both tools is used for decision making.

Still TODO:

• second set of linalg functions
• second set of array creation & manipulation functions
• constants
• PRs for topics for which content is being discussed in an issue:
  • mutability (see Add section on copy-view behaviour and mutability #66)
  • static typing (partly addressed in section Function and method signatures)
  • parallelism (see Parallelism - what do libraries offer, and is there an API aspect to it #4)
  • future API standard evolution & versioning
• Topics that we don't yet have content for:
  • Indexing (likely complex)
  • Array object (should be straightforward)
  • Purpose (take from announcement blog post)
  • C API (see WIP: strategies for C/Cython API usage #51)
• Check what's in array libraries but not in the standard - did we leave out anything we shouldn't have? --> see List of APIs currently in (or proposed to be included in) the specification #43

[rgommers]

A text search for "TODO" on master:

$ grin -C2 TODO

./API_specification/index.rst:
   28 - .. _api-versioning:
   29 - 
   30 : TODO: add `__array_api_version__` (or similar) version here, and the way to
   31 +       retrieve it - see https://github.com/data-apis/workgroup/issues/24
./API_specification/set_functions.md:
   37 -     -   If `True`, the function must sort the unique elements in ascending order before returning as output. If `False`, the function must sort the unique elements in the same order that they occur in `x`. Default: `False`.
   38 - 
   39 :         _TODO: sort order needs discussion. See [gh-40](https://github.com/data-apis/array-api/issues/40)_
   40 + 
   41 + #### Returns
./design_topics/data_interchange.md:
   57 - 
   58 - 
   59 : TODO: design an appropriate Python API for DLPACK (`to_dlpack` followed by `from_dlpack` is a little clunky, we'd like it to work more like the buffer protocol does on CPU, with a single constructor function).
   60 + 
   61 : TODO: specify the expected behaviour with copy/view/move/shared-memory semantics in detail.
   62 + 
   63 + 
./design_topics/device_support.md:
    3 - # Device support
    4 - 
    5 : TODO. See https://github.com/data-apis/array-api/issues/39
    6 + 
./future_API_evolution.md:
    6 - the process documented at:
    7 - 
    8 : TODO: link to specific document in https://github.com/data-apis/governance once
    9 + it is available.
   10 + 
./purpose_and_scope.md:
   42 - 
   43 - The concrete set of functionality that is in scope for this version of the
   44 : standard is shown in this diagram (_TODO: update after deciding on how optional
   45 + extensions are dealt with_):
   46 + 
./verification_test_suite.md:
    5 - ## Measuring conformance
    6 - 
    7 : TODO: explain how to use the test suite to measure and report on the level of
    8 + (non-)conformance.

EDIT: some of these are already decided on and can easily be fixed up. The important ones that need significant work are:

• DLPACK: design a new Python API, specify move/copy/shared-mem semantics better
• device support
• how to deal with optional extensions

EDIT 2: down to:

$ grin -C2 TODO
./API_specification/set_functions.md:
   37 -     -   If `True`, the function must sort the unique elements in ascending order before returning as output. If `False`, the function must sort the unique elements in the same order that they occur in `x`. Default: `False`.
   38 - 
   39 :         _TODO: sort order needs discussion. See [gh-40](https://github.com/data-apis/array-api/issues/40)_
   40 + 
   41 + #### Returns
./design_topics/data_interchange.md:
   57 - 
   58 - 
   59 : TODO: design an appropriate Python API for DLPACK (`to_dlpack` followed by `from_dlpack` is a little clunky, we'd like it to work more like the buffer protocol does on CPU, with a single constructor function).
   60 + 
   61 : TODO: specify the expected behaviour with copy/view/move/shared-memory semantics in detail.
   62 + 
   63 + 
./design_topics/device_support.md:
    3 - # Device support
    4 - 
    5 : TODO. See https://github.com/data-apis/array-api/issues/39
    6 + 
./purpose_and_scope.md:
   42 - 
   43 - The concrete set of functionality that is in scope for this version of the
   44 : standard is shown in this diagram (_TODO: update after deciding on how optional
   45 + extensions are dealt with_):
   46 + 

[rgommers]

I think we're good here, all PRs are merged and there are TODO comments for the things that are left to do. No need for this tracking issue anymore.