gh-105201: Add PyIter_NextItem()

[erlend-aasland]

Return -1 and set an exception on error; return 0 if the iterator is
exhausted, and return 1 if the next item was fetched successfully.

Prefer this API to PyIter_Next(), which requires the caller to use
PyErr_Occurred() to differentiate between iterator exhaution and errors.

Co-authered-by: Irit Katriel [email protected]

• Issue: PyIter_Next has ambiguous return value  #105201

---

📚 Documentation preview 📚: https://cpython-previews--122331.org.readthedocs.build/

[picnixz]

Small nitpicks but those are mostly documentation related. By the way, is there any plan to have something working like next(iter, sentinel) with a sentinel support? or is there some already existing? (or is it even useful for CPython & the universe? I like using next(..., sentinel) when I don't know whether I have an item or not, but I'm not sure whether this pattern has a simple equivalent in CPython).

[dg-pb]

lgtm

[erlend-aasland]

Thanks for your helpful remarks, everyone. I found a bug and updated the PR. I also noticed we should add the tests to Modules/_testcapi/abstract.c and Lib/test/test_capi/test_abstract.py instead of their current location. I'll update the PR shortly with this adjustment.

[vstinner]

LGTM

[picnixz]

I'm very sorry for the late review Erland but I was a bit busy this w-e. Those are nitpicking comments, so you can ignore them if you want (just me being compiled in -Wextra -Wpedantic).

[markshannon]

The return value should the sole means of determining subsequent control flow, and the output parameter should provide only necessary data, IMO.

In other words, item should only be guaranteed to be set if PyIter_NextItem returns 1, and the control flow in the loop should only test the return value, not item.

Which means the example use needs to rewritten from:

      PyObject *item = NULL;
      while (PyIter_NextItem(iter, &item)) {
          if (item == NULL) {
              Py_DECREF(iter);
              goto error;
          }
          do_something(item);
          Py_DECREF(item);
      }
      Py_DECREF(iter);

      int err;
      while (1) {
          PyObject *item;
          int ret = PyIter_NextItem(iter, &item)) {
          if (err <= 0) break;
          do_something(item);
          Py_DECREF(item);
      }
      Py_DECREF(iter);
      if (err < 0) {
          goto error;
      }

which might be a couple of lines longer, but has only one test in the loop, not two.

We have three options, regarding the value of item if PyIter_NextItem returns <= 0:

1. Set it to NULL
2. Do not touch it
3. Leave it undefined

1 requires work, but adds no additional information.

2 provides options for the user to provide some default and have it unchanged once the iterator exhausted. I don't see much value in this, but it allows a use case that 1 does not.

3 is best for performance and it leaves us free to implement PyIter_NextItem efficiently if the underlying mechanisms change

I'd prefer to specify 3 for now, as it doesn't prevent us from choosing 1 or 2 later on.

[erlend-aasland]

We have three options, regarding the value of item if PyIter_NextItem returns <= 0:

The C API WG already decided to go for alternative 1: set it explicitly to NULL.

Which means the example use needs to rewritten from [...]

Actually, I think I'll drop the example from this PR. We can add it in a following PR, if it is needed. I added @zooba's example from the C API WG repo.

[vstinner]

LGTM

[erlend-aasland]

Thanks for all the reviews!

[encukou]

That failure is unrelated, the iOS buildbot is green now.