Iterator.toIndexedSeq calls hasNext() multiple times in Scala 2.13.0-RC1 (preview)

[xerial]

Problem:

• Iterator.toIndexedSeq skips the first element if Iterator.hasNext changes some internal state.
• Iterator's hasNext() will be called multiple times, so if the code is expecting that hasNext() will be called only once and changing some internal state, some elements will be skipped in the generated IndexedSeq.
  • For example, JDBC ResultSet.next() will move the internal cursor, so Iterators for traversing JDBCResultSet will be broken.
  • There has been no guarantee nor any specification that Iterator.hasNext() must return the same result for subsequent calls before calling next() https://docs.oracle.com/javase/8/docs/api/java/util/Iterator.html#hasNext--
• This behavior is introduced after Scala 2.13.0-M5.
• Iterator.haxNext() implementation that are not idempotent will produce broken results with this change, even though ideally all of the implementations of Iterator.haxNext() should be idempotent.

Code Example

class MyIterator extends Iterator[Int] {
   private var i = 0
   def hasNext: Boolean = {
      val flag = i < 2
      i += 1
      flag
   }
   def next:Int = i
}

object Example {
  def main(args:Array[String]): Unit = {
    val s = new MyIterator().toIndexedSeq
    println(s.mkString(", ")) // Expected 1,2, but 2.13.0-pre-59975bb produces 2
  }
}

This code is in https://github.com/xerial/scala-bug-repro

How to reproduce:

$ git clone https://github.com/xerial/scala-bug-repro
$ cd scala-bug-repo
$ sbt run
...
2 

For the other Scala versions

$ sbt 
> ++ 2.12.8!
> run
1, 2

This behavior is found at #11453

Cause

VectorBuilder.addAll calls the Iterator.hasNext to check whether it should advance to the next block or not:

while (it.hasNext) { // The first call is here
  advanceToNextBlockIfNecessary()
  lo += it.copyToArray(xs = display0, start = lo, len = display0.length - lo) // The next call happens here
}

Related change: scala/scala#7588

[Sciss]

It should probably be said though that the contract of Iterator does not forbid hasNext to be called any number of times. I would say if your iterator's behaviour changes if hasNext is called multiple times, it is a bug in your code.

[xerial]

@Sciss There has been no guarantee nor any specification that Iterator.hasNext() must return the same result for subsequent calls before calling next() https://docs.oracle.com/javase/8/docs/api/java/util/Iterator.html#hasNext--

[xerial]

It also means hasNext() is checked multiple times in toIndexedSeq, so some redundancy is introduced in the recent change.

[Sciss]

@xerial I don't disagree that calls might be redundant, but I think you are mistaken to assume hasNext can return different answers when called multiple times (without popping elements through next()). Indeed you must even be able to call next() without prior calls to hasNext, this is not forbidden by the spec. Iterator of course is a tricky structure because for any non-trivial iterator, hasNext must advance the internal state.

There are many discussions on the Internet, e.g. https://coderanch.com/t/378826/java/Iterator-hasNext-idempotent

I insist your code is buggy if hasNext cannot be safely called multiple times.

[xerial]

Actually fixing my code is not a big deal, but I don't know how many Iterator implementations are assuming that hasNext() will be called only once for each move.

[xerial]

I'm not arguing Iterator.haxNext() must be called only once.

Rather I mean releasing Scala 2.13.0-RC1 with this behavior change will reveal how many Iterator.hasNext() implementations are non-idempotent, and we need to ask all dependent Scala/Java library maintainers to fix such Iterator implementations.

I don't think this is a good timing to do so, because Scala 2.12 to 2.13 is already a big jump and has been causing many pains for library maintainers.

[xerial]

The first hasNext call happens in VectorBuilder:

        while (it.hasNext) { // The first call is here
          advanceToNextBlockIfNecessary()
          lo += it.copyToArray(xs = display0, start = lo, len = display0.length - lo)
        }

And IterableOnce.copyToArray will call hasNext again:

  def copyToArray[B >: A](xs: Array[B], start: Int, len: Int): Int = {
    val it = iterator
    var i = start
    val end = start + math.min(len, xs.length - start)
    while (i < end && it.hasNext) { // the second call 
      xs(i) = it.next()
      i += 1
    }
    i - start
  }

[xerial]

@joshlemer Do we have any workaround to avoid calling Iterator.hasNext() multiple times in Vector? This optimization is introduced at scala/scala#7588

[Ichoran]

I don't consider this a bug. Iterators that can't consistently present what elements they have aren't necessarily going to end up with a maximal representation as a sequence. Relying on any particular series of calls to hasNext and next isn't specified in the API and shouldn't be supported. It's too fragile.

In this case there's a simple recursive function or do-while loop that avoids the extra call (at the cost of a bit of extra work if the iterator starts out empty), but I don't think we should promise anything. Relying on fragile undocumented behavior for correctness is almost never a good way to go.

[xerial]

@Ichoran Yes. It's not a bug, rather it's a behavior change that was introduced just 7 days ago scala/scala#7588 that will affect all poorly implemented Iterators.

And finding the cause of such behavior change will be tricky because only the first element will be missing without any error. Some DBMS implementation that might be relying on the previous behavior will cause incorrect results. This kind of bug will be hard to find.

[Ichoran]

Workaround

class SingleCallIterator[A](underlying: Iterator[A]) extends Iterator[A] {
  private[this] var state = 0
  def hasNext = state > 0 || {
    val ans = underlying.hasNext
    state = if (ans) 1 else -1
    ans
  }
  def next: A = {
    val ans = underlying.next
    state = 0
    ans
  }
}

Wrap your unruly iterators in that.

[Ichoran]

Another workaround:

Replace

myUnrulyIterator.toIndexedSeq

with

val b = IndexedSeq.newBuilder[Foo]
while (myUnrulyIterator.hasNext) b += myUnrulyIterator.next
b.result

[xerial]

@Ichoran My point is how many such fixes we need to make against the existing Scala/Java libraries because of this behavior change.

It's not only about my code.

I don't have any numbers about how many libraries will start to produce missing results silently.

[joshlemer]

@xerial actually the regression/change is a result of https://github.com/scala/scala/pull/7110/files#diff-59f3462485b74027de4fd5e9febcc81bR744, not the PR merged a few days ago.

I don't have a very strong opinion whether this should be considered a bug or not, but if it will be considered a bug I would be willing to help think/write an alternative implementation which calls hasNext just once.