SimpleStreamReader cannot read multibyte characters on buffer boundary

[gbrdead]

When reading character by character using UTF-8, SimpleStreamReader returns -1 (EOF) when it encounters an emoticon. The problem exists even when reading a lot of characters at once but is harder to reproduce (the four bytes of the emoticon must get misaligned with the buffer).

[gbrdead]

input.txt

The following code:

		try (Reader r = new SimpleStreamReader(new FileInputStream("input.txt"), StandardCharsets.UTF_8))
		{
			int c;
			while ((c = r.read()) != -1)
			{
				System.out.print((char)c);
			}
		}

does not create the same output as the input. I cannot simplify the example for the moment.

[jhy]

Good find, thanks!

I updated it so that when in this condition (a surrogate pair is straddling the buffer, and the requested len is not enough to read both characters), returns read length of zero. And handle that in the CharacterReader.

This saves having to track the last emitted high/low surrogate etc.

Note that reads of length 1 then may return 0 characters, and that r.read() is not implemented and won't work for surrogates correctly.

[jhy]

@gbrdead can you validate this?

[gbrdead]

Yes, it works for all the example data I have.