unicode encoding error callbacks

[doerwalter]

BPO	432401
Nosy	@malemburg, @doerwalter
Files	diff7.txt test_codeccallbacks.py: test script diff8.txt: PyUnicode_EncodeDecimal done diff9.txt diff10.txt test_codeccallbacks.py: test script which catches the bugs fixed in diff10.txt diff11.txt speedtest.py: test speed for encoding diff12.txt test_codeccallbacks.py: test script for diff12.txt test_codeccallbacks.py: Adds parameter and result test for the callbacks diff13.txt diff.txt diff.txt: Revised patch diff.txt: Patch V3: Renamed the encode function to include "unicode". a few fixes in Lib/encodings diff.txt: Patch V4: (enc, uni, pos, state) -> (out, newpos) communication and speedups diff.txt: Patch V5: Unicode encoding error handling callback registry diff.txt: Patch V6: Encoding and decoding for string and unicode done

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/malemburg'
closed_at = <Date 2002-09-02.13:19:15.000>
created_at = <Date 2001-06-12.13:43:03.000>
labels = ['interpreter-core']
title = 'unicode encoding error callbacks'
updated_at = <Date 2002-09-02.13:19:15.000>
user = 'https://github.com/doerwalter'

bugs.python.org fields:

activity = <Date 2002-09-02.13:19:15.000>
actor = 'doerwalter'
assignee = 'lemburg'
closed = True
closed_date = None
closer = None
components = ['Interpreter Core']
creation = <Date 2001-06-12.13:43:03.000>
creator = 'doerwalter'
dependencies = []
files = ['3365', '3366', '3367', '3368', '3369', '3370', '3371', '3372', '3373', '3374', '3375', '3376', '3377', '3378', '3379', '3380', '3381', '3382']
hgrepos = []
issue_num = 432401
keywords = ['patch']
message_count = 42.0
messages = ['36773', '36774', '36775', '36776', '36777', '36778', '36779', '36780', '36781', '36782', '36783', '36784', '36785', '36786', '36787', '36788', '36789', '36790', '36791', '36792', '36793', '36794', '36795', '36796', '36797', '36798', '36799', '36800', '36801', '36802', '36803', '36804', '36805', '36806', '36807', '36808', '36809', '36810', '36811', '36812', '36813', '36814']
nosy_count = 2.0
nosy_names = ['lemburg', 'doerwalter']
pr_nums = []
priority = 'high'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue432401'
versions = []

[doerwalter]

This patch adds unicode error handling callbacks to the
encode functionality. With this patch it's possible to
not only pass 'strict', 'ignore' or 'replace' as the
errors argument to encode, but also a callable
function, that will be called with the encoding name,
the original unicode object and the position of the
unencodable character. The callback must return a
replacement unicode object that will be encoded instead
of the original character.

For example replacing unencodable characters with XML
character references can be done in the following way.

u"aäoöuüß".encode(
"ascii",
lambda enc, uni, pos: u"&#x%x;" % ord(uni[pos])
)

[malemburg]

Logged In: YES
user_id=38388

Thanks for the patch -- it looks very impressive !.

I'll give it a try later this week.

Some first cosmetic tidbits:

• please don't place more than one C statement on one line
  like in:
  """
  + unicode = unicode2; unicodepos =
  unicode2pos;
  + unicode2 = NULL; unicode2pos = 0;
  """

• Comments should start with a capital letter and be
  prepended
  to the section they apply to

• There should be spaces between arguments in compares
  (a == b) not (a==b)

• Where does the name "...Encode121" originate ?

• module internal APIs should use lower case names (you
  converted some of these to PyUnicode_...() -- this is
  normally reserved for APIs which are either marked as
  potential candidates for the public API or are very
  prominent in the code)

One thing which I don't like about your API change is that
you removed the Py_UNICODE*data, int size style arguments --
this makes it impossible to use the new APIs on non-Python
data or data which is not available as Unicode object.

Please separate the errors.c patch from this patch -- it
seems totally unrelated to Unicode.

Thanks.

[doerwalter]

Logged In: YES
user_id=89016

• please don't place more than one C statement on one line
  like in:
  """

•           unicode = unicode2; unicodepos =
  

unicode2pos;

•           unicode2 = NULL; unicode2pos = 0;
  

"""

OK, done!

• Comments should start with a capital letter and be
  prepended
  to the section they apply to

Fixed!

• There should be spaces between arguments in compares
  (a == b) not (a==b)

Fixed!

• Where does the name "...Encode121" originate ?

encode one-to-one, it implements both ASCII and latin-1
encoding.

• module internal APIs should use lower case names (you
  converted some of these to PyUnicode_...() -- this is
  normally reserved for APIs which are either marked as
  potential candidates for the public API or are very
  prominent in the code)

Which ones? I introduced a new function for every old one,
that had a "const char *errors" argument, and a few new ones
in codecs.h, of those PyCodec_EncodeHandlerForObject is
vital, because it is used to map for old string arguments to
the new function objects. PyCodec_RaiseEncodeErrors can be
used in the encoder implementation to raise an encode error,
but it could be made static in unicodeobject.h so only those
encoders implemented there have access to it.

One thing which I don't like about your API change is that
you removed the Py_UNICODE*data, int size style arguments > --
this makes it impossible to use the new APIs on non-Python
data or data which is not available as Unicode object.

I look through the code and found no situation where the
Py_UNICODE*/int version is really used and having two
(PyObject *)s (the original and the replacement string),
instead of UNICODE*/int and PyObject * made the
implementation a little easier, but I can fix that.

Please separate the errors.c patch from this patch -- it
seems totally unrelated to Unicode.

PyCodec_RaiseEncodeErrors uses this the have a \Uxxxx with
four hex digits. I removed it.

I'll upload a revised patch as soon as it's done.

[doerwalter]

Logged In: YES
user_id=89016

One thing which I don't like about your API change is that
you removed the Py_UNICODE*data, int size style arguments

this makes it impossible to use the new APIs on non-Python
data or data which is not available as Unicode object.

Another problem is, that the callback requires a Python
object, so in the PyObject *version, the refcount is
incref'd and the object is passed to the callback. The
Py_UNICODE*/int version would have to create a new Unicode
object from the data.

[malemburg]

Logged In: YES
user_id=38388

About the Py_UNICODE*data, int size APIs:
Ok, point taken.

In general, I think we ought to keep the callback feature as
open as possible, so passing in pointers and sizes would not
be very useful.

BTW, could you summarize how the callback works in a few
lines ?

About _Encode121: I'd name this _EncodeUCS1 since that's
what it is ;-)

About the new functions: I was referring to the new static
functions which you gave PyUnicode_... names. If these are
not supposed to turn into non-static functions, I'd rather
have them use lower case names (since that's how the Python
internals work too -- most of the times).

[doerwalter]

Logged In: YES
user_id=89016

How the callbacks work:

A PyObject * named errors is passed in. This may by NULL,
Py_None, 'strict', u'strict', 'ignore', u'ignore',
'replace', u'replace' or a callable object.
PyCodec_EncodeHandlerForObject maps all of these objects to
one of the three builtin error callbacks
PyCodec_RaiseEncodeErrors (raises an exception),
PyCodec_IgnoreEncodeErrors (returns an empty replacement
string, in effect ignoring the error),
PyCodec_ReplaceEncodeErrors (returns U+FFFD, the Unicode
replacement character to signify to the encoder that it
should choose a suitable replacement character) or directly
returns errors if it is a callable object. When an
unencodable character is encounterd the error handling
callback will be called with the encoding name, the original
unicode object and the error position and must return a
unicode object that will be encoded instead of the offending
character (or the callback may of course raise an
exception). U+FFFD characters in the replacement string will
be replaced with a character that the encoder chooses ('?'
in all cases).

The implementation of the loop through the string is done in
the following way. A stack with two strings is kept and the
loop always encodes a character from the string at the
stacktop. If an error is encountered and the stack has only
one entry (during encoding of the original string) the
callback is called and the unicode object returned is pushed
on the stack, so the encoding continues with the replacement
string. If the stack has two entries when an error is
encountered, the replacement string itself has an
unencodable character and a normal exception raised. When
the encoder has reached the end of it's current string there
are two possibilities: when the stack contains two entries,
this was the replacement string, so the replacement string
will be poppep from the stack and encoding continues with
the next character from the original string. If the stack
had only one entry, encoding is finished.

(I hope that's enough explanation of the API and implementation)

I have renamed the static ...121 function to all lowercase
names.

BTW, I guess PyUnicode_EncodeUnicodeEscape could be
reimplemented as PyUnicode_EncodeASCII with a \uxxxx
replacement callback.

PyCodec_RaiseEncodeErrors, PyCodec_IgnoreEncodeErrors,
PyCodec_ReplaceEncodeErrors are globally visible because
they have to be available in _codecsmodule.c to wrap them as
Python function objects, but they can't be implemented in
_codecsmodule, because they need to be available to the
encoders in unicodeobject.c (through
PyCodec_EncodeHandlerForObject), but importing the codecs
module might result in an endless recursion, because
importing a module requires unpickling of the bytecode,
which might require decoding utf8, which ... (but this will
only happen, if we implement the same mechanism for the
decoding API)

I have not touched PyUnicode_TranslateCharmap yet,
should this function also support error callbacks? Why would
one want the insert None into the mapping to call the callback?

A remaining problem is how to implement decoding error
callbacks. In Python 2.1 encoding and decoding errors are
handled in the same way with a string value. But with
callbacks it doesn't make sense to use the same callback for
encoding and decoding (like codecs.StreamReaderWriter and
codecs.StreamRecoder do). Decoding callbacks have a
different API. Which arguments should be passed to the
decoding callback, and what is the decoding callback
supposed to do?

[doerwalter]

Logged In: YES
user_id=89016

One additional note: It is vital that errors is an
assignable attribute of the StreamWriter.

Consider the XML example: For writing an XML DOM tree one
StreamWriter object is used. When a text node is written,
the error handling has to be set to
codecs.xmlreplace_encode_errors, but inside a comment or
processing instruction replacing unencodable characters with
charrefs is not possible, so here codecs.raise_encode_errors
should be used (or better a custom error handler that raises
an error that says "sorry, you can't have unencodable
characters inside a comment")

BTW, should we continue the discussion in the i18n SIG
mailing list? An email program is much more comfortable than
a HTML textarea! ;)

[malemburg]

Logged In: YES
user_id=38388

How the callbacks work:

A PyObject * named errors is passed in. This may by NULL,
Py_None, 'strict', u'strict', 'ignore', u'ignore',
'replace', u'replace' or a callable object.
PyCodec_EncodeHandlerForObject maps all of these objects
to
one of the three builtin error callbacks
PyCodec_RaiseEncodeErrors (raises an exception),
PyCodec_IgnoreEncodeErrors (returns an empty replacement
string, in effect ignoring the error),
PyCodec_ReplaceEncodeErrors (returns U+FFFD, the Unicode
replacement character to signify to the encoder that it
should choose a suitable replacement character) or
directly
returns errors if it is a callable object. When an
unencodable character is encounterd the error handling
callback will be called with the encoding name, the
original
unicode object and the error position and must return a
unicode object that will be encoded instead of the
offending
character (or the callback may of course raise an
exception). U+FFFD characters in the replacement string
will
be replaced with a character that the encoder chooses ('?'
in all cases).

Nice.

The implementation of the loop through the string is done
in
the following way. A stack with two strings is kept and
the
loop always encodes a character from the string at the
stacktop. If an error is encountered and the stack has
only
one entry (during encoding of the original string) the
callback is called and the unicode object returned is
pushed
on the stack, so the encoding continues with the
replacement
string. If the stack has two entries when an error is
encountered, the replacement string itself has an
unencodable character and a normal exception raised. When
the encoder has reached the end of it's current string
there
are two possibilities: when the stack contains two
entries,
this was the replacement string, so the replacement string
will be poppep from the stack and encoding continues with
the next character from the original string. If the stack
had only one entry, encoding is finished.

Very elegant solution !

(I hope that's enough explanation of the API and
implementation)

Could you add these docs to the Misc/unicode.txt file ? I
will eventually take that file and turn it into a PEP which
will then serve as general documentation for these things.

I have renamed the static ...121 function to all lowercase
names.

Ok.

BTW, I guess PyUnicode_EncodeUnicodeEscape could be
reimplemented as PyUnicode_EncodeASCII with a \uxxxx
replacement callback.

Hmm, wouldn't that result in a slowdown ? If so, I'd rather
leave the special encoder in place, since it is being used a
lot in Python and probably some applications too.

PyCodec_RaiseEncodeErrors, PyCodec_IgnoreEncodeErrors,
PyCodec_ReplaceEncodeErrors are globally visible because
they have to be available in _codecsmodule.c to wrap them
as
Python function objects, but they can't be implemented in
_codecsmodule, because they need to be available to the
encoders in unicodeobject.c (through
PyCodec_EncodeHandlerForObject), but importing the codecs
module might result in an endless recursion, because
importing a module requires unpickling of the bytecode,
which might require decoding utf8, which ... (but this
will
only happen, if we implement the same mechanism for the
decoding API)

I think that codecs.c is the right place for these APIs.
_codecsmodule.c is only meant as Python access wrapper for
the internal codecs and nothing more.

One thing I noted about the callbacks: they assume that they
will always get Unicode objects as input. This is certainly
not true in the general case (it is for the codecs you touch
in the patch).

I think it would be worthwhile to rename the callbacks to
include "Unicode" somewhere, e.g.
PyCodec_UnicodeReplaceEncodeErrors(). It's a long name, but
then it points out the application field of the callback
rather well. Same for the callbacks exposed through the
_codecsmodule.

I have not touched PyUnicode_TranslateCharmap yet,
should this function also support error callbacks? Why
would
one want the insert None into the mapping to call the
callback?

1. Yes.
2. The user may want to e.g. restrict usage of certain
   character ranges. In this case the codec would be used to
   verify the input and an exception would indeed be useful
   (e.g. say you want to restrict input to Hangul + ASCII).

A remaining problem is how to implement decoding error
callbacks. In Python 2.1 encoding and decoding errors are
handled in the same way with a string value. But with
callbacks it doesn't make sense to use the same callback
for
encoding and decoding (like codecs.StreamReaderWriter and
codecs.StreamRecoder do). Decoding callbacks have a
different API. Which arguments should be passed to the
decoding callback, and what is the decoding callback
supposed to do?

I'd suggest adding another set of PyCodec_UnicodeDecode...()
APIs for this. We'd then have to augment the base classes of
the StreamCodecs to provide two attributes for .errors with
a fallback solution for the string case (i.s. "strict" can
still be used for both directions).

One additional note: It is vital that errors is an
assignable attribute of the StreamWriter.

It is already !

Consider the XML example: For writing an XML DOM tree one
StreamWriter object is used. When a text node is written,
the error handling has to be set to
codecs.xmlreplace_encode_errors, but inside a comment or
processing instruction replacing unencodable characters
with
charrefs is not possible, so here
codecs.raise_encode_errors
should be used (or better a custom error handler that
raises
an error that says "sorry, you can't have unencodable
characters inside a comment")

Sure.

BTW, should we continue the discussion in the i18n SIG
mailing list? An email program is much more comfortable
than
a HTML textarea! ;)

I'd rather keep the discussions on this patch here --
forking it off to the i18n sig will make it very hard to
follow up on it. (This HTML area is indeed damn small ;-)

[doerwalter]

Logged In: YES
user_id=89016

> [...]
> raise an exception). U+FFFD characters in the
replacement
> string will be replaced with a character that the
encoder
> chooses ('?' in all cases).

Nice.

But the special casing of U+FFFD makes the interface
somewhat
less clean than it could be. It was only done to be 100%
backwards compatible. With the original "replace" error
handling the codec chose the replacement character. But as
far as I can tell none of the codecs uses anything other
than '?', so I guess we could change the replace handler
to always return u'?'. This would make the implementation a
little bit simpler, but the explanation of the callback
feature *a lot* simpler. And if you still want to handle
an unencodable U+FFFD, you can write a special callback for
that, e.g.

def FFFDreplace(enc, uni, pos):
if uni[pos] == "\ufffd":
return u"?"
else:
raise UnicodeError(...)

> The implementation of the loop through the string is
done
> in the following way. A stack with two strings is kept
> and the loop always encodes a character from the string
> at the stacktop. If an error is encountered and the
stack
> has only one entry (during encoding of the original
string)
> the callback is called and the unicode object returned
is
> pushed on the stack, so the encoding continues with the
> replacement string. If the stack has two entries when an
> error is encountered, the replacement string itself has
> an unencodable character and a normal exception raised.
> When the encoder has reached the end of it's current
string
> there are two possibilities: when the stack contains two
> entries, this was the replacement string, so the
replacement
> string will be poppep from the stack and encoding
continues
> with the next character from the original string. If the
> stack had only one entry, encoding is finished.

Very elegant solution !

I'll put it as a comment in the source.

> (I hope that's enough explanation of the API and
implementation)

Could you add these docs to the Misc/unicode.txt file ? I
will eventually take that file and turn it into a PEP
which
will then serve as general documentation for these things.

I could, but first we should work out how the decoding
callback API will work.

> I have renamed the static ...121 function to all
lowercase
> names.

Ok.

> BTW, I guess PyUnicode_EncodeUnicodeEscape could be
> reimplemented as PyUnicode_EncodeASCII with a \uxxxx
> replacement callback.

Hmm, wouldn't that result in a slowdown ? If so, I'd
rather
leave the special encoder in place, since it is being
used a
lot in Python and probably some applications too.

It would be a slowdown. But callbacks open many
possiblities.

For example:

Why can't I print u"gürk"?

is probably one of the most frequently asked questions in
comp.lang.python. For printing Unicode stuff, print could be
extended the use an error handling callback for Unicode
strings (or objects where __str__ or tp_str returns a
Unicode object) instead of using str() which always returns
an 8bit string and uses strict encoding. There might even
be a
sys.setprintencodehandler()/sys.getprintencodehandler()

[...]
I think it would be worthwhile to rename the callbacks to
include "Unicode" somewhere, e.g.
PyCodec_UnicodeReplaceEncodeErrors(). It's a long name,
but
then it points out the application field of the callback
rather well. Same for the callbacks exposed through the
_codecsmodule.

OK, done (and PyCodec_XMLCharRefReplaceUnicodeEncodeErrors
really is a long name ;))

> I have not touched PyUnicode_TranslateCharmap yet,
> should this function also support error callbacks? Why
> would one want the insert None into the mapping to call
> the callback?

1. Yes.
2. The user may want to e.g. restrict usage of certain
   character ranges. In this case the codec would be used to
   verify the input and an exception would indeed be useful
   (e.g. say you want to restrict input to Hangul + ASCII).

OK, do we want TranslateCharmap to work exactly like
encoding,
i.e. in case of an error should the returned replacement
string again be mapped through the translation mapping or
should it be copied to the output directly? The former would
be more in line with encoding, but IMHO the latter would
be much more useful.

BTW, when I implement it I can implement patch bpo-403100
("Multicharacter replacements in
PyUnicode_TranslateCharmap")
along the way.

Should the old TranslateCharmap map to the new
TranslateCharmapEx
and inherit the "multicharacter replacement" feature, or
should I leave it as it is?

> A remaining problem is how to implement decoding error
> callbacks. In Python 2.1 encoding and decoding errors
are
> handled in the same way with a string value. But with
> callbacks it doesn't make sense to use the same callback
> for encoding and decoding (like
codecs.StreamReaderWriter
> and codecs.StreamRecoder do). Decoding callbacks have a
> different API. Which arguments should be passed to the
> decoding callback, and what is the decoding callback
> supposed to do?

I'd suggest adding another set of PyCodec_UnicodeDecode...
()
APIs for this. We'd then have to augment the base classes
of
the StreamCodecs to provide two attributes for .errors
with
a fallback solution for the string case (i.s. "strict" can
still be used for both directions).

Sounds good. Now what is the decoding callback supposed to
do?
I guess it will be called in the same way as the encoding
callback, i.e. with encoding name, original string and
position of the error. It might returns a Unicode string
(i.e. an object of the decoding target type), that will be
emitted from the codec instead of the one offending byte. Or
it might return a tuple with replacement Unicode object and
a resynchronisation offset, i.e. returning (u"?", 1) means
emit a '?' and skip the offending character. But to make
the offset really useful the callback has to know something
about the encoding, perhaps the codec should be allowed to
pass an additional state object to the callback?

Maybe the same should be added to the encoding callbacks to?
Maybe the encoding callback should be able to tell the
encoder if the replacement returned should be reencoded
(in which case it's a Unicode object), or directly emitted
(in which case it's an 8bit string)?

> One additional note: It is vital that errors is an
> assignable attribute of the StreamWriter.

It is already !

I know, but IMHO it should be documented that an assignable
errors attribute must be supported as part of the official
codec API.

Misc/unicode.txt is not clear on that:
"""
It is not required by the Unicode implementation to use
these base classes, only the interfaces must match; this
allows writing Codecs as extension types.
"""