utf8toUnicode encodes to ambigious hex sequences

The utf8toUnicode transformation function outputs hex sequences of the form %uXXXX and %uXXXXXX. Other characters are passed through as-is. For some inputs, these sequences are indigustinguable from its encoded output produces by different inputs.

This ticket is to report two separate such encoding ambiguities: no escaping for a literal %, and trailing literal hex digits after four-digit sequences.

Escaping literal % characters

This function outputs hex sequences for non-ASCII codepoints. Other characters are passed through as-is.

The % character is passed through as-is, too, and so input of the form abc%uXXXXxyz will produce output which is indistinguishable from a legitimate hex sequence generated by the function.

This is possibly also a security risk, in that a consumer reading hex sequences would then treat the following few characters as digits (if they are legal hex characters), and then convert the sequence to a single codepoint. So this allows for a bypass by way of "sneaking through" those characters.

Variable length sequences

utf8toUnicode encodes to four hex digits for some codepoints, and six hex digits for other codepoints.

For example, the input \xc4\x80-\xf4\x8f\xbf\xbf is encoded to: %u0100-%u10ffff

This gives ambiguity for output, for example å00, because the output there would be %uXXXX00 where the 00 are low-ASCII bytes (literal hex digits) passed through as-is. That's indistinguishable from a single codepoint of ``%uXXXXXX`.

This could also be a possible bypass: For a rule looks to match %u1234 but would allow %u123456, an attacker would find some character for the 56 part which is syntactically permissible (whitespace, for example), and arrange for that to be after the disallowed character. Rather like \0-free shellcode, but the other way around.

My suggested fix for both issues is to to always encode to six-digit hex sequences, and also to encode % as a hex squence.

This breaks compatibility for rules using utf8toUnicode, and so notice would need to be given to rule authors to update those rules. Hence I would suggest replacing utf8toUnicode with a new function (which I have called utf8toHex), and to update rules accordingly. Then utf8toUnicode may be removed.

I forgot to mention in my proposed changes above, that while the interface is changing, I'd also take the opportunity to escape control codes.

The guts of my implementation are, for each codepoint:

        if (codepoint >= 32 && codepoint < 127 && codepoint != '%') {
            if (p + 1 >= buf_end) {
                goto error;
            }

            *p = (unsigned char) codepoint;
            p++;
        } else {
            int r;

            if (p + 2 + 6 + 1 >= buf_end) {
                goto error;
            }

            r = sprintf(p, "%%u%06lx", (unsigned long) codepoint);
            if (r == -1) {
                goto error;
            }

            assert(r == 2 + 6);
            p += r;
        }

where invalid codepoints are also treated as an error.

@katef

Hi @katef,

Thanks for the issue report, want to make a contribution on a format of a patch ? I think it will be nice to have the utf8toHex transformation. If you want to catchup about it we are on slack: http://modsecurity.slack.com

@katef

Revisiting this one. Still true in 3.x: t:utf8toUnicode passes a literal % through unchanged (the single-byte branch in src/actions/transformations/utf8_to_unicode.cc), so abc%u0041 in the input is indistinguishable from an encoded A. @katef's proposed change (escape % and control characters) never became a patch.

This is a behaviour change for any rule matching on the transformed output, so it needs a deliberate call: change the encoding in both branches, add a new transformation with the unambiguous output as suggested in the thread, or document the ambiguity and close. Needs a maintainer decision.

While looking at this again I found a separate, newer defect in the same v2 function: since 2.9.9 (#3284) t:utf8toUnicode drops the last hex digit of every code point above U+FFFF, so U+1F600 comes out as %u1f60. Tracked in #3684 with a fix PR to follow; the 4-vs-6 digit ambiguity described here is unchanged by that fix and stays open as this issue.

Navigation Menu

Repository navigation

`utf8toUnicode` encodes to ambigious hex sequences #1211

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

utf8toUnicode encodes to ambigious hex sequences #1211

Description

Activity

katef commented on Sep 11, 2016

zimmerle commented on Sep 26, 2016

fzipi commented on Oct 12, 2026

fzipi commented on Oct 13, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`utf8toUnicode` encodes to ambigious hex sequences #1211