RE: Encoding Standard (mostly complete)

[Doug Ewell <doug@ewellic.org>]

Copying the Unicode mailing list.

Masatoshi Kimura <VYV03354 at nifty dot ne dot jp> wrote:

> (2012/04/19 9:33), Doug Ewell wrote:
>> Given the sequence F8 80 80 80 80, the Unicode Standard specifies
>> that a decoder should recognize F5 as an invalid UTF-8 code unit, do
>> whatever it does on an error condition, and then continue with the
>> next byte. This will generate 5 error conditions if handling of
>> errors includes trying to continue.
>
> Where TUS defines this? It seems to contradict TUS 6.1.0 p.96:
> http://www.unicode.org/versions/Unicode6.1.0/ch03.pdf#page=42
> |Although a UTF-8 conversion process is required to never consume
> |well-formed subsequences as part of its error handling for ill-formed
> |subsequences, such a process is not otherwise constrained in how it
> |deals with any ill-formed subsequence itself. An ill-formed
> |subsequence consisting of more than one code unit could be treated as
> |a single error or as multiple errors. For example, in processing the
> |UTF-8 code unit sequence <F0 80 80 41>, the only formal requirement
> |mandated by Unicode conformance for a converter is that the <41> be
> |processed and correctly interpreted as <U+0041>. The converter could
> |return <U+FFFD, U+0041>, handling <F0 80 80> as a single error, or
> |<U+FFFD, U+FFFD, U+FFFD, U+0041>, handling each byte of <F0 80 80> as
> |a separate error, or could take other approaches to signalling <F0 80
> |80> as an ill-formed code unit subsequence.

I remembered reading a statement from UTC that interpretation of an ill-
formed sequence was supposed to terminate as soon as the sequence was
determined to be ill-formed. Conformance definition C10 does say:

> For example, in UTF-8 every byte of the form 110xxxxx₂ must be
> followed with a byte of the form 10xxxxxx₂. A sequence such as
> <110xxxxx₂ 0xxxxxxx₂> is illegal, and must never be generated. When
> faced with this illegal byte sequence while transforming or
> interpreting, a UTF-8 conformant process must treat the first byte
> 110xxxxx₂ as an illegal termination error: for example, either
> signaling an error, filtering the byte out, or representing the byte
> with a marker such as FFFD (REPLACEMENT CHARACTER). In the latter two
> cases, it will continue processing at the second byte 0xxxxxxx₂.

A lead byte of 11111000₂ is ill-formed.

And in fact, the section of TUS that Masatoshi quoted goes on to say:

> Using the definition for maximal subpart, the best practice can be
> stated simply as:
>
> Whenever an unconvertible offset is reached during conversion of a
> code unit sequence:
>
> 1. The maximal subpart at that offset should be replaced by a single
>    U+FFFD.
>
> 2. The conversion should proceed at the offset immediately after the
>    maximal subpart.

However, this description does use the word "should," not "must," and it
goes on (on the same page) to offer a table with three "possible
alternative approaches" for mapping an ill-formed UTF-8 sequence into
characters. It recommends the method described above, but allows the
other two.

So the bottom line is that Masatoshi is right: the Unicode Standard does
not specify that a decoder *must* respond to an invalid lead byte as I
said, only that it *should*. I agree that this is unnecessarily vague.

Whether this calls for a complete recasting of the definition of UTF-8
by WHATWG, or by any individual contributors therein, is of course a
different matter.

> It is exactly a purpose of Encoding Standard to avoid these kind of
> vagueness.

Again, I'm not sure whether it is within the authority or responsibility
of WHATWG or any individual to provide a "better" definition of a
Unicode encoding form than that provided by Unicode. I do understand the
desire to nail down the various legacy encodings, such as Shift-JIS,
that have been interpreted over the years in very flexible and confusing
ways. I don't think UTF-8 falls into this category at all.

--
Doug Ewell | Thornton, Colorado, USA
http://www.ewellic.org | @DougEwell ­