case-_sensitive_ ordering, case-_insensitive_ ordering , code point order, etc.

[TallTed]

Originally posted by @TallTed in #289 (comment)

I think we need to say something about case-sensitive ordering being the norm in some environments, and case-insensitive ordering in others, as well as how to override both norms, assuming that is possible anywhere if not everywhere. I know we've said in some place(s) that lexical ordering is to be code point order, which is effectively case-sensitive, but I don't think this will suffice for all use cases. I believe code point order will also prove problematic when trying to compare, for instance, ä, as a single character and as a composite character (i.e., a combination of a and ¨), and in making all the as (uppercase, lowercase, with diacriticals, etc.) appear as a group and be in a predictable order within that group.

[Tpt]

I fear that moving beyond the dumb code point ordering into the world of natural language ordering will require the full Unicode Collation Algorithm complexity. Everything smaller is likely to fall short. The good news is that it's well understood and specified already.

[TallTed]

it's well understood and specified already

By some. I submit that if our work is successful, an increasing number of users who do not understand such things will generate SPARQL queries that include ORDER BY clauses, and run into surprise walls when A is unexpectedly found far away from Å and å and a.

Note that I did not suggest that we need to specify implementation of anything, only that we include some advisory text, stating that "natural" order does not necessarily resemble Unicode point order.

[afs]

@TallTed would you care to find and provide the W3C i18n advice for this please?
I provide the F&O information on #289 - included below.

I think we need to say something about case-sensitive ordering being the norm in some environments, and case-insensitive ordering in others,

which environments?

I believe code point order will also prove problematic when trying to compare, for instance, ä, as a single character and as a composite character (i.e., a combination of a and ¨)

That isn't to do with case-sensitivity.
See NFC https://unicode.org/reports/tr15/

---

Functions and operators: fn:compare, fn:collation-key

fn:collation-key($key as xs:string, $collation as xs:string) as xs:base64Binary

ORDER BY fn:collationKey(string, collation)