Unicode 6:XML Entities draft

Unicode 6.0 was published last month, and the proposals for Unicode 6.1 are firming up, both of these releases have significant new characters for mathematical use, so I have updated the Editors' draft of “XML Entity Definitions for Characters ”.

The main source file, unicode.xml has been updated to contain information for all characters in Unicode 6.0, and the provisional allocations for the Arabic Math Alphabets in Unicode 6.1. There is no change to the set of entity names or the MathML or HTML dtd derived from these sources.

Although this document is styled as an editors' draft for an update to the current recommendation, there are no immediate plans to publish a formal update to the W3C recommendation. However I hope to track changes to Unicode in this editors' draft, and perhaps once the proposals to add Arabic mathematical characters to Unicode are all processed, we may try to submit this for formal review as a Proposed Edited Recommendation.

Unicode 6.0

Most of the new characters in Unicode 6.0 are not directly related to Mathematics, although the large collection of “emoji” derived from characters used in the Japanese mobile phone industry provides some interesting characters that I'm sure could be used for mathematical operators (U+1F4A9 perhaps?). However there are some specifically mathematical characters including new heavy (ultra bold) plus and minus (U+2795 and U+2796) which may find use either in display contexts or as additional operators distinct from the usual plus and minus.

Unicode 6.1 (proposals)

The mathematical alphabets (bold, fraktur, double-struck, etc. ) that are in Unicode, and available as values in MathML 2's mathvariant attribute fit well with the mathematical traditions using the Roman and Greek alphabets but don't really work with other alphabets, notably Arabic.

Azzeddine Lazrek proposed that MathML and Unicode be extended with additional math alphabets corresponding to conventions used in Arabic typeset Mathematics. (initial, tailed, looped, stretched) these were added to MathML in the recently finalised MathML 3,0, and the corresponding code points have been allocated to Unicode (all in the block 1EE??) and planned to be standardised in Unicode 6.1. I have provisionally added this data to unicode.xml, and added a table showing the characters to the entities draft.

XML Entity Definitions for Characters

The W3C today published XML Entity Definitions for Characters as a Recommendation (i.e., Web standard). It's taken a long time (over a decade) but I hope you find them useful...

XML Entities: W3C Vote

As noted in a previous post, XML Entity Definitions for Characters is now a Proposed Recommendation.

To help the W3C make the decision to make the final transition to Recommendation status the Advisory Committee members are asked to submit comments. If you are in a W3C member company, please ask your AC representative to comment via the form
http://www.w3.org/2002/09/wbs/33280/PR-xml-entity-names-20100211/
by the 11th of March!

Entities: Proposed Recommendation

I'm pleased to be able to report that today the W3C has published
XML Entity Definitions for Characters
as a Proposed Recommendation.

It's only really a list of names and numbers, but it seems to have taken up a large chunk of my life since I accidentally fell into supporting these things.

XML Entity Definitions for Characters

I'm pleased to announce that the last call draft of XML Entity Definitions for Characters has been published:

http://www.w3.org/TR/2009/WD-xml-entity-names-20091117/

Comments are welcome on www-math@w3.org, please put the text
[last-call...
in the subject line, preferably like this:
[Entities-last-call] - summary of comment

DSDL: initial thoughts

Prompted by some questions of Rick Jelliffe on the schematron list, I've been looking at DSDL lately. This is a multipart ISO/IEC standard defining various aspects of XML validation. The parts are at various stages; from fully ratified international standard to (I think) no existing draft at all. Below is a summary of collected thoughts on the various parts.

1: Overview

No great surprises, just an index into the following parts...

2: Regular-grammar-based validation - RELAX NG

For anyone working with document oriented XML, Relax NG is the gold standard of schema languages. More expressive than either XSD or DTD, more naturally namespace aware and (especially in its compact syntax form) particularly easy to write.

3: Rule-based validation - Schematron

I think this is probably the oldest of the languages that were pulled together to form DSDL. It's stood the test of time. I've ever really had occasion to use it much but I have from time to time made experiments with schematron, including the first implementation of the html based "schematron-report" version, and somewhat more recently some experiments on this blog.

4: Namespace-based validation dispatching language - NVDL

NVDL, or something like it, should perhaps form the basis of a solving the perennial problem of forming "Compound Documents" such as including MathML in a host document type such as XHTML, or DocBook, or TEI, etc. Nagging doubts that it sometimes seems rather heavyweight just for validation, for any particular case (such as MathML in XHTML) it's usually possible to construct a compound schema directly, and as we found while looking at W3C CDF, in many contexts defining a compound schema is the easy part of defining a compound document format, the real problems lurk elsewhere defining the behaviour and inheritance of properties across the interface between the languages. Such questions as should the current font, or font size be inherited from the host document type to the embedded fragment.

However problems of property inheritance and event propagation are rightly out of scope for a validation language, so this really is just musing on a perennial problem the we have with MathML in (anything), which NVDL doesn't really address

5: Data Type Library Language - DTLL

I don't know, this is more or less the right thing, although I sent some minor comments re use of xpath2 (which the current draft avoids in favour of xpath 1).

But perhaps it's just too late. XSD datatypes are rather horrible and rather inflexible but they more or less do the job, most of the time, and even Relax NG users are by now in the habit of using xs:boolean and friends, as of course are Xpath 2 users. perhaps it is too late for this to ever gain traction, but perhaps not...

6: Path-based integrity constraints

This part appears to be on hold, with no public draft.

7: Character Repertoire Description Language - CRDL

CRDL, or CREPDL as it appears to be known now is what sparked my current interest in DSDL. specifically Rick Jelliffe asked on the schematron list for code to convert a crepdl specification into a schematron (which is effectively the same as converting it into one or more xpath expressions.

I sketched out a rough implementation in that thread, but actually I think that this is perhaps harder than it need be as CREPDL is using too powerful a technology to express character ranges. Regular expressions are highly efficient mechanism for specifying substrings, but CREPDL, as currently specified just really specifies single characters. A character repertoire is just a partition of the Unicode code range into three (characters that are definitely in, definitely out, or maybe in the repertoire.) However if regexp were not used, a different syntax would have to be invented for character ranges, and I don't have any good suggestions here, so perhaps using regex is OK, perhaps...

8: Document Schema Renaming Language - DSRL

It's difficult to know what to say about this section. WG1 recently published a Defect Report detailing some of the comments I'd raised on the public comment list. But really that list just scratches the surface. The specification as it stands is completely contradictory and unimplementable.

9: Datatype- and Namespace-aware DTDs

This seems to be technically sound, but doomed attempt to give a veneer of namespace respectability to DTD. Perhaps in 1998 this might have had a chance of taking off but now, post Relax NG, I can't see the point. DTD are not going to go away any time soon despite predictions in some quarters, at NAG for example we use DTD extensively, but if I want a namespace aware grammar language I'd use Relax NG every time rather than a DTD syntax with a collection of processing instructions giving namespace bindings.

10: Validation Management

This part appears to be on hold, with no public draft.

XML 1.0 Fifth Edition

The W3C is about to publish XML 1.0 5th edition, that is assuming that my (and others') objections are overruled.

This sets a really terrible precedence, and sadly puts XML into a similar state as HTML, where the specification will be widely ignored (as it will be inconsistent) and people will have no choice other than just to test against a collection of major implementations and do whatever they do. The position for HTML is so bad that the editor for HTML 5 is on record as saying that the HTML 4 specification is essentially irrelevant to HTML5, and HTML5 is instead based on a formalisation of existing implementation behaviour. XML was intended to move markup languages away from such "tag soup" and base everything on a well specified foundation.

XML 1.1 changed the rules for XML names (in a good way) allowing a very much more open set of characters to be used in XML names. However XML 1.1 has not had wide take-up, and so the XML core WG has decided to use subterfuge of changing the XML Recommendation in place by introducing a fake errata that changes the Name production.

There is an attempt to trivialise my and similar objections as process objections. Clearly it is a gross abuse of process but that is not the main point of the objection. It is a technical issue. 5th edition places every specification that refers to XML into a completely unspecified status. Do the features of the language use the original XML 1.0 production or the incompatible one in the 5th edition? I asked for a simple yes/no answer to the question of whether it would be conformant to use the new characters in xpath. It is clear from the reply that even members of the XML Core (and W3C TAG) groups can not say definitively whether a single xpath step using such a character is conformant or not. If Henry Thompson can't answer this, how can anyone expect a normal developer to know the answer? The issue is not restricted to XPath, the same lack of clarity surrounds simple questions as to whether IDs using such a character are valid in SVG, or DocBook, or any other language you care to name.

No doubt the development community will recover and make things work, but as I said above users will have to go by what the implementors do, they will no longer be able to go by the specifications, which is a shame, that might yet have bad consequences.

At the very least the TAG ought to update its finding on versioning strategies to explain how, if a user community shows some resistance to using a new version, a useful approach is to remove choice by making incompatible changes in place, but without changing either the major or minor version number.

XML position at NAG

NAG is recruiting again.

There are several positions available, including one in my group, "XML Technologies".

XML Entity definitions for Characters

In many contexts people find it convenient to enter characters that are not on the keyboard as entity references, such as &rightarrow; to get an arrow rather than remembering what keyboard shortcut or numeric reference (&#x2192;) would produce this. In many cases, life would be simpler if people did not do this: having entity references means that not only do you need a <!DOCTYPE declaration to reference a DTD that defines the entities, you need your XML parser to read the DTD, and it makes processing fragments of XML much harder, as either the fragments do not have a <!DOCTYPE (in which case they are not, themselves, well formed) and the fragment pasting operation needs to ensure that a suitable DTD reference is placed on the target document, or the fragments do have a doctype, and the fragment pasting needs to strip this off, and still ensure that the target document has a compatible DTD.

If fragments are being moved from one place to another this can be difficult. Consider moving a fragent of MathML from an XHTML document to Docbook for example. XHTML and Docbook define entities with in several case the same name but different definition (the original ISO definitions of the entity names did not give definitions in terms of Unicode characters) and older versions of Unicode did not have sufficient technical symbols to give sensible definitions for most of these names.

All of which preamble is just leading up to say Unicode 5.1 (Beta) does have suitable characters for all the ISO and MathML entities..

The Entity draft at http://www.w3.org/2003/entities has thus been updated to a new "2007" version, which we (the W3C Math WG) hope to submit to W3C as a new Recommendation track document shortly, but you can view my Editor's Draft here.

MathML3 will hopefully use these by reference, if (X)HTML (and possibly other non-W3C systems such as Docbook) could do the same, then hopefully we would finally have a set of entity names with widespread consistent use across multiple languages. Hopefully.

Over the years I've been maintaining these sets we've kept fairly regular contact with the STIX group and the tables of characters in the above document include characters typeset with the STIX Fonts (if you have them installed, and they work in your browser. (The plane 1 characters still fail for me in all browsers on windows).

Comments are welcome, either in this blog, or better on www-math@w3.org.

XML position at NAG

NAG is looking for someone to work in the XML Technologies Group, that is, "my" group. So if you are interested, please contact the address specified in the above posting.