David Carlisle

Robert Miner

2011-12-07T22:29:00.001Z

Robert Miner, my friend and colleague on the W3C Math Working Group died yesterday from liver cancer.

Robert has Chaired or Co-Chaired the group since it started, and has been active in developing standards for mathematics on the web at the W3C since at least 1996. He will be sorely missed.

html5mathml

2011-09-23T00:46:00.000+01:00

As some of you may have seen on google+ With a bit of help, I have MathML working in Firefox Mobile on Android. The key observation (of Karl Tomlinson) was that I needed a recent version of the font I was using (dejavu serif) as older ones did not have the necessary glyphs to build up “stretchy” characters such as large brackets needed for mathematical display.

So I have added support for Firefox/Android to my javascript library for enabling mathml-in-html5 (Firefox all versions, IE+MathPlayer 6-10, Chrome, Safari, Opera are all supported to some extent).

html5mathml on googlecode

This reminded me that I never actually announced its existence, hence this posting. The javascript is maintained in a google code svn repository at html5mathml, although you might want to start with the test/example file at test1.html

The code is all freely available, It is all under the MIT licence, except for the copy of the dejavu serif font which is available under its own custom free licence, included in the distribution.

Usage

The idea is that you write a conforming html5+mathml document and just add the line:

<script src="html5mathml.js"></script>

into the head of the document, adjusting the src attribute to point to a local copy of the files downloaded from the above location. Please do use local copies, googlecode is a source repository not intended for serving production code to be used. Depending on the client browser being used, this JavaScript file may use one of two css files or the above mentioned dejavu font, all of which are available from the source repository.

Comments fixes and experiences using more browsers all welcome, either here, or in email or on the www-math list.

Comparison with MathJax

Currently the primary javascript library for enabling MathML support in browsers is MathJax, so perhaps I should offer some comparison. MathJax does far more than html5mathml, and generally speaking produces better MathML display, however it is somewhat slower and perhaps harder to set up locally (although it now has a public CDN server distribution which simplifies things greatly if access to the server can be assumed).

MathJax includes parsers for Tex-like syntax as well as MathML; this library assumes the input is MathML and relies on the browser to parse it (although includes some fixup for pre-html5 legacy browser parsing.) MathJax can be configured to do its own MathML rendering using CSS, or uses the native MathMl rendering in the browser if available. html5mathml essentially just assumes that the browser has MathML support although it does include css similar to that used in the the CSS profile for MathML. As such it loads much more quickly and reveals the MathML capabilities of the browser.

Future Plans

I try to track browser developments as far as possible (IE10 preview + MathPlayer for example is supported) but this is a spare time activity and I don't have access to all platforms, so any comments or code contributions for other platforms is always welcome. Currently the largest component is the dejavu font, probably I should subset that to just the characters needed for stretchy symbols. IE10 will no doubt require some changes once the full version comes out, and hopefully there will be a version of Chrome using the webkit MathML rendering (as now used in Safari). Similar techniques may be used to enable svg-in-html in legacy browsers that only support svg in xml. An early version of this code, handling MathML and SVG on a smaller range of browsers was discussed in a posting I made to the NAG blog last year.

slinky canvas

2011-07-18T19:15:00.001+01:00

The NAG Blog, to which I contribute from time to time, has a permanent link to Mike Croucher's blog, Walking Randomly. I happened to notice his current article on parametric plotting. The parametric plots are demonstrated with an applet providing Mathematica generated plots and slider controls. Mathematica is a nice piece of software but I couldn't help but think that a modern web browser ought to be able to do this without help from an external application.

HTML (5) provides a canvas drawing API that ought to be able to plot a curve or two, and also built in slider form controls.

I've not used canvas before, but it turns out to be pretty trivial to use, and my attempt is shown below. If you have a canvas enabled browser the plot should appear looking like the plot shown in Mike Croucher's article. Below that should appear some input forms that allow you to change the parameters. These appear as slider controls if a browser understands the HTML5 markup for this (Chrome does in its dev channel version at least). In other browsers they appear as text boxes, which allow you to change the values. (Firefox (4+) and IE (9+) work this way.) The plot should be redrawn if you move the mouse off the form controls.

Actually I prefer the text entry control to the slider, as it is easier to exactly control the value. I'd hoped that the arrow keys could be used to step the slider, but as implemented in chrome at least, that isn't the case. No doubt with a bit more scripting such keyboard shortcuts could be added, but this will do as a first attempt.

e: 0 80

f: 0 80

g: 0 80

h: 0 80

i: 0 80

Converting to Strict Content MathML

2011-07-08T21:36:00.003+01:00

Some OpenMath History

MathML and OpenMath have always had a shared history and more or less documented ways of converting between them. Conceptually the conversion is very simple and OpenMath symbol abc could be expressed as the Content MathML symbol <csymbol>abc</csymbol>. Successive versions of MathML have in fact added features that made this conversion simpler and better information preserving. MathML2 added csymbol which is a better fit for OpenMath symbols than ci. In MathML1 and MathML2 further information about the OpenMath Symbol would have to be packaged into the definitionURL attribute. In MathML3 we added explicit support for recording Content Dictionaries by adding a cd attribute to csymbol.

Although the basic idea of the transformation was simple, the details of the transformation were complicated by a desire to map to the pre-defined MathML elements where available. So <OMS name="sin" cd="trans1"/> should map to the MathML element <sin/> rather than <csymbol>sin</csymbol>. The relationship between the predefined MathML Content MathML forms and the simpler, more regular, but much more verbose, OpenMath syntax was not formally specified by MathML so had to be specified as part of the transformation to OpenMath. An early version of such a transformation description is this 10 year old document still available from the OpenMath site. Conversion between MathML and OpenMath. Around the same time, the conversions were implemented in XSLT. The original versions predated XSLT 1, although the versions currently available from the OpenMath site use XSLT 2. Converting from OpenMath to MathML: om2cmml and from MathML to OpenMath: cmml2om.

MathML3 Strict Content MathML

The relationship between <sin/> and <csymbol>sin</csymbol> may rightfully be seen as a purely MathML issue and not something that should be a by-product of converting to the OpenMath form <OMS name="sin"/> and so MathML3 introduced, for each of its Content layout forms, an explicit rewrite rule expressing the construct in Strict Content MathML which is a restricted form just using csymbol. Section 4.6 of Chapter 4 of the MathML 3 spec specifies an explicit multi-pass algorithm applying these rewrite rules to convert any valid Content MathML expression into an expression just using the restricted Strict Content MathML vocabulary. 4.6 The Strict Content MathML Transformation.

Conceptually a conversion to Strict Content MathML could be made which first converted to OpenMath, then converted back to MathML using a stylesheet that removed all the special case rules originally added to om2cmml and documented in the OpenMath report referenced above. This was in fact implemented and is how the majority of the Strict Content MathML examples in Chapter 4 of the MathML spec were constructed. However there were some choices to be made in the mapping specification and in some cases the old OpenMath stylesheets made different choices. Where feasible I updated the stylesheets to match, but one essential structural difference remained. cmml2om implements a typical XSLT depth first walk over the input document, applying whichever template matches at that point. However the working group felt that the exposition of the algorithm was clearer if it was expressed as a multi-pass algorithm where each rule is applied in order over the whole tree, with the result being passed on to the next stage, to be rewritten by the next rewrite rule. These two approaches usually produce the same result, but in edge cases where the order of transformations matter they produce results that are (baring bugs) mathematically equivalent, but are structurally different.

As a requirement for MathML3 to proceed to W3C Recommendation status, we needed to show that the algorithm in Section 4.6 was implementable and did the right thing. It was clear that while my conversion via OpenMath was fairly reasonable it wasn't implementing the algorithm as specified and didn't produce the specified results in all cases.

C2S Implementation

Fortunately Robert Miner (co-chair of the Math WG) stepped up and offered to implement the algorithm as specified. It was good that he did, as inevitably, implementation experience showed some gaps or inconsistencies in the first drafts of the algorithm, and the final published form is much improved as a result of this implementation.

The initial home of the new stylesheet was in the W3C member area of the W3C CVS repository. Recently Robert suggested that we make it public, and asked if I'd host it at my google code web-xslt project site, since it is there and hosts other MathML related XSLT stylesheets.

So Robert's implementation is now available (under W3C or MIT licen[cs]e) from google code: c2s.

Comments on the stylesheet are probably best addressed to the www-math mailing list, but comments may also be dropped here on this blog or on the google code wiki pages.

MathML for CSS Profile

2011-06-08T01:08:00.000+01:00

I'm delighted to announce that the MathML for CSS profile has been published as a W3C Recommendation.

http://www.w3.org/TR/2011/REC-mathml-for-css-20110607/

This had a dependency on CSS 2.1 which was also published today.

This completes the three REC-track documents produced by the MathML3-era Working Group, joining:

http://www.w3.org/TR/2010/REC-MathML3-20101021/

and

http://www.w3.org/TR/2010/REC-xml-entity-names-20100401/

The MathML for CSS profile (most of the technical work for which was done by George Chavchanidze of Opera) defines how a subset of MathML may be implemented using CSS. This is useful for two reasons: Firstly, it highlights possible places where CSS could be extended to improve its expressive power (so allowing a larger subset of MathML to be implemented) or secondly, it may be viewed directly as an implementation of a useful subset of MathML (and is similar to the CSS stylesheet for MathML which is used for MathML support in Opera). As usual, comments are welcome here or on www-math@w3.org list.

Unicode 6:XML Entities draft

2010-11-07T00:44:00.003Z

Unicode 6.0 was published last month, and the proposals for Unicode 6.1 are firming up, both of these releases have significant new characters for mathematical use, so I have updated the Editors' draft of “XML Entity Definitions for Characters ”.

The main source file, unicode.xml has been updated to contain information for all characters in Unicode 6.0, and the provisional allocations for the Arabic Math Alphabets in Unicode 6.1. There is no change to the set of entity names or the MathML or HTML dtd derived from these sources.

Although this document is styled as an editors' draft for an update to the current recommendation, there are no immediate plans to publish a formal update to the W3C recommendation. However I hope to track changes to Unicode in this editors' draft, and perhaps once the proposals to add Arabic mathematical characters to Unicode are all processed, we may try to submit this for formal review as a Proposed Edited Recommendation.

Unicode 6.0

Most of the new characters in Unicode 6.0 are not directly related to Mathematics, although the large collection of “emoji” derived from characters used in the Japanese mobile phone industry provides some interesting characters that I'm sure could be used for mathematical operators (U+1F4A9 perhaps?). However there are some specifically mathematical characters including new heavy (ultra bold) plus and minus (U+2795 and U+2796) which may find use either in display contexts or as additional operators distinct from the usual plus and minus.

Unicode 6.1 (proposals)

The mathematical alphabets (bold, fraktur, double-struck, etc. ) that are in Unicode, and available as values in MathML 2's mathvariant attribute fit well with the mathematical traditions using the Roman and Greek alphabets but don't really work with other alphabets, notably Arabic.

Azzeddine Lazrek proposed that MathML and Unicode be extended with additional math alphabets corresponding to conventions used in Arabic typeset Mathematics. (initial, tailed, looped, stretched) these were added to MathML in the recently finalised MathML 3,0, and the corresponding code points have been allocated to Unicode (all in the block 1EE??) and planned to be standardised in Unicode 6.1. I have provisionally added this data to unicode.xml, and added a table showing the characters to the entities draft.

MathML 3.0 Recommendation

2010-10-21T23:21:00.001+01:00

I'm pleased to report that MathML 3.0 is today published as a W3C Recommendation. Seven years to the day since the last MathML Recommendation, MathML 2.0 2nd edition was published on 21st October 2003.

So, what have we been doing for seven years?

My Working Group colleague, Neil Soiffer has just posted a blog entry describing some of the main features and linking to a nice summary of the main new features in MathML3, so I won't list them all in detail here, however some of the headline features are listed below.

Additions to control bi-directional layout, for Arabic styles in particular

The Arabic Note detailed some extensions to MathML 2.0 that would enable a richer variety of right-to-left layouts as used in typesetting Arabic mathematics. In MathML 3.0 this bi-directional control is fully integrated into the language, allowing the effect of each of the presentation layout elements to be specified in RTL and LTR modes.

Elementary math layouts (long division, etc)

Previously long division and long multiplication etc could be typeset using table layout and a lot of spacing adjustment but this was difficult to produce and very hard to process in an accessible way, by for example a speech renderer. MathML 3 introduces new elements (mstack and mlongdiv, principally) which enable a much more natural and accessible encoding of these layout schemes.

Linebreaking of mathematics

For the first time MathML 3 specifies control over automatic linebreaking and provides improved features for manual forced linebreaking and alignment of expressions.

Officially registered MIME Types

The mime type application/mathml+xml has been unofficially used for some time, but it is now officially registered (along with two more mime types specific to Presentation and Content MathML). These Mime types might prove particularly useful in systems that use MIME to label clipboard fragments.

Closer alignment between Content MathML and OpenMath

Chapter 4, describing Content MathML has been totally rewritten to provide a direct, explicit alignment with OpenMath. This alignment was always in the background (the two languages being developed at roughly the same time, by overlapping groups of people) but making the alignment explicit, and making small changes on both the MathML and OpenMath side has allowed many rough edges to be removed, and I think gives a much clearer presentation of the “semantics” underlying the Content MathML elements.

RelaxNG Schema

The normative DTD used for MathML 1 and 2 is replaced in MathML 3 by a Relax NG schema. this is much more expressive than a DTD, and allows many of the constraints that previously could only be expressed in words to be built in to the grammar. A DTD (and XSD schema) are still provided in the Math Working Group pages, as a convenience.

Many clarifications and improvements throughout the spec

If you stare at the MathML2 spec for 7 years you may notice that some parts are clearer than others.

Integration with HTML5

This is briefly mentioned in Chapter 6, but mainly specified in the HTML 5 draft specification. One of the main difficulties of using MathML on the web has always been that it was designed as an XML application to fit with XHTML, and using XHTML has proved to be far more difficult than envisaged in 1998 when XML started. Some notable browsers are only now starting to support XHTML in beta releases, and even in browsers with good XML/XHTML support, it is difficult to integrate XHTML with HTML based document systems (such as the blogger system hosting this blog). HTML5, allowing MathML (and SVG) in text/html systems will be a massive boost to getting Mathematics into web based systems.

A MathML for CSS Profile 2nd PR

2010-10-21T22:00:00.000+01:00

A MathML for CSS Profile Has today been edited in place to confirm that it has passed all the ballots and implementation requirements to proceed to full W3C Recommendation status.

The final hurdle is that it has a normative dependency on CSS 2.1 which is currently at Candidate Recommendation stage, so MathML for CSS is blocked at Proposed Recommendation stage until CSS 2.1 reaches Recommendation. Hopefully early next year.

There are apparently several W3C Working drafts similarly blocked by CSS 2.1 (which has been at Candidate Recommendation stage for over a year). Hopefully the CSS Working Group will get the remaining issues resolved and testing reports done this year!

MathML3 PR

2010-08-10T21:44:00.000+01:00

The W3C today published MathML3 and the MathML for CSS Profile as proposed recommendations.

PR is almost the final hurdle, and assuming none of the W3C members object at this stage, then we should be on course to reach Recommendation status in September.

As always, comments are welcome on www-math@w3.org, or via this blog.

MathML3 2nd Last Call

2010-06-10T22:55:00.000+01:00

A new draft of MathML3 has been released: http://www.w3.org/TR/2010/WD-MathML3-20100610/.

This is (again) a Last Call draft. There were sufficient changes and clarifications resulting from the Candidate Recommendation review draft that we decided to issue a Last Call again, to allow people to comment on the changes before (hopefully) we progress to Proposed Recommendation status.

We have provided a Diff marked version and a List of changes to make it easier to review the changes since the CR draft.

STIX Fonts 1.0

2010-05-28T20:52:00.001+01:00

15 years in the making, they finally released the fonts!
http://www.stixfonts.org/

XML Entity Definitions for Characters

2010-04-01T21:57:00.000+01:00

The W3C today published XML Entity Definitions for Characters as a Recommendation (i.e., Web standard). It's taken a long time (over a decade) but I hope you find them useful...

F#

2010-03-11T13:41:00.001Z

Mostly these days I suppose I'm known for MathML and XSLT stuff. Before that I used TeX rather a lot (I must check one of these days whether I have yet posted as often to xsl-list as I have to comp.text.tex). Around the time I started to use TeX (1987 or thereabouts) I was mainly programming in the functional programming language Standard ML. I haven't done so much with ML since, but the ML family of languages have been having something of a renaissance recently, especially with Microsoft's F# language.

So I've been returning to my roots and doing a bit of functional programming over the last few days, writing an article hosted at NAG on calling the NAG library from F#.

XML Entities: W3C Vote

2010-02-21T13:01:00.000Z

As noted in a previous post, XML Entity Definitions for Characters is now a Proposed Recommendation.

To help the W3C make the decision to make the final transition to Recommendation status the Advisory Committee members are asked to submit comments. If you are in a W3C member company, please ask your AC representative to comment via the form
http://www.w3.org/2002/09/wbs/33280/PR-xml-entity-names-20100211/
by the 11th of March!

Entities: Proposed Recommendation

2010-02-11T23:15:00.001Z

I'm pleased to be able to report that today the W3C has published
XML Entity Definitions for Characters
as a Proposed Recommendation.

It's only really a list of names and numbers, but it seems to have taken up a large chunk of my life since I accidentally fell into supporting these things.

MathML on the Clipboard

2010-01-21T23:31:00.003Z

I got a new machine at work with Windows 7 on it.

One of the more interesting applications coming with Windows 7 is the Math Input Panel. This is designed for pen input on a tablet-style device and performs pretty impressively accurate recognition of mathematical expressions. While designed for a tablet, it also works pretty well if you are just “writing” the expression with a finger on a small laptop trackpad, which is how I have been using it.

The Math Input Panel is designed with a very simple interface with virtually no customisation options. It offers no way of saving the expressions generated and just offers a simple insert button that tries to insert the math expression at the insertion point in a currently open application. This works well for Word 2007 which accepts MathML from the clipboard and transparently converts it to its internal form and renders it, but other more generic tools such as XML editors that could use the MathML do not accept MathML from the clipboard in this way. Unlike MathPlayer or Word, The Math Input Panel doesn't offer fallback text representations of the XML markup on the clipboard. Marko Panic, the program manager for the development of this tool confirmed to me that this was a design decision as they didn't want the end user to be faced with raw XML. This is not unreasonable but not what I wanted personally (I like to see my XML raw:-). Marko confirmed that the MathML is on the clipboard and it should be possible to extract it with a few lines of code, or if I wanted a more extensive customisation there was documentation of the API offered by the underlying DLL available at
http://msdn.microsoft.com/en-us/library/dd317324(VS.85).aspx
and
http://msdn.microsoft.com/en-us/library/dd317311(VS.85).aspx.

I decided to brush up my C# forms programming and produced a small form that shows any MathML on the clipboard. The main code (everything apart from the boilerplate Visual Studio files) is available on google code While it's particularly useful to see the MathML generated by the Math Input panel, it also works with other applications, notably MathPlayer and Word, that place MathML on the clipboard.

While looking via Google for some programming tips on my form, I came across a very similar blog posting from last year. That form had some differences though (displaying the IE folding tree view of the XML) so I completed my form here. The screenshot shows the Math Input Panel interpreting my appalling handwriting, and the mmlclipboard form displaying the generated MathML.

XSLT for Presentation MathML in a Browser

2009-12-20T23:22:00.005Z

Back in 2002 at the MathML Conference I presented a stylesheet, pmathml.xsl, that detected the browser being used and inserted whatever was necessary to enable MathML on that platform. The stylesheet is distributed, along with the slides from the conference presentation, from the W3C Math/XSL area.

The stylesheet is showing its age, in particular a large part of its complexity is due to the fact that that it encodes two completely separate languages in one file, XSLT, and the ‘wd-xslt’ language used in IE5. IE 5 is not an important platform these days, and for some years MathPlayer has been able to act as a MIME Filter in Internet Explorer, and so automatically insert the needed object references into the document without using an XSL stylesheet. However the XSL stylesheet mechanism still has uses, and may be adapted to other browsers. An updated version of this stylesheet has been in use for the NAG Library documentation.

I have placed an updated version of pmathml.xsl on google code. This loses all the wd-xsl code so requires IE6 or later, but it detects Opera and Safari and in that case inserts some extra elements to make it more likely that the documents meets the conditions of the MathML profile for CSS, and links to some CSS to render the resulting elements. The CSS is a combination of some old CSS of mine and George Chavchanidze's CSS developed as part of the MathML profile for CSS.

Unfortunately xlink links no longer work in Firefox 3, and so this version uses some javascript to give link behaviour to any element with an xlink:href attribute. Since it is doing this it also enables links for unprefixed href attributes, in anticipation of MathML3's href attribute for linking.

I hope to extend this to enhance MathML3 support, rendering the new MathML3 features as far as possible in existing browsers. The current check in is mainly to form a baseline in google code before adding any new MathML3 features.

MathML as ASCII Art

2009-12-18T19:34:00.006Z

Normally one wants to use a full typesetting system, and access to fonts with a rich variety of symbols to typeset mathematics. But sometimes you don't.

Plain text, restricted to ASCII, still has its uses. It is possible to lay out the 2-dimensional mathematical notations as a kind of ‘ASCII-art’. Historically in the days before GUI workbook-style interfaces most Computer Algebra systems would use this kind of layout, and they can usually still be persuaded to use this style even in current versions.

I've just made available a very old stylesheet, it was I think one of the first XSLT 2 stylesheets that I wrote, which renders MathML in this style. It still does not do all of MathML3 (or 2) presentation MathML, but I intend to extend it as time permits.

The stylesheet is available on google code and some examples running against the MathML test suite are shown below.

The google code project also has the htmlparse styesheet discussed previously in this blog, and a perhaps more serious XSLT stylesheet to render MathML via TeX, again that will be extended to cover most of MathML3, as part of the MathML CR implementation phase – something to do over the Christmas break. Documentation of exactly what's in the project will be added eventually, it's only been running a day.

sum1

infinity  
  ----    
   \      
    )   x 
   /     i
  ----    
 i = 0

msubsup1


 1    
/  x  
| e dx
/     
 0

mtable2


                                                     2          2                          
1                     x + y                         x  + 2yx + y                           
                       2          2                  3      2     2     3                  
x + y                 x  + 2yx + y                  x  + 3yx  + 3y x + y                   
 2          2          3      2     2     3          4      3     2 2     3     4          
x  + 2yx + y          x  + 3yx  + 3y x + y          x  + 4yx  + 6y x  + 4y x + y           
 3      2     2     3  4      3     2 2     3     4  5      4      2 3      3 2     4     5
x  + 3yx  + 3y x + y  x  + 4yx  + 6y x  + 4y x + y  x  + 5yx  + 10y x  + 10y x  + 5y x + y

mfrac4

       _ 
1 +  \|5 
---------
    2

MathML3 article at IBM developerWorks

2009-12-16T17:38:00.003Z

IBM developerWorks have just published an article that I wrote on on MathML3, http://www.ibm.com/developerworks/xml/library/x-mathml3. I doubt that it has any information new to readers of this blog, but perhaps it may reach a new audience and spread the word about MathML3.

Candidate Recommendations

2009-12-15T20:41:00.002Z

I'm pleased to be able to announce that the Candidate Recommendation drafts of MathML3 and the MathML for CSS profile are available at http://www.w3.org/TR/2009/CR-MathML3-20091215/ and http://www.w3.org/TR/2009/CR-mathml-for-css-20091215/.

It is during this Candidate Recommendation phase that implementation work needs to begin in earnest as we need to show via the test suite that all the features are implemented by two implementations.

Thoughts on Revising DSRL

2009-12-11T13:27:00.003Z

A while ago now, I posted an article on DSDL. In it I commented that part 8, DSRL, had a lot of problems. The new Project Editor, Francis Cave, has recently posted the first cut at a revised draft of DSRL, and also posted some thoughts from me suggesting some more extensive revisions that could be made.

The posting can be seen in the public archives of dsdl-discuss at http://lists.dsdl.org/dsdl-discuss/2009-12/0002.html. dsdl-discuss is a closed list for members of SC 34/WG 1, however comments are welcome on the public list dsdl-comment 'at' dsdl.org archived at http://lists.dsdl.org/dsdl-comment/.

XML Entity Definitions for Characters

2009-11-17T17:57:00.002Z

I'm pleased to announce that the last call draft of XML Entity Definitions for Characters has been published:

http://www.w3.org/TR/2009/WD-xml-entity-names-20091117/

Comments are welcome on www-math@w3.org, please put the text
[last-call...
in the subject line, preferably like this:
[Entities-last-call] - summary of comment

MathML CSS Profile

2009-10-06T21:26:00.002+01:00

The W3C has just published the Last Call draft of A MathML for CSS profile.

2 down 1 to go!

MathML3 Last Call

2009-09-25T00:19:00.004+01:00

The Last Call draft of MathMl3 has just been published at http://www.w3.org/TR/2009/WD-MathML3-20090924/

Please send comments to www-math@w3.org, starting your subject line with
[MathML3-last-call]

The has been more refinement of the elementary math layout markup in chapter 3, a lot of work cleaning up and normalising chapter 4 on Content Markup, and a new Relax NG Schema, which is now almost all generated (via XSLT) from the text of the spec, as attributes and content models are now displayed in a tabular form more amenable to machine processing.

Also for the first time, we are publishig DTD and XSD versions of the schema, both of which are automatically converted from the Relax NG schema by a combination of XSLT (to simplify the schemas) and trang (to do the actual conversion) and (in the case of dtd) sed processing to massage the dtd into xhtml modularisation form, so that it supports namespaces as far as is possible with a dtd.

New MathML Drafts

2009-06-04T21:31:00.005+01:00

I'm pleased to announce another draft of both MathML 3 and the MathML profile for CSS.

"Mathematical Markup Language (MathML) Version 3.0"
http://www.w3.org/TR/2009/WD-MathML3-20090604/

"A MathML for CSS profile"
http://www.w3.org/TR/2009/WD-mathml-for-css-20090604/

Comments to www-math please.

MathML3 Draft

2008-11-17T22:56:00.002Z

We just released a new working draft of MathML3

http://www.w3.org/TR/2008/WD-MathML3-20081117/

Comments to www-math@w3.org please!

DSDL: initial thoughts

2008-10-17T00:17:00.001+01:00

Prompted by some questions of Rick Jelliffe on the schematron list, I've been looking at DSDL lately. This is a multipart ISO/IEC standard defining various aspects of XML validation. The parts are at various stages; from fully ratified international standard to (I think) no existing draft at all. Below is a summary of collected thoughts on the various parts.

1: Overview

No great surprises, just an index into the following parts...

2: Regular-grammar-based validation - RELAX NG

For anyone working with document oriented XML, Relax NG is the gold standard of schema languages. More expressive than either XSD or DTD, more naturally namespace aware and (especially in its compact syntax form) particularly easy to write.

3: Rule-based validation - Schematron

I think this is probably the oldest of the languages that were pulled together to form DSDL. It's stood the test of time. I've ever really had occasion to use it much but I have from time to time made experiments with schematron, including the first implementation of the html based "schematron-report" version, and somewhat more recently some experiments on this blog.

4: Namespace-based validation dispatching language - NVDL

NVDL, or something like it, should perhaps form the basis of a solving the perennial problem of forming "Compound Documents" such as including MathML in a host document type such as XHTML, or DocBook, or TEI, etc. Nagging doubts that it sometimes seems rather heavyweight just for validation, for any particular case (such as MathML in XHTML) it's usually possible to construct a compound schema directly, and as we found while looking at W3C CDF, in many contexts defining a compound schema is the easy part of defining a compound document format, the real problems lurk elsewhere defining the behaviour and inheritance of properties across the interface between the languages. Such questions as should the current font, or font size be inherited from the host document type to the embedded fragment.

However problems of property inheritance and event propagation are rightly out of scope for a validation language, so this really is just musing on a perennial problem the we have with MathML in (anything), which NVDL doesn't really address

5: Data Type Library Language - DTLL

I don't know, this is more or less the right thing, although I sent some minor comments re use of xpath2 (which the current draft avoids in favour of xpath 1).

But perhaps it's just too late. XSD datatypes are rather horrible and rather inflexible but they more or less do the job, most of the time, and even Relax NG users are by now in the habit of using xs:boolean and friends, as of course are Xpath 2 users. perhaps it is too late for this to ever gain traction, but perhaps not...

6: Path-based integrity constraints

This part appears to be on hold, with no public draft.

7: Character Repertoire Description Language - CRDL

CRDL, or CREPDL as it appears to be known now is what sparked my current interest in DSDL. specifically Rick Jelliffe asked on the schematron list for code to convert a crepdl specification into a schematron (which is effectively the same as converting it into one or more xpath expressions.

I sketched out a rough implementation in that thread, but actually I think that this is perhaps harder than it need be as CREPDL is using too powerful a technology to express character ranges. Regular expressions are highly efficient mechanism for specifying substrings, but CREPDL, as currently specified just really specifies single characters. A character repertoire is just a partition of the Unicode code range into three (characters that are definitely in, definitely out, or maybe in the repertoire.) However if regexp were not used, a different syntax would have to be invented for character ranges, and I don't have any good suggestions here, so perhaps using regex is OK, perhaps...

8: Document Schema Renaming Language - DSRL

It's difficult to know what to say about this section. WG1 recently published a Defect Report detailing some of the comments I'd raised on the public comment list. But really that list just scratches the surface. The specification as it stands is completely contradictory and unimplementable.

9: Datatype- and Namespace-aware DTDs

This seems to be technically sound, but doomed attempt to give a veneer of namespace respectability to DTD. Perhaps in 1998 this might have had a chance of taking off but now, post Relax NG, I can't see the point. DTD are not going to go away any time soon despite predictions in some quarters, at NAG for example we use DTD extensively, but if I want a namespace aware grammar language I'd use Relax NG every time rather than a DTD syntax with a collection of processing instructions giving namespace bindings.

10: Validation Management

This part appears to be on hold, with no public draft.

XML 1.0 Fifth Edition

2008-10-15T23:04:00.002+01:00

The W3C is about to publish XML 1.0 5th edition, that is assuming that my (and others') objections are overruled.

This sets a really terrible precedence, and sadly puts XML into a similar state as HTML, where the specification will be widely ignored (as it will be inconsistent) and people will have no choice other than just to test against a collection of major implementations and do whatever they do. The position for HTML is so bad that the editor for HTML 5 is on record as saying that the HTML 4 specification is essentially irrelevant to HTML5, and HTML5 is instead based on a formalisation of existing implementation behaviour. XML was intended to move markup languages away from such "tag soup" and base everything on a well specified foundation.

XML 1.1 changed the rules for XML names (in a good way) allowing a very much more open set of characters to be used in XML names. However XML 1.1 has not had wide take-up, and so the XML core WG has decided to use subterfuge of changing the XML Recommendation in place by introducing a fake errata that changes the Name production.

There is an attempt to trivialise my and similar objections as process objections. Clearly it is a gross abuse of process but that is not the main point of the objection. It is a technical issue. 5th edition places every specification that refers to XML into a completely unspecified status. Do the features of the language use the original XML 1.0 production or the incompatible one in the 5th edition? I asked for a simple yes/no answer to the question of whether it would be conformant to use the new characters in xpath. It is clear from the reply that even members of the XML Core (and W3C TAG) groups can not say definitively whether a single xpath step using such a character is conformant or not. If Henry Thompson can't answer this, how can anyone expect a normal developer to know the answer? The issue is not restricted to XPath, the same lack of clarity surrounds simple questions as to whether IDs using such a character are valid in SVG, or DocBook, or any other language you care to name.

No doubt the development community will recover and make things work, but as I said above users will have to go by what the implementors do, they will no longer be able to go by the specifications, which is a shame, that might yet have bad consequences.

At the very least the TAG ought to update its finding on versioning strategies to explain how, if a user community shows some resistance to using a new version, a useful approach is to remove choice by making incompatible changes in place, but without changing either the major or minor version number.

25

2008-07-30T13:40:00.003+01:00

As a rule, this blog restricts to technical items, but rules are meant to be broken...

Joanna and I got married 25 years ago today.

XSLT in the browser

2008-05-14T10:57:00.004+01:00

XSLT in the browser, its original motivating use case, has had a somewhat difficult time, but with support finally appearing in Opera and Safari, and the support in IE and Firefox/mozilla being fairly stable it looked like perhaps its time had come (even if stuck in an XSLT 1 era).

But then came Firefox 3 beta 5.....

Firefox has decided that relative links that traverse the filesystem out of the current directory are a security error. They are (pretty confusingly) reported as XML parse error rather than a security error.

https://bugzilla.mozilla.org/show_bug.cgi?id=427333

This is pretty fatal to browsing XML files on the filesystem, at NAG for instance, the documentation is distributed as a directory tree of a few thousand XML files, at the top level is a styles directory and so the individual files have xml-stylesheet links linking to ../../styles/foo.xsl or whatever is needed. This works in all browsers that support XSLT at all, except firefox 3 beta 5. Where apparently the only solution is to change the link to href="foo.xsl" and then copy the stylesheet into dozens of directories. Blurgh....

I hope bug 427333 gets fixed before FF3 gets out of beta....

Update: ../ stylesheets work again in FF3 release candidate 1, yipee!

MathML3 Draft

2008-04-09T23:15:00.003+01:00

The W3C Math WG has just published a new working draft of MathML3

http://www.w3.org/TR/2008/WD-MathML3-20080409

As always, comments are welcome on the www-math@w3.org mailing list

XML position at NAG

2008-01-10T23:20:00.000Z

NAG is recruiting again.

There are several positions available, including one in my group, "XML Technologies".

Three new W3C Working Drafts

2007-12-14T17:12:00.000Z

Today the W3C published three new working drafts

MathML3 Several new features worked on this time, more Content MathML improvements, more information on layouts for elementary mathematics (long division, etc.), and the first draft of a Relax NG Schema. Also, we have re-instated the XHTML+MathML version of the spec.

A MathML for CSS profile I'm down as co-editor of this, but all the main credit should go to George Chavchanidze of Opera Software, who's continuing his long standing work of getting mathematical rendering using pure CSS declarations.

XML Entity definitions for Characters The latest iteration of the the definitions of Characters. This was formerly part of the MathML spec (Chapter 6) but had been separated out and extended to include all the ISO entity sets, and the HTML Entity sets. I still hope that eventually this can be a joint ISO/W3C publication, updating ISO/IEC TR 9773-13. We'll see...

Mathematics in PowerPoint 2007

2007-12-02T11:36:00.000Z

I've never used PowerPoint, but I've been investigating recently the mathematical and in particular MathML capabilities of the Office 2007 suite.

It's been noted in several places that PowerPoint doesn't support the new Word 2007 math zones and that if you cut and paste a math expression from Word to PowerPoint, the result is an image, which means you can't edit it or search on it in that form, and it looks horrible on screen, especially if you have background colour or textures applied to your slides.

This note is just to mention a mechanism of getting correctly rendered editable mathematical text into PowerPoint, in a form which has the full oomml XML markup in the pptx file, so you can extract that and convert to MathML if needed using the Microsoft supplied stylesheet. I suspect that this mechanism is (or ought to be) well known by anyone (not me!) who's used PowerPoint, but a google search didn't show up anyone else mentioning it in this context, so I thought I'd post...

To get a math expression inserted, don't cut and paste a math zone from word, in Powerpoint choose
insert / Object /MicroSoft Word Document
then make a 'document' consisting of the equation you need using the embedded copy of Word. the resulting equation will be saved as an embedded object, and if you unzip the pptx PowerPoint file you will find a docx version of the embedded object in the embeddings directory, which you can further unzip to locate the oomml math XML.

The resulting equation renders as text rather than an image and may be edited at any time later, just click on it and you get thrown into a copy of Word.

Screen shot with one of Word's example equations rendered twice in a PowerPoint slide, once as an embedded object and once as an image.

XML Entity definitions for Characters

2007-11-29T21:44:00.000Z

In many contexts people find it convenient to enter characters that are not on the keyboard as entity references, such as &rightarrow; to get an arrow rather than remembering what keyboard shortcut or numeric reference (→) would produce this. In many cases, life would be simpler if people did not do this: having entity references means that not only do you need a <!DOCTYPE declaration to reference a DTD that defines the entities, you need your XML parser to read the DTD, and it makes processing fragments of XML much harder, as either the fragments do not have a <!DOCTYPE (in which case they are not, themselves, well formed) and the fragment pasting operation needs to ensure that a suitable DTD reference is placed on the target document, or the fragments do have a doctype, and the fragment pasting needs to strip this off, and still ensure that the target document has a compatible DTD.

If fragments are being moved from one place to another this can be difficult. Consider moving a fragent of MathML from an XHTML document to Docbook for example. XHTML and Docbook define entities with in several case the same name but different definition (the original ISO definitions of the entity names did not give definitions in terms of Unicode characters) and older versions of Unicode did not have sufficient technical symbols to give sensible definitions for most of these names.

All of which preamble is just leading up to say Unicode 5.1 (Beta) does have suitable characters for all the ISO and MathML entities..

The Entity draft at http://www.w3.org/2003/entities has thus been updated to a new "2007" version, which we (the W3C Math WG) hope to submit to W3C as a new Recommendation track document shortly, but you can view my Editor's Draft here.

MathML3 will hopefully use these by reference, if (X)HTML (and possibly other non-W3C systems such as Docbook) could do the same, then hopefully we would finally have a set of entity names with widespread consistent use across multiple languages. Hopefully.

Over the years I've been maintaining these sets we've kept fairly regular contact with the STIX group and the tables of characters in the above document include characters typeset with the STIX Fonts (if you have them installed, and they work in your browser. (The plane 1 characters still fail for me in all browsers on windows).

Comments are welcome, either in this blog, or better on www-math@w3.org.

More STIX Experiments

2007-11-27T22:05:00.000Z

Having posted a small test file to the stix comment page , I noticed that the stix site has a larger test file (choose "STIX Font Glyph Tables" from the menu options on the STIX site, which will take you to
http://www.stixfonts.org/allGlyphs.html.

This differs from my test (apart from being rather larger) in that each cell is individually assigned an appropriate font with separate CSS classes rather than having a single CSS font list and relying on the font choice to fall through fonts that do not have the appropriate glyph. The results are different but still a bit disappointing.

Three images of my Windows XP setup, one of the Font directory just to show they are there, one of the top of the allGlyphs file showing side by side in Firefox 2.0.0.9, Opera 9.21 and IE 7.0.5730.11, note that only FF actually shows the bold and italic variants. then the same three browsers showing the bottom of the file. here FF is all white, Opera is white for some characters and two missing glyph markers for others, and IE is mainly white but strangely enough actually renders the last few entries, which are monospace digits.

The images link to larger screen dumps which show (or rather do not show) the glyphs more clearly.

Opera joins the party

2007-11-19T15:05:00.000Z

http://my.opera.com/desktopteam/blog/2007/11/16/even-more-work

In this build, we added MathML support out of the box

Stix fonts: Initial comments

2007-11-07T16:43:00.000Z

After a wait of many years the STIX fonts have finally been released as public beta! These offer the promise of much better, more portable support for scientific documents on the web and elsewhere. The STIX fonts provide a uniform set of fonts that provide the glyphs for almost all the mathematical characters that have been added to unicode in recent years.

Initial testing suggests there may still be some problems with plane 1 characters (I just submitted the following comment to the stix beta test comment form...

I tested browser support (windows XP: Firefox, IE 7, Opera) with similar results in all three browsers. The quadruple integral displayed in all three browsers. The plane 1 alphabetic character did not display at all, FF displays 2 missing glyph boxes, IE shows 1 missing glyph box, and opera just shows white space..
I am not able to tell at this stage whether this is user error (I have not specified a sufficiently large set of the STIX fonts in CSS ?) or if this is due to lack of plane 1 support in the browsers, or if there is a problem with the unicode tables in the stix fonts, so I supply the small test file below, which I expect to render as as
QUADRUPLE INTEGRAL OPERATOR
MATHEMATICAL BOLD CAPITAL A
<html>
<head>
<title>stix</title>
<style>
p {
font-family: STIXGeneral, STIXGeneral-Italic, STIXGeneral-Bold, STIXGeneral-BoldItalic;
}
</style>
</head>
<body>
<p>x2a0c &#x2a0c;</p>
<p>x1d400 &#x1d400;</p>
</body>
</html>

New MathML Draft

2007-10-05T22:23:00.000+01:00

The W3C just published the latest draft of MathML 3. As ever, comments welcome on www-math@w3.org mailing list.

XML position at NAG

2007-06-12T14:15:00.000+01:00

NAG is looking for someone to work in the XML Technologies Group, that is, "my" group. So if you are interested, please contact the address specified in the above posting.

The Big Switch

2007-06-05T23:06:00.000+01:00

There was a thread a while ago on xsl-list discussing when was a good time to switch from XSLT1 to XSLT2. The consensus seemed to be, "it depends...."

For NAG, the answer is NOW!

After a lot of regression testing, staring at diff files and crossing of fingers, we just switched the processor for our main stylesheets over from saxon 6 to saxon 8.

This is quite a large (set of) stylesheets (around 26K lines of XSLT) so is quite a substantial test of XSLT2's compatibility with XSLT1. The results were pretty good, although it's not exactly surprising as an earlier version of these stylesheets were used to request improvements to backward compatibility mode as defined in an earlier draft, all the main changes requested in that report were made.

We are taking a multi-stage process to switching to version 2:

Import an XSLT2 stylesheet that defines a few extension functions in the saxon 6 extension namespace using xsl:function and standard function calls. saxon:node-set defined as identity function, saxon:tokenize defined using XPath2 tokenize, saxon:distinct defined using XPath2 distinct-values etc.
Process the stylesheets with saxon 8 over the NAG Library documentation for Fortran and C and check the results were the same.
The only significant difference was due not so much to a change of language but to the change of processor. Saxon has changed its default ordering for xsl:sort. As documented, adding lang="en" reverted the behaviour.
Start using XSLT2 features in the stylesheet, and start removing the existing calls to saxon6 extension functions. (This is where we are now.) This will eventually remove the need for the functions defined in step 1.
(Some time soon.) Change the specified version on the stylesheets from 1.0 to 2.0. This will turn off BC mode and will require another round of regression testing. As noted in the old email above, the stylesheets often pass parameters to named templates that do not define those parameters, which will become an error. this will be easy to find as the error is fatal and so you just fix them. Harder will be finding all the places where XSLT1's implicit "first node in document order" semantics have been used. Fortunately in our case we have a large, but relatively stable document collection to process, so it's feasible to make this change and then run automated comparisons over the results of processing the entire collection in the two modes.

This is far from being the first use of XSLT2 at NAG, but it's the first time we've switched a stylesheet collection from version 1 to version 2, so far things have gone pretty smoothly...

The EXSLT node-set function

2007-05-29T22:43:00.003+01:00

Following on from a thread on xsl-list which turned to the question of support for the useful xx:node-set() extension function in the XSLT engines used by popular browsers.

Using XPath extension functions in a cross-browser environment is greatly simplified if all the XSLT engines use the same extension namespace. This was one of the motivating reasons behind the community initiative to standardise on the EXSLT extension namespaces.

Opera's XSLT engine supports exslt:node-set, Mozilla/Firefox's XSLT engine doesn't in the current release, but does in the "Gran Paradiso" alpha tests for Firefox 3. Internet Explorer (6 and 7) don't support EXSLT, but do support the functionally identical msxsl:node-set function in the usual msxsl extension namespace.

The usual way to handle exslt:node-set and msxsl:node-set in the same document is to use xsl:choose blocks, with tests on function-available('exslt:node-set') but that is often inconvenient if you want to use xx:node-set in the middle of an XPath.

In the above XSL-List thread I casually suggested that an alternative would be to just always use exslt:node-set in the body of the stylesheet and use the msxsl:script extension to define exslt:node-set for IE. That turned out not to be as easy as I thought as node-set isn't a valid function name in either of the supported extension languages in msxsl (JScript or VBScript). However Julian Reschke came up with the construct needed, use associative array syntax so you can use ['node-set'] to define the function. A complete stylesheet using this technique is shown below

<xsl:stylesheet
  version="1.0" 
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:exslt="http://exslt.org/common"
  xmlns:msxsl="urn:schemas-microsoft-com:xslt"
  exclude-result-prefixes="exslt msxsl">
  

<msxsl:script language="JScript" implements-prefix="exslt">
 this['node-set'] =  function (x) {
  return x;
  }
</msxsl:script>


<xsl:variable name="x">
  <y/>
</xsl:variable>

<xsl:template match="x">
  <html>
    <head><title>test exslt node set</title></head>
    <body>
      <xsl:apply-templates select="exslt:node-set($x)/*"/>
    </body>
  </html>
</xsl:template>

<xsl:template match="y">
  <p>node set!</p>
</xsl:template>

</xsl:stylesheet>

The same stylesheet is online together with a test file which just consists of a stylesheet reference and an empty element <x/>.

If your browser supports EXSLT (either natively or using the technique above) viewing the test file link should show "node set!". This appears to work in Firefox Gran paradiso, Internet Explorer 7, Opera 9.21. It doesn't work in Firefox 2 (you get an error message about an unsupported extension function). Users of other browsers that support XSLT (Safari?) feel free to report any results in the comments!

Updated 2008-08-06 to move test file to a different server.

Updated 2009-12-17 to move to google code server. The file now reports that Safari 3.2 supports node-set

schematron updates

2007-05-09T02:22:00.000+01:00

Ken Holman posted a problem with the skeleton which also applied to my version described in an earlier post so I updated the code, at the same URI. There have also been some updates in the last few days removing saxon dependencies, as described in the coments, thanks to Colin Adams for picking those up. [updated again 2007/05/09] A couple of errors had crept into the schematron-get-full-path mode used to display the XPath of the current node.

XHTML and MathML from OpenOffice.org 2.2

2007-04-29T23:16:00.000+01:00

In the comments on Murray's blog I commented that (given a fixed stylesheet that does the conversion) the workflow to extract MathML from an XML file that contains MathML, and one that contains some XML transformable to MathML is pretty much the same.

This was in reply to a comment that Science (and it turns out, Nature) don't accept Office 2007 native format, partly at least due to a perceived lack of MathML support.

So, I thought I'd repeat my recent experiment Using OpenOffice.org rather than MS Office 2007.

I took the test1.docx file used as the test file in the earlier post, saved it (using Office 2007) as .doc (which warned me that the mathematics would get turned into pictures) and then imported it into OpenOffice.org, regenerated the mathematics using the formula editor provided, deleting the Word generated images, and saved as .odt. Now the aim of the game is to get an XHTML+MathML document, starting from here...

Step 1: Save as html. This produces a passable approximation to HTML (not valid but not quite so fanciful as the output from MS Word). The Mathematics is all images, and unlike Word there is no comment or other markup (not even an alt attribute) to give an alternative format. However there is a name attribute which gives the original object name, so we can retrieve the MathML from the .odt file.
Step 2: Unzip the odt file.
Step 3: Add an (empty) file math.dtd to several subdirectories extracted from the zip file, to make the content.xml files containing MathML well formed. (A catalogue would be a more sensible alternative to supplying multiple copies of the dtd).
Step 4: Run a small stylesheet that imports htmlparse.xsl to convert the HTML to XHTML, replaces the math images by their matching MathML fragments, and does a bit more cleanup on the xhtml to make it valid.
Step 5: There is no step 5:-)

The stylesheet in step 4 looks like this:

<xsl:stylesheet 
    version="2.0"
    xmlns="http://www.w3.org/1999/xhtml"
    xmlns:h="http://www.w3.org/1999/xhtml"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:d="data:,dpc"> 
  <xsl:import href="../htmlparse/htmlparse.xsl"/>
  <xsl:template name="main">
    <xsl:processing-instruction name="xml-stylesheet"
     >type="text/xsl" href="pmathml.xsl"</xsl:processing-instruction>
    <xsl:apply-templates select="d:htmlparse(
     unparsed-text('test1oo.html','windows-1252'))"/>

  </xsl:template>
  <xsl:template match="@*|node()">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
  </xsl:template>

  <xsl:template match="@align|@valign|@dir|@frame|@rules">
    <xsl:attribute name="{name()}" select="lower-case(.)"/>
  </xsl:template>

  <xsl:template match="h:img[starts-with(@name,'Object')]">
    <xsl:copy-of select="doc(concat('odt-unzip/Object ',
                             substring-after(@name,'Object'),
                             '/content.xml'))"/>
  </xsl:template>
</xsl:stylesheet>

The result of the transformation looks like this.

Comparing the stylesheet here with the one in the earlier post you'll see that the complexity in either case is about the same. Word writes all the information into a single file (as comments) which can be a bit more convenient, but OpenOffice.org places the MathML directly in the zip file which may also have advantages. Word's HTML output takes a lot more cleaning up to be valid HTML (or XHTML) but there are tools around that do that, more or less.

Rob Weir has a recent blog entry picking up on the MathML support as a reason to go with ODF. I don't buy the argument, to be honest. It's not surprising that publications are not set up to accept the zipped xml formats from any of these systems. Publisher's in house document processes are ususally somewhat complicated and fine tuned. Tooling up to accept any new format will take them time. There are plenty of reasons to argue over which format is better in any given situation, but a general statement that storing MathML is good, and storing something transformable to MathML is bad doesn't really convince me of anything, sorry!

MathML 3 .0 and MathML for CSS

2007-04-27T09:56:00.000+01:00

Two new working drafts issued today: MathML 3.0 and A MathML for CSS profile. These drafts are the first public working drafts by the current incarnation of the Math Working group. No doubt I will be posting more about those later, comments of course welcome either here or (better) on the www-math list.

htmlparse, updated

2007-04-23T00:48:00.001+01:00

There are two well known open source tools for cleaning up and fixing HTML to produce XML suitable for further processing, tidy and TagSoup.

Back in 2004 I posted some XSLT code, htmlparse.xsl, to do a similar job and used it in a few examples on the xsl-list.

When I needed such a tool for the MathML from Office post, my first thought was to use the htmlparse stylesheet. Unfortunately the output from Word was too much for the stylesheet, which couldn't cope with the Microsoft conditional declarations.

However it turned out not to be hard to extend the stylesheet, so I have updated htmlparse.xsl to fix a couple of bugs in attribute handling, and to extend it to cope with <![if !vml]> and friends. This now gives a pure XSLT route of getting from Word output to XHTML+MathML. Using the same test file as the earlier post, the stylesheet:

<xsl:stylesheet 
   version="2.0"
   xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
   xmlns:d="data:,dpc"> 
  <xsl:import href="../htmlparse/htmlparse.xsl"/>
  <xsl:import href="xhtml-mathml.xsl"/>
  <xsl:template name="main">
    <xsl:apply-templates select="d:htmlparse(
              replace(
                  unparsed-text('test1.htm','windows-1252'),
                  'http://www.w3.org/TR/REC-html40',
                  'http://www.w3.org/1999/xhtml'))"/>
  /xsl:template>
</xsl:stylesheet>

Produces an XHTML+MathML document as shown.

Although it's used here to convert an entire document, TagSoup, which interfaces to XSLT as a SAX parser, is probably preferable in this context. htmlparse (originally written just as an excercise in XSLT2 regular expressions) is probably more useful for parsing small fragments of html, as often found in query strings or embedded in CDATA in XML files. It has the advantage of being pure XSLT, not requiring any extension functions or non standard parser usage.

2009-12-18 updated location of htmlparse to google code

Testing the emacs interface

2007-04-18T23:06:00.001+01:00

Posting from emacs, editing existing post from emacs

MathML in HTML

2007-04-18T01:43:00.000+01:00

As mentioned in an earlier post, I'm experimenting with Peter Jipsen's script for using MathML in posts served as text/html. The original script assumed a MathML enabled browser, I've just added some conditional code so that if Gecko or MathPlayer are not detected, some CSS, mainly due to George Chavchanidze, is inserted. It doesn't do all of presentation MathML yet but it renders fractions, msqrt and superscripts enough to make the above post more or less legible in Opera. Things need a bit more tuning yet, I seem to have introduced a noticeable delay before the mathematics is rendered (in all browsers), but progress, I think...

Schematron Experiments

2007-04-14T00:32:00.000+01:00

Rick Jelliffe, acting as editor of ISO Schematron has a call for corrections and improvements to schematron, and also is working on updating the XSLT skeleton implementation, see this thread on the schematron-love list. This got me looking at schematron again, after a gap of some years...

The one suggestion that I made for a specification change is to allow schematron contexts to match text nodes. The current situation is very strange, schematron contexts (in XSLT1 or XSLT2 bindings) have (exactly) the syntax of XSLT patterns, but with an extra semantic restriction (not made explicitly in the specification) that any text nodes that are matched are ignored. The reference schematron implementation (current and previous version as far as I have seen) have implemented this inconsistently, applying schematron rules to text nodes (due to the fact that the default template for each pattern uses selects on node(), but not if the parent of the text node is the context of a schematron rule as then the recursion is via *|comment()|processing-instruction(). Rick has put this forward as an official change request to the working group, but seems unconvinced by the change, we'll see!

Meanwhile, looking at the code, I've not looked at the schematron implementation since I made the schematron-report (which google tells me was back in 1999, doesn't time fly:-) I think it's changed quite a bit since then, especially to accommodate the "skeleton" paradigm, but the basics are the same.

A trial implementation

There's been quite a bit of discussion on the schematron-love list about implementation changes, and posted code, but a mailing list isn't always the best place to discuss code samples, so I thought I'd try posting here.

Schematron allows you to specify several patterns and each rule has a context at which point several assertions can be made about XPath expressions. As noted above a context is more or less identical to an XSLT pattern, and the implementation of each rule in a pattern is as an XSLT template with a unique priority and a mode corresponding to the pattern to be used. For each pattern in the schematron schema, a full recursive tree walk is made via templates which process a node and then recursively process attributes and children. There are optimisation flags to try to speed this up, in particular by omitting attribute processing, and the specification that text nodes should not be visited, is partly due to a concern over this processing speed.

I argued on the list that visiting text nodes were not a serious time concern compared to other issues, and certainly not enough to distort the language. But for once I decided to heed Mike Kay's oft cited advice on xsl-list that performance questions should be answered by measuring timings.

I have made a modified schematron skeleton implementation that takes a number of parameters that control the way that the schematron patterns are implemented. Also available is a test document and test schema used in the test below.

The parameters controlling the generation of the XSLT are described below.

visit-text: This parameter defaults to 'false' at which setting text nodes are not visited when establishing the contexts for schematron patterns. This is the behaviour currently mandated by the specification. If it is set to 'true' text nodes are visited which is the behaviour but forward as a suggestion for a future schematron update. In some cases omitting text nodes is implemented by selecting *|comment()|processing-instruction() rather than node() (this does appear to save some time, but in other cases it is necessary to add a filter [not(self::text())] to explicitly skip text nodes, in which case not visiting text nodes may take extra time due to the extra check. Compare the times for 2 and 3 and 6 and 7 below. In all cases though on this test file the difference is only around 50ms in 1000, so not really significant enough to affect the language design.
attributes: Normally if recursing down the tree both the attribute and child nodes are selected, setting this to 'false' will cause the attribute to be omitted (the existing iso_schematron_skeleton has a similar parameter) one difference is that this defaults to 'true' if it is safe to do so (if no context in the schema has contains '@' or 'attribute').
only-child-elements: Similar to the attributes parameter above, this causes a test of node() to be changed to * so that just elements are visited on the child axis. Again this defaults to 'true' if it is safe to do so (no context contains a '(').
select-contexts: Perhaps the most radical option. Traditionally schematron has been implemented by a traditional walk over the tree where each step is accomplished by a template which process a node, and then recursively applies templates on child nodes and attributes. However this recursion is tail recursive and so is effectively an iteration, so an alternative implementation strategy is to first select all the nodes that need to be processed, and then process them with a non-recursive template that process a single node and does not call apply-templates.; This defaults to '', which enables the classic recursive behaviour.; If set to '//' then a an XPath is constructed from all the rule contexts in a pattern and used in a single select, so if a pattern has rules with contexts a b and c a select="//(a|b|c)" is generated. This method (on this test) leads to the quickest processor, but the difference over the default behaviour is not that great, and unfortunately it requires XSLT2 binding to allow the general step using (). To generate an XPath 1 expression would require too much parsing of the context than is really convenient in XSLT.; If set to 'key' the behaviour is similar to the behaviour with '//' but rather than use an XPath using // an XSLT key is generated. This has the advantage that it is much easier to generate legal XSLT1, and my intuition was that this would be the fastest so long as memory for the key did not get out of hand. It seems that my intuition here was wildly, spectacularly wrong. As can be seen from the timings below, runs 4 and 5, using this key setting take around 20 times longer.

One unrelated and largely cosmetic change, the schematron implementation needs to generate unique priorities for each template that it generates. The exact numbers do not matter, just the relative ordering. The existing ISO implementation counts down from 4000. the 4000 always bothered me (I don't know why, really) so I have changed it here to count up from 1000 (keeping the relative ordering). Strictly speaking, counting down from 4000 isn't safe as if you have 4000 templates the priorities specified would be less than the priorities specified on default templates in the same mode, however the real reason is cosmetic. (I was tempted to count up from 1 and lower the default templates more, but I stared from 1000, just to give them all 4 digit numbers.

Results for one test document

The following script was used, it's a bash script but can just be viewed as a sequence of command line instructions and equivalent code could be used in any operating system

Run 1 uses the current schematron beta skeleton, as a base comparison., runs 2 to 7 uses the modified implementation using various combinations of the parameters described above.

the schematron output (log1.txt ... log7.txt) is checked with the unix diff command: no output here is good, which means that the output is the same in each case. the -3 flag to saxon causes it to run the code three times and report the average time (thus avoiding measuring JVM startup and stylesheet compilation times).

#!/bin/bash

rm temp?.xsl
saxon8 -novw -o temp1.xsl dpc.sch
   iso_schematron_skeleton.xsl
saxon8 -novw -o temp2.xsl dpc.sch
   dpc_schematron_skeleton.xsl
saxon8 -novw -o temp3.xsl dpc.sch
   dpc_schematron_skeleton.xsl visit-text=true
saxon8 -novw -o temp4.xsl dpc.sch
   dpc_schematron_skeleton.xsl select-contexts=key
saxon8 -novw -o temp5.xsl dpc.sch
   dpc_schematron_skeleton.xsl visit-text=true
                               select-contexts=key
saxon8 -novw -o temp6.xsl dpc.sch
   dpc_schematron_skeleton.xsl select-contexts=//
saxon8 -novw -o temp7.xsl dpc.sch
   dpc_schematron_skeleton.xsl visit-text=true
                               select-contexts=//

rm log?.txt logg?.txt
saxon8 -3 -novw -o log1.txt book.xml temp1.xsl 2> logg1.txt
saxon8 -3 -novw -o log2.txt book.xml temp2.xsl 2> logg2.txt
saxon8 -3 -novw -o log3.txt book.xml temp3.xsl 2> logg3.txt
saxon8 -3 -novw -o log4.txt book.xml temp4.xsl 2> logg4.txt
saxon8 -3 -novw -o log5.txt book.xml temp5.xsl 2> logg5.txt
saxon8 -3 -novw -o log6.txt book.xml temp6.xsl 2> logg6.txt
saxon8 -3 -novw -o log7.txt book.xml temp7.xsl 2> logg7.txt

echo diff
diff log1.txt log2.txt
diff log1.txt log3.txt
diff log1.txt log4.txt
diff log1.txt log5.txt
diff log1.txt log6.txt
diff log1.txt log7.txt

echo grep
grep Average logg*txt

And the results are:

diff
grep
logg1.txt:*** Average execution time over 3 runs: 1078ms
logg2.txt:*** Average execution time over 3 runs: 1052ms
logg3.txt:*** Average execution time over 3 runs: 1062ms
logg4.txt:*** Average execution time over 3 runs: 21843ms
logg5.txt:*** Average execution time over 3 runs: 21885ms
logg6.txt:*** Average execution time over 3 runs: 1047ms
logg7.txt:*** Average execution time over 3 runs: 995ms

The default behaviour (2) of this script is a little faster than (1), probably because it visits fewer text nodes to more fully implement the current standard, When all text nodes are visited in run (3) it's slightly slower. As noted above (4) and (5) are unusable, unless this turns out to be a bug in either my code or the XSLT processor, this indicates that the 'key' option should be removed from any final release. 6 and 7 using // and not visiting or visiting text nodes turns out to be the fastest here, but probably not sufficiently different to make it worth having this as an option in a final release, unless perhaps other people with real documents who have found the classic recursive method slow report this as faster. (I suspect some documents will be faster, some slower, depending greatly on the documents and the underlying XSLT processor.

Licence

Most of the code in the modified skeleton is Copyright Rick Jelliffe, and the code is distributed under the same licence as his implementation of which this is just a minor modification. The comments at the top of the file say:

<!-- DPC
Modified schematron skeleton code by David Carlisle.
http://dpcarlisle.blogspot.com/search/label/schematron

All modifications are marked with XML comments starting
with " DPC".

The majority of the code is copyright
Rick Jelliffe and Academia Sinica Computing Center
distributed under the license below, see "LEGAL NOTICE".
The modified sections of code are the work of David Carlisle and are
distributed under the same licence. Rick Jelliffe or others are free to
incorporate David Carlisle's code back in to other schematron implementations
which use this licence, for any such incorporation, specific attribution is
not required, although the attribution style used for earlier contribution
is appreciated.

This is a test implementation for discussion on the schematron mailing list
suggesting some possible changes and enhancements for a schematron implementation.
People are welcome to try the code and comment, but for production use you are
strongly advised to use the "unofficial reference" implementation from schematron.org
This is not intended to be a long term "competing" implementation.
-->

XHTML and MathML from Office 2007

2007-04-10T02:44:00.001+01:00

xhtml-mathml stylesheet updated 2007/05/09

I commented on Murray's blog That it ought to be possible to get XHTML/MathML documents out of Word. Having speculated that it ought be possible, conscience dictated that I try to do it. The results are described below.

The first problem was that I'd never used Word (or any other WYSIWYG editor). It seems strange to me, but then I've been using emacs so long I'm probably corrupted.

Word 2007 has MathML input/output (via an XSL stylesheet installed with the system), and has HTML input/output (via its save as web page file menu), so the plan of action is: save the document as html, clean it up to xhtml, using the stylesheet to convert the mathematics to MathML at the same time.

Write your document in Word 2007, save as web page file.htm .
Use tagsoup to get some usable XML from this output. java -jar tagsoup-1.1.jar --lexical --output-encoding=iso-8859-1 file.htm > temp.xml
Use the supplied xhtml-mathml stylesheet to do some further cleanup and apply the Microsoft supplied omml2mml.xsl stylesheet to the math fragments. java -jar saxon8.jar -o file.xml temp.xml xhtml-mathml.xsl

Example:

Comments welcome!

testing the interface

2007-04-09T22:41:00.001+01:00

Just testing...

I probably won't be posting too much MathML here, but I wanted to check it was possible. Unfortunately I think blogger always serves as text/html, but Peter Jipsen's script seems to work well in this context. Currently it requires a Gecko browser such as Firefox, or IE with MathPlayer installed. Hopefully I can extend the script to allow some basic CSS rendering in other browsers later.

I also needed to check I could write posts in emacs of course!

Finally some examples using the MathML in HTML script. $\frac{1}{2}$

x = \frac{- b \pm \sqrt{b^{2} - 4 a c}}{2 a}

[update, continuous Relax NG validation of the mathml in xhtml in atom markup as I type into this emacs buffer. Isn't emacs wonderful. Will post schemas used shortly.]

Hello World

2007-03-25T00:10:00.000Z

I thought I'd start a blog.....