XML position at NAG

NAG is looking for someone to work in the XML Technologies Group, that is, "my" group. So if you are interested, please contact the address specified in the above posting.

The Big Switch

There was a thread a while ago on xsl-list discussing when was a good time to switch from XSLT1 to XSLT2. The consensus seemed to be, "it depends...."

For NAG, the answer is NOW!

After a lot of regression testing, staring at diff files and crossing of fingers, we just switched the processor for our main stylesheets over from saxon 6 to saxon 8.

This is quite a large (set of) stylesheets (around 26K lines of XSLT) so is quite a substantial test of XSLT2's compatibility with XSLT1. The results were pretty good, although it's not exactly surprising as an earlier version of these stylesheets were used to request improvements to backward compatibility mode as defined in an earlier draft, all the main changes requested in that report were made.

We are taking a multi-stage process to switching to version 2:

1. Import an XSLT2 stylesheet that defines a few extension functions in the saxon 6 extension namespace using xsl:function and standard function calls. saxon:node-set defined as identity function, saxon:tokenize defined using XPath2 tokenize, saxon:distinct defined using XPath2 distinct-values etc.
2. Process the stylesheets with saxon 8 over the NAG Library documentation for Fortran and C and check the results were the same.
3. The only significant difference was due not so much to a change of language but to the change of processor. Saxon has changed its default ordering for xsl:sort. As documented, adding lang="en" reverted the behaviour.
4. Start using XSLT2 features in the stylesheet, and start removing the existing calls to saxon6 extension functions. (This is where we are now.) This will eventually remove the need for the functions defined in step 1.
5. (Some time soon.) Change the specified version on the stylesheets from 1.0 to 2.0. This will turn off BC mode and will require another round of regression testing. As noted in the old email above, the stylesheets often pass parameters to named templates that do not define those parameters, which will become an error. this will be easy to find as the error is fatal and so you just fix them. Harder will be finding all the places where XSLT1's implicit "first node in document order" semantics have been used. Fortunately in our case we have a large, but relatively stable document collection to process, so it's feasible to make this change and then run automated comparisons over the results of processing the entire collection in the two modes.

This is far from being the first use of XSLT2 at NAG, but it's the first time we've switched a stylesheet collection from version 1 to version 2, so far things have gone pretty smoothly...

The EXSLT node-set function

Following on from a thread on xsl-list which turned to the question of support for the useful xx:node-set() extension function in the XSLT engines used by popular browsers.

Using XPath extension functions in a cross-browser environment is greatly simplified if all the XSLT engines use the same extension namespace. This was one of the motivating reasons behind the community initiative to standardise on the EXSLT extension namespaces.

Opera's XSLT engine supports exslt:node-set, Mozilla/Firefox's XSLT engine doesn't in the current release, but does in the "Gran Paradiso" alpha tests for Firefox 3. Internet Explorer (6 and 7) don't support EXSLT, but do support the functionally identical msxsl:node-set function in the usual msxsl extension namespace.

The usual way to handle exslt:node-set and msxsl:node-set in the same document is to use xsl:choose blocks, with tests on function-available('exslt:node-set') but that is often inconvenient if you want to use xx:node-set in the middle of an XPath.

In the above XSL-List thread I casually suggested that an alternative would be to just always use exslt:node-set in the body of the stylesheet and use the msxsl:script extension to define exslt:node-set for IE. That turned out not to be as easy as I thought as node-set isn't a valid function name in either of the supported extension languages in msxsl (JScript or VBScript). However Julian Reschke came up with the construct needed, use associative array syntax so you can use ['node-set'] to define the function. A complete stylesheet using this technique is shown below

<xsl:stylesheet
  version="1.0" 
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  xmlns:exslt="http://exslt.org/common"
  xmlns:msxsl="urn:schemas-microsoft-com:xslt"
  exclude-result-prefixes="exslt msxsl">
  

<msxsl:script language="JScript" implements-prefix="exslt">
 this['node-set'] =  function (x) {
  return x;
  }
</msxsl:script>


<xsl:variable name="x">
  <y/>
</xsl:variable>

<xsl:template match="x">
  <html>
    <head><title>test exslt node set</title></head>
    <body>
      <xsl:apply-templates select="exslt:node-set($x)/*"/>
    </body>
  </html>
</xsl:template>

<xsl:template match="y">
  <p>node set!</p>
</xsl:template>

</xsl:stylesheet>

The same stylesheet is online together with a test file which just consists of a stylesheet reference and an empty element <x/>.

If your browser supports EXSLT (either natively or using the technique above) viewing the test file link should show "node set!". This appears to work in Firefox Gran paradiso, Internet Explorer 7, Opera 9.21. It doesn't work in Firefox 2 (you get an error message about an unsupported extension function). Users of other browsers that support XSLT (Safari?) feel free to report any results in the comments!

Updated 2008-08-06 to move test file to a different server.

Updated 2009-12-17 to move to google code server. The file now reports that Safari 3.2 supports node-set

schematron updates

Ken Holman posted a problem with the skeleton which also applied to my version described in an earlier post so I updated the code, at the same URI. There have also been some updates in the last few days removing saxon dependencies, as described in the coments, thanks to Colin Adams for picking those up. [updated again 2007/05/09] A couple of errors had crept into the schematron-get-full-path mode used to display the XPath of the current node.

XHTML and MathML from OpenOffice.org 2.2

In the comments on Murray's blog I commented that (given a fixed stylesheet that does the conversion) the workflow to extract MathML from an XML file that contains MathML, and one that contains some XML transformable to MathML is pretty much the same.

This was in reply to a comment that Science (and it turns out, Nature) don't accept Office 2007 native format, partly at least due to a perceived lack of MathML support.

So, I thought I'd repeat my recent experiment Using OpenOffice.org rather than MS Office 2007.

I took the test1.docx file used as the test file in the earlier post, saved it (using Office 2007) as .doc (which warned me that the mathematics would get turned into pictures) and then imported it into OpenOffice.org, regenerated the mathematics using the formula editor provided, deleting the Word generated images, and saved as .odt. Now the aim of the game is to get an XHTML+MathML document, starting from here...

Step 1

Save as html. This produces a passable approximation to HTML (not valid but not quite so fanciful as the output from MS Word). The Mathematics is all images, and unlike Word there is no comment or other markup (not even an alt attribute) to give an alternative format. However there is a name attribute which gives the original object name, so we can retrieve the MathML from the .odt file.

Step 2

Unzip the odt file.

Step 3

Add an (empty) file math.dtd to several subdirectories extracted from the zip file, to make the content.xml files containing MathML well formed. (A catalogue would be a more sensible alternative to supplying multiple copies of the dtd).

Step 4

Run a small stylesheet that imports htmlparse.xsl to convert the HTML to XHTML, replaces the math images by their matching MathML fragments, and does a bit more cleanup on the xhtml to make it valid.

Step 5

There is no step 5:-)

The stylesheet in step 4 looks like this:

<xsl:stylesheet 
    version="2.0"
    xmlns="http://www.w3.org/1999/xhtml"
    xmlns:h="http://www.w3.org/1999/xhtml"
    xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
    xmlns:d="data:,dpc"> 
  <xsl:import href="../htmlparse/htmlparse.xsl"/>
  <xsl:template name="main">
    <xsl:processing-instruction name="xml-stylesheet"
     >type="text/xsl" href="pmathml.xsl"</xsl:processing-instruction>
    <xsl:apply-templates select="d:htmlparse(
     unparsed-text('test1oo.html','windows-1252'))"/>

  </xsl:template>
  <xsl:template match="@*|node()">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()"/>
    </xsl:copy>
  </xsl:template>

  <xsl:template match="@align|@valign|@dir|@frame|@rules">
    <xsl:attribute name="{name()}" select="lower-case(.)"/>
  </xsl:template>

  <xsl:template match="h:img[starts-with(@name,'Object')]">
    <xsl:copy-of select="doc(concat('odt-unzip/Object ',
                             substring-after(@name,'Object'),
                             '/content.xml'))"/>
  </xsl:template>
</xsl:stylesheet>

The result of the transformation looks like this.

Comparing the stylesheet here with the one in the earlier post you'll see that the complexity in either case is about the same. Word writes all the information into a single file (as comments) which can be a bit more convenient, but OpenOffice.org places the MathML directly in the zip file which may also have advantages. Word's HTML output takes a lot more cleaning up to be valid HTML (or XHTML) but there are tools around that do that, more or less.

Rob Weir has a recent blog entry picking up on the MathML support as a reason to go with ODF. I don't buy the argument, to be honest. It's not surprising that publications are not set up to accept the zipped xml formats from any of these systems. Publisher's in house document processes are ususally somewhat complicated and fine tuned. Tooling up to accept any new format will take them time. There are plenty of reasons to argue over which format is better in any given situation, but a general statement that storing MathML is good, and storing something transformable to MathML is bad doesn't really convince me of anything, sorry!

MathML 3 .0 and MathML for CSS

Two new working drafts issued today: MathML 3.0 and A MathML for CSS profile. These drafts are the first public working drafts by the current incarnation of the Math Working group. No doubt I will be posting more about those later, comments of course welcome either here or (better) on the www-math list.

htmlparse, updated

There are two well known open source tools for cleaning up and fixing HTML to produce XML suitable for further processing, tidy and TagSoup.

Back in 2004 I posted some XSLT code, htmlparse.xsl, to do a similar job and used it in a few examples on the xsl-list.

When I needed such a tool for the MathML from Office post, my first thought was to use the htmlparse stylesheet. Unfortunately the output from Word was too much for the stylesheet, which couldn't cope with the Microsoft conditional declarations.

However it turned out not to be hard to extend the stylesheet, so I have updated htmlparse.xsl to fix a couple of bugs in attribute handling, and to extend it to cope with <![if !vml]> and friends. This now gives a pure XSLT route of getting from Word output to XHTML+MathML. Using the same test file as the earlier post, the stylesheet:

<xsl:stylesheet 
   version="2.0"
   xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
   xmlns:d="data:,dpc"> 
  <xsl:import href="../htmlparse/htmlparse.xsl"/>
  <xsl:import href="xhtml-mathml.xsl"/>
  <xsl:template name="main">
    <xsl:apply-templates select="d:htmlparse(
              replace(
                  unparsed-text('test1.htm','windows-1252'),
                  'http://www.w3.org/TR/REC-html40',
                  'http://www.w3.org/1999/xhtml'))"/>
  /xsl:template>
</xsl:stylesheet>

Produces an XHTML+MathML document as shown.

Although it's used here to convert an entire document, TagSoup, which interfaces to XSLT as a SAX parser, is probably preferable in this context. htmlparse (originally written just as an excercise in XSLT2 regular expressions) is probably more useful for parsing small fragments of html, as often found in query strings or embedded in CDATA in XML files. It has the advantage of being pure XSLT, not requiring any extension functions or non standard parser usage.

2009-12-18 updated location of htmlparse to google code