VTD-XML Benchmark Report for XPath Evaluation

Objective

This articles reports the measurement of XPath evaluation performance for VTD-XML version 1.9, and compares with Xerces 2.7.1 DOM and XPath functions bundled with JDK 1.5 (which is Xalan), as well as Jaxen 1.1.

Methodology Discussion

What is measured?

This benchmark report focuses on XPath evaluation performance only, i.e. the test code parses the document into internal structure, then repetitively evaluates a single pre-compiled XPath over the entire document. Our test measures the time (aka. latency) it takes to evaluate the XPath over the entire document. 

How to measure?

The benchmark code performs the following steps:

1. Read XML file into an in-memory buffer.
2. Parse the document once.
3. Evaluate a single pre-compiled XPath expression for a large number of iterations.

All benchmark programs first loop thru the XPath evaluation code a number of iterations so the server JVM compile them into native code to obtain optimal performance, before the real measurement of the average latency of subsequent XPath evaluations.

Test Setup

Hardware

• Processor:  Pentium M 1.7GHz (2MB L2 integrated cache). Large cache size is typical among multi-core server processors.
• Memory: 512 MB 

Software

• Windows XP
• JDK version 1.506; default set to server JVM.
• XML parsers: Xerces DOM version 2.7.1and XPath bundled with JDK 1.5 (Xalan), Xerces DOM 2.7.1 and Jaxen, VTD-XML 1.9

The XML files

Three XML files of similar structure, but different sizes, are used for the test.

<?xml version="1.0"?>
<purchaseOrder orderDate="1999-10-20">
    <shipTo country="US">
        <name>Alice Smith</name>
        <street>123 Maple Street</street>
        <city>Mill Valley</city>
        <state>CA</state>
        <zip>90952</zip>
    </shipTo>
    <billTo country="US">
        <name> Robert Smith </name>
        <street>8 Oak Avenue</street>
        <city>Old Town</city>
        <state>PA</state>
        <zip>95819</zip>
    </billTo>
    <comment>Hurry, my lawn is going wild!</comment>
    <items>
        <item partNum="872-AA">
            <productName>Lawnmower</productName>
            <quantity></quantity>
            <USPrice>148.95</USPrice>
            <comment>Confirm this is electric</comment>
        </item>
        <item partNum="926-AA">
            <productName>Baby Monitor</productName>
            <quantity>1</quantity>
            <USPrice>39.98</USPrice>
            <shipDate>1999-05-21</shipDate>
        </item>
        <item partNum="872-AA">
            <productName>Lawnmower</productName>
            <quantity><![CDATA[1]]></quantity>
            <USPrice>148.95</USPrice>
            <comment>Confirm this is electric</comment>
        </item>
        <item partNum="926-AA">
            <productName>Baby Monitor</productName>
            <quantity>1</quantity>
            <USPrice>39.98</USPrice>
            <shipDate>1999-05-21</shipDate>
         </item>
               ...
    </items>
</purchaseOrder>

The respective file sizes are:

• "po_small.xml" ----  6780 bytes
• "po_medium.xml" ---- 112,238 bytes
• "po_big.xml" -----   1,219,388 bytes

The following XPath expressions are used for the test

• /*/*/*[position() mod 2 = 0]
• /purchaseOrder/items/item[USPrice<100]
• /*/*/*/quantity/text()
• //item/comment
• //item/comment/../quantity
• /purchaseOrder/items/item[USPrice > 100]/comment/text()
• /*/*/*/* [position() mod 2 = 0]

The Code Sample

A snippet of the DOM/JDK benchmark code is shown below:

Document d = null;
k = total;
d = parser.parse(bais);
while (k > 0) {
        NodeList nodeList = (NodeList) xPathExpression.evaluate(d,
        XPathConstants.NODESET);
        k--;
}
long l=0,lt=0;
for (int j = 0; j < 10; j++) {
        l = System.currentTimeMillis();
        for (int i = 0; i < total; i++) {
            NodeList nodeList = (NodeList) xPathExpression.evaluate(d,
            XPathConstants.NODESET);
}
long l2 = System.currentTimeMillis();
lt = lt + (l2 - l);
}
System.out.println(" average XPath latency ==> "
+ ((double) (lt) / total / 10) + " ms");

 

Jaxen's code looks like the following:

 XPath expression = new org.jaxen.dom.DOMXPath(xpe);
System.out.println("====> update_jaxen1 ==>"+xpe);
//parse and xpath eval

Document d = null;
d = parser.parse(bais);
k = total;
while (k > 0) {
   List results = expression.selectNodes(d);
   k--;
}
long l=0,lt=0;
for (int j = 0; j < 10; j++) {
l = System.currentTimeMillis();
for (int i = 0; i < total; i++) {
   List results = expression.selectNodes(d);
}
long l2 = System.currentTimeMillis();
lt = lt + (l2 - l);
}
System.out.println(" average XPath latency ==> "
+ ((double) (lt) / total / 10) + " ms");

In comparison, a sample snippet of VTD-XML benchmark code is shown below:

k=total;
VTDNav vn = null;
while(k>0){
        while((i1=ap.evalXPath())!=-1){
        }
        ap.resetXPath();
        k--;
}
for (int j=0;j<10;j++){
        l = System.currentTimeMillis();
        for(int i = 0;i<total;i++)
        {
            while((i1=ap.evalXPath())!=-1){
            }
            ap.resetXPath();
        }
        long l2 = System.currentTimeMillis();
        lt = lt + (l2 - l);
        //System.out.println("j ==> "+j);
}
System.out.println(" average XPath latency ==> "+
((double)(lt)/total/10) + " ms");

Results

Notice that the table includes # of nodes returned from XPath evaluation.

Latency Number Comparison

/*/*/*[position() mod 2 = 0]

/purchaseOrder/items/item[USPrice <100]

/*/*/*/quantity/text()

//item/comment

//item/comment/../quantity

/*/*/*/* [position() mod 2 = 0]

/purchaseOrder/items/item[USPrice > 100]/comment/text()

Graphical View

/*/*/*[position() mod 2 = 0]

/purchaseOrder/items/item[USPrice <100]

/*/*/*/quantity/text()

//item/comment

//item/comment/../quantity

/*/*/*/* [position() mod 2 = 0]

/purchaseOrder/items/item[USPrice > 100]/comment/text()

Conclusion

As readers can see from the benchmark, it is particularly troubling that the JDK's bundled XPath engine (rumored to be Xalan) is quite slow, especially for small documents, whose XPath performance is over twice as slow as parsing. Something seems to be extraordinarily wrong because although DOM's parsing is slow, one would expect that the random access nature of DOM would enable a speedy XPath evaluation. But it is not the case. In contrast, VTD-XML's XPath is far better!