XProc is the XML Processing pipelining language.  It is "a language designed for describing operations to be performed on XML documents." Callimachus uses XProc for a variety of functions; XProc is used when an operation needs to tranform XML data or fetch data from remote sources.  XProc pipelines can, like Unix pipelines, consist of multiple steps and can also branch into multiple streams.

Our goal will be to allow individuals to represent their interests in an XML file, publish it somewhere and then provide the URL to the file in their address book entries.  The Person View template will be modified to show a list of that person's interests.

The final result will look like this:

Person with interests

We will show you how to retrieve an XML file from a remote server, transform the file into an HTML fragment and use JavaScript to insert the fragment into the Person View template.  The Create and Edit templates will be updated to allow the collection of a URL to an XML document.

Overview of the Process

The following figure shows what we will do:

XProc application data flow summary

The steps are:

  1. Retrieve an XML document
  2. Use XSLT to transform the XML contents into an HTML unordered list
  3. Call the XProc pipeline from JavaScript to get the unordered list
  4. Insert the list into the Person View template for a given Person

Define an XML Document Representing a Person's Interests

We want to represent a person's interests in XML format, so we will use a simple XML structure like this:


<person>
  <name>
    <firstname>...</firstname>
    <lastname>...</lastname>
  </name>
  <interest>
    <label>...</label>
    <url>...</url>
  </interest>
</person>

Further, we want a person to be able to list as many interests as they like, so we will allow the <interest> tag to be used multiple times.

Those of you who are already familiar with RDF and Linked Data might be wondering why we didn't use Friend-of-a-Friend (FOAF) files for this since they are in RDF, many people already have them published and the FOAF specification includes an interest element.  We could have done that (in fact, you can view David Wood's FOAF file that includes interests), but our one of our goals was to demonstrate how Callimachus can easily make use of XML data since it is so commonly available from enterprise software.

We have provided a sample XML interests file that is served live on the Web for you to use until you publish your own.  It is located at http://3roundstones.com/dave/interests.xml.  The file's contents are:


<?xml version="1.0"?>
<person>
 <name>
   <firstname>David</firstname>
   <lastname>Wood</lastname>
 </name>
 <interest>
	<label>Linked Data</label>
	<url>https://en.wikipedia.org/wiki/Linked_Data</url>
 </interest>
 <interest>
	<label>Resource Description Framework (RDF)</label>
	<url>https://en.wikipedia.org/wiki/Resource_Description_Framework</url>
 </interest>
 <interest>
	<label>Metadata</label>
	<url>https://en.wikipedia.org/wiki/Metadata</url>
 </interest>
 <interest>
	<label>Semantic Web</label>
	<url>https://en.wikipedia.org/wiki/Semantic_Web</url>
 </interest>
 <interest>
	<label>Origins of agriculture</label>
	<url>https://en.wikipedia.org/wiki/History_of_agriculture</url>
 </interest>
 <interest>
	<label>Theory of Memes</label>
	<url>https://en.wikipedia.org/wiki/Meme</url>
 </interest>
 <interest>
	<label>Egyptian hieroglyphs</label>
	<url>https://en.wikipedia.org/wiki/Hieroglyphics</url>
 </interest>
 <interest>
	<label>Software maintenance</label>
	<url>https://en.wikipedia.org/wiki/Software_maintenance</url>
 </interest>
 <interest>
	<label>Cognition</label>
	<url>https://en.wikipedia.org/wiki/Cognition</url>
 </interest>
</person>

Write an XProc Pipeline to Retrieve, Transform and Render XML data

The first step is to create the XProc pipeline.  XProc has its own unique syntax that takes some getting used to.  Don't be too concerned about trying to learn XProc at the same time as you are learning Callimachus.  Rather, try to follow how Callimachus is using XProc for now and go on to learn more about XProc later.

Navigate to the initial-app/pipelines folder and select "Pipeline" from the Create menu:

Create an XProc pipeline

Callimachus will give you a dialog box showing a simple (non-functional) XProc pipeline.  You will need to define some meaningful XProc steps and save your pipeline.  We want to define an XProc pipeline that will:

  1. Return an XML document, but omit the XML header declaration;
  2. Require a parameter that represents the URL to a remote XML document;
  3. Retrieve the XML document from that URL;
  4. Pass the document through an XSLT transformation to turn it into an HTML unordered list.

The top of any XProc pipeline in Callimachus defines some handy namespaces:


<?xml version="1.0" encoding="UTF-8" ?>
<p:pipeline version="1.0"
        xmlns:p="http://www.w3.org/ns/xproc"
        xmlns:c="http://www.w3.org/ns/xproc-step"
        xmlns:l="http://xproc.org/library"
        xmlns:calli="http://callimachusproject.org/rdf/2009/framework#">

Next, we want to tell the pipeline to return an HTML document without the XML declaration.  The <p:serialization> step is used for that:


<p:serialization port="result" media-type="text/html" method="xml" omit-xml-declaration="true" />

If you wanted to return a different media type such as text/plain, you could do that here.

Our pipeline will require a parameter that is a URL.  We will call that parameter "href", but you can name it anything you like.  The <p:option> step is used to define parameters:


<p:option name="href" required="true" />

 In order for Callimachus to process our pipeline, it will need to load a library.  You need to load libraries after providing namespaces, serialization information and parameter definitions:


<p:import href="/callimachus/library.xpl" />

Now we need to fetch the URL that we have been given.  The <p:load> step will fetch the URL and return its results.  The URL to fetch was put into the $href variable earlier:


<p:load> 
    <p:with-option 
        name="href" 
        select="$href" 
    /> 
</p:load>

At this point, the pipeline has the contents of the remote XML document.  Now we need to pass it through an XML Stylesheet Language Transformation, or XSLT file, to turn it into an HTML list:


<p:xslt name="find-interests">
    <p:input port="stylesheet">
        <p:document href="../transforms/find-interests.xsl" />
    </p:input>
</p:xslt>

Finally, we end the pipeline definition:


</p:pipeline>

The entire pipeline looks like this:


<?xml version="1.0" encoding="UTF-8" ?>
<p:pipeline version="1.0"
        xmlns:p="http://www.w3.org/ns/xproc"
        xmlns:c="http://www.w3.org/ns/xproc-step"
        xmlns:l="http://xproc.org/library"
        xmlns:calli="http://callimachusproject.org/rdf/2009/framework#">

<p:serialization port="result" media-type="text/html" method="xml" omit-xml-declaration="true" />

<p:option name="href" required="true" /> 

<p:import href="/callimachus/library.xpl" />

<p:load> 
    <p:with-option 
        name="href" 
        select="$href" 
    /> 
</p:load> 

<p:xslt name="find-interests">
    <p:input port="stylesheet">
        <p:document href="../transforms/find-interests.xsl" />
    </p:input>
</p:xslt>

</p:pipeline>

You can save your pipeline now in the /initial-app/pipelines/ folder.  Give it the name find-interests.xpl so it matches the name we used in the finished application.

But what about that XSLT file we snuck into the last step?  Navigate to the /initial-app/transforms/ folder and create an XSLT transform file:

Create an XSLT transformation file

XSLT is yet another language in the XML tool chain.  You are probably familiar with it if you work with XML documents.  If not, you might want to start by reading the XSLT page on Wikipedia.

Our XSLT file will:

  1. Find any <person> tags in the file that it is operating on;
  2. Print an HTML unordered list tag (<ul>);
  3. Find any <interest> tags and for each one:
    1. Print an HTML list item tag (<li>);
    2. Print an HTML anchor tag (<a>) with an href attribute consisting of the value of the XML <url> tag;
    3. Print the value of the XML <label> tag;
    4. Close the anchor (</a>);
    5. Close the list item (</li>).

The contents of the XSLT file should look like this:


<?xml version="1.0"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" indent="yes"/>
<xsl:strip-space elements="*" />

  <xsl:template match="person">
    <div id="interestlist">
      <p class="lead">Interests</p>
      <ul>
        <xsl:for-each select="interest">
          <li>
            <a>
              <xsl:attribute name="href">
                <xsl:value-of select="descendant::url" />
              </xsl:attribute>
              <xsl:value-of select="descendant::label" />
            </a>
          </li>
        </xsl:for-each>
      </ul>
    </div>
  </xsl:template>

</xsl:stylesheet>

Save your XSLT file in the /initial-app/transforms/ folder using the name find-interests.xsl.

Test the Pipeline

You can test the pipeline by calling its URL with the ?results pragma.  Adding ?results to the end of the URL will execute the pipeline and return its results.  Try it on the finished application's pipeline first:  ../finished-app/pipelines/find-interests.xpl?results

You should see an error that looks like this:

Pipeline error

Oops!  What happened?  Did you remember that the href parameter was defined as mandatory?  The pipeline won't execute without that parameter being present.

Try it again, this time by adding the parameter to the query string of the URL:  ../finished-app/pipelines/find-interests.xpl?results&href=http://3roundstones.com/dave/interests.xml

Now you should see the results, an HTML unordered list, as in the following screenshot:

Interests list

You can also call the pipeline from the command line using a utility such as curl, which is available for most operating systems.  The curl command line will need provide the entire URL and use authentication for your user account.  For example, the Callimachus user david who signs in using the HTTP Digest authentication method could access the pipeline results like this:


curl --digest --user david 'https://wiki.3roundstones.com/Development/Projects/Tutorial/finished-app/pipelines/find-interests.xpl?results&href=http://3roundstones.com/dave/interests.xml'

The curl utility would then ask for david's password and present the results:


<ul><li><a href="https://en.wikipedia.org/wiki/Linked_Data">Linked Data</a></li><li><a href="https://en.wikipedia.org/wiki/Resource_Description_Framework">Resource Description Framework (RDF)</a></li><li><a href="https://en.wikipedia.org/wiki/Metadata">Metadata</a></li><li><a href="https://en.wikipedia.org/wiki/Semantic_Web">Semantic Web</a></li><li><a href="https://en.wikipedia.org/wiki/History_of_agriculture">Origins of agriculture</a></li><li><a href="https://en.wikipedia.org/wiki/Meme">Theory of Memes</a></li><li><a href="https://en.wikipedia.org/wiki/Hieroglyphics">Egyptian hieroglyphs</a></li><li><a href="https://en.wikipedia.org/wiki/Software_maintenance">Software maintenance</a></li><li><a href="https://en.wikipedia.org/wiki/Cognition">Cognition</a></li></ul>

The white space was stripped from the results by using this command in the XSLT file:


<xsl:strip-space elements="*" />

Modify the Create, Edit and View Templates

Now that we have a working XProc pipeline, we will need to add the ability to collect URLs to XML interests files to our address book application.  The Create and Edit templates will need to provide the new URL field and the View template will need to make use of it.

Our steps will be:

  1. Add a field for entering URL to XML document to Create and Edit templates
  2. Add the property to the View template

On the Person Create and Edit templates, add this code between the "FOAF Profile Page" text field and the "Status" pulldown:


<div class="form-group">
  <label for="interests">XML interests document</label>
  <input type="url" class="form-control" id="interests" value="{tute:interests}" placeholder="http://3roundstones.com/dave/interests.xml" />
</div>

That will create (or edit) a new property using the RDF predicate tute:interests.  We just made that up; you can call it anything you like.  Of course, you will also need to add the definition of the tute namespace to the top of both templates.  We just assigned a unique URI to that namespace:


xmlns:tute="http://callimachusproject.org/2014/03/tutorial#"

As always, have a look at the finished app's Create and Edit templates if you need to see exactly where this information goes.

The Person View template will also need to have the tute namespace defined at the top of the file.  Callimachus will complain if you attempt to use a namespace that is not defined.

Add a reference to the tute:interests property in the View template.  Callimachus templates will only render properties that have been defined in their HTML, so we must provide a reference if we want the interests to appear.  We could hide the HTML division if we were worried about it, but we intend for JavaScript to operate on the URL on page load, so we didn't hide it for this tutorial.  The reference should look like this:


<div id="interests" class="col-sm-2">{tute:interests}</div>

We gave the division an id attribute so the JavaScript could easily locate it.  The CSS class tells Bootstrap how to lay out the division.  The first row of the View template now looks like:


<div class="row">
  <div class="col-sm-4 col-sm-offset-4 text-center">
    <img src="{foaf:depiction}" class="img-circle" />
  </div>
  <div class="col-sm-2">
    <p class="lead">Currently <span id="status" rel="foaf:status" resource="?status" class="label label-info">{skos:prefLabel}</span></p>
  </div>
  <div id="interests" class="col-sm-2">{tute:interests}</div>
</div>

You may want to play around with the Bootstrap column numbers to allocate more or less space (or change the layout entirely if you prefer).

You might also want to adjust the CSS at the top of the View template so the interests list lays out differently from the lists below the person's name. Something like this will help:

Assign a class called em on the existing unordered lists:


<ul class="em">

...and update the CSS to:


<style>
        ul.em {
            list-style: none;
            padding-left: 0;
        }
        ul {
            padding-left: 20px;
        }
        .em {
            font-size: 18px;    
        }
</style>

That ensures that the existing lists do not get bullet points added to them, but the new interests list will.

Remember, you can always compare your work to the finished application.

Add JavaScript to Call the Pipeline

There is already a JavaScript file associated with the Person View template called scripts/person-view.js.  We will want to modify that file to execute the pipeline and insert the new list into place if the template includes the tute:interests property.  Specifically, our steps will be to write some JavaScript that:

  1. Retrieves the XML URL if present
  2. Passes the XML URL as a parameter to the XProc Pipeline
  3. Places the results into a container on the View template

Our code looks like this:


// Find interests, if any
var url = $("#interests").html();
$( "#interests" ).load( "../pipelines/find-interests.xpl?results&href=" + url );

Pretty simple, right? The code will only do anything if the tute:interests property is present on the page returned by the view template. If so, the pipeline is called and the results placed into the HTML division we added to the Person View template.

Finally, we place this code so it will execute when the document is ready:


$(document).ready(function() {
    //...
    
    // Find interests, if any
    var url = $("#interests").html();
    $( "#interests" ).load( "../pipelines/find-interests.xpl?results&href=" + url );
});

That's it!  You should now be able to make your own XML files, publish them and get Callimachus to render them in your address book when they are associated with Person entries.

A Word About Serving XML files from Callimachus

Some of you might want to publish XML interest files from your Callimachus instance and will be disappointed to note that Callimachus won't accept them.  Why not?  Because Callimachus is not a general-purpose Web server like Apache or even Tomcat.  There are almost 1500 media types and Callimachus has to date only allowed the ones that are necessary for it to function.  We are naturally open to feedback about this decision on the Callimachus-discuss mailing list or the #callimachus IRC channel on Freenode.

One way to publish XML files to Callimachus is to trick Callimachus into thinking that they are just plain text by giving them a .txt file extension when you upload them.  They will serve to the Web with a media type of text/plain, which will be acceptable to JavaScript and other forgiving client applications.

Wrap Up & Review

Even though it can be a steep learning curve, you can do a lot of powerful things with XProc. If your application can not be implememented using Classes or Named Queries, it can be done via XProc. Learning XProc is well worth the effort.

A good source of XProc examples is Callimachus itself:  Look in the /webapp/pipelines directory in the source code and the Callimachus XProc examples document.  There is also an XProc tutorial and the official XProc specification has all the details.