Digital Earth Weblog

As many people have noted, the Yahoo! Internet Location Platform has been released. After looking at the examples it all made sense, but two things stood out:

1. Request: The use of a custom query mechanism rather than using HTTP query strings. (This is addressed in Part 1 of this post.)
2. Response: The lack of URIs to identify related resources.

What follows is just a quick exploration of the second of these two issues.

URIs for Related Resources

The response documents provided by the Yahoo! Internet Location Platform do not contain any URI’s to identify related resources. This seems like a lost opportunity to make client applications less fragile and make life easier for developers. It could also make scaling on the server-side easier. (Yes, I’m talking about using REST.)

The current approach provides some of the information required to construct URIs, but the client application needs to be aware of the URI pattern to used to construct the URI for each type of resource.

There can be three types of related resources implied in a location search result:

1. Related locations. e.g. Country, state, admin, locality, etc.
2. Pages of matching records from the same collection of search results.
3. Neighbouring geographies.

1. Related Locations:

Using the Sydney Opera House landmark search example, the related country is returned as:

<country type="Country" code="AU">Australia</country>

 

If an application wishes to take any action that requires knowing the centroid or bounding box of Australia it must know how to construct the correct URI to query the server. e.g.

http://where.yahooapis.com/v1/places.q('Australia')

 

This creates a fragile client application. If the URI construction rules, the host domain, or the URI path change then the client applications will break. The client application needs to know too much about the internal workings of the service. i.e. It is more tightly coupled that it needs to be.

An alternative solution is to identify each related resource with a fully formed URI, in the original response document, which the client application could then dereference:

<country yahoo:uri="http://where.yahooapis.com/v1/place/23424748"
  type="Country" code="AU">Australia</country>

 

Yahoo! can then make all the changes they want to the URI that identifies the Australia resource representation without breaking client applications. It doesn’t matter if they change the URI to:

http://where07.yahoo.com/location/australia

 

The client application will still work provided it still understands the structure of the response document. (Data coupling is a topic for another day.)

2. Paging:

Using the Springfield place search example, when a place search matches more records than it returns in the firts response document this is indicated using:

<places ... yahoo:start="0" yahoo:count="5" yahoo:total="61">

 

These three attributes provide useful information for the client application, but again, in order to get the next five matching results the client needs to know how to construct the appropriate URI:

http://where.yahooapis.com/v1/places.q('springfield');start=5;count=5

 

It would be helpful to developers if the original response included a URI for both the next five and the previous set of records (if applicable):

<next count="5" 
  yahoo:uri="http://where.yahooapis.com/v1/places.q('springfield');start=10;count=5"/>
<previous count="5" 
  yahoo:uri="http://where.yahooapis.com/v1/places.q('springfield');start=0;count=5"/>

 

This is a less compelling example, because it assumes that the client application always wishes to obtain matching results in fixed size sets, whereas the approach taken by the Yahoo! Internet Location Platform allows the client application to make that choice on-the-fly.

3. Neighbouring Geographies:

The current response documents do not include any details about neighbouring geographies, but there is a neighbouring geographies query. Client applications must know how to construct yet another URI to obtain this information:

http://where.yahooapis.com/v1/place/12795711/neighbors 

 

Why not simply add a new element to the response document that provides that URI?

<neighbors type="State" 
  yahoo:uri="http://where.yahooapis.com/v1/place/12795711/neighbors"/>

 

Conclusion:

Taking a more standardised approach to adding query parameters to URIs, and providing URI’s to related resources, would enable developers to create less fragile client applications. The Yahoo! Internet Location Platform also appears to have missed an opportunity to deliver a true REST interface in an application where it would be very well suited, however…

Caveat: All the above has been written knowing nothing about the business or design constraints that shaped the Yahoo! Internet Location Platform. There are likely to be many perfectly sensible reasons for the design choices that were made.

As many people have noted, the Yahoo! Internet Location Platform has been released. After looking at the examples it all made sense, but two things stood out:

1. Request: The use of a custom query mechanism rather than using “HTTP query strings”:http://en.wikipedia.org/wiki/Query_string.
2. Response: The lack of URIs to identify related resources. (This is addressed in Part 2 of this post.)

What follows is just a quick exploration of the first of these two issues.

Query Mechanisms

The Yahoo! Internet Location Platform uses an interesting mixture of three different query syntax styles.

1. The place name search syntax looks like an object method call.

/places.q('sydney%20opera%20house')

 

Yahoo!‘s function-like syntax places.q(‘place%20name’) makes it look like q is a method on the places object. That would suggest the underlying implementation may have leaked into the interface (surely not!).

2. The paging parameters, start and count, are semicolon delimited strings.

/places.q('springfield');start=0;count=5

 

Avoiding the use of HTTP query strings would allow HTTP caches to cache the results of the place name searches. That would improve performance on common searches while reducing the load on the Yahoo! servers. However, that would prevent Yahoo! from controlling access and accurately metering usage.

3. And, query strings are still used to determine language and format of the resource:

/places.q('usa')?lang=fr
/place/2487956?format=json

 

There goes the caching theory, so why not just use plain vanilla query strings for all parameters and keep it consistent? Example:

/places?q=springfield&start=0&count=5&format=json&

 

It’s dead easy to construct a query string from a map, or dictionary, of name/value pairs and append it to a GET request. Using a non-standard and inconsistent approach creates more work for developers.

Continue to Part 2

The simple Netkernel application that I had previously put together was quite basic, and not that interesting from a real world perspective. Here’s a more practical example that also uses PostGIS to perform some spatial processing.

Requirement: Given an X/Y coordinate, find me the nearest piece of equipment.

Request:
http://hostname/webdb/nearest-equip?x=9555555&y=4555555

Application:

(Apology: My syntax highlighter has trouble with XML namespaces, so I’ve had to present the plain text version.)

<idoc>
  <seq>
    <instr>
      <type>SQLEscapeXML</type>
      <operand>this:param:param</operand>
      <target>var:param</target>
    </instr>
    <instr>
      <type>xslt</type>
      <operand>var:param</operand>
      <operator>
        <xsl:stylesheet 
          xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
          version="1.0">
          <xsl:template match="/nvp">
            <sql>
              SELECT equip.equipment_id, 
              Distance(
              GeomFromText(
              'POINT(<xsl:value-of select="x"/>
              <xsl:text> </xsl>
              <xsl:value-of select="y"/>
              )'
              ), equip.the_geom
              ) as distance
              FROM equip
              ORDER BY distance ASC
              LIMIT 1;
            </sql>
          </xsl:template>
          </xsl>
      </operator>
      <target>var:sql</target>
    </instr>
    <instr>
      <type>sqlQuery</type>
      <operand>var:sql</operand>
      <target>var:queryResult</target>
    </instr>
    <instr>
      <type>xslt</type>
      <operand>var:queryResult</operand>
      <operator>
        <xsl:stylesheet 
          xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
          version="1.0">
          <xsl:output indent="no" method="xml"/>
          <xsl:template match="/results/row">
            <equipment>
              <xsl:copy-of select="./*"/>
            </equipment>
          </xsl:template>
          </xsl>
      </operator>
      <target>this:response</target>
    </instr>
  </seq>
</idoc>

Normally I’d store the XSL in separate files, but have included it here inline so you can see what’s going on.

The result is a small XML document that looks like:

<?xml version="1.0" encoding="UTF-8"?>
<equipment> 
  <equipment_id>123456</equipment_id>
  <distance>78349.5745430395</distance>
</equipment>

Of course, all the cool cats will want JSON output rather than “boring old XML”, so here’s how to do it. The new <instr>uction at the bottom of the DPML document adds a whopping 20% to the amount of code in the application. :-)

<idoc>
  <seq>
    <instr>
      <type>SQLEscapeXML</type>
      <operand>this:param:param</operand>
      <target>var:param</target>
    </instr>
    <instr>
      <type>xslt</type>
      <operand>var:param</operand>
      <operator>
        <xsl:stylesheet 
          xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
          version="1.0">
          <xsl:template match="/nvp">
            <sql>
              SELECT equip.equipment_id, 
              Distance(
              GeomFromText(
              'POINT(<xsl:value-of select="x"/>
              <xsl:text> </xsl:text>
              <xsl:value-of select="y"/>
              )'
              ), equip.the_geom
              ) as distance
              FROM  equip
              ORDER BY distance ASC 
              LIMIT 1;
            </sql>
          </xsl:template>
          </xsl>
      </operator>
      <target>var:sql</target>
    </instr>
    <instr>
      <type>sqlQuery</type>
      <operand>var:sql</operand>
      <target>var:queryResult</target>
    </instr>
    <instr>
      <type>xslt</type>
      <operand>var:queryResult</operand>
      <operator>
        <xsl:stylesheet 
          xmlns:xsl="http://www.w3.org/1999/XSL/Transform" 
          version="1.0">
          <xsl:output indent="no" method="xml"/>
          <xsl:template match="/results/row">
            <equipment>
              <xsl :copy-of select="./*"/>
            </equipment>
          </xsl:template>
          </xsl>
      </operator>
      <target>var:xmlResult</target>
    </instr>
    <instr>
      <type>JSONFromXML</type>
      <operand>var:xmlResult</operand>
      <target>this:response</target>
    </instr>
  </seq>
</idoc>

Spatially enabled database. 100% declarative code. Love it!

Update: The above spatial query takes 80ms to execute on my 2Ghz laptop, although the equip table only contains 270 records.

I’ve been interested in Netkernel for a while. It offers a decarative development approach, like using SQL to query a database instead of writing the code that performs the actual query processing. Netkernel also gets extra bonus points for embracing REST (but I digress).

Netkernal lets you focus on logical resources that are loosely coupled. You can use declarative and imperative languages where they make sense.

As a simple example, I set out to build the simplest possible application that could accept an HTTP GET request, query a database, and return the results of the query as an XML document. Here’s the core application, minus the configuration of the module and database connection:

<idoc>
  <seq>
    <instr>
      <type>sqlQuery</type>
      <operand>
        <sql>
          SELECT id, name 
          FROM items;
        </sql>
      </operand>
      <target>var:queryResult</target>
    </instr>
    <instr>
      <type>xslt</type>
      <operand>var:queryResult</operand>
      <operator>ffcpl:/resources/style/get-item-list.xsl</operator>
      <target>this:response</target>
    </instr>
  </seq>
</idoc>

This is a Declarative Process Markup Language (DPML) document. It contains a <seq>uence of three two <instr>uctions. The first instruction queries the database and stores the result in the variable “queryResult”. If you view an XML representation of the query result it looks like:

<results>
  <row>
    <id>1</id>
    <name>Foo</name>
  </row>
  <row>
    <id>2</id>
    <name>Bar</name>
  </row>
</results>

This could be returned as the response document, but elements named results/row don’t exactly describe the sematics of the data. Therefore, the next instruction uses a small XSL stylesheet to change the document structure to:

<itemlist>
  <item>
    <id>1</id>
    <name>Foo</name>
  </item>
  <item>
    <id>2</id>
    <name>Bar</name>
  </item>
</itemlist>
<pre>

The third instruction “casts” the result to a MIME type of text/xml. As Peter pointed out in the comments, if I use the stylesheet to define the output method as XML, using <xsl:output method=“xml”/>, I don’t need the third instruction, which was:

<instr>
  <type>cast</type>
  <operand>var:serviceResult</operand>
  <operator>
    <cast>
      <mimetype>text/xml</mimetype>
    </cast>
  </operator>
  <target>this:response</target>
</instr>

Notice that I haven’t had to worry about object types. Netkernel takes care of that for me.

Production Ready

The use of a declarative development approach means that if it works as desired it’s ready for production. There is no imperative code in this application so there isn’t a whole lot to test. The only optimisation would be to use Netkernel’s built in caching to minimise the load of the database.

This example plays to Netkernel’s strengths, but contrast it with the imperative code required to build this application in Java or C#.Net.

The Backstory

A while ago I ran some code metrics on a Java servlet Web application that I’d written. I was surprised at just how much code was involved. 5,500 non-comment source statements (NCSS) for an application that had about 20 data entry forms. That seemed like a lot.

Stripes had been used for the Web framework and Hibernate was used for the database layer. These two tools had reduced the amount of Java code that had to be be written, and about 500 NCSS were to manage an archaic interface with another system. That leaves 5,000 NCSS for user interface logic and domain model.

This exercise prompted me to start looking at alternative ways of developing applications. What’s that? Ruby on Rails, Django, and friends. Sure, they are options, but I was looking for something different. I wanted something declarative, not imperative. Netkernel fits that requirement.

Update: A simple Netkernel Spatial application

Flex’s HTTPService class supports “GET, POST, HEAD, OPTIONS, PUT, TRACE and DELETE”. Mmm…interesting.

The REST community has long bemoaned the lack of native browser support for HTTP verbs other than GET and POST. The XMLHttpRequest object has provided one option for AJAX applications. Flex provides similar capabilities for Flash “rich Internet applications”.

[Note: First post after upgrade to WordPress 2.1.3.]

Mmm…may be a little premature to make that call, but it’s good to see the alternatives getting a run. Following on from my last post, here are some more links:

Nelson Minar tells us Why SOAP Sucks, based on his experience helping Google implement SOAP APIs.

Duncan Cragg’s is writing a 9 part series titled “The REST Dialogues”:

Part 1: Getting Data.
Part 2: Setting Data.

Peter Lacey’s take on SOAP, The S stands for Simple, is funny and contains more than a little truth.

Via Mark Baker.

Sean Gillies asked Why Does WFS Dislike the Web?. I haven’t played with WFS in years, but I’d like to explore a point raised by Allan Doyle in a the comment on Sean’s post.

Allan mentioned that adoption of a technology depends on how easy it is to get something working quickly. He was comparing KML with OGC specifications.

In the case of WMS, if it had been designed to follow REST principles I don’t think it would have been as successful because it would have been more difficult to use.

REST says that a URI identifies an abstract resource. HTTP headers are used in the request to convey metadata about the desired resource representation (e.g. Accepts), and in the response to describe the actual resource representation (e.g. Content-Type).

If you look through the WMS query string parameters some of them are metadata that describe the request, the user agent’s desired resource representation, and MIME type to be used for exceptions. Not all are used to identify the abstract resource itself:

• VERSION
• WIDTH
• HEIGHT
• FORMAT
• BGCOLOR
• EXCEPTIONS

In a sense, CRS is also a candidate for being metadata. It’s the same abstract resource regardless of the CRS used for the resource representation (but the CRS is tied to BBOX which is used to describe the resource, so it’s not clear cut).

Anyway, if all those name/value pairs had to be passed through as HTTP headers you would have needed a client other than just a plain old browser to send a WMS request. Not the best way to encourage adoption.

The value of being able to put a URI in your browser’s address bar and see what comes back cannot be understated. Instant gratification counts on the path to getting a quick result.

Update, 2006-10-22: The conversation continues at import cartography.

[tags]WMS, REST, HTTP[/tags]

Public APIs for mapping services are common. Lately, the trend seems to be to call these APIs “RESTful”, because REST (REpresentational State Transfer) has become a bit of a buzzword. However, I’ve yet to see one that actually follows REST principles.

REST is an architectural style. REST is not a toolkit, a push button option in your Integrated Development Environment (IDE), or an API. In fact, if you see parameter name like operation (op), action (actn), or request in the URI query string it is usually an indicator that the REST style is not being used.

You have to consciously build your applications to follow REST principles. Resources, resource collections, links, content types, and state transfer are key ingredients. Paul James does a great job of explaining the issues.

Most mapping services are just HTTP GET requests that return an image. They are “Web-style” and Web friendly. They are useful. But, they are not REST, and calling them such is misleading.

[tags]rest, web-style, map service[/tags]

Here’s some Friday afternoon humour from Elliotte Rusty Harold.

REST vs. WS-*: A Parable

My view on this is that if you really do need the enterprise middleware/complexity required by WS-* then go for it. Just keep in mind that there are many cases where that level of complexity is not needed, and you can do a lot using plain old HTTP and XML. Horses for courses.