Digital Earth Weblog

The opportunity recently presented itself to reverse engineer a PostGIS database using three different data modelling tools. Heere are some first impression on these tools, and results of how they handled the PostGIS geometry data type.

» Read more...

The web services catalogue space seems a little empty at the moment. UDDI and OGC Catalog Services are the only two standards that I’m aware of, and implementations are thin on the ground. I’ve been wondering how to fill the gap.

The interest in catalogues came from real world requirements at one of my clients. They have a few web services now, and are creating more. There is a growing need for a catalogue to help discover and manage their web services in a 24/7 operational environment. For example, four concrete requirements are:

• Publishing the location of primary and backup services. The intent is that applications that consume services can automatically “fail-over” to alternate services if the primary service is not responding.
• Pro-active monitoring of web services. An application somewhere should be able to obtain a list of services and regularly check availability and response times.
• Manual testing of services. If something goes wrong IT staff need to be able run simple tests to isolate the problem.
• Manual discover of services. As the number of web services, and web resources, grows a catalogue facilitates discovery.

In some ways all the above deal with managing change in the network topology. In theory, Cool URIs don’t change, but in practice that’s not easy to achieve.

Existing Standards

Standards like UDDI and OGC Catalog Services are different beasts. They are design by committee with vendor involvement, and the hope is that someone will provide an implementation. (On a related note, have a look at The Rise and Fall of CORBA.)

Frankly, I’m much more interested in standards that codify industry practice. In this case, a lightweight web resource catalogue that helps manage The Eight Fallacies of Distributed Computing in a heterogeneous environment.

Web Resources

To avoid confusion, the term “web resource” is being abused here to refer to:

• Any resource in a REST architecture,
• Web-style services (e.g. “XML over HTTP”, Web Map Services, ArcXML, etc), and
• SOAP web services.

A catalogue that only does SOAP/WSDL is not practical in the real world. I want to be able to use the same system for managing Web Map Services, or even just a single web page.

I’m thinking: simple, practical and web architecture. URIs. Behind the firewall.

Out of Scope

Personally, I don’t find the following of much interest in this domain:

• Storing extensive service descriptions, if any.
• Harvesting extensive metadata, if any.
• Support for automated discovery (UDDI tried that).
• Distributed searches using protocols like Z39.50.

URIs can point to service descriptions and other metadata. Automated discovery relies on the first two, and is nice theory, but in business people are involved. Distributed search can be an overlay/aggregation layer, if required.

Prior Art

Tim Bray advises not to invent new XML languages unless you have to. After Googling around, and giving this some thought, it seems that the following could be a good place to start:

• Atom Publishing Protocol for reading and writing to the catalogue.
• Atom Syndication Format for the message format.
• Opensearch for querying.
• GeoRSS for describing location.
• Google Data APIs for inspiration.
• Mapdex for inspiration.

Phil Windley has also mentioned LDDI – Lightweight Description, Discovery, and Integration. LDDI uses microformats, and is the subject of a PhD thesis, which should be interesting.

Web Services Inspection Language (WSIL) also lives in this space. Uptake has been slow, possibly due to it being related to WSDL and UDDI.

Where to From Here

For me the first step is looking at the interfaces to such a catalogue, and defining the document formats. An implementation is then on the table.

If anyone is interested in collaborating on this I’d be interested to hear from you. A GeoRSS-style (without the recent drama) effort would be cool.

Update, 2006-06-25: Added MapDex to the prior art list. Apologies to Jeremy for leaving out an obvious influence.

Update, 2006-06-28: Added GeoRSS to the prior art list. Thanks Mikel.

[tags]xml, catalog, rest, web services, uddi, opensearch, gdata, georss, mapdex[/tags]

Just found out that the NSW Lambert projection has an official EPSG code – EPSG:3308. It is in latest version of the EPSG Geodetic Parameter Database (v6.8).

Note to software developers: It would be appreciated if you could add this projection to the list of supported projections in your products, especially any product that provides a WMS interface.

[tags]projection, geodetic, SRS, CRS[/tags]

Most people in the spatial industry have some experience with metadata (ANZLIC, FGDC, ISO 19115, Dublin Core, etc). That experience is not always positive. Creating metadata is time consuming. The quality of metadata can be frustrating. Mostly it just sits in a repository and becomes out of date.

Metadata is a powerful tool, but only when it is actually used. Creating metadata just in case someone, somewhere, one day, maybe, might need it does not inspire good quality content (but there are some good reasons to do it anyway). Metadata that provides benefit to someone today is the metadata that gets the attention.

I came across a great example in a recent Inside the Net podcast. The Pandora music service, and the Music Genome Project.

The Pandora website lets you create your own personal radio stations. You train the Pandora application by telling it which artists or music tracks you like. It then uses metadata to select similar tracks from other artists which it then streams to you across the Internet. You can further refine the track selection by giving individual tracks the thumbs up or thumbs down.

It works, very well. I’ve discovered a lot of music that I didn’t know existed. The really interesting part is what makes this system work. Three factors stand out:

1. A metadatabase that stores about 400 discrete criteria describing each music track.
2. Around 40 formally educated musicians, on staff, who are trained to analyse each music track and populate the metadatabase in a consistent manner.
3. A software application that can exploit the metadata to provide value to the users of the service.

Wouldn’t it be great if spatial metadata projects had the same focus on providing immediate value, and the resourcing to achieve it.

[tags]metadata, music, software, radio[/tags]

The JPEG 2000 image format is an ISO standard. There are now several desktop GIS applications that can read JPEG 2000 image files. However, there is no open standard that describes the metadata required to allow those images to be georeferenced. So right now, even though JPEG 2000 is a standard image format, it is not an interoperable format within the geospatial community.

Around the middle of 2004 (I think) the OpenGeospatial Consortium (OGC) received several submissions on how the georeferencing metadata could be encoded using GML. The submissions seemed simple enough. At time the general feeling was that a standard would be forthcoming by the end of 2004.

Well, 2005 is mostly history and no standard had been sighted. Meanwhile, anyone who wants to use JPEG 2000 in the geospatial realm is unable to do so unless they working in a single vendor environment (with some exceptions).

A colleague pointed me to a discussion paper available on the OGC website entitled “GML in JPEG 2000 for Geographic Imagery (GMLJP2)”. The scope of the standardisation effort seems to have grown to include:

• Coverage encoding – descriptions of the coverage geometry and content values.
• Coverage metadata – arbitrary metadata content.
• Image annotation – drawing attention to some region of an image.
• Geographic features – any GML feature collection.
• Feature and annotation styling – visual presentation of geographic features.
• Coordinate reference systems.
• Units of measure.

I can see the value in these capabilities. They also make sense if you look at the list of contributors to the discussion paper. They are mostly from companies interested in image data creation, management and high end processing.

My question: What happened to enabling simple interoperability between applications for the vast majority of people who simply want to read JPEG 2000 images into their GIS and have them display in the correct location? I thought that would have been the logical first step. It involves minimal effort and solves a simple problem for a lot of people. 80/20.

Note: I’m not an OGC member so I have no visibility of the standardisation process.

The exercise of creating a few KML files helped identify some of the issues around displaying WMS layers in the free version of Google Earth. There are improvements that can be made to Google Earth, and the WMS community also needs to make some changes if they want to see more wide-spread use of WMS.

Chicken and Egg

The ideal result would be for Google Earth to provide tools that would let any user discover and display WMS services. That means Google Earth would need enhanced KML editing and display tools that let any user:

1. Discover WMS services (requires service level metadata – Abstract, Keywords, etc),
2. Select the layers to display from the full list of available layers (requires layer level metadata – Title, Abstract, LegendURL),
3. Set the default visibility of each layer, and then
4. Have Google Earth communicate directly with the WMS (no “reflector” scripts required).

However, even if Google made it easy to find the WMS, my perception is that the metadata available in most public WMSs is not sufficient to make the display of their data layers a good user experience. Just seeing a long list of layer titles is not much use unless the user can accurately guess, or already knows, what the layers represent. From what I’ve seen the same issue applies to viewing WMS layers in desktop GIS like ArcMap, so this is not just a client issue.

Until there is a large user base demanding better WMS services I don’s see the quality of the WMS metadata improving. Google Earth has a large user base and could definitely drive demand for WMS, and that would be a very good thing.

Given that we have a chicken and egg situation, this wishlist tries to focus on the changes that could make a difference in the shorter term. My assumption is that the people who are creating KML that uses WMS layers are likely to be the people who publishing the WMS, and therefore creating a few reflector scripts is not beyond the realm of possibility.

Google Earth

Legend Icons

Right now, the best bang for buck enhancement to Google Earth would be to allow (legend) images to be displayed in description area of each network link. If included in the KML it would display the image below the <Snippet> or <description>:

The KML might look something like:
<LegendIcon width="100" height="200">   <href>http://www.example.org/legend.png</href> </LegendIcon>

Loading Overlays

Personally, I find the red “loading” image that appears, when overlays are being requested, to be a bit over the top. I’d prefer to see a one or two pixel semi-transparent line drawn around the extent of the overlay and the status text displayed outside the map display pane.

Refreshing Overlays

There may be a bug in the refresh code when the browser pane is activated (it may be my refresh settings). Google Earth 3.0.0529 beta. To repeat it:

1. View the WMS Example 6 KML file in Google Earth. Wait until the map display stops moving and the network link has refreshed.
2. Click on the “National Parks Estate Layer” hyperlink, in the Places pane.
3. In the pop-up description window, click on the “ANZLIC Metadata” hyperlink. A web page is then displayed in the browser pane.
4. Close the browser pane by clicking on the [X] button (top right of the pane).
5. The map then refreshes the overlay image using the extent that was visible while the browser pane was open. Screenshot [296KB, opens new window].

Spatial Accuracy of Imagery

Several people have noticed that the high resolution imagery displayed in Google Earth does not align with vector data they know to be accurate. I’ve seen 10m-20m errors between vectors that represent street centrelines and the underlying Google Earth imagery.

This level of spatial error is not an issue if you are marking a place by hand, because the relative accuracy is all that counts. However, when data from external sources is converted to KML (either as KML vectors or through ground overlays) it is a big issue.

Direct Access to WMS

Once the state of the WMS metadata world improves then I’d like to see Google Earth offer direct access to WMSs. Although, as mentioned, there is an opportunity for Google create the demand.

WMS Publishers

WMS Discovery

It has got to be easier to find public WMSs! There needs to be a low tech approach, because we’ve all been waiting (at least in Australia) for centralised registries that always seems to be just over the horizon.

My suggestion is to run an XSL stylesheet over each of your capabilities file to produce an HTML document. Put that document on a public web server and link to it from a well known page. If Google, and other search engines, index these pages at least mere mortals have half a chance of finding your WMS. Hyperlinks are great things!

Of course, for that to work we need better quality…

Capabilities Metadata

Most people hate writing metadata, and WMS capabilities files are no exception. It’s usually the last thing that gets done when a WMS is published.

The WMS specification dictates what is the minimum set of elements required to create a valid capabilities document. Unfortunately, as discovered, that minimum set is not that useful in the Google Earth type scenario.

A few people in the OpenGeospatial community have been pushing for a commonly accepted “WMS capabilities profile”. The idea is to specify which of the optional elements must be provided in a capabilities document in order for it to be useful for a particular purpose.

I wholeheartedly agree with the idea of a WMS capabilities profile. From a Google Earth perspective the profile needs to enable me to run an XSL stylesheet over a WMS capabilities document and produce a functional, and usable, KML file. No editing should be required.

Repurposing capabilities metadata as KML has another significant benefit. It encourages the publishing of useful metadata content. There is no point defining a WMS capabilities profile if the textual content in the capabilities documents does not support discovery, or let the reader make decisions about whether a particular WMS fits their needs.

To those writing capabilities metadata, my suggestion is to treat it like writing content for a web page:

• You need to focus on writing for your audience, not your peers.
• Write it, then edit it until it is less than half the original size (I’m trying, really).
• Provide hyperlinks to more details information, if required.

Publish KML Files

If you publish a WMS consider publishing some KML files. The experience will be worthwhile. If you provide your own WMS reflector you can easily track how may hits on your WMS are coming from Google Earth.

Feel free to use any on the files that are part of my previous post.

Styling WMS Layers

As demonstrated in the previous post, WMS layers that were styled for display on white browser pages can be difficult to see when overlaid on satellite imagery. Consider providing a separate WMS for Google Earth that provides alternate styling.

I know this could be done using Styled Layer Descriptors, but I haven’t had time to think it through. The WMS reflector script could be used to pass that information back to Google Earth in the WMS GetMap URL.

Wishlist Ends

Well, for now anyway. :-) Thanks again to DEC for providing the WMS (you know who you are).

It took a big chunk of my free time to write this wishlist series. I now better appreciate what the free version of Google Earth is capable of, and when it makes sense to use it.

Thanks for reading. I hope you got something out of it as well. If you use any of the files provided in this write-up it would be great to get an email from you.

Previous posts:
» Part 1 – KML
» Part 2 – Network Links
» Part 3a – WMS, the process

While playing with the free version of Google Earth I started a list of enhancements that I’d like to see in the product. The desired outcome is to make it easier to use Google Earth as a spatial data delivery platform, without impacting on the collective end user experience.

Adding features increases complexity so I’m conscious of the impact that the items on my wishlist might have on the end user. Given that the vast majority of users of the free version of Google Earth are not familiar with GIS, and nor should they be, maintaining ease of use is a top of the list requirement.

KML

The KML editing tools in Google Earth enable anyone to create useful KML files without having to see a line of KML. I’d like to see a few enhancements to KML that would improve what can be done behind the scenes.

KML Schema: A post on the Keyhole BBS suggested that a schema for KML is forthcoming, which is great news. I can only add voice to the chorus. Relax NG or XML Schema would be fine, thanks. :-)

Having a comprehensive XML schema for KML will make it a lot easier to create KML documents using schema aware XML editors. A KML schema will also make it easier to create XSL stylesheets that transform other XML formats into KML. Those of us who write code that generates KML will really appreciate the ability to validate our output.

Documentation: The documentation and tutorial were useful to get started, but I found the descriptions of some of the elements a little brief. I’d like to see some more detail on the thinking behind the each element and how it can be exploited.

For instance, the <Schema> element is a mechanism for extending KML. This element may have the potential to raise KML from a presentation format to a useful data storage/transfer format, but there is very little information on how to use it.

There are also a few simple errors and omissions in “Version 2.0 for Google Earth Client Version 3.0” of the KML documentation. Examples:

1. The name of the <snippet> element should be <Snippet>,
2. There is no description for <viewBoundScale>, and
3. When viewed in Firefox there are broken internal links in the documentation page (e.g. href=”#styleUrl” and name=”StyleUrl” don’t connect due to case sensitivity).

Metadata: There is no provision for storing metadata within KML files that describes the “what, when, who, where and how” of the KML content. The geospatial world has traditionally tried to solve this problem with formal metadata specifications, and centralised metadata registries. Experience has shown that it is very difficult to get quality metadata within a single organisation, so attempting to do so on an Internet scale would be a waste of time.

I must admit that I’ve got mixed feelings about asking for metadata elements in KML. Everyone agrees that metadata is useful, but few people actually supply it, and even fewer provide high quality metadata, so why bother. Well, as the number of KML file available on the Internet grows it will become more difficult to find those that fit your requirements (fitness for purpose), to identify their origin (attribution), and the legal beagles in some organisations will demand the inclusion of copyright and disclaimers.

Given the way KML files are passed around like hyperlinks it would be better to include the metadata in the KML itself, rather than store it separately.

Fitness for purpose: Now, Google seem to be OK at this search thing, so I’m sure they’ll make it easier to find KML files. However, right now it’s not that easy. There are a few well known websites that offer downloads. If you use the Google search file type hack you’ll see that there is very little descriptive information about KML files. You get even less if you change the file type to “kmz”.

http://www.google.com/search?q=kmz+filetype:kmz

The simple approach would be for Google to index the content of the <description> elements within a KML file. The addition of a top level summary element might also be useful. Sometimes the text that you wish to display to the user, via the <description> element, is not what you’d use as an abstract or summary.

Attribution: When I’m looking at spatial data I always like to know where it came from. I think the metadata elements in the Atom format would be a good start for KML. If a link to formal metadata is required it can be included in the document metadata (see atom:link).

Transparency: Looking at some of the KML files that are floating around the Web it is obvious that people have created GroundOverlays using images that were designed for display on web pages with while, or pale, backgrounds. That’s not a complaint, I’ve done this myself.

The problem is that in order to create a nice looking overlay you have to use transparency for the entire GroundOverlay image, and that reduces the visual impact of the parts of the image that are of interest. If I’m interested in roads I probably don’t want those roads to be semi-transparent by default. It would be ideal if the <GroundOverlay> would allow you to specify one colour that Google Earth would then convert 100% transparent before displaying the overlay image. i.e. If I specify <transparentColor>ffffff</transparentColor> all white pixels in my overlay image would be made transparent, and the remaining pixels could be displayed at 100% opacity. Not totally, ideal, but it should cover most cases.

KML in Atom: I’ve been thinking about embedding KML in Atom feeds. We’ve already seen some examples of being able to subscribe to Network Links that are generated from news feeds. This lets you visualise what is happening where. Combining the news content with the spatial content would seem to be a useful next step.

The grey matter washing machine is still in spin cycle on this one, so if anything useful is extracted I’ll let you know.

I’ll post some more wishlist items on Network Links, and support for WMS, over the next few days.

Later posts:
» Part 2 – Network Links
» Part 3a – WMS Support, the process
» Part 3b – WMS Support, the wishlist

I recently mentioned an audio version of Clay Shirky’s Taxomomies to Tags presentation. He has refined his thoughts and published a written version entitled Ontology is Overrated: Categories, Links, and Tags. Lots of diagrams. Well worth a read if you’re into classification.

If you want more insight into tagging then don’t read my stuff, listen to an MP3 of Clay Shirky’s presentation Ontology is Overrated. It runs for 44 minutes, but it’s worth it!

Courtesy of IT Conversations.

In the previous post I suggested that people who work with ANZLIC metadata may be interested in the discussion on tagging, but I didn’t mention why. A “tag” is a keyword that means something to the person publishing a resource, and which is attached to that resource to enable discovery.

Well known sites that use tags, like del.iciuo.us and Flickr, have some similarities to ANZLIC metadata collections:

• They are collection of resources – bookmarks and photos respectivly.
• Their content is created by a diverse community of users, rather than a centralised group of specialists.

They also differ in significant ways:

• ANZLIC records are classified using a fixed set of Search Words and qualifiers, whereas the tags applied to the bookmarks on del.icio.us and photos on Flickr are created by the person who adds the resource to the collection.
• Collections that rely on tagging do not have to spend any time creating taxonomies attempt to be universal (but they can still identify related resources).
• Taxonomies define subject areas up front, whereas tagging allows those subject areas to form as time goes on. It’s central control vs the wisdom of the crowd.

When tags are used taxonomies and thesauri can still provide plenty of value by providing relationships between tags (parent/child and siblings), and synonyms. The difference is that they move into the background and become enabling devices rather than in your face tools that users have to deals with.