Digital Earth Weblog

A surprise request arrived from a client yesterday. “We need a wiki, can you help?”

Sure. What they didn’t know was that I had already installed JSPWiki one of their development servers a while ago. Another developer and I used it for simple development notes and maintaining lists of links to other websites. Nothing complex.

At the time I had told a few people in the organisation about the “developer wiki”, but it never got any traction. For others, at the time, it was a solution looking for a problem. This time someone mentioned wikis to a manager who had a business need driven by cost saving. He was looking for a way to replace an externally hosted online help service, and to create document oriented content.

The manager had done his research. The original email request came with a link to WikiMatrix. The requirements included multiple sub-sites, individual user accounts and permission management, visibility of all changes, WYSIWYG editor, etc, and in bold “it must be free and open source”. OK, no problem.

Off to the Web I went looking for the latest open source wiki software that would run on the web application platforms used by this organisation – Windows, SQL Server, Java or .NET. There were not many options that looked like they could meet the client’s requirements. The latest versions of JSPWiki and XWiki seemed like the best candidates.

“What about [insert favourite wiki] that runs on PHP/Python/Perl and MySQL?”, you ask. Yes, WAMP would have provided a lot more options, some of which would have been better candidates. However, this organisation only has one part time network/systems administrator, and relies on contract developers, so my recommendation was to not introduce yet another platform.

I created a list of tasks that I’d have to complete to get the open source wikis to meet the client’s requirements. The list included items like:

• Learn custom JSP/Velocity template system and create site template(s).
• Learn security model and config files, create user accounts, find out how/if that works across multiple sub-sites.
• Set up multiple sub-sites.
• Obtain open source WYSIWYG editor, integrate it and remove unwanted options/tools.
• etc…

A quick look through the documentation of the candidate products revealed that some of this would be easy and some might be difficult. Difficulty/risk was usually proportional to the availability and quality of documentation. Some was good, some was thin, some did not exist.

By the time I finished doing a quick feasibility on both products I was thinking that there might be 3-5 days work involved in getting an open source product to the point where it would meet the client’s requirements. And, there was still some risk that those requirements might not be able to be met.

I presented the client with two options:

1. JSPWiki plus 3-5 days of my time (and some risk), or
2. Buy a 25 user licence of Confluence.

Confluence is an enterprise wiki with very good documentation.

After a quick discussion with the client about total cost, technical risk, limiting dependency on myself to adminster and maintain the wiki software, and why I had not offered more options (the PHP/Python/Perl issue) I got the go ahead to install a 30 day trial of Confluence. In about 45 minutes Confluence was downloaded, installed on Tomcat (two small config files), licensed, plus I had set up some trial Spaces (sub-sites) and added a few user accounts.

A few more minutes were required to add the organisation logo and change some of the colours to the organisational intranet standard. While I was doing that the manager was already creating test pages. Sweet!

(Note: This post is not a “commercial software is better than open source software” rant. In this case, for this particular client’s requirements, I thought Confluence was the better option.)

Wiki Software Comparisons

• WikiMatrix
• Wikipedia

Disclosure: I am a customer of Atlassian Software. I own an Jira licence, but other than that I have relationship with the company.

[tags]wiki, confluence, open source[/tags]

This weblog has just been upgraded from WordPress 1.5 to WordPress 2.0. All the changes in version 2.0 are in the back end. There should not be any visible changes.

All seems to have gone smoothly except the SimpleTags plug-in doesn’t seem to be working (note to self: err, RTFM), but please let me know if you notice anything broken. Thanks.

[tags]WordPress[/tags]

Recently I was asked to assist in debugging a SOAP integration issue. A large desktop application was being enhanced to obtain data from a web service created by my client. They were not interoperating as advertised. This post looks at what happens when SOAP interop goes bad.

My sweepingly general theory is that developers like SOAP because it’s easy for them to use. Just create some classes and let the SOAP tools take care of generating WSDL and providing serialisation to/from the XML wire format. Same on the client end. Use the SOAP tools to read the WSDL and generate the required stub classes.

Developers never have to see what travels across the wire, or even understand it, until it doesn’t work. Then they suddenly need to know a lot about XML – SOAP, WSDL, namespaces, and XML Schema. It’s a whole new ballgame.

In this particular case the web service was created using Microsoft Visual Studio and ASP.Net. The desktop application was developed by a company in another country. It was written in C++ so the Microsoft SOAP Toolkit was used as the SOAP client. The challenge was to get the two to work together.

Requests to the web service were quite simple, just a few parameters, but for some reason the request was failing due to a data type error. The error message was not very helpful so the first step was to work out where the data type problem was occurring – on the client or on the server.

We tried two free SOAP monitors without any success. No SOAP traffic was showing up. Ethereal to the rescue. By monitoring HTTP traffic we discovered that the test client application was requesting, and receiving, the WSDL from the web service. But, it was failing before it could issue a SOAP request. Suspected problem at the client end.

A quick test with one of my XML editors (<oXygen/>) helped prove that the web service was working as expected. <oXygen/> has a SOAP client that reads a WSDL file and generates a SOAP request document. We filled in some test values and fired off a request. The SOAP response contained all the right values in the right elements. Problem definitely at the client end.

To cut a long story short, it turned out the test client application was getting the order of the request parameters mixed up and the SOAP Toolkit was complaining about data type errors. I thought this would be difficult to do if you used classes generated from the WSDL, but it turned out that the developer was using low level functions in the SOAP Toolkit, not generated classes. It took quite a few iterations, over the course of several weeks, to get it all working.

In the middle of all this the web service developer and myself had written our own test clients (VB.Net and Delphi) just to prove to ourselves that it was possible to get this working in a few hours. Apache Axis was also used to generate C++ classes from the WSDL.

For better or worse, SOAP has become the industry standard for web services. For me, the take away from this saga was not to assume that every developer out there knows what SOAP and XML are, or how to leverage the benefits that SOAP provides. The tools make it pretty easy to use, but SOAP is only useful if the humans at both ends really understand how to exploit it.

Just in case anyone is waiting for part three of my Google Earth wishlist (WMS Support), it is coming. I’ve had to dig a little deeper than I initially expected.

In between having a competition with my daughter on who could finish the latest Harry Porter novel first (she did) I started building KML files with network links to WMS’s. I found a bunch of issues on both the Google Earth and WMS sides. The result that I’m no longer sure that Google Earth is a good fit for displaying WMS. The write-up, KML files and some code are in progress.

Over the past week I’ve been playing with the free version of Google Earth. This has involved creating some proof of concept KML files for a client, and writing a prototype Java servlet application that exposes ArcIMS Image Services as KML. I love my job!

Several organisations on my radar are looking at how Google Earth might be used as a spatial delivery platform both inside and outside their enterprise. Herewith some notes from my experience so far.

Google Earth is a very cool 3D viewer, but under that shiny exterior it is basically a data presentation tool. In many ways, it has more in common with Adobe Acrobat Reader than it does with desktop GIS. Like PDF, KML is also a presentation format. It is not a suitable format for spatial data storage or transfer.

Google Earth is a thick client application, so you have to play within it’s boundaries. The good news is that a published data product (a KML or KMZ file) can be created in a very short period of time if the data that you need are already available. The trade-off is that you have less control over the way your information is displayed, and how the user interacts with it.

The default layout of the Google Earth interface is optimised to promote Google’s services. For example, the Search panel contains Local Search, Directions, and Fly To options. Fly To only works on town names, not location names like “Mt Everest”. You can turn off the panels that you don’t want to see by using items under the Tools menu, but in practice the vast majority of people will use it in its default configuration.

The Layers list contains a lot of vector data layers that may be completely irrelevent to your application. You cannot remove layers from the Layer list. Some of the data is also quite coarse.

Another key point is that when a user loads a KML file it shows up in the Places panel. Everything in the Places panel is editable by the user. Anyone can edit the KML file and save it under a new name.

OK, time to get into some more technical issues.

For those who are familiar with desktop GIS there are some fundamental differences between GIS project files and KML files:

• A KML file is not a single “project” level document that defines everything that you can see in the map display. Think of it as a bundle of one or more “layers” that are displayed as a “layer group”. The user is free to view your KML file at the same time as they are viewing other KML files.
• The vector and raster layers provided by Google are always available. You can turn off the vector layers using the checkboxes in the Layers panel, but you have no control over them from within a KML file.
• Each individual spatial feature in a loaded KML file is listed in the Places panel. Groups of like features are ususally contained in a single folder (similar to a GIS layer). This can make the Places panel almost unusable if a KML file contains a large number of features. Example: The 3D model of the Sydney Harbour Bridge is made up of 18,000 faces. When you open that folder each face is listed, and you’re in for a lot of scrolling!
• There is no concept of minimum and maximum display scales. All data is displayed all the time. This is significant for network links (see below).

Network Links

KML’s element is simple yet quite powerful, and deserves special mention. It lets you bring in data from external sources into Google Earth. If you have existing web map services you can bring in dynamically generated map images and have them overlaid on the terrain model. This works quite well, but the display of that data in 3D on planetary scale raises a few new, and not so new, issues.

Data extent: If you are viewing the entire planet Google Earth will ask for a map image covering the entire planet (-180,-90,180,90). If your data does not cover the entire planet you can use KML to restrict the output of your map server to a smaller extent.

Scale limits: KML has no concept of view scale limits so Google Earth will make requests to all active network links regardless of the current user’s view. Setting the minimum display scale on your map services is recommended.

Example: I set up a map service that provided all the roads in the state of New South Wales, and didn’t include a minimum scale limit. This map service was also reprojecting on-the-fly. Google Earth asked for a state-wide map and ArcIMS dutifully consumed 100% of available CPU on the server for several minutes while it produced a map image.

Response times: Google Earth likes its network links to respond quickly. If it doesn’t get a response in less that about 5 seconds it complains with a modal dialog box. Not very user friendly.

Server load: A 3D viewer invites the user to pan and zoom. If you have network links set up they will most likely use a view based refresh. When the camera stops moving Google Earth makes requests the network link. This can put a lot of stress on the map server. If I use my mouse to pan-stop-pan-stop-pan-stop I’ve just sent three requests to the map service, but I’ll probably only see the last one. You can prevent this by adding a one or two second delay after the camera stops.

Fixed layers: When you set up a network link to bring in a map image from an external map service the map image will display a fixed list of layers. There is no ability to switch layers on and off from the Google Earth end. The only value it passes through is the bounding box of the current view.

Querying overlays: There is no way to do a feature information query on an overlay obtained via a network link. Display only.

Styling overlays: The styling of the data provided by your map service needs to be significantly different to the styling used in a 2D web mapping application that uses a light coloured background. The imagery in Google Earth is generally quite dark so light coloured vectors, and light text with a dark ghosting, seem to work best.

Today is the end of the financial year in Australia. This time of year is interesting for organisations that provide services to government clients. There is always work that must be done before the immovable deadline of 30 June. It is also a time when product vendors are looking for clients who still have money in their budget, and when clients don’t have time to evaluate those products properly.

Digital Earth Pty Ltd was one of those vendors. We benefited from end of financial year deals, several times. Some of the software was purchased with the best of intentions, but still sits on the shelf. No vendor likes to see their software go unused. It means no ongoing business relationship, and no happy customers to use as a reference. It just doesn’t sound good to say “Agency X bought our stuff but they haven’t used it”.

A few weeks ago I visited a client to help them evaluate some enterprise geospatial software. It was great to see a government agency taking the time to give competing solutions a thorough evaluation.

Government organisations often make major technology purchases near the end of the financial year, if they have money left in their budget. However, this is possibly the worst time to make such decisions. There is pressure to spend the money before the books close (“use it ot lose it”). Staff are often busy finishing off other projects, their real work. Nobody has time to do a thorough evaluation.

Doing a thorough evaluation takes a lot of effort. Due to the complexity of today’s enterprise environments the best way to do it is to run a pilot project:

• Install the software in your environment;
• Load it with realistic volumes of data;
• Do at least proof-of-concept integration with other systems;
• Train people how to use it and manage it;
• Use it for real world tasks;
• Load test all server applications;
• See what works, see what breaks;

You’ll be glad you did.

I spent a few days last week updating a web mapping application, built around Image Web Server, that I wrote for a client about 5 years ago. It was like…“Who wrote this rubbish!?”.

There was a lot of Javascript spread across multiple frames. Some of it was being generated by server-side ASP code. There was also a lot of code mixed in with HTML. i.e. It was messy.

It would have been nice to be able to rewrite the application completely using XHTML, CSS positioning and refactored Javascript, but it works and the client has more important thing to be done. It’s all about ROI and getting things done.

A quick inventory today revealed what is probably a well known fact to some. Open source software components are becoming more common in corporate information systems. I kind of knew that, but it wasn’t until I tallied up the components that will be used to develop a Java web application for a client that it hit home:

1. Tomcat
2. Hibernate
3. Several Jakata Commons libraries
4. NetBeans, which also includes…
5. ANT, and
6. JUnit

We could also be using Struts, but have chosen JavaServer Faces because it has been bundled with ArcGIS Server (which will be used extensively by this client). Two decidedly non-open source databases will be used – SQL Server and Oracle.

It will be interesting to get a feel for the value the open source components provide as the project continues. Regardless of what that value is, my sincere thanks to everyone who contributed to those components.