Digital Earth Weblog

Anyone else find the rules for the new OGC Individual Membership kind of…well…restrictive?

» Read more...

Settings for using a Minimax MM-5500U EVDO modem, on Telstra’s EVDO network, from Ubuntu Feisty. Possibly only of interest to Australian EVDO wireless broadband users.

» Read more...

There is only one reason why I carry a Palm Pilot. Times for Palm OS is an excellent timesheet application, especially if you have multiple clients on the go at one time. However, synchronising my Palm Pilot with Windows XP running inside VMware has been troublesome.

Aside: It looks like Comit now also have a Pocket PC version of Times.

Since upgrading to VMware Workstation 5.5.4, on Ubuntu 7.04 Fiesty Fawn, this is what I do:

1. Run: sudo mount -t usbfs usbfs /proc/bus/usb
2. Start VMware Workstation.
3. Start Windows XP virtual machine, and log in.
4. Make sure XP has the focus.
5. Put the Palm Pilot in the cradle.
6. Start Palm Desktop (which I’ve configured to start the HotSync applet).
7. Synchronise.

If someone has a better approach please leave a comment. Also, it looks like USB will be handled differently in the coming VMware Workstation 6.0.

A significant part of my current role as Remover of Mystery is learning. A lot of learning. At present I’m on the steep end of these learning curves:

• Off-site disaster recovery,
• Storage Area Networks (SANs)*,
• Server virtualisation*,
• Enterprise architecture,
• Enterprise architecture management tools*,
• Business intelligence tools*, and
• Two way database replication.

(* Likely to result in product selection.)

It’s all interesting stuff, and thankfully there are a few people who can help me out, but the context switching is just insane. If you’ve been in a similar situation I’d love to hear how you managed.

This week has been spent digging into the details of various SAN and server virtualisation products. It has involved conversations with various vendors. Vendor perspectives were then cross-checked with experiences listed in user forums (love the Web). So much information has been absorbed that it hurts.

Some information has fallen out of the buffer due to capacity limitations. However, the week has been defined by several other frustrations:

1. Even after explaining what we want to achieve some vendors were still trying to sell us complex tools that we don’t need. Only one salesperson suggested that their product was not the right tool for a certain requirement.
2. Opinions in web forums against using server virtualisation in production environments are strong but rarely backed up with useful facts.
3. Even after a week of questions, listening, and research it was realised that we still don’t have enough knowledge to make a truly informed selection.

The first point above is no surprise. However, the latter two are interesting.

Server Virtualisation

The idea of server virtualisation makes some people cringe. It just sounds wrong, but when faced with the prospect of having fifty 1RU servers all doing very little, or five larger servers working harder, it’s difficult to ignore the economics of products like VMware.

While digging around in the VMware forums you see comments like:

• “It’s way slower than just a physical server.”
• “It’s great for dev and test, but I’d never use it for production.”

Details on the basis for these assertions are usually missing. Everyone’s workload profiles, software configurations, and hardware configurations are going to be different so such statements are essentially meaningless.

Sure, a virtualised environment will have overhead. The real question is how that overhead affects your applications in your environment. Even if there might be a 30% networking overhead when using iSCSI to connect to external storage from a Windows virtual machine executing on VMware ESX 3.0 running on a physical server without a TCP/IP Offload Engine (TOE) card, how does that really affect your application when it’s running on a Gigabit Ethernet connection shared by several other virtual machines?

One thing that was obvious this week is that there are opportunities for bottlenecks in virtual server infrastructure, and many of them are nothing to do with the virtualisation layer. e.g. SAN, physical servers, and networking. So how do you decide whether to use a product like VMware?

Making Informed Choices

This week was all about learning more about what current products can do. We needed to validate the physical architecture that we think we need to build. Rather than just ticking off supported features from a check-list, we were also looking at ease of management and alternative ways of meeting our overall business objectives.

Right now the feeling is that we know enough to make a justifiable decision, maybe. The only way to really find out if a product meets our needs is to test it in our own environment, and ensure that it is configured correctly. That takes a lot of time and effort, but the alternatives are to take pot luck or over provision for everything.

If you haven’t got time to evaluate a product in a restricted environment you probably don’t have time to configure and administer it properly in a production environment.

Wow, 8 GB DIMMs are awefully expensive! This was discovered recently while pricing a 2 RU (rack unit) server from a well known vendor. The server contained 4 DIMM slots, and when equipped with 4 x 4 GB DIMMs came out at about AUD20k. Not too bad for a fast dual Xeon quad-core box with 16 GB of RAM. However, changing the RAM to 4 x 8 GB DIMMs increased the price to over AUD60k. Actually, the RAM alone cost about AUD50k. Youch!!!

So, it’s cheaper to buy two servers from this vendor with 16 GB of RAM each than to buy one server with 32 GB of RAM. Thinking that this was a blatant rip-off, servers with similar specifications were priced from other well known vendors. The massive price differential between 16 GB and 32 GB versions were almost identical for all vendors. So, 8 GB DIMMs really are expensive.

Now, you might be asking: Why in the world do you need so much RAM? Server virtualisation. Each of these physical servers would be running quite a few virtual servers, including applications like SQL Server and ArcGIS Server. Having enough RAM becomes very important for performance reasons, but luckily 16 GB per dual processor server is probably enough.

We have a G5 iMac running MacOS 10.3.9 at home. After installing the latest security update, and rebooting, Internet access was pathetically slow to non-existent. On a whim I ran the Disk Utility and Repaired Permissions. The iMac is now back to its normal self.

Paul Ramsey has put together three frank blog posts on spatial data infrastructures (SDIs). Well worth a read if you have anything to do with an SDI effort.

1. Why SDIs Fail
2. Must SDIs Fail?
3. Making SDIs Work

I’ve only participated in one spatial data infrastructure effort, at a technical level. The problems Paul describes ring true. Here are some of my experiences (generalised):

• Services: Asking people to expose public services because it’s the right thing to do (what Paul calls “karmic reward”) guarantees that it will be on the bottom of their to do list, if they have the infrastructure and skills.
• Data: If data needs to be prepared (cleaned, denatured, etc) to make it “safe” for publishing then don’t count on it happening without a lot of cajoling. Expect resistance to making data public, especially any data that can be perceived to impact on financial interests (e.g. salinity on farmland).
• Metadata: Always the last thing to get done, and getting it usually required a lot of hassling. When we did get metadata it was either too detailed, or too brief.
• Performance: One slow or unreliable service, especially if it is supplying the base map, will kill the user experience, and the client application cops the blame.
• Client complexity: Any web client that can do everything required by an SDI will be complex, and therefore difficult to use for non-GIS users.
• Cost/benefit: The user community was, well, out there somewhere. We had a rough idea who they were but without decent metrics it was difficult to know which maps and individual datasets were popular. But, having metrics would have challenged the “publish everything” ideal.
• Politics: The more people/organisations involved the harder it gets. Sharing some money around can help provide focus, but when people are involved… An SDI is a political/people/business issue, not a technical issue.

Despite all the issues, I still think that this project was a worthwhile effort. At the time there were ideas, standards and (a few too many bleeding edge) technologies that had to be tested in the real world.

The most valuable output of the project might be obtained by shining a cold hard light on the entire process and discovering what really worked, and what didn’t.

It has been quiet here in the Digital Earth Dungeon. I’ve been doing a lot of software development work. Apart from prototyping a Delphi desktop application, and gearing up for a VB.Net application, I’ve been working on a Java web application that uses the excellent Stripes framework and Hibernate object/relational mapper.

After a bad experience with JavaServer Faces (JSF) it was decided that JSF would be removed from this application, and that another web framework should not be introduced. A custom presentation layer was written using home grown controllers and JSP views. This worked reasonably well, but a lot of time was spent writing code to do mundane repetitive things. Copying parameter values from the HTTP request to backing beans, the associated conversions to non-String data types, and common validation routines come to mind.

Application developers should be focused on providing business value, not the plumbing. This sort of stuff should just work, or be dead simple to implement. As the application got larger more and more duplicate code appeared, and a lot of it was doing mundane stuff. The duplication should have been abstracted out, but the time was not available to write yet another web framework. Others have done a better job of that already.

So, it was off to the Web, yet again, looking for a suitable Java web framework. There are many to choose from, and there were two core criteria:

1. Not having a learning curve that resembled Mt Everest. Immediate productivity was required.
2. It had to be a request-based framework.

Component/event-based frameworks like JSF do their best to abstract away the Web. This additional complexity was not wanted. A framework that worked with the web was the way to go.

After some late nights surfing the web, reading framework documentation, and sifting through the opinions and biases of many commenters on websites and blogs we had a winner: WebWork. What! Isn’t this about Stripes? Patience Grasshopper.

WebWork’s learning curve looked steeper than desired, but there were many positives. There were books available, indicating maturity and a solid community. The product seemed to have a good future, being slated as the basis for the next generation of Struts. Some well know web applications use WebWork. Etc.

At the last minute a post to a website was noticed, written by a guy named Tim Fennell, which mentioned a framework called Stripes. “Stripes? Haven’t heard of that one. Oh well, one more can’t hurt.”

The homepage of the Stripes website struck a chord. “The main driver behind Stripes is that web application development in Java is just too much work!” Exactly! Some more reading revealed that Stripes was definitely worth a decent look. Version 1.2.2 was downloaded. It was small, with minimal dependencies. A nice change. Exploration of the sample application began, along with some more reading of the well written online docs.

To cut a long story short (oops, too late, sorry) Stripes was implemented. It lived up to high expectations. The application developer (ahem) took a short while to get their head around some key concepts, and to work out how to best use Stripes with Hibernate, but after that it was plain sailing.

Stripes took care of all the mundane tasks and let the developer focus on business logic. It also gives the developer a bit of flexibility. It doesn’t force them to follow “the one and only true way”. In short, it was a pleasure to use. It also substantially reduced the amount of code in the presentation layer of this web application, and made it more maintainable.

While all this was going on Tim Fennell, and a growing developer community, were working on Stripes 1.3. It has just been released, so if you’re on the lookout for a Java web framework do yourself a favour and have a look at Stripes.

Very nice work, Tim!

[tags]java, stripes, web frameworks, hibernate[/tags]

Recently my notbook has been performing slower than it used to. High CPU usage, jerky mouse cursor, music stuttering while compiling Java code, not good. I though it was a disk issue (had a small “data lost” scare a few days ago), but no, it was problem between chair and keybord. I’d misconfigured Winamp after upgrading to version 5.21.

In earlier versions of Winamp there was an option to set the size of the file read buffer. I had this set quite high in an attempt to reduce disk access while running on battery. In Winamp 5.21 this setting didn’t seem to exist. I looked all through Preferences and couldn’t find it, but I hadn’t looked hard enough.

While stuffing around in Preferences I had accidently configured it to use the Nullsoft Disk Writer plug-in for output. Turns out that this plug-in writes your MP3/Ogg files to disk as rather large uncompressed WAV files while they play.

A post on the Winamp Forum suggested that I should use the DirectSound plug-in for output on Windows XP. After taking the time to select the DirectSound plug-in (see image), and configure it, my user experience is as expected.

[tags]winamp, directsound, cpu[/tags]