Digital Earth Weblog

Via Chris, and as he says, Wow. My company was a reseller of ERM products for several years, so it will be interesting to see how the ERM product line evolves under Leica’s stewardship. I too, hope the ERM team stays in Perth (good luck guys and gals).

Leica press release dated 21 May 2007.

Private companies have been capturing aerial imagery and making it available for general sale for a few years. However, this is the first time I’ve heard of a company doing the same with infra-red thermal imagery.

Hotmapping.co.uk have complete aerial heat-loss data sets of most of London and all of Norwich.

I wonder if Google or Microsoft would make this sort of imagery available via their respective mapping products? There are some privacy issues but, on balance, it seems like a good thing. It would look very nice in Google Earth.

This script might be useful to anyone looking to inventory ECW files. Given a target directory it will list all ECW file in that directory, and its subdirectories. Output is CSV format, and includes file path and name, TLX, TLY, BRX, BRY, width (cols), height (rows), cell size X, and cell size Y. Prerequisites and command line usage detailed in the file header.

Quick Google Earth news:

• Google Earth 4.0 beta has been released. There is an updated user interface, a Linux version, and support for textured 3D models.
• KML 2.1 offers a few enhancements over KML 2.0. Region-based network links look interesting, especially for WMS overlays.
• New Australian high resolution imagery. So far I’ve seen new data around Sydney extending to Nowra in the south, the base of the Blue Mountains in the west, and beyond Newcastle in the north.

[tags]google earth, imagery, kml[/tags]

Recent posts on the Scrum Master Certification course may not have been of much interest to those of you who are into geospatial. However, if you are into aerial and satellite imagery there was a tenuous link. Three of the participants were from a movie visual effects shop.

This company has worked on titles like Harry Potter and Lord of the Rings. Their render farm generates a huge amount of imagery, and uses a massive amount of processing power. Here are some rough figures that I picked up during conversations:

• They render 30 frames per second, sometimes for an entire movie (I forgot to ask how large each frame image is).
• A single frame containing 3D artwork can take 5 hours to render.
• Automated processes combine frames into Quicktime sequences.
• They also have automated QA processes.
• They upgrade their storage capacity in about 5 terabyte chunks.

It would be interesting to see if some of the infrastructure and processes used in visual effects could be applied to earth imagery.

[tags]imagery, visual effects, aerial, satellite[/tags]

The Enhanced Compressed Wavelet (ECW) technology is great stuff. It makes it possible for a lot more people across an organisation to get access to valuable imagery, and that imagery can be Gigabytes or even Terabytes in size.

Disclaimer: This post has been recreated from an article that was on the old Digital Earth Pty Ltd website. The original article was last updated on 30 June 2001. Some minor changes have been made in this version, but some of this information may be out of date. All the products mentioned have definitely evolved.

This article aims to help you get the best results from your ECW compression. It is aimed at users of colour (RGB) imagery. It does not attempt to cover detailed system requirements for processing and compressing large mosaics.

ECW Compression Tools

There are two options for ECW compression:

Free ECW Compressor

The Free ECW Compressor can be used to compress single files of up to 500 Mbytes in size (uncompressed). It does not support the compression of mosaics or batch compression, but is suitable for manually compressing small files.

You can download the Free ECW Compressor, and free ECW viewing plug-ins for many common GIS, CAD and office applications, from the ER Mapper download page.

ER Mapper

If you want to compress larger files, perform batch compression of multiple files, or follow the:

1. orthorectify,
2. mosaic,
3. colour balance, and
4. ECW compress

processing steps then the software you need is ER Mapper. ER Mapper is available resellers around the world.

The ECW compressor is constantly being improved by Earth Resource Mapping. Before starting a compression project please make sure you have the latest version of the software you are using.

Target Compression Ratio vs Actual Compression Ratio

Before we get into the details of getting the best from your ECW compression it is important to understand compression ratios. The ECW compressor asks the user for a “target compression ratio” (labelled below as “Desired compression ratio”), not a resulting “actual compression ratio”.

ECW is not a lossless compression technique. It achieves its very high compression ratios by discarding some of the high frequency information in your image. The numeric value you enter for the target compression ratio is used during the compression process to determine how much of that high frequency information is discarded.

The ECW compressor will produce ECW files of consistent image quality if you use the same target compression ratio. The actual compression ratio achieved depends on the amount of information in your image. An image with few changes in texture and contrast (e.g. an aerial photograph of grasslands) will compress to a higher actual compression ratio than a very complex image (e.g. an aerial photograph of a dense urban area). Therefore, the resulting ECW images will be different sizes.

It may help to think of the target compression ratio as an “inverse quality index”. The higher the target compression ratio the lower the quality.

Maximising Image Quality

Compressing Once is Best

Since ECW compression is not lossless it is always best to compress your image data only once to minimise the loss of quality. Always compress the original data whenever possible. (Please do not delete any of your original data after producing ECW files! Archive them on reliable media.)

Multiple Compression is Sometimes Required

Sometimes there are reasons for having to compress imagery more than once. The classic case is when you are working on a very large mosaic but do not have the hard disk space to load all the source images at once. To get around this you can use ER Mapper to create and compress “sub-mosaics”, and then re-compress the sub-mosaics into the final large mosaic.

The trick to minimising loss of quality is to re-compress the sub-mosaics at an integer multiple of their idividual target compression ratios. For instance, if you want to use a final target compression ratio of 20:1 then you would compress the sub-mosaics at 5:1 (n = 4), 10:1 (n = 2), or 20:1 (n = 1).

Example: Suppose that you had 200 Gbytes of data that you wanted to mosaic, at a target ratio of 20:1, but you only had about 80 Gbytes of disk space. I would split the mosaic into four sub-mosaics of about 50 Gbytes each and then process them as follows:

For each sub-mosaic:

1. Load source images onto your hard disk.
2. Use ER Mapper’s Image Display and Mosaic Wizard to mosaic the source data.
3. Colour balance, if required.
4. Compress the mosaic at 10:1.
5. Delete the source data, keep the ECW file.

Use ER Mapper’s Image Display and Mosaic Wizard to mosaic the ECW files produced by compressing the sub-mosaics. Compress the final mosaic with a target ratio of 20:1. Note: Re-compressing ECW data takes longer than compressing TIFF or ERS files because the data has to be decompressed from the sub-mosaic images before it can be compressed into the final mosaic.

Different Data Types

To get the best quality ECW files you need to consider the type of image data you are compressing. You can break image types into two groups — earth observation and cartographic.

Earth Observation

These are images that have texture, where adjacent pixels are likely to have different values. For example:

• Aerial photography — colour and grayscale
• Satellite imagery — pseudo-colour and other non-classified RGB derivatives.

The ECW compressor is optimised for these type of images. Selecting a target ratio of 20:1 or 25:1 for colour imagery, and 10:1 for grayscale, will result in good quality images.

The ECW compression process tends to visually “flatten” RGB images when higher compression ratios are used. To improve the visual display the ECW decompressing application will add random visual noise to the image during the image viewing process — if it was compressed at a target ratio of 10:1 or higher. This is designed to improve the perception of image texture. Note: The “visual noise” is not actually present in the ECW file, is introduced via the decompression/viewing process.

Note: This is a problem for some cartographic style images. (See “Cartographic” below.)

Since the visual noise is added by the decompressing application it will not be added twice to an ECW file that is the result of multiple compressions. An example of how you would have the visual noise added twice is:

1. Compress the image to ECW.
2. View the image (visual noise gets added) and then save the image to another format (the saved image has the visual noise saved as well).
3. Compress the newly saved image to a new ECW file.
4. View the new ECW file (visual noise gets added again).

Cartographic

These are images that often have large areas containing uniform colour values, and where adjacent pixels are most likely to have the same values.

For example:

• Scanned topographic maps
• Rasterised vector data. e.g. cadastre
• Classified satellite imagery

These type of images require additional care during compression in order to maximise display quality. We do not want the decompressor to add random noise to these images if they are compressed at 10:1 or higher. We want nice big blocks of uniform colour.

There are two ways to avoid the random visual noise issue:

• Compress as RGB and use a target compression ratio of less than 10:1, or
• Compress the data as Multiband instead of RGB.

You should also find that these style of images compress very well. It is not uncommon to get actual compression ratios more than double the target compression ratio.

Notes on Multiband: Typically, multiband ECW images containing 3 bands are approximately 1/3 larger than their equivalent RGB version. When compressing an RGB image the compressor takes advantages of special techniques applicable to red/green/blue imagery which result in an additional reduction in file size.

Calculating the Size of Mosaics

Input Size

If you are compressing a mosaic using ER Mapper calculating the size of that mosaic becomes important for Image Web Server users. Their licence may restrict them to serving images of a certain size before they were compressed.

The ECW compressor works on a line-by-line basis, which is why it does not require a lot of memory to compress very large mosaics. The compressed data is stored in the ECW file as a pyramid of data blocks (ECW is not a tile based format — there is a whole bunch of mathematics involved which I don’t fully understand, and is therefore beyond the scope of this article.)

Each line of image data, that is being compressed, needs to be the same length. Therefore, the extent of the input mosaic is its bounding box.

ER Mapper can handle the display and compression of images with different cell sizes. For example you can display aerial photography and satellite imagery in one mosaic, and you can compress it. The point to note is that in order to do this
the compressor must have a uniform cell size across all images in the mosaic. This must be the minimum cell size used by all images in the mosaic.

Calculating the size of an input mosaic requires:

• Width of the bounding box (width).
• Height of the bounding box (height).
• Minimum cell size in the mosaic (cellSizeX and cellSizeY).
• Number of bands to be created in the ECW file (numOfBands).

The following factors are ignored:

• Image overlap. (Only data that contributes to the mosaic is included.)
• Compressed images — all compressed data (e.g. LZW TIFF, ECW files) are decompressed and then recompressed so you may have a saving in disk space but there is no saving in the size of the input image.

Green = 10cm cell size, Yellow = 20cm cell size, White = no data.

In the above image the yellow images will be sub-samples to 10cm cell sizes during the compression process. This effectively increases the size of those images by a factor of four, but they will compress quite well.

To calculate size of a mosaic image before compression use this formula (the same linear unit of measurement must be used for all dimensions):

numOfBytes = ((width / cellSizeX) x (height / cellSizeY)) x numOfBands

size in Gigabytes = numOfBytes / (1024 x 1024 x 1024)

Output Size

As mentioned above, the final size of an ECW file depends on he target compression ratio, the information in the image, and several other factors:

• Compression type: RGB or Multiband (also mentioned above)
• Whether it was optimised for Internet use.

If you are going to serve your ECW files using Image Web Server then you can increase performance by telling the compressor to optimise for Internet use. This reduces the size of the data blocks in the ECW file from 512×128 to 64×64, but increases the size of the “Block Offset Table” in the header of the ECW file. The result is approximately a 10% increase in file size.

Credits

Thanks to Stuart Nixon (CEO, Earth Resource Mapping), and the Technical Support and Development teams at Earth Resource Mapping in Perth, Australia. I learnt everything in this article from them.

[tags]ECW, imagery, aerial, satellite, compresison, ER Mapper, Image Web Server[/tags]

In a comment on my previous post James Fee suggests that ArcGIS Explorer will provide access to the public ArcWeb Services. As far as I can tell that means DOQQ imagery in the USA and lower resolution satellite data for everyone else. (Could be wrong here. Found nothing specific on the ArcWeb Services website.)

The BYO high resolution imagery approach essentially resticts ArcGIS Explorer to use behind the firewall. The licensing of higher resolution imagery will prevent most organisations, other than Google and Microsoft, from publishing it to the Internet. If you don’t have high resolution imagery you may as well publish your data in a web browser.

If, as suggested at OZRI, ArcGIS Explorer does require ArcGIS Server then an ArcGIS Explorer application is definitely not free. ArcGIS Explorer may provide incentive for organisations to purchase ArcGIS Server, I’d suggest that very few organisations will publish data on the public Internet using ArcGIS Explorer. The cost of the infrastruture to support a potentially large number of users is likely to be significant. Google provides most of that infrastructure for free (other than the yet to be sighted advertising).

My take, at this point in time, is that ArcGIS Explorer is going to remain a niche product. It looks like it could be a very useful product within that niche. It may even be good enough to kick Google Earth off ESRI’s turf, but I’ve seen nothing to suggest that ArcGIS Explorer is going to make anything but a tiny ripple in the Google Earth ocean. Google are too good at providing free tools that lots of people want to use.

Update

James fee has responsed to this post on his blog. I left a comment clarifying that…

Roughly speaking, I

The new Google Earth imagery, that is. These vectors are coming from ArcIMS which is reprojecting them on-the-fly from a custom Lambert Conformal Conic projection to WGS84/Geographic.

Nice work Google Earth crew. Keep providing imagery this good and I’m prepared to recommend Google Earth as a spatial viewing platform (but only if any advertising stays out of the way).

Now, where’s ArcGIS Explorer and what does it have to offer Australian users in the way of imagery?

Back in October 2005 I wrote about what appeared to be an image rectification issue in the Google Earth imagery in the Sydney (Australia) area. This problem had been enough to discourage me from using Google Earth to display high resolution vector data.

Well, I have to review my position because Google have provided some new raster data over Sydney, and it is higher resolution. The edge matching issue has gone away, and I’m keen to see how well the vectors match with the new imagery.

The old:

The new:

Nice!

Perhaps this explains the mismatch we are seeing between the Google Earth imagery and high quality vector data. There does appear to be an issue with image orthorectification or registration.

Download KMZ file.

Update, 2005-12-03: New imagery over Sydney. The problem shown above seems to have been fixed.