Archive for the Data Category

Useful geodata links

Posted in Data with tags , on February 22, 2010 by hansvandermaarel

I’ve been asked recently about sources of useful (free) geodata. Since I thought it might be nice for other people to have as well I’ve decided to make a blog post about it. So here goes.

Natural Earth Vector – Deserves to be mentioned first. A global cartographic (!) dataset at scales 1:10M, 1:50M and 1:110M, both vector and raster. Made possible by Tom Patterson, Nathaniel Kelso and a small army of volunteers. Made by cartographers, for cartographers.

Geocommons Finder – Offers a wide range of thematic datasets.

GIS Data Depot – Requires registration but again a good source of data. VMAP / DCW per country, many US datasets per county.

SRTM data – Elevation data in 90 meter resolution (30 meters in the continental US) data in 1×1 degree tiles, or go to the CGIAR site for corrected data in 5×5 degree tiles. SRTM only covers the earth from 60 North to 60 South and does not offer bathymetry. Viewfinder offers manually corrected data for significant mountain areas as well as manually created data for areas not covered by SRTM.

Aster Global DEM – Higher resolution than SRTM, but I’ve heard mixed reports about the data quality.

MapStor – Old USSR topographic maps, either free (download one by one) or dirt cheap even for large areas. While most of these maps are by now quite outdated (they were, after all, produced before the collapse of the Soviet Union), in many parts of the world they are the only easily obtainable source of reference data. If you speak Russian of course… I’ve personally used this to track down villages in India that Google doesn’t show. The maps come with a georeference in OziExplorer format. Global Mapper can read this and can also automatically remove the collars.

Global Land Cover Facility – Offers various raster datasets, including Landsat, for download. Much of the data is offered in seperate bands, so you’ll need to do a bit of research to use it.

Geonames – USGS board on geographic names. Note that the data is not without errors, so always doublecheck.

True Marble – 250 meter (free) and 15 meter (paid) satellite data. Automatically generated out of Landsat data. Some artifacts here and there but overall a good dataset.

David Rumsey collection – Not geodata per-se, but a vast collection of old maps.

Perry-Castañeda Library – Large collection of reference maps offered by the University of Texas.

Cloudmade and Geofabrik offer shapefile extracts from the OpenStreetmap database by continent/country/state.

EarthExplorer and the Seamless Server are two USGS sites offering data for download. Mainly US data as far as I can tell.

US TIGER – vector data of the US offered by the US Cencus Bureau. Its quality is not always 100%, see this post on Cartotalk for an explanation.

US National Atlas – Many thematic datasets for the US.

Advertisements

Natural Earth Vector is here!

Posted in Data, Neogeography with tags on December 3, 2009 by hansvandermaarel

Those of you who follow the cartographic news will undoubtedly have heard of this already, but Natural Earth Vector has finally been released. This is a royalty-free vector dataset for the entire world at scales 1:10M, 1:50M and 1:110M. Existing datasets tend to be way too detailed for these scale levels (VMAP0, for example, is meant for scale 1:1M… OpenStreetmap is aimed at even smaller scales).

Many volunteers (including yours truly) have put in a lot of time and effort to create this dataset. Many thanks to everybody who contributed.

If you want to take a look at the data, please visit www.naturalearthdata.com

Natural Earth Vector

Posted in Data, FME with tags , , on July 16, 2009 by hansvandermaarel

Ever since Tom Patterson started offering Natural Earth, a public domain raster dataset of the entire Earth, I’ve been a big fan of it. In fact, the base layer for the Oolaalaa globes is based on Natural Earth I (with added bathymetry and increased contrast).

After releasing 3 raster datasets this way, Tom has recently teamed up with Nathaniel V. Kelso to create a vector product that also will be released into the public domain: Natural Earth Vector. Nathaniel has invited me to participate in this as well, and to be honest I think it’s a wonderful project. So far I’ve been involved in extracting road data from OpenStreetmap and identifying rivers in Europe.

Processing the OpenStreetmap data, using FME, reinforced my slight aversion of XML/GML based data. When downloading the data, you have the option of getting the .osm data, which is in an XML format, or shapefiles. FME does support the .osm format, but processing the files takes a terrible amount of time. Processing the shapefiles for Europe, Asia, Oceania, Africa and South America, just filtering out the major highways, takes about 11 minutes. I don’t dare to do that with the .osm data, but I fear it may take days. It’s nice that XML is so open and free, but for large amounts of data it’s simply unsuitable.

Anyway, to get back to Natural Earth Vector, it will be officially unveiled at the NACIS conference in Sacramento this october and I for one am looking very much forward to that.

Radiohead anaglyph

Posted in Data, FME, Software with tags , , , , , , on August 12, 2008 by hansvandermaarel

Rock band Radiohead’s latest music video, House of Cards, features a LIDAR visualisation of singer Thom Yorke. This is a quite cool thing to see, but the really cool thing is that they open sourced all of the raw LIDAR data, it’s available for download at code.google.com.

The software that’s provided there lets you play with the data and create your own animation, but I thought I’d do something different. So I downloaded the raw data, played with it for a bit and came up with this:

This is an anaglyph, so you’re going to need some red/blue glasses to see it in its full 3D glory.

So how did I produce this?

  1. I downloaded the data and picked a random frame.
  2. Every frame is supplied as a CSV file with X, Y, Z coordinates. I used FME to create a 3D shapefile out of this.
  3. I loaded that 3D shapefile into a new VNS project as control points and gridded a DEM from it. Added some background terrain (otherwise it’d just be a floating head) and positioned a camera right above it.
  4. Rendered with the stereo option in VNS. This actually renders a stereopair of the scene.
  5. With the help of an easy tutorial for Photoshop I composited both images into a single image.

So there you have it. Amazing to see all that high-end graphics and GIS software being put to this use… I have to admit this was all a lot easier than I imagined it to be, so I’m certainly going to try out the anaglyph technique some more.

Creating custom Google Map tilesets

Posted in Data, Maps, Neogeography with tags , on July 9, 2008 by hansvandermaarel

Based on a few questions coming in through Cartotalk over the past few months, I’ve decided to look into offering some services around the creation of custom Google Map tilesets. This is in effect and extra layer with any georeferenced data you like, appearing inside Google Maps, running on your own website.

Global Mapper, which I’ve raved about before, offers a nifty way to export any georeferenced raster image to a Google Maps tileset, it will even generate the html and javascript for you. Unfortunately, in the case of maps it’s generally not a matter of pressing a button and it will magically work. There’s a few things to keep in mind and those are the things I hope to add value to my customers with.

  • Resolution
  • Projection
  • Design for screen, rather than print

Basically it boils down to making sure the image that is read by Global Mapper is as close to the correct projection and resolution as possible, to avoid resampling. I’ve put a few samples online on the Red Geographics website.  As always, feel free to contact me if there’s questions.

In addition to the design, another important thing to keep in mind is the optimization of the images. I’m partnering with Derek Tonn of GraphicsOptimization for this. It’s amazing how much smaller he can squeeze the images and still have them look the same (at least to the naked eye). A 20% reduction in the case of b/w aerial photography for example (not bad if you consider the 2 sets together weighed in at 120 Mb before optimization). This saves time, energy and ultimately, money.

Google Map Maker

Posted in Data, Neogeography with tags , , on June 25, 2008 by hansvandermaarel

Slightly confusing title perhaps, but it seems Google is going the OpenStreetMap route with their new Google Map Maker service. This lets users add and/or edit features (points, lines and areas) for Google Maps.

Currently, the only areas supported are the ones that are sparsely covered by Google Maps in the first place, a number of Caribbean islands, Iceland, Pakistan, Cyprus, Vietnam. Edits and additions are moderated and of course you’re signing away all your rights to the edits you made to Google 🙂

It reminds me a bit of Wikimapia, or the “community” layer in Google Earth. You’d be amazed at what you can find in there!

New stuff on OpenStreetmap.org and Global Mapper

Posted in Data, Software with tags , on April 29, 2008 by hansvandermaarel

In the past I’ve been quite unsuccesful trying to get data from OpenStreetmap.org into another geo-application. Any other geo-application for that matter. There was some documentation on how to get it into a PostgreSQL database, including an executable that would take data directly from the main OSM database and put it into a local one, but that was lacking some important information (like, the passwords and table structure it was expecting…). The good people at Safe got a long way in supporting the xml-format that JOSM exports to, somebody wrote a Manifold script to download the data and load it into Manifold and there was somebody in Germany kind enough to make regular exports of various European countries and offer them online as shapefiles. None of those options were really “complete”.

For something that’s supposed to be ‘open’, it sure is hard to get data out of it…

Then two things happened in a short amount of time:

  • The OpenStreetmap website was changed so that exports to xml could now be made directly from the site. No need to install and run JOSM anymore.
  • Global Mapper 9.03 came out, with native OpenStreetmap support.

Just tried that combination and it works like a charm. It loads all information from OSM, including more point features than the shapefiles provided. Exporting from Global Mapper to shapefiles is a breeze too, so this is a very big step in opening up OpenStreetmap.