Archive for the FME Category

Spatial databases once more

Posted in FME, MAPublisher, Software with tags , , on January 20, 2010 by hansvandermaarel

For a while I’ve been meaning to try out more stuff with spatial databases. I’ve been using them for a few years actually, but so far that’s always been MySQL Spatial thanks to a simple install package produced by the good people at Apachefriends.org. It’s easy to set up that way but… MySQL Spatial isn’t very widely supported, in fact the only application I have that actually reads/writes it is FME…

So, wanting to try something different I went ahead and installed PostgreSQL with PostGIS. This should be wider supported, of the software that I use on a regular basis FME, Manifold and ArcGIS should be able to read/write it. Well, FME certainly does, no problem there. Having a bit more trouble getting Manifold and Arc to talk to the database though. Apparently they require an ODBC connection and that seems to be tricky to set up. Hopefully I can get that sorted out soon.

— Update January 21st —

Well, as Jeff has pointed out, I should not worry about Arc…

I’ve continued hacking away at this and managed to get both FME and Manifold talking to a PostGIS database. Funny enough Manifold only does that in the 32 bit version, though it appears there’s some extra dll’s for the 64 bit version that can be used to get that one working too. Also, veteran Manifold user (rock star I should say) Art Lembo of Gisadvisor.com has a training video on setting up an enterprise spatial database with Manifold and PostGIS that looks rather interesting.

Anyway, I have opted to install the databases on a brand new (and rather cute) Compaq ultraportable. For playing around with databases in a network environment it appears to be just fine (which is good, because I’m short on space)

Advertisements

Natural Earth Vector

Posted in Data, FME with tags , , on July 16, 2009 by hansvandermaarel

Ever since Tom Patterson started offering Natural Earth, a public domain raster dataset of the entire Earth, I’ve been a big fan of it. In fact, the base layer for the Oolaalaa globes is based on Natural Earth I (with added bathymetry and increased contrast).

After releasing 3 raster datasets this way, Tom has recently teamed up with Nathaniel V. Kelso to create a vector product that also will be released into the public domain: Natural Earth Vector. Nathaniel has invited me to participate in this as well, and to be honest I think it’s a wonderful project. So far I’ve been involved in extracting road data from OpenStreetmap and identifying rivers in Europe.

Processing the OpenStreetmap data, using FME, reinforced my slight aversion of XML/GML based data. When downloading the data, you have the option of getting the .osm data, which is in an XML format, or shapefiles. FME does support the .osm format, but processing the files takes a terrible amount of time. Processing the shapefiles for Europe, Asia, Oceania, Africa and South America, just filtering out the major highways, takes about 11 minutes. I don’t dare to do that with the .osm data, but I fear it may take days. It’s nice that XML is so open and free, but for large amounts of data it’s simply unsuitable.

Anyway, to get back to Natural Earth Vector, it will be officially unveiled at the NACIS conference in Sacramento this october and I for one am looking very much forward to that.

FME vs Global Mapper

Posted in FME, Software with tags , , on July 9, 2009 by hansvandermaarel

So for a long time I’ve been saying I generally don’t use FME for raster work because it’s slow. This was all based on some bad experiences way back with FME 2006 (the first version to support raster data if I recall correctly) and the infamous 37-Gigabyte temporary file that was left on my hard drive after a mosaicking operation crashed. So I’d mostly been using FME for dynamically making a clip out of a larger raster file, which I would invariably produce with Global Mapper.

Recently, Dale Lutz of Safe Software had suggested that I’d give FME another chance, because they had made a lot of progress in the handling of raster data. So I decided to give it another go, pitting the FME 2010 “UC” beta against Global Mapper 10.02.All tests were done in Windows Vista x64 running through Bootcamp on my Mac Pro with 10 Gb of ram. All data was present on the local hard drive.

Test 1 : tiling a raster

I took a TIFF file of Tom Patterson’s Natural Earth. Specifially, I had already produced a combined version. Tom offers it in “west” and “east” files, I put those together to get the entire earth in a single, almost 2 Gb, GeoTIFF. I set up both FME and Global Mapper to tile this file into 90 x 90 degree chunks and output each of them as GeoTIFF. FME blazed through this in a mere 32 seconds. Global Mapper took close to 18 minutes.

Test 2 : mosaicking the tiles back to an ECW

Another common procedure, mosaicking a number of raster tiles and creating a single large image out of them. I did try to set the compression factor in both FME and Global Mapper to the same value but since they specify them in different ways, it’s difficult. I got close, but not quite. In any case, compression generally isn’t going to give very nice results on map images, where you often have areas with solid colors. Also, I did struggle a bit with FME to get it to deal with the raster bands (the alpha channel) in the GeoTIFF’s correctly. Still, I only measured actual processing time, not how much time it took me to set it up (which, to be honest, is roughly the same in both FME and Global Mapper). I fed both applications their own tiles that were output in test 1. FME got through this in about 1:30, Global Mapper took 3:30

Final verdict

Easy one really, FME took a combined total of 2 minutes of processing, against 21 minutes for Global Mapper. Need I say more?

I do think the two products each have their unique strengths though, so I will certainly not stop using Global Mapper. But I will use FME much more often for this kind of work.

FME User Conference – day 2

Posted in FME with tags , on June 13, 2009 by hansvandermaarel

Well, wrapping up the FME User Conference, I can say it’s been another interesting one. This morning Don and Dale showed us a bit about the new things coming in FME 2010, some of which I had already seen in beta’s or was revealed earlier in the event.

Certainly an important improvement is security on FME Server. This was not in place before and users would have to rely on other, 3rd party, ways to secure the access to the server. Now that it comes bundled with FME Server itself, it’ll be easier for the average user to set it up.

One thing I’m personally looking forward to to try out is the improved generalisation options that were mentioned by Daniel Pilon of NR-Can in his presentation about generalising a 1:50k topo dataset to 1:250k. This is something I’ve been looking into myself in the past (though a less dramatic scale range) and never really got far with it.

More room for improvement comes with the new 3D stuff that I have already mentioned. I think it will be a good idea to review the workflow for the Virtueel Apeldoorn environment and see if we can cut some corners. In fact, after 3ds and obj writing was announced last december, we have already started to look into those as a possible replacement for DXF in the whole process.

FME User Conference – day 1

Posted in FME with tags , on June 12, 2009 by hansvandermaarel

Day one of the FME User Conference brought us, as always, the big Don and Dale introduction. I never thought I’d hear AC/DC’s “Thunderstruck” at a conference, but they managed to do so. As usual, their introduction once again proves that Safe is a fun company to work at (or, in my case, do business with).

The keynote, delivered by James Fee, touched upon some interesting concepts on how data is being managed and distributed nowadays. One of his main points was that it’s not just about publishing the data, but also about your potential users being able to find it.

The presentations, 2 tracks, covered a wide range of subjects, though it’s surprising to see a very high proportion of FME Server related presentations being done. For me the highlight was Brendan Cunningham of Kilkenny County Council in Ireland, who talked about a SMS notification system powered by FME. My own presentation, about a current project where I have to combine Dutch and Belgian data into one coherent set was well-received and I’m pretty pleased with how it went.

In the afternoon it was time for FME Idol and FMEopardy (an FME version of jeopardy), which once again were great fun to laugh. The day was ended with the conference social up in the Roundhouse Lodge at Whistler Blackcomb Mountain (not a bad place to be, to be honest).

FME User Conference – day 0

Posted in FME with tags , on June 11, 2009 by hansvandermaarel

Well, it’s technically not the first day of the conference, that doesn’t start until tomorrow, but I participated in some of the technical sessions today and already saw some interesting things. This is all based on what we’ve seen in the FME 2010 beta build used at the conference:

  • PDF writing is apparently improved quite a bit. This is good news for me because I occasionally get questions along the lines of “can you do 5000 maps with driving directions?” (yes, an actual question…) and FME is a perfect tool for doing the processing. If the output can be made to look fairly decent, this would be a very serious option.
  • There is a replacement for the FME Viewer on the way. Still in a very early stage, but it’s looking pretty good. This new tool, the DataInspector, will do 3D as well.
  • I’ve noticed a couple of new transformers that seem to indicate that network analisys is an upcoming frontier for FME.
  • Many 3D formats can now also be read (OBJ, 3DS, Sketchup) and have support for textures. Getting a textured model from Sketchup into 3D PDF is a breeze now.
  • Dynamic reading/writing is another big thing. I haven’t had much change to use this in real-world scenarios yet, apart from using a Generic Writer once, but based on what I’ve seen this could be a very useful thing in environments where a data model may slightly change over time. Also, clever use of a SchemaMapper can save a lot of repetitive use of AttributeCreators, AttributeSetters and AttributeCopiers.

Aside from all that, it’s nice to meet other FME users and see what they’re up to. The atmosphere at the FME events is always very relaxed.

Not immediately conference-related, but worth a mention, is the fact that over the past years I’ve not used FME to mosaick large amounts of raster data, based on some bad experiences way back with FME 2006, preferring Global Mapper instead. Dale Lutz recently told me that I really ought to give FME another chance, so when I get back from this trip I’ll see if I can set up some benchmarking tests to figure this out once and for all.

FME user conference 2009

Posted in FME with tags on June 8, 2009 by hansvandermaarel

The 2009 FME User Conference starts on Thursday (with pre-conference workshops and training sessions commencing today) and I’m currently on my way to Whistler to attend. This time, I’m set to do one lightning talk (processing 3D data for an online virtual city) and one presentation (processing topographic data from 2 different national mapping agencies, combining them into a single dataset).

The rest of the programme lookes very interesting and there seems to be a fairly heavy focus on 3D this time. I’m looking forward to it!