OS Meridian Roads Data

I have been doing a lot of work processing OS OpenData recently and whilst loading Meridian 2 onto a PostGIS system, I have encountered a couple of issues that I thought were worthy of a post.

 

The source data files I am working with are the latest MapInfo TAB files (Version: 07/2013) downloaded from the OpenData website.  My working system is a Kubuntu 13.04 laptop and the Server a virtual machine running PostgreSQL 9.1.  The actual data transfer is performed by the Geospatial Data Abstraction Library (GDAL).  I have written a short BASH script that calls the GDAL ogr2ogr utility, iterating over the extracted files from the downloaded archive:

#!/bin/bash 
function UploadFiles
{
 # Upload TAB files to PostGIS with OGR.
 for File in "$1"/*.TAB 
   do
     if test -f "$File"
     then
       echo "$File"'...'
       export PGCLIENTENCODING=LATIN1
       ogr2ogr -f "PostgreSQL" PG:"host=XXX.XXX.XXX.XXX 
       user=postgres dbname=OpenData password=XXXXX port=5432"
       "$File" -a_srs "EPSG:27700" -skipfailures -progress
     fi
   done
 # recurse directories
 for SubDir in "$1"/*
   do
     if test -d "$SubDir"
     then
       UploadFiles "$SubDir"
     fi
   done
}
UploadFiles "/media/Archive/GIS Data/OS OpenData/Meridian"
However, I did encounter a few snags.  Firstly, the A road section ‘O13RDLWRD2HTV’ caused the following warning:
Warning 1: Geometry to be inserted is of type Multi Line String, whereas the layer geometry type is Line String.

Converting the geometry for this record into well known text format revealed:MULTILINESTRING((430708.0001 431916.9998,430793.00005 431966.00005),(430791.00015 431966.00005,430837.9999 431994.0001))The Meridian road network is intended to consist of continuous lines, so this two part record is, in effect, a broken line.  The work around is to remove the inner line ends and convert the remaining end points into a single line string:LINESTRING(430708.0001 431916.9998, 430837.9999 431994.0001)To help visualise this I have prepared a graphic using Quantum GIS. The original broken line is rendered in red and my suggested solution in green.

MeridianBug

The fixed record can be added back onto the server with the following SQL command:

INSERT INTO "a_road" 
("wkb_geometry" , "feature_code", "osodr", "route_number", 
 "road_name", "length_metres", "trunk_indicator") 
VALUES 
(ST_GeomFromEWKT('SRID=27700;LINESTRING(430708.0001 431916.9998, 
 430837.9999 431994.0001)'), 3001, 'O13RDLWRD2HTV', 'A61', '', 
 152, '');

There is a second similar problem in the minor roads layer for road section ‘O16A01KU1ABTW’.  The solution for this is exactly the same, stitching the two segments back into a single line.  I have reported these two issues to Ordnance Survey and  I have received feedback from the Product Manager that both errors have now been fixed. The changes will be issued in the next release of Meridian 2 Open Data.

One last thing to mention, on my first attempt at the data upload, I did find some accented characters had not been correctly loaded.  The PGCLIENTENCODING define in the BASH script hints PostgreSQL to perform the correct character re-encoding during the import.

Lazarus on the Raspberry Pi

[EDIT: The build steps detailed in this post will no-longer work with current (10/07/14) versions of Raspbian and Free Pascal. Please see the more recent post here for a newer set of build instructions for Free Pascal 2.7.1 and here for Free Pascal 3.1.1 with Lazarus 1.7]

I have been investigating the use of Free Pascal and Lazarus on the Raspberry Pi. The Pi is a minimal, low cost, ARM based computer, designed primarily for educational uses. As a Pascal advocate, I would love to see a modern Pascal compiler made widely available on a platform like this.

The official operating system for the Pi is called Raspbian which, as it’s name suggests, is a derivative of Debian Linux. Free Pascal and Lazarus packages are available on Debian, so I wanted to see if they had successfully transferred to Raspbian in a usable form.

I have one of the new Raspberry Pi Model B’s that has 512Mb of RAM and I have been using it with the current Raspbian image (2012-10-28). After spending a little while getting to grips with Raspbian, I was able to progress to the business of testing Free Pascal. The good news is that the standard Debian packaging commands worked a treat:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install fpc
sudo apt-get install lazarus

The only snag from a visibility point of view is that there is no automatic integration of Lazarus into the LXDE menu. From the command-line though, ‘fpc’ will invoke the compiler and ‘lazarus’ the GTK2 based IDE. However, as with Debian, the versions of the available packages are a little behind the current releases. On my main Kubuntu system, I habitually build Free Pascal and Lazarus from source, using Subversion to obtain the latest changes. This process requires a seed compiler, which in this case, can easily be supplied from the default repository.

So I started again with a fresh Raspbian SD card, but this time, I only installed FPC from the repository. Free Pascal additionally requires the GNU build tools and fortunately these are already installed in Raspbian, so I only had to add Subversion before I could attempt to use my usual build process on the Pi with a few of the options tweaked for ARM output:

sudo apt-get install subversion

Whilst researching this I have found that others have already attempted this with varying degrees of success. Common problems encountered seem to be caused by running out of memory, or card storage space during the build process. Indeed, I hit both issues even with the Model B’s 512Mb RAM. In the case of RAM usage, I had to reduce the amount of RAM dedicated to the GPU from 128Mb to 64MB. I have seen reports for Model A Pi’s where this needed to be reduced to 16Mb to avoid problems. I also ran out of space on a 4Gb SD card and had to swap to an 8Gb one to provide enough head-room.

After a couple of hours solid processing I had a working build of Free Pascal 2.7.1 with ARM hard float support and Lazarus 1.1. For thoses interested in repeating this process you can download the scripts here: FPC Build Script and Lazarus Build Script. It is worth pointing out that the two scripts behave in slightly different ways. The Free Pascal process is more elaborate for two reasons. Firstly, it always makes a temporary working copy to perform the build, this is a brute force approach to preventing changes over time causing build issues for the compiler or runtime libraries (admittedly at the cost of build time). Secondly, it also performs a full install to the system to superceed the Raspbian 2.6.0 seed compiler. This means that it becomes the new default ‘fpc’ system command. The Lazarus script, by contrast, is a much simpler affair, building its binary in the subversion working folder. I usually run it from this personal folder, from which I can make changes such as installing additional packages, without having to use sudo permissions.

Now I wanted to evaluate the capability of this new setup. Firstly I built several of the standard example programs supplied with Lazarus. All of the ones I tested executed without problems and they looked pretty good on the LXDE desktop. It is worth commenting on the fact that the general operation feels sluggish compared to an Intel based system. However, it is not that slow that you would find it unusable and I have seen Eclipse run slower on a much more capable PC!

Much of my work with Lazarus involves creating graphical output, so I wanted to see how it coped with using some of the graphics libraries available. The built in FPGraphics and LCL TBitmap work without issue, but are often much slower than more specialised options. Next I tried the Lazarus OpenGL package. This unfortunately, did not compile. I will investigate this further and it will probably become the subject of my next post on the Raspberry Pi as the Pi has very good hardware support for OpenGL. I also tried Graphics32 and BGRABitmap which are both highly optimised software renderers for 32bit raster images. Unfortunately, both of them seem to have too many low-level Intel dependencies to port easily to an ARM based system.

To test the system’s graphics capability, I have written a small Mandelbrot set renderer, as shown below, (Source Code). This uses the simplest form of TCanvas.Pixels method for writing the image. This project takes around 2 seconds to generate the image shown below on my Intel i7 based system, but nearly 2 minutes on the Pi! I haven’t performed any detailed bench marking yet, but I will investigate this further.

There is a very interesting Mandelbrot set program called ‘triangle2.c’ supplied on the Raspbian SD image in the /opt/vc/src/hello_pi/ folder. This is written in C and uses OpenGL and fragment shaders to achieve very fast access to the Broadcom GPU hardware present on the Raspberry Pi. It produces stunning results, calculating a full screen Mandelbrot set with overlaid Julia set in real-time. So there should be plenty of scope to reduce my program’s 2 minute render times! It will be especially interesting to see how fast a pure ARM CPU program can operate before having to resort to direct GPU calls for further improvements.

One last comment about Free Pascal is to mention that it can be compiled as a cross compiler, meaning it can be an Intel based program that I can run on my desktop system, but generate ARM binary output for execution on the Pi. Whilst running a native Lazarus desktop on the Pi is perfectly feasible (in fact, it would be a very good platform for introducing people to visual programming in Pascal), large projects would certainly feel more comfortable when worked on with a much faster CPU, especially whilst compiling larger projects and I will definitely be looking into this over the coming weeks.

Lazarus 1.0 Released!

After a 14 year development process the Lazarus project has finally released version 1.0 of this excellent Object Pascal IDE. The length of this extraordinary release cycle is not due to slow progress, but rather a reflection of the very demanding quality standards the core team have held to, refusing to release until they have fixed all bugs classed as required for system stability (many commercial projects could learn a thing or two from this!). Even before this ‘official’ release I have preferred this as my development platform of choice for many years now. If you have used Delphi, you will immediately be at home with this development system. More details are available from the official announcement.

WoodlandCalc Updated

The Woodland Potential Calculator project has just been updated to include the first draft copy of the national dataset:

http://sourceforge.net/projects/woodlandcalc/

There are still a number of areas that have not been individually assessed and some of the NCA descriptions are still only draft place holders. None-the-less, this is a first chance to see the application running with a realistic set of national data. Further updates will follow as the project being lead by the Forestry Commission and Natural England to assess the scope of the potential for increasing the extent of tree cover progresses.

VPR added to Graphics32

Two of my favourite graphics libraries have been merged together. The Vector Polygon Rasteriser (VPR), a very fast polygon scanline converter, has been merged into Graphics32, the software rendering library I used for the display rendering in the Woodland Calculator project.

The old polygon rendering code has been completely replaced and the results are very interesting. The rendering performance is slightly reduced compared with the original fixed point base code, but it is faster than the old floating point, high quality code; whilst managing to produce equally impressive results. In fact, it can perform some tasks, such as rendering thick outlines, much more easily than in the prior version.


Old Renderer

New Renderer

If you would like to see the difference for yourself, you can download the new WoodlandCalc.exe from SourceForge and compare it with the previous one also on SourceForge.

 

 

 

A dash of PHP

It turns out that it takes longer to write this post, than it does to add the PHP to list the recent WordPress posts, thanks to a few simple function calls wrapped in a loop to fetch the posts:

<div>
  <h3>Latest News</h3>
  <ul>
    <?php require('./blog/wp-blog-header.php'); ?>
    <?php query_posts('showposts=8'); ?>
    <?php while (have_posts()): the_post(); ?>
      <li>
        <span><?php the_date('j/n'); ?></span>
        <a href="<?php the_permalink(); ?>">
          <?php the_title();?>
        </a>
      </li>
    <?php endwhile; ?>
  </ul>
</div>

This is a vanilla WordPress loop construct with a few tweaks to suit my use here. The ‘showposts=8’ parameter limits the number of posts to be listed to 8 and the more cryptic ‘j/n’ parameter for query_posts, formats the date as day/month without leading zeros.

WordPress Progress

It is nearly a month of evenings and weekends later. As predicted, the real effort was in integrating and synchronising my static HTML pages with the WordPress PHP content. However, to be fair, it wasn’t all one way.  I have changed some of the original website to match the TwentyEleven theme used by the blog as I preferred some of its styling over my original design!

I have to say, I like the clean use WordPress makes of cascading style sheets, which enabled a lot of cosmetic customisation without having to edit the PHP code directly. My last task is to replace the old news links on the index page with dynamic content from this blog. Hopefully there is a tidy way to do this!

WordPress up and running!

I have been meaning to add a blog to the Michell Computing website for some time and now I have finally got round to trying out the WordPress install. It actually was surprisingly easy to add it to an existing website. The install wasn’t 5 minutes, more like 15, but very smooth none-the-less, so well done WordPress.

I suspect the real effort will have to go into harmonising the style with the main web site.  I will post progress here and then be able to get on with the real business of developing GIS database systems in FreePascal and PostGIS!