This article could also be titled „How not to implement a national open data policy for massive geospatial data sets“ or „Forget single-photon LiDAR, England already has single-quantum LiDAR“ … (-:
But here comes the shocker and I would to make this a learning experience for those planning similar download portals. Again, the horizontal resolutions of the DTM and DSM rasters is 50 cm, 1 m, and 2 m. But what vertical resolution was chosen? I can still not quite believe it. It is more than micrometer, more than nanometers, and even more than picometers. I had to look up the name. The vertical resolution ranges from femtometers to attometers. This means that the ASCII numbers that specify the elevation for each grid cell are written down with 15 to 17 digits after the decimal point. Here an overview of units and the corresponding number of digits after the decimal point:
0 - meters: 1.0 1 - decimeters: 0.1 2 - centimeters: 0.01 3 - millimeters: 0.001 6 - micrometers: 0.000001 9 - nanometers: 0.000000001 12 - picometers: 0.000000000001 15 - femtometers: 0.000000000000001 18 - attometers: 0.000000000000000001
No seriously. This is a gigantic waste of network bandwidth, storage, and – more importantly – people’s time. Please fix this as soon as possible. Here an example: I downloaded LIDAR-DSM-1M-SP37.zip (237.96 MB compressed) and a quick look at one DSM after unzipping the 100 tiles (1891.13 MB uncompressed) was reason enough for this article:
D:\LAStools\bin>more LIDAR-DSM-1M-SP37\sp3070_DSM_1m.asc ncols 1000 nrows 1000 xllcorner 430000.000000000000 yllcorner 270000.000000000000 cellsize 1.000000000000 NODATA_value -9999 79.9499969482421875 80.23999786376953125 80.95999908447265625 80.9199981689453125 80.90000152587890625 81.44000244140625 80.3300018310546875 79.68000030517578125 79.76000213623046875 79.69000244140625 79.56999969482421875 [...]
If you look at these numbers more carefully you see that they really only ought to have centimeter resolution. I quickly changed the resolution to centimeter with a run of lasgrid on 4 cores:
D:\LAStools\bin>lasgrid -i LIDAR-DSM-1M-SP37\*.asc ^ -step 1 -use_bb ^ -odir LIDAR-DSM-1M-SP37-NO-FLUFF -oasc ^ -cores 4
The result is a DSM that is identical for all practical purposes … just compare the first ten elevations below with those ones above.
D:\LAStools\bin>more LIDAR-DSM-1M-SP37-NO-FLUFF\sp3070_DSM_1m.asc ncols 1000 nrows 1000 xllcorner 430000.000000 yllcorner 270000.000000 cellsize 1.000000 NODATA_value -9999.0 79.95 80.24 80.96 80.92 80.90 81.44 80.33 79.68 79.76 79.69 79.57 [...]
The resulting 100 *.asc tiles use only 580.45 MB uncompressed on disk: an instant storage saving of nearly 70 percent over those tiles with the insanely high resolution. After compressing them back into a single zipped archive I get a compressed file of size 161.99 MB – still a whopping 32 percent less than the zipped archive that I had originally downloaded.
Environment Agency, please lower the vertical resolution of all your DSM and DTM rasters to centimeters. This will directly translate into enourmous storage and bandwidth savings for you over the coming years with each download being around 30 percent smaller and faster. It will also allow your users to work more efficient with the rasters as decompressing and parsing the files will be quicker. In the future I will happily work with you to pick the perfect format for distributing your soon-to-be-open raw LiDAR points and with all the money you will safe for the storage and tranmission of the rasters you could easily become the third Gold Sponsor of the LASzip LiDAR compressor … (-;
This the most precise DSM and DTM ever!.
This is just typical of using software or database double precision floats without thinking about the consequences or relevance to the real world. Something we spent a lot of time talking about at University – any decent engineering, computing or science course should cover this in detail.
What kind of disclaimer do they have on the website?
I just downloaded a ZIP yesterday. (Struggling with LAStools – but thats by the way – just a novice on a steep learning curve).
The file I am ‚experimenting‘ with has the following data (truncated), which appears to be in centimetre heights.
ncols 2000
nrows 2000
xllcorner 369500
yllcorner 302500
cellsize 0.25
NODATA_value -9999
101.27 101.18 101.09 101 101 100.96 100.94 100.92 100.9 100.86 100.86 100.87 100.87 100.87 100.84 100.7 100.63 100.55 100.41 100.32 100.16 99.91 99.74 99.62 99.61 99.61 99.52 99.52 99.46 99.4 99.41 99.54 99.58 99.62 99.63 99.56 99.49 99.47 99.57 99.48
Can you provide the link? That 0.25 cm raster looks perfect. Yes, it has centimeter resolution. Not so much any of the 50cm, 1m, and 2m rasters we had looked at the day after the release. Those had femtometer to attometer resolution … (-:
d:\LAStools\bin>more sp2970_DSM_50cm.ascncols 2000
nrows 2000
xllcorner 429000.000000000000
yllcorner 270000.000000000000
cellsize 0.500000000000
NODATA_value -9999
-9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 -9999 90.09999847412109375 90.6399993896484375 91.19000244140625 91.87000274658203125 92.43000030517578125 92.95999908447265625 92.720001220703125 92.12000274658203125 91.48999786376953125 90.910003662109375 90.31999969482421875 88.1699981689453125 88.3300018310546875 84.3300018310546875 85.20999908447265625 85.589996337890625 87.3899993896484375 87.12000274658203125 86.220001220703125 88.23999786376953125 90.45999908447265625 90.970001220703125 91.01999664306640625 90.98999786376953125 91.410003662109375 91.56999969482421875 92.0800018310546875 92.18000030517578125 92.279998779296875 91.970001220703125 91.68000030517578125 90.8899993896484375 91.3000030517578125 91.3600006103515625 90.87999725341796875 90.70999908447265625 91.3600006103515625 92.15000152587890625 90.339996337890625 88.48000335693359375 89.660003662109375 89.5500030517578125 91.0800018310546875 91.45999908447265625 96.05999755859375 93.19000244140625 87.59999847412109375 89.089996337890625 91.4199981689453125 90.160003662109375 86.51000213623046875 88.19000244140625 84.779998779296875 84.54000091552734375 84.59999847412109375 84.720001220703125 85.1399993896484375 85.73000335693359375 84.51000213623046875 84.43000030517578125 84.37000274658203125 84.37999725341796875 84.8000030517578125 85.410003662109375 85.34999847412109375 85.3600006103515625 85.37999725341796875 85.40000152587890625 85.4499969482421875 85.18000030517578125 84.45999908447265625 84.59999847412109375 84.5800018310546875 85.1100006103515625 85.98999786376953125 87.84999847412109375 87.94000244140625 88.029998779296875 87.5500030517578125 86.3000030517578125 84.8300018310546875 84.720001220703125 84.8300018310546875 84.8000030517578125 84.9499969482421875 84.8300018310546875 84.8000030517578125 [...]
I went to http://environment.data.gov.uk/ds/survey#/ searched for ‘Coalport, Telford’ And then selected 25cm DTM. The downloaded file was LIDAR-DTM-25CM-SJ60.zip
http://environment.data.gov.uk/ds/rest/product/download/11188
It’s a large file, 655MB, and took absolutely ages to download. The zip has 32 files. I unzipped them and then selected ‘SJ6902ne_DSM_25CM.asc’ to work with as it is quite small. ‘sj6704_DTM_25cm.asc’ is one of the included files in that zip and it has the same 1cm resolution, I haven’t checked the others.
It’s a femtometer-parade and an attometer-celebration all the way. (-: I have just downloaded the file you talk about via the link you sent:
http://environment.data.gov.uk/ds/rest/product/download/11188
d:\LAStools\bin\LIDAR-DTM-25CM-SJ60> dir LIDAR-DTM-25CM-SJ60.zip09/04/2015 05:02 PM 686,843,311 LIDAR-DTM-25CM-SJ60.zip
d:\LAStools\bin\LIDAR-DTM-25CM-SJ60>..\unzip.exe LIDAR-DTM-25CM-SJ60.zipArchive: LIDAR-DTM-25CM-SJ60.zip
inflating: sj6005_DTM_25cm.asc
inflating: sj6104_DTM_25cm.asc
inflating: sj6105_DTM_25cm.asc
inflating: sj6204_DTM_25cm.asc
inflating: sj6205_DTM_25cm.asc
inflating: sj6303_DTM_25cm.asc
inflating: sj6304_DTM_25cm.asc
inflating: sj6305_DTM_25cm.asc
inflating: sj6306_DTM_25cm.asc
inflating: sj6403_DTM_25cm.asc
inflating: sj6404_DTM_25cm.asc
inflating: sj6405_DTM_25cm.asc
inflating: sj6502_DTM_25cm.asc
inflating: sj6503_DTM_25cm.asc
inflating: sj6504_DTM_25cm.asc
inflating: sj6505_DTM_25cm.asc
inflating: sj6602_DTM_25cm.asc
inflating: sj6603_DTM_25cm.asc
inflating: sj6604_DTM_25cm.asc
inflating: sj6605_DTM_25cm.asc
inflating: sj6701_DTM_25cm.asc
inflating: sj6702_DTM_25cm.asc
inflating: sj6703_DTM_25cm.asc
inflating: sj6704_DTM_25cm.asc
inflating: sj6801_DTM_25cm.asc
inflating: sj6802_DTM_25cm.asc
inflating: sj6803_DTM_25cm.asc
inflating: sj6804_DTM_25cm.asc
inflating: sj6900_DTM_25cm.asc
inflating: sj6901_DTM_25cm.asc
inflating: sj6902_DTM_25cm.asc
inflating: sj6903_DTM_25cm.asc
d:\LAStools\bin\LIDAR-DTM-25CM-SJ60>more sj6902_DTM_25cm.ascncols 4000
nrows 4000
xllcorner 369000.000000000000
yllcorner 302000.250000000000
cellsize 0.250000000000
NODATA_value -9999
53.90000152587890625 53.84999847412109375 53.79000091552734375 53.75 53.68000030517578125 53.59999847412109375 53.549999237060546875 [...]
Yes – I checked you are quite correct – what a !! up.
I have a couple of files that are OK but these were supplied before they went ‚free‘.
Thanks for the feedback on the data release – really useful. As with all the openDEFRA data releases, we’re first releasing what we use internally as alpha quality data, to get exactly this kind of feedback on how we could improve the releases. Keep the feedback coming & let’s get more into the mode of working together, so we can end up with high quality open data that creates new opportunities. Thanks, Simon Redding, openDEFRA programme.
Martin, thank you for the feedback. When we announced the release of LIDAR data back in June 2015, we set ourselves an ambitious target of making the data available by September. The complexity and size of the data meant that we needed to develop a delivery service that was robust, easy to use and able to deliver more data to a wider range of people than our current platform could manage.
The issue with the double precision data values being used in the ASCII files, which you have raised in your article, was noted three weeks ago. It would however have been impractical to fix it at the time and still hit our deadline of releasing the data on 1 September. In the coming weeks the zip file of many 1x1km ASCII tiles will be replaced by a single 10x10km zipped ASCII tile, and the Z values in the new tiles will be to 3 decimal places (no more femtometers and attometers).
Over the next few months there will be many more open datasets available from the site, including survey specific DSM and DTM tiles dating back to 1996. From these data change analysis will be possible, as there are locations with many repeat surveys over the years. LIDAR point data in compressed format, metadata layers, and several other derivative layers (our favourite being solar radiance layers, which could be used to help locating solar panels!), will also be available. These are a small part of the many hundreds of datasets being released as open data by the Environment Agency over the next year, and in turn a small part of the 8,000 datasets being released by the Department of the Environment, Farming and Rural Affairs (DEFRA).
We would be more than happy to discuss the LIDAR open data release with you and your readers, and to hear any suggestions on how to improve the service. You can contact us directly on alastair.duncan@environment-agency.gov.uk. We are very keen to make our fantastic LIDAR and other datasets as useful and usable as possible, and we hope that by removing the cost barriers to the data will make it possible for businesses and local communities to develop new innovative tools and techniques to further benefit the environment.
Alastair Duncan
Geomatics Technical Specialist
Environment Agency