This article could also be titled “How not to implement a national open data policy for massive geospatial data sets” or “Forget single-photon LiDAR, England already has single-quantum LiDAR” … (-:
But here comes the shocker and I would to make this a learning experience for those planning similar download portals. Again, the horizontal resolutions of the DTM and DSM rasters is 50 cm, 1 m, and 2 m. But what vertical resolution was chosen? I can still not quite believe it. It is more than micrometer, more than nanometers, and even more than picometers. I had to look up the name. The vertical resolution ranges from femtometers to attometers. This means that the ASCII numbers that specify the elevation for each grid cell are written down with 15 to 17 digits after the decimal point. Here an overview of units and the corresponding number of digits after the decimal point:
0 - meters: 1.0 1 - decimeters: 0.1 2 - centimeters: 0.01 3 - millimeters: 0.001 6 - micrometers: 0.000001 9 - nanometers: 0.000000001 12 - picometers: 0.000000000001 15 - femtometers: 0.000000000000001 18 - attometers: 0.000000000000000001
No seriously. This is a gigantic waste of network bandwidth, storage, and – more importantly – people’s time. Please fix this as soon as possible. Here an example: I downloaded LIDAR-DSM-1M-SP37.zip (237.96 MB compressed) and a quick look at one DSM after unzipping the 100 tiles (1891.13 MB uncompressed) was reason enough for this article:
D:\LAStools\bin>more LIDAR-DSM-1M-SP37\sp3070_DSM_1m.asc ncols 1000 nrows 1000 xllcorner 430000.000000000000 yllcorner 270000.000000000000 cellsize 1.000000000000 NODATA_value -9999 79.9499969482421875 80.23999786376953125 80.95999908447265625 80.9199981689453125 80.90000152587890625 81.44000244140625 80.3300018310546875 79.68000030517578125 79.76000213623046875 79.69000244140625 79.56999969482421875 [...]
If you look at these numbers more carefully you see that they really only ought to have centimeter resolution. I quickly changed the resolution to centimeter with a run of lasgrid on 4 cores:
D:\LAStools\bin>lasgrid -i LIDAR-DSM-1M-SP37\*.asc ^ -step 1 -use_bb ^ -odir LIDAR-DSM-1M-SP37-NO-FLUFF -oasc ^ -cores 4
The result is a DSM that is identical for all practical purposes … just compare the first ten elevations below with those ones above.
D:\LAStools\bin>more LIDAR-DSM-1M-SP37-NO-FLUFF\sp3070_DSM_1m.asc ncols 1000 nrows 1000 xllcorner 430000.000000 yllcorner 270000.000000 cellsize 1.000000 NODATA_value -9999.0 79.95 80.24 80.96 80.92 80.90 81.44 80.33 79.68 79.76 79.69 79.57 [...]
The resulting 100 *.asc tiles use only 580.45 MB uncompressed on disk: an instant storage saving of nearly 70 percent over those tiles with the insanely high resolution. After compressing them back into a single zipped archive I get a compressed file of size 161.99 MB – still a whopping 32 percent less than the zipped archive that I had originally downloaded.
Environment Agency, please lower the vertical resolution of all your DSM and DTM rasters to centimeters. This will directly translate into enourmous storage and bandwidth savings for you over the coming years with each download being around 30 percent smaller and faster. It will also allow your users to work more efficient with the rasters as decompressing and parsing the files will be quicker. In the future I will happily work with you to pick the perfect format for distributing your soon-to-be-open raw LiDAR points and with all the money you will safe for the storage and tranmission of the rasters you could easily become the third Gold Sponsor of the LASzip LiDAR compressor … (-;