NRW Open LiDAR: Download, Compression, Viewing

UPDATE: (March 6th): Second part merging Bonn into proper LAS files

This is the first part of a series on how to process the newly released open LiDAR data for the entire state of North Rhine-Westphalia that was announced a few days ago. Again, kudos to OpenNRW for being the most progressive open data state in Germany. You can follow this tutorial after downloading the latest version of LAStools as well as a pair of DGM and DOM files for your area of interest from these two download pages.

We have downloaded the pair of DGM and DOM files for the Federal City of Bonn. Bonn is the former capital of Germany and was host to the FOSS4G 2016 conference. As both files are larger than 10 GB, we use the wget command line tool with option ‘-c’ that will restart where it left off in case the transmission gets interrupted.

The DGM file and the DOM file are zipped archives that contain the points in 1km by 1km tiles stored as x, y, z coordinates in ETRS89 / UTM 32 projection as simple ASCII text with centimeter resolution (i.e. two decimal digits).

>> more dgm1l-lpb_32360_5613_1_nw.xyz
360000.00 5613026.69 164.35
360000.00 5613057.67 164.20
360000.00 5613097.19 164.22
360000.00 5613117.89 164.08
360000.00 5613145.35 164.03
[...]

There is more than one tile for each square kilometer as the LiDAR points have been split into different files based on their classification and their return type. Furthermore there are also synthetic points that were used by the land survey department to replace certain LiDAR points in order to generate higher quality DTM and DSM raster products.

The zipped DGM archive is 10.5 GB in size and contains 956 *.xyz files totaling 43.5 GB after decompression. The zipped DOM archive is 11.5 GB in size and contains 244 *.xyz files totaling 47.8 GB. Repeatedly loading these 90 GB of text data and parsing these human-readable x, y, and z coordinates is inefficient with common LiDAR software. In the first step we convert the textual *.xyz files into binary *.laz files that can be stored, read and copied more efficiently. We do this with the open source LASzip compressor that is distributed with LAStools using these two command line calls:

laszip -i dgm1l_05314000_Bonn_EPSG5555_XYZ\*.xyz ^
       -epsg 25832 -vertical_dhhn92 ^
       -olaz ^
       -cores 2
laszip -i dom1l_05314000_Bonn_EPSG5555_XYZ\*.xyz ^
       -epsg 25832 -vertical_dhhn92 ^
       -olaz ^
       -cores 2

The point coordinates are is in EPSG 5555, which is a compound datum of horizontal EPSG 25832 aka ETRS89 / UTM zone 32N and vertical EPSG 5783 aka the “Deutsches Haupthoehennetz 1992” or DHHN92. We add this information to each *.laz file during the LASzip compression process with the command line options ‘-epsg 25832’ and ‘-vertical_dhhn92’.

LASzip reduces the file size by a factor of 10. The 956 *.laz DGM files compress down to 4.3 GB from 43.5 GB for the original *.xyz files and the 244 *.laz DOM files compress down to 4.8 GB from 47.8 GB. From here on out we continue to work with the 9 GB of slim *.laz files. But before we delete the 90 GB of bulky *.xyz files we make sure that there are no file corruptions (e.g. disk full, truncated files, interrupted processes, bit flips, …) in the *.laz files.

laszip -i dgm1l_05314000_Bonn_EPSG5555_XYZ\*.laz -check
laszip -i dom1l_05314000_Bonn_EPSG5555_XYZ\*.laz -check

One advantage of having the LiDAR in an industry standard such as the LAS format (or its lossless compressed twin, the LAZ format) is that the header of the file stores the number of points per file, the bounding box, as well as the projection information that we have added. This allows us to very quickly load an overview for example, into lasview.

lasview -i dgm1l_05314000_Bonn_EPSG5555_XYZ\*.laz -GUI
The bounding boxes of the DGM files quickly display a preview of the data in the GUI when the files are in LAS or LAZ format.

The bounding boxes of the DGM files quickly give us an overview in the GUI when the files are in LAS or LAZ format.

Now we want to find a particular site in Bonn such as the World Conference Center Bonn where FOSS4G 2016 was held. Which tile is it in? We need some geospatial context to find it, for example, by creating an overview in form of KML files that we can load into Google Earth. We use the files from the DOM folder with “fp” in the name as points on buildings are mostly “first returns”. See what our previous blog post writes about the different file names or look at the second part of this series. We can create the KML files with lasboundary either via the GUI or in the command line.

lasboundary -i dom1l_05314000_Bonn_EPSG5555_XYZ\dom1l-fp*.laz ^
            -gui
Only the "fp" tiles from the DOM folder loaded the GUI into lasboundary.

Only the “fp” tiles from the DOM folder loaded the GUI into lasboundary.

lasboundary -i dom1l_05314000_Bonn_EPSG5555_XYZ\dom1l-fp*.laz ^
            -use_bb -labels -okml

We zoom in and find the World Conference Center Bonn and load the identified tile into lasview. Well, we did not expect this to happen, but what we see below will make this series of tutorials even more worthwhile. There is a lot of “high noise” in the particular tile we picked. We should have noticed the unusually high z range of 406.42 meters in the Google Earth pop-up. Is this high electromagnetic radiation interfering with the sensors? There are a number of high-tech government buildings with all kind of antennas nearby (such as the United Nations Bonn Campus the mouse cursor points at).

Significant amounts of high noise are in the first returns of the DOM tile we picked.

Significant amounts of high noise are in the first returns of the DOM tile we picked.

But the intended area of interest was found. You can see the iconic “triangulated” roof of the building that is across from the World Conference Center Bonn.

The World Conference Center Bonn is across from the building with the "triangulated" roof.

The World Conference Center Bonn is across from the building with the “triangulated” roof.

Please don’t think it is the responsibility of OpenNRW to remove the noise and provide cleaner data. The land survey has already processed this data into whatever products they needed and that is where their job ended. Any additional services – other than sharing the raw data – are not in their job description. We’ll take care of that … (-:

Acknowledgement: The LiDAR data of OpenNRW comes with a very permissible license. It is called “Datenlizenz Deutschland – Namensnennung – Version 2.0” or “dl-de/by-2-0” and allows data and derivative sharing as well as commercial use. It only requires us to name the source. We need to cite the “Land NRW (2017)” with the year of the download in brackets and specify the Universal Resource Identification (URI) for both the DOM and the DGM. Done. So easy. Thank you, OpenNRW … (-:

New ‘laspublish’ creates Web Portals for 3D Viewing and Downloading of LiDAR

PRESS RELEASE (for immediate release)
February 18, 2016
rapidlasso GmbH, Gilching, Germany

Just in time for ILMF 2016, the makers of the open LASzip LiDAR compressor annouce the latest addition to their LiDAR processing software LAStools. The new ‘laspublish‘ from rapidlasso GmbH creates stand-alone Web Portals for interactive 3D viewing of LiDAR points and for selective downloading of LAZ or LAS files. The new tool is based on the cutting-edge streaming point cloud viewing technology of Potree that optimizes large LiDAR point clouds for streaming via the Web such that anyone can visualize, explore, and (optionally) download them with any modern browser.

3D viewer and download portal created with 'laspublish'

3D viewer and download portal created with ‘laspublish’

The interactive 3D viewer streams on demand only the relevant parts of the point clouds. It not only visualizes the LiDAR in many useful and intuitive ways, but is also equipped with measurement tools to calculate distances or areas and profile or clipping tools for close-up inspections. With its integrated LASzip compression, options to color LiDAR by classification, return type, intensity, and point source ID, stunning visuals via Eye Dome Lighting (EDL), and the optional 2D download map, the new ‘laspublish‘ empowers professional and novice users alike to create stand-alone LiDAR Webportals with just a few button clicks.

a few clicks and 'laspublish' creates a professional LiDAR portal

a few clicks and ‘laspublish’ creates a professional LiDAR portal

The new ‘laspublish‘ is now an integral part of the LAStools software and is bundled together with all necessary components of the Potree software. It provides an instant and cost-effective solution for generating a set of Web pages that realize a self-contained LiDAR portal offering interactive online visualization and exploration as well as easy and intuitive distribution of large LiDAR data sets via the optional download map.

About rapidlasso GmbH:
Technology powerhouse rapidlasso GmbH specializes in efficient LiDAR processing tools that are widely known for their high productivity. They combine robust algorithms with efficient I/O and clever memory management to achieve high throughput for data sets containing billions of points. The company’s flagship product – the LAStools software suite – has deep market penetration and is heavily used in industry, government agencies, research labs, and educational institutions. Visit http://rapidlasso.com for more information. As the only Diamond sponsor, rapidlasso GmbH has been the main financial supporter of the open source Potree package by Markus Schütz over the past two years.

About Potree:
Potree is a WebGL based viewer for large point clouds. The project evolved as a Web based viewer from the Scanopy desktop point cloud renderer by TU Wien, Institute of Computer Graphics and Algorithms. It will continue to be free and open source with a FreeBSD license to enable anyone to view, analyze and publicly share their large datasets. Visit http://potree.org for more information.

The dArc Force Awakens: ESRI escalates LiDAR format war

The empire has not changed their evil ways, despite an encouraging email from ESRI’s founder and president Jack Dangermond in response to the Open Letter by the OSGeo that was delivered to ESRI, OGC, and the ASPRS. Facing an incredible backlash by the LiDAR community over the release of their “LAZ clone” there was a new hope that unnecessary format fragmention could be avoided by working together within the Point Cloud Domain Working Group of the OGC. In fact only one thing happened: ESRI went silent on the controversy. They temporarily stopped promoting their “LAZ clone” and focused on locking in more content.

dArc_force_awakens

The message of the rebellion has been consistent and clear like in these two videos from the TC meeting of the OGC in Nottingham and the ASPRS side bar in Reno: a roadmap forward to avoid format fragmentation by exploiting the “natural break” in the format due to LAS 1.4. But there was zero technical contribution from ESRI during the past three PC-DWG meetings of the OGC. The slide sets that bored the audiences in Boulder and in Nottingham were not meant to contribute but merely stalled for time. Recently in Sydney ESRI was awefully quiet, knowing they were doing the exact opposite of what the OGC stands for. And now the empire strikes back.

laztozlas

There is a dArc force awakening that threatens the peace within the LiDAR community. ESRI has just released a new tool (see above) that enslaves point clouds by converting them from the open LAZ format to the near-identical but closed “LAZ clone” that they call “zLAS” or “Optimized LAS”. This comes just a few months after an entire nation‘s LiDAR was enslaved in this proprietary format. We have repeatedly warned about the ramifications of locking up Petabytes of LiDAR data in a closed format that is controlled by a single vendor.

ESRI is one of the largest GIS training organizations. By instructing LiDAR novices to “optimize” their LiDAR files and pushing LiDAR providers to switch from open LAS or open LAZ to closed zLAS, they effectively destroy the current success of our open formats. ESRI’s command of the GIS market can – little by little – turn their own proprietry format into the dominant way in which LiDAR point clouds are stored. Then we loose our open exchange formats. Hence, ESRI’s proprietary format threatens all that we have achieved with LAS (and LAZ) over the past years: compatible LiDAR data exchange and incredible LiDAR software interoperability.

ESRI is now escalating the LiDAR format wars. Join the rebellion, Jedis: download your lazer sabers and liberate some LiDAR.

This is not an anti-ESRI campaign. For the past three years we have been trying to resolve this situation. We have repeatedly reached out to ESRI to prevent format fragmentation. We have repeatedly offered to create a joint compressed format. We have plead, begged, and bargained for the sake of our LiDAR community and the sake of their ArcGIS user community not to promote a near-identical yet incompatible way for storing massive amounts of point cloud data.

Potree puts Big and Beautiful LiDAR in Your Browser

PRESS RELEASE (for immediate release)
September 14, 2015
rapidlasso GmbH, Gilching, Germany

Just in time for INTERGEO 2015, the Potree software was released in its latest 1.3 version. Potree is a WebGL based point cloud viewer for very large datasets. The Potree software allows to publish large LiDAR point clouds on the Web such that anyone can explore the data with nothing more but a modern browser. The interactive 3D viewer not only visualizes the LiDAR in many useful and intuitive ways but also comes with tools to perform various measurements. As its only Gold Sponsor, rapidlasso GmbH is the main supporter of this powerful open source package by Markus Schütz.

17.7 billion points around San Simeon,, CA courtesy of Open Topography

17.7 billion points from San Simeon, CA courtesy of Open Topography

The long-term sponsorship of rapidlasso GmbH has directly supported a number of useful features such as the integration of our award-winning LASzip compressor using the pure javascript version contributed by Hobu Inc, optimization for massive airborne LiDAR data, profile selection, tools for distance and area measurements, options to color by classification, return type, and point source ID, and a clipping tool. The particular features sponsored in the most recent 1.3 release of Potree are the incredible Eye Dome Lighting (EDL) and faster data conversion for large data sets. A number of interesting showcases (including the CA13 example shown here) are available on the Potree page.

In the near future the Potree software will be distributed together with the LAStools package to offer a one-click solution for generating Webportals that host and distribute large LiDAR data sets and offer interactive online visualization and exploration. Potree is open source software that is free for anyone to acquire and to deploy. Please remember that using open source software is not the same as supporting open source software. Given the positive experience that rapidlasso GmbH has had with Potree we can only encourage other geospatial companies to support with time or money those open source projects that help your business.

About rapidlasso GmbH:
Technology powerhouse rapidlasso GmbH specializes in efficient LiDAR processing tools that are widely known for their high productivity. They combine robust algorithms with efficient I/O and clever memory management to achieve high throughput for data sets containing billions of points. The company’s flagship product – the LAStools software suite – has deep market penetration and is heavily used in industry, government agencies, research labs, and educational institutions. Visit http://rapidlasso.com for more information.

About Potree:
Potree is a WebGL based viewer for large point clouds. The project evolved as a Web based viewer from the Scanopy desktop point cloud renderer by TU Wien, Institute of Computer Graphics and Algorithms. It will continue to be free and open source with a FreeBSD license to enable anyone to view, analyze and publicly share their large datasets. Visit http://potree.org for more information.

England Releases National LiDAR DEM with Insane (!) Vertical Resolution

This article could also be titled “How not to implement a national open data policy for massive geospatial data sets” or “Forget single-photon LiDAR, England already has single-quantum LiDAR” … (-:

You may have heard about the amazing open data release by the Environment Agency. So far LiDAR-derived DTM and DSM rasters have been released for 72% of the entire English territory at horizontal resolutions of 50 cm, 1 m, and 2 m. They can be downloaded here. The rasters are distributed as zipped archives of tiles in textual ASC format (*.asc). While easy to parse it would not be our first format of choice for such a large release as it loads slower than a comparable binary format like GeoTIFF or BIL … but so far so good.
Open data download portal for DSM and DTM rasters

Open data download portal for DSM and DTM rasters of England

But here comes the shocker and I would to make this a learning experience for those planning similar download portals. Again, the horizontal resolutions of the DTM and DSM rasters is 50 cm, 1 m, and 2 m. But what vertical resolution was chosen? I can still not quite believe it. It is more than micrometer, more than nanometers, and even more than picometers. I had to look up the name. The vertical resolution ranges from femtometers to attometers. This means that the ASCII numbers that specify the elevation for each grid cell are written down with 15 to 17 digits after the decimal point. Here an overview of units and the corresponding number of digits after the decimal point:

 0 - meters:      1.0
 1 - decimeters:  0.1
 2 - centimeters: 0.01
 3 - millimeters: 0.001
 6 - micrometers: 0.000001
 9 - nanometers:  0.000000001
12 - picometers:  0.000000000001
15 - femtometers: 0.000000000000001
18 - attometers:  0.000000000000000001
Wikipedia states that “The picometre’s length is of an order such that its application is almost entirely confined to particle physics, quantum physics, chemistry and acoustics. Atoms are between 62 and 520 pm in diameter, and the typical length of a carbon-carbon single bond is 154 pm.” and the “femtometer […] was so named in honour of physicist Enrico Fermi, as it is a typical length-scale of nuclear physics. […] For example, the charge radius of a proton is approximately 0.84–0.87 femtometres while the radius of a gold nucleus is approximately 8.45 femtometres.” There is no individual Wikipedia entry for attometers because it’s just too small for most practical use … except for specifying the elevations in the DSM and DTM rasters across England … (-; … this interactive animation gives you a sense of those scales.
a Helium atom has a diameter of about 62 picometers.

diameter of Helium atom =  62 picometers

No seriously. This is a gigantic waste of network bandwidth, storage, and – more importantly – people’s time. Please fix this as soon as possible. Here an example: I downloaded LIDAR-DSM-1M-SP37.zip (237.96 MB compressed) and a quick look at one DSM after unzipping the 100 tiles (1891.13 MB uncompressed) was reason enough for this article:

D:\LAStools\bin>more LIDAR-DSM-1M-SP37\sp3070_DSM_1m.asc
ncols        1000
nrows        1000
xllcorner    430000.000000000000
yllcorner    270000.000000000000
cellsize     1.000000000000
NODATA_value  -9999
 79.9499969482421875 80.23999786376953125 80.95999908447265625 80.9199981689453125 80.90000152587890625 81.44000244140625 80.3300018310546875 79.68000030517578125 79.76000213623046875 79.69000244140625 79.56999969482421875 [...]

If you look at these numbers more carefully you see that they really only ought to have centimeter resolution. I quickly changed the resolution to centimeter with a run of lasgrid on 4 cores:

D:\LAStools\bin>lasgrid -i LIDAR-DSM-1M-SP37\*.asc ^
                      -step 1 -use_bb ^
                      -odir LIDAR-DSM-1M-SP37-NO-FLUFF -oasc ^
                      -cores 4

The result is a DSM that is identical for all practical purposes … just compare the first ten elevations below with those ones above.

D:\LAStools\bin>more LIDAR-DSM-1M-SP37-NO-FLUFF\sp3070_DSM_1m.asc
ncols 1000
nrows 1000
xllcorner 430000.000000
yllcorner 270000.000000
cellsize 1.000000
NODATA_value -9999.0
79.95 80.24 80.96 80.92 80.90 81.44 80.33 79.68 79.76 79.69 79.57 [...]

The resulting 100 *.asc tiles use only 580.45 MB uncompressed on disk: an instant storage saving of nearly 70 percent over those tiles with the insanely high resolution. After compressing them back into a single zipped archive I get a compressed file of size 161.99 MB – still a whopping 32 percent less than the zipped archive that I had originally downloaded.

Environment Agency, please lower the vertical resolution of all your DSM and DTM rasters to centimeters. This will directly translate into enourmous storage and bandwidth savings for you over the coming years with each download being around 30 percent smaller and faster. It will also allow your users to work more efficient with the rasters as decompressing and parsing the files will be quicker. In the future I will happily work with you to pick the perfect format for distributing your soon-to-be-open raw LiDAR points and with all the money you will safe for the storage and tranmission of the rasters you could easily become the third Gold Sponsor of the LASzip LiDAR compressor … (-;

PS: Just curious … which software did you use to generate those insanely high vertical resolutions in the first place?

RIEGL Becomes LASzip Sponsor for LAS 1.4 Extension

PRESS RELEASE (for immediate release)
August 31, 2015
rapidlasso GmbH, Gilching, Germany

We are happy to announce that RIEGL Laser Measurement Systems, Austria has become a sponsor of the award-winning LASzip compressor. Their contribution at the Silver level will kick-off the actual development phase of the “native LAS 1.4 extension” that had been discussed with the LiDAR community over the past two years. This “native extension” for LAS 1.4 complements the existing “compatibility mode” for LAS 1.4 that was supported by Gold sponsor NOAA and Bronze sponsors Quantum Spatial and Trimble Geospatial. The original sponsor who initiated and financed the open sourcing of the LASzip compressor was USACE – the US Army Corps of Engineers (see http://laszip.org).

The existing “LAS 1.4 compatibility mode” in LASzip was created to provide immediate support for compressing the new LAS 1.4 point types by rewriting them as old point types and storing their new information as “Extra Bytes”. As an added side-benefit this has allowed legacy software without LAS 1.4 support to readily read these newer LAS files as most of the important fields of the new point types 6 to 10 can be mapped to fields of the older point types 1, 3, or 5.

In contrast, the new “native LAS 1.4 extension” of LASzip that is now sponsored in part by RIEGL will utilize the “natural break” in the format due to the new point types of LAS 1.4 to introduce entirely new features such as “selective decompression”, “rewritable classifications and flags”, “integrated spatial indexing”, … and other functionality that has been brain-stormed with the community since rapidlasso GmbH had issued the open “call for input” on native LASzip compression for LAS 1.4 in January 2014. We invite you to follow the progress or contribute to the development via the discussions in the “LAS room“.

silverLASzip_m60_512_275

About rapidlasso GmbH:
Technology powerhouse rapidlasso GmbH specializes in efficient LiDAR processing tools that are widely known for their high productivity. They combine robust algorithms with efficient I/O and clever memory management to achieve high throughput for data sets containing billions of points. The company’s flagship product – the LAStools software suite – has deep market penetration and is heavily used in industry, government agencies, research labs, and educational institutions. Visit http://rapidlasso.com for more information.

About RIEGL:
Austrian based RIEGL Laser Measurement Systems is a performance leader in research, development and production of terrestrial, industrial, mobile, bathymetric, airborne and UAS-based laser scanning systems. RIEGL’s innovative hard- and software provides powerful solutions for nearly all imaginable fields of application. Worldwide sales, training, support and services are delivered from RIEGL‘s Austrian headquarters and its offices in Vienna, Salzburg, and Styria, main offices in the USA, Japan, and in China, and by a worldwide network of representatives covering Europe, North and South America, Asia, Australia and Africa. Visit http://riegl.com for more information.

Trimble joins LASzip sponsors USACE, NOAA, and Quantum Spatial

PRESS RELEASE (for immediate release)
July 13, 2015
rapidlasso GmbH, Gilching, Germany

We are happy to announce that Trimble’s Geospatial Division has become a sponsor of the LASzip compressor. Their contribution as a Bronze sponsor will improve the existing “LAS 1.4 compatibility mode” of LASzip whose creation and maintenance is already being supported by Gold sponsor NOAA and Bronze sponsor Quantum Spatial. The original Gold sponsor of the open source LASzip compressor was USACE – the US Army Corps of Engineers (see http://laszip.org).

The “LAS 1.4 compatibility mode” was created to provide immediate support for compressing the new LAS 1.4 point types by rewriting them as old point types and storing their new fields as “Extra Bytes”. As an added benefit this allows older software (without LAS 1.4 support) to access the newpoint types of LAS 1.4 files that would otherwise be unreadable. All important fields of the new point types 6 to 10 (i.e. those fields that matter to older software) are mapped to the corresponding fields of the older known point types 1, 3, or 5.
bronze_m60_512_275The Bronze sponsorship of Trimble’s Geospatial Division will pay for on-going improvements in the LASzip DLL and – in particular – add support for writing the new LAS 1.4 points in a streaming manner followed by an automated update of the bounding box and the point counters in the header.

About rapidlasso GmbH:
Technology powerhouse rapidlasso GmbH specializes in efficient LiDAR processing tools that are widely known for their high productivity. They combine robust algorithms with efficient I/O and clever memory management to achieve high throughput for data sets containing billions of points. The company’s flagship product – the LAStools software suite – has deep market penetration and is heavily used in industry, government agencies, research labs, and educational institutions. Visit http://rapidlasso.com for more information.

About Trimble’s Geospatial Division:
Trimble’s Geospatial Division provides solutions that facilitate high-quality, productive workflows and information exchange, driving value for a global and diverse customer base of surveyors, engineering and GIS service companies, governments, utilities and transportation authorities. Trimble’s innovative technologies include integrated sensors, field applications, real-time communications and office software for processing, modeling and data analytics. Using Trimble solutions, organizations can capture the most accurate spatial data and transform it into intelligence to deliver increased productivity and improved decision-making. Whether enabling more efficient use of natural resources or enhancing the performance and lifecycle of civil infrastructure, timely and reliable geospatial information is at the core of Trimble’s solutions to transform the way work is done. Visit http://trimble.com/Industries/Geospatial/ for more information.

About Trimble:
Trimble applies technology to make field and mobile workers in businesses and government significantly more productive. Solutions are focused on applications requiring position or location – including surveying, construction, agriculture, fleet and asset management, public safety and mapping. In addition to utilizing positioning technologies, such as GPS, lasers and optics, Trimble solutions may include software content specific to the needs of the user. Wireless technologies are utilized to deliver the solution to the user and to ensure a tight coupling of the field and the back office. Founded in 1978, Trimble is headquartered in Sunnyvale, California. Visit http://trimble.com for more information.