First Open LiDAR in Germany

UPDATE: (January 6th): Our new tutorial „downloading Bonn in LiDAR„.
UPDATE: (January 9th): Now a second state went open LiDAR as well.
UPDATE: (March 6th): New tutorial „merging Bonn into proper LAS files„

Kudos to OpenNRW for offering online download links for hundreds of Gigabytes of open LiDAR for the entire state of North Rhine-Westphalia (Nordrhein-Westfalen) as announced a few months ago:

• download raw LiDAR points describing the DTM (DGM)
• download raw LiDAR points describing the DSM (DOM)

More and more countries, states, and municipalities are deciding to make their LiDAR archives accessible to the general public. Some are doing entirely for free with instant online download and a generous open license that allows data sharing and commercial use. Others still charge a „small administrative fee“ and require filling our actual paperwork with real signatures in ink and postal mailing of hard drives that can easily take half a year to complete. Some licences are also stricter in terms of data sharing and commercial use.

And then there was Germany where all the LiDAR data has traditionally been locked up in some cave by the 16 state survey departments and was sold for more than just a fee. Financial reasons would usually prohibit residents in Germany from making, for example, an elevation profile for their favorite mountain bike trail for hobby purposes. Or starting a small side business that (for 5 Euro a sheet) sells „Gassi Maps“ with elevation-optimized dog walking paths of low incline that go past suitable potty spots and dog-friendly coffee shops.

The reason that many national and state mapping agencies have opened their LiDAR holdings for free and unencumbered access are manifold. A previous blog post had looked at the situation in England whose Environment Agency also used to sell LiDAR data and derivatives. The common argument that government agencies have been using to justify selling data (paid for with taxes) to those very same tax payers was that this would be used to finance future surveys.

It was the „freedom of information“ request by Louise Huby on November 21st, 2014 that exposed this as a flawed argument. The total amount of revenue generated for all LiDAR and derivatives sales by the Environment Agency was just around £323,000 per year between 2007 and 2014. This figure was dwarfed by its annual operational budget of £1,025,000,000 in 2007/08. The revenue from LiDAR sales was merely 0.03 percent of the agencies’s budget. That can maybe pay for free coffee and tea in the office, but not for future airborne LiDAR flights. The reaction was swift. In September 2015 the Environment Agency made all their DTM and DSM rasters down to 0.25 meter resolution available online for open access and in March 2016 added the raw point clouds for download in LAZ format with a very permissible license. It has since been an incredible success story and the Environment Agency has been propelled into the role of a „champion for open data“ as sketched in my ACRS 2016 keynote talk that is available on video here.

Frustrated with the situation in Germany and inspired by the change in geospatial data policy in England, we have been putting in similar „Frag den Staat“ freedom of information requests with 11 of the 16 German state mapping agencies, asking about how much sales revenue was generated annually from their LiDAR and derivative sales. These four states denied the request:

• Bavaria (Bayern)
• Hesse (Hessen)
• Saxony (Sachsen)
• Baden-Württemberg (Baden-Württemberg)

We never heard back from Lower Saxony (Niedersachsen) and Thuringia (Thüringen) wanted more fees than we were willing to spend on this, but our information requests were answered by five states that are here listed with the average amount of reported revenue per year in EUR:

• Rhineland-Palatinate (Rheinland-Pfalz), 58,012 / year (2005 – 2015)
• Saxony-Anhalt (Sachsen-Anhalt), 132,350 / year (2011 – 2015)
• Saarland (Saarland), 4,448 / year (2008 – 11/2015)
• Schleswig-Holstein (Schleswig-Holstein), 18,458 / year (2012 – 2014)
• State of Bremen (Land Bremen), 597 / year (10/2012 – 09/2015)

LiDAR acquisitions are expensive and while it would be interesting to also find out how much each state has actually spent on airborne surveys over the past years (another „Frag den Staat“ request anyone?), it is obvious that the reported revenues are just a tiny fraction of those costs. Exact details of the reported revenue per year can be accessed via the links to the information requests above. The cost table of each answer letter is copied below.

However, it’s not all bad news in Germany. Some of you may have seen my happy announcement of the OpenNRW initiative that come January 2017 this would also include the raw LiDAR points. And did it happen? Yes it did! Although the raw LiDAR points are maybe a little tricky to find, they are available for free download and they come with a very permissible license.

The license is called „Datenlizenz Deutschland – Namensnennung – Version 2.0“ or „dl-de/by-2-0“ and allows data and derivative sharing as well as commercial use. It merely requires you to properly name the source. For the LiDAR you need to list the „Land NRW (2017)“ with the year of the download in brackets as the source and specify the data set that was used via the respective Universal Resource Identification (URI) for the DOM and/or the DGM.

Follow this link to get to the open data download portal. Now type in „punktwolke“ (German for „point cloud“) into the search field and on this January 3rd 2017 that gives me 2 „Ergebnisse“ (German for „results“). The LiDAR point cloud representation is a bit unusual by international standards. One link is for the DGM (German for DTM) and the other for the DOM (German for DSM). Both links are eventually leading you to unstructured LiDAR point clouds that are describing these surfaces. But it’s still a little tricky to find them. First click on the „ATOM“ links that get you to XML description with meta information and a lot of links for the DTM and the DSM. Somewhere hidden in there you find the actual download links for hundreds of Gigabytes of LiDAR for the entire state of North Rhine-Westphalia (Nordrhein-Westfalen):

• download raw LiDAR points describing the DTM (DGM)
• download raw LiDAR points describing the DSM (DOM)

We download the two smallest zipped files DGM and DOM for the municipality of Wickede (Ruhr) to have a look at the data. The point cloud is in EPSG 5555 which is a compound datum of horizontal EPSG 25832 aka ETRS89 / UTM zone 32N and vertical EPSG 5783 aka the „Deutches Haupthohennetz 1992“.

The DGM zip file has a total of 212 *.xyz files that list the x, y, and z coordinate for each point in ASCII format. We first compress them and add the EPSG 25832 code with laszip. The compressed LAZ files are less than half the size of the zipped XYZ files. Each file corresponds to a particular square kilometer. The name of each tile contains the lower left coordinate of this square kilometer but there can be multiple files for each square kilometer:

• 14 files with „ab“ in the name contain very few points. They look like additional points for under bridges. The „b“ is likely for „Brücke“ (German for „bridge“).
• 38 files with „ag“ in the name contain seem to contain only points in areas where buildings used to cover the terrain but with ground elevation. The „g“ is likely for „Gebäude“ (German for „building“).
• 30 files with „aw“ in the name contain seem to contain only points in areas where there are water bodies but with ground elevation. The „w“ is likely for „Wasser“ (German for „water“).
• 14 files with „brk“ in the name also contain few points. They look like the original bridge point that are replaced by the points in the files with „ab“ in the name to flatten the bridges. The „brk“ is also likely for „Brücke“ (German for „bridge“).
• 42 files with „lpb“ in the name. They look like the last return LiDAR points that were classified as ground. The „lpb“ is likely for „Letzter Pulse Boden“ (German for „last return ground“).
• 42 files with „lpnb“ in the name. They look like those last return LiDAR points that were classified as non-ground. The „lpnb“ is likely for „Letzter Pulse Nicht Boden“ (German for „last return not ground“).
• 32 files with „lpub“ in the name contain very few points. They look like the last return points that are too low and were therefore excluded. The „lpub“ is likely for „Letzter Pulse Unter Boden“ (German for „last return under ground“).

It is left to an exercise to the user for figure out which of those above sets of files should be used for generating a raster DTM. (-: Give us your ideas in the comments. The DOM zip file has a total of 72 *.xyz files. We also compress them and add the EPSG 25832 code with laszip. The compressed LAZ files are less than half the size of the zipped XYZ files. Again there are multiple files for each square kilometer:

• 30 files with „aw“ in the name contain seem to contain only points in areas where there are water bodies but with ground elevation. The „w“ is likely for „Wasser“ (German for „water“).
• 42 files with „fp“ in the name. They look like the first return LiDAR points. The „fp“ is likely for „Frühester Pulse“ (German for „first return“).

If you use all these points from the DOM folder your get the nice DSM shown below … albeit not a spike-free one.

Kudos to OpenNRW for being the first German state to open their LiDAR holdings. Which one of the other 15 German state survey departments will be next to promote their LiDAR as open data. If you are not the last one to do so you can expect to get featured here too … (-;

UPDATE (January 5th): The folks at OpenNRW just tweeted us information about the organization of the zipped archives in the DTM (DGM) and DSM (DOM) folders. We guessed pretty okay which points are in which file but the graphic below (also available here) summarizes it much more nicely and also tells us that „a“ was for „ausgefüllt“ (German for „filled up“). Maximally two returns per pulse are available: either a single return or a first return plus a last return. There are no intermediate returns, which may be an issue for those interested in vegetation mapping.