LASmoons: Elia Palop-Navarro

Elia Palop-Navarro (recipient of three LASmoons)
Research Unit in Biodiversity (UO-PA-CSIC)
University of Oviedo, SPAIN.

Background:
Old-growth forests play an important role in biodiversity conservation. However, long history of human transformation of the landscape has led to the existence of few such forests nowadays. Its structure, characterized by multiple tree species and ages, old trees and abundant deadwood, is particularly sensible to management practices (Paillet et al. 2015) and requires long time to recover from disturbance (Burrascano et al. 2013). Within protected areas we would expect higher proportions of old-growth forests since these areas are in principle managed to ensure conservation of natural ecosystems and processes. Nevertheless, most protected areas in the EU sustained use and exploitation in the past, or even still do.

lasmoons_elia_palopnavarro_0

Part of the study area. Dotted area corresponds to forest surface under protection.

Goal:
Through the application of a model developed in the study area, using public LiDAR and forest inventory data (Palop-Navarro et al. 2016), we’d like to know how much of the forest in a network of mountain protected areas retains structural attributes compatible with old-growth forests. The LiDAR processing tasks which LAStools will be used for involve a total of 614,808 plots in which we have to derive height metrics, such as mean or median canopy height and its variability.

Vegetation profile colored by height in a LiDAR sample of the study area.

Vegetation profile colored by height in a LiDAR sample of the study area.

Data:
+ Public LiDAR data that can be downloaded here with mean pulse density 0.5 points per square meter. This data has up to 5 returns and is already classified into ground, low, mid or high vegetation, building, noise or overlapped.
+ The area covers forested areas within protected areas in Cantabrian Mountains, occupying 1,207 km2.

LAStools processing:
1) quality checking of the data as described in several videos and blog posts [lasinfo, lasvalidate, lasoverlap, lasgrid, las2dem]
2) use existing ground classification (if quality suffices) to normalize the elevations of to heights above ground using tile-based processing with on-the-fly buffers of 50 meters to avoid edge artifacts [lasheight]
3) compute height-based forestry metrics (e.g. ‘-avg’, ‘-std’, and ‘-p 50’) for each plot in the study area [lascanopy]

References:
Burrascano, S., Keeton, W.S., Sabatini, F.M., Blasi, C. 2013. Commonality and variability in the structural attributes of moist temperate old-growth forests: a global review. Forest Ecology and Management 291:458-479.
Paillet, Y., Pernot, C., Boulanger, V., Debaive, N., Fuhr, M., Gilg, O., Gosselin, F. 2015. Quantifying the recovery of old-growth attributes in forest reserves: A first reference for France. Forest Ecology and Management 346:51-64.
Palop-Navarro, E., Bañuelos, M.J., Quevedo, M. 2016. Combinando datos lidar e inventario forestal para identificar estados avanzados de desarrollo en bosques caducifolios. Ecosistemas 25(3):35-42.

Prototype for “native LAS 1.4 extension” of LASzip LiDAR Compressor Released

PRESS RELEASE (for immediate release)
February 13, 2017
rapidlasso GmbH, Gilching, Germany

Just in time for ILMF 2017 in Denver, the makers of the popular LiDAR processing software LAStools announce that the prototype for the “native LAS 1.4 extension” of their award-winning open source LASzip LiDAR compressor is ready for testing. An update to the compressed LAZ format had become necessary due to a core change in the ASPRS LAS 1.4 specification which had introduced several new point types.

A new feature of the updated LASzip compressor is the ability to selectively decompress of only those attributes of each point that really are needed by the application that is reading the LAZ file. Minimally this will be the x and y coordinate of each point and the return counts, which are sufficient to – for example – calculate the exact extend of the survey area. Most applications will also want to access z coordinate. However, the intensities, the GPS times, the RGB or NIR colors, and the new “Extra Bytes” are often not needed. As the updated LAZ format compresses these different attributes into separate layers, their decompression can then be skipped. Therefore sometimes only 40% of a compressed LAZ file needs to be decompressed to access the coordinates of points with many attributes.

percentage of bytes in a compressing LAZ file corresponding to different point attributes

The percentages of a compressed LAZ file used to encode different point attributes for two example LAS 1.4 files.

The new LASzip prototype is currently being crowd-tested. Interested parties who already have holdings of LAS 1.4 files with point types 6 to 10 may send an email to ‘lasproto@rapidlasso.com’ to participate in these tests.

The release of the new LASzip compressor comes more than a year late as development had been intentionally delayed to give ESRI an opportunity to contribute their needs and ideas to create a joint open format with the community and avoid LiDAR format fragmentation. Sadly, this effort ultimately failed.

About rapidlasso GmbH:
Technology powerhouse rapidlasso GmbH specializes in efficient LiDAR processing tools that are widely known for their high productivity. They combine robust algorithms with efficient I/O and clever memory management to achieve high throughput for data sets containing billions of points. The company’s flagship product – the LAStools software suite – has deep market penetration and is heavily used in industry, government agencies, research labs, and educational institutions. Visit http://rapidlasso.com for more information.

LASmoons: Jesús García Sánchez

Jesús García Sánchez (recipient of three LASmoons)
Landscapes of Early Roman Colonization (LERC) project
Faculty of Archaeology, Leiden University, The Netherlands

Background:
Our project Landscapes of Early Roman Colonization (LERC) has been studying the hinterland of the Latin colony of Aesernia (Molise region, Italy) using several non-destructive techniques, chiefly artefactual survey, geophysics, and interpretation of aerial photographs. Nevertheless large areas of the territory are covered by the dense forests of the Matese mountains, a ridge belonging the Apennine chain, or covered by bushes due to the abandonment of the countryside. The project won’t be complete without integrating the marginal, remote and forested areas into our study of the Roman hinterland. Besides, it’s also relevant to discuss the feasibility of LiDAR data sets in the study of Mediterranean landscapes and its role within contemporary Landscape Archaeology.

some clever caption

LiDAR coverage in Molise region, Italy.

Goal:
+ to study in detail forested areas in the colonial hinterland of Aesernia.
+ to found the correct parameters of the classification algorithm to be able to locate possible archaeological structures or to document appropriately those we already known.
+ to document and create new visualization of hill-top fortified sites that belong to the indigenous population and are currently poorly studied due to inaccessibility and forest coverage (Monte San Paolo, Civitalla, Castelriporso, etc.)
+ to demonstrate the archaeological potential of LiDAR data in Italy and help other scholars to work with that kind of data, explaining basic information about data quality, where and how to acquire imagery and examples of application in archaeology. A paper entitled “Working with ALS – LiDAR data in Central South Italy. Tips and experiences”, will be presented in the International Mediterranean Survey Workshop by the end of February in Athens.

Civitella hillfort (Longano, IS) and its local context: ridges and forest belonging to the Materse mountains and the Appenines.

Data:
Recently the LERC project has acquired a large LiDAR dataset created by the Italian Geoportale Nazionale and the Minisstero dell’Ambiente e della Tutella del Territorio e del Mare. The data was produced originally to monitor land-slides and erosive risk.
The average point resolution is 1 meter.
+ The data sets were cropped originally in 1 sq km. tiles by the Geoportale Nazionale for distribution purposes.

LAStools processing:
1) data is provided in *.txt files thus the first step is to create appropriate LAS files to work with [txt2las]
2) combine areas of circa 16 sq km (still fewer than 20 million points to be processed in one piece with LAStools) in the surroundings of the colony of Aesernia and in the Matese mountains [lasmerge]
3) assign the correct projection to the data [lasmerge or las2las]
4) extract the care-earth with the best-fitting parameters [lasground or lasground_new]
5) create bare-earth terrain rasters as a first step to visualize and analyze the area [lasdem]

LASmoons: Rachel Opitz

Rachel Opitz (recipient of three LASmoons)
Center for Virtualization and Applied Spatial Technologies
Department of Anthropology, University of South Florida, USA

Background:
In Spring 2017 Rachel Opitz will be teaching a course on Remote Sensing for Human Ecology and Archaeology at the University of South Florida. The aim of the course is to provide students with the practical skills and knowledge needed to work with contemporary remote sensing data. The course focuses on airborne laser scanning and hyper-spectral data and their application in Human Ecology and Archaeology. Through the course students will be introduced to a number of software packages commonly used to process and interpret these data, with an emphasis on free and/or open source tools.

Classification parameters and the resolution at which the DTM is interpolated both have a significant impact on our ability to recognize anthropogenic features in the landscape. Here we see a small quarry. More aggressive filtering and a coarser DTM resolution (left) makes it difficult to recognize that this is a quarry. Less aggressive filtering and a higher resolution (right) leaves some vegetation behind, but makes the edges of the quarry and some in-situ blocks clearly visible.

Goal:
The students will develop practical skills in applied remote sensing through hands-on exercises. Learning to assess, manage and process large data sets is essential. In particular, the students in the course will learn to:
+ Identify the set of techniques needed to solve a problem in applied remote sensing
+ Find public imagery and specify acquisitions
+ Assess data quality
+ Process airborne LiDAR data
+ Combine complementary remote sensing data sources
+ Create effective data visualizations
+ Analyze digital topographic and spectral data to answer questions in human ecology and archaeology

Data:
The course will include case studies that draw on a variety of publicly available data sets that will all be used in the exercises:
+ the PNOA data from Spain
+ data held by NOAA
+ data collected using NASA’s GLiHT platform

LAStools processing:
LAStools will be used throughout the course, as students learn to assess the quality of LiDAR data, classify raw LiDAR point clouds, create raster terrain and canopy models, and produce visualizations. The online tutorials and videos available via the company website and the over 50 hours of video on YouTube as well as the LAStools user forum will be used as resources during the course.

NRW Open LiDAR: Download, Compression, Viewing

This is the first part of a series on how to process the newly released open LiDAR data for the entire state of North Rhine-Westphalia that was announced a few days ago. Again, kudos to OpenNRW for being the most progressive open data state in Germany. You can follow this tutorial after downloading the latest version of LAStools as well as a pair of DGM and DOM files for your area of interest from these two download pages.

We have downloaded the pair of DGM and DOM files for the Federal City of Bonn. Bonn is the former capital of Germany and was host to the FOSS4G 2016 conference. As both files are larger than 10 GB, we use the wget command line tool with option ‘-c’ that will restart where it left off in case the transmission gets interrupted.

The DGM file and the DOM file are zipped archives that contain the points in 1km by 1km tiles stored as x, y, z coordinates in ETRS89 / UTM 32 projection as simple ASCII text with centimeter resolution (i.e. two decimal digits).

>> more dgm1l-lpb_32360_5613_1_nw.xyz
360000.00 5613026.69 164.35
360000.00 5613057.67 164.20
360000.00 5613097.19 164.22
360000.00 5613117.89 164.08
360000.00 5613145.35 164.03
[...]

There is more than one tile for each square kilometer as the LiDAR points have been split into different files based on their classification and their return type. Furthermore there are also synthetic points that were used by the land survey department to replace certain LiDAR points in order to generate higher quality DTM and DSM raster products.

The zipped DGM archive is 10.5 GB in size and contains 956 *.xyz files totaling 43.5 GB after decompression. The zipped DOM archive is 11.5 GB in size and contains 244 *.xyz files totaling 47.8 GB. Repeatedly loading these 90 GB of text data and parsing these human-readable x, y, and z coordinates is inefficient with common LiDAR software. In the first step we convert the textual *.xyz files into binary *.laz files that can be stored, read and copied more efficiently. We do this with the open source LASzip compressor that is distributed with LAStools using these two command line calls:

laszip -i dgm1l_05314000_Bonn_EPSG5555_XYZ\*.xyz ^
       -epsg 25832 -vertical_dhhn92 ^
       -olaz ^
       -cores 2
laszip -i dom1l_05314000_Bonn_EPSG5555_XYZ\*.xyz ^
       -epsg 25832 -vertical_dhhn92 ^
       -olaz ^
       -cores 2

The point coordinates are is in EPSG 5555, which is a compound datum of horizontal EPSG 25832 aka ETRS89 / UTM zone 32N and vertical EPSG 5783 aka the “Deutsches Haupthoehennetz 1992” or DHHN92. We add this information to each *.laz file during the LASzip compression process with the command line options ‘-epsg 25832’ and ‘-vertical_dhhn92’.

LASzip reduces the file size by a factor of 10. The 956 *.laz DGM files compress down to 4.3 GB from 43.5 GB for the original *.xyz files and the 244 *.laz DOM files compress down to 4.8 GB from 47.8 GB. From here on out we continue to work with the 9 GB of slim *.laz files. But before we delete the 90 GB of bulky *.xyz files we make sure that there are no file corruptions (e.g. disk full, truncated files, interrupted processes, bit flips, …) in the *.laz files.

laszip -i dgm1l_05314000_Bonn_EPSG5555_XYZ\*.laz -check
laszip -i dom1l_05314000_Bonn_EPSG5555_XYZ\*.laz -check

One advantage of having the LiDAR in an industry standard such as the LAS format (or its lossless compressed twin, the LAZ format) is that the header of the file stores the number of points per file, the bounding box, as well as the projection information that we have added. This allows us to very quickly load an overview for example, into lasview.

lasview -i dgm1l_05314000_Bonn_EPSG5555_XYZ\*.laz -GUI
The bounding boxes of the DGM files quickly display a preview of the data in the GUI when the files are in LAS or LAZ format.

The bounding boxes of the DGM files quickly give us an overview in the GUI when the files are in LAS or LAZ format.

Now we want to find a particular site in Bonn such as the World Conference Center Bonn where FOSS4G 2016 was held. Which tile is it in? We need some geospatial context to find it, for example, by creating an overview in form of KML files that we can load into Google Earth. We use the files from the DOM folder with “fp” in the name as points on buildings are mostly “first returns”. See what our previous blog post writes about the different file names if you can not wait for the second part of this series. We can create the KML files with lasboundary either via the GUI or in the command line.

lasboundary -i dom1l_05314000_Bonn_EPSG5555_XYZ\dom1l-fp*.laz ^
            -gui
Only the "fp" tiles from the DOM folder loaded the GUI into lasboundary.

Only the “fp” tiles from the DOM folder loaded the GUI into lasboundary.

lasboundary -i dom1l_05314000_Bonn_EPSG5555_XYZ\dom1l-fp*.laz ^
            -use_bb -labels -okml

We zoom in and find the World Conference Center Bonn and load the identified tile into lasview. Well, we did not expect this to happen, but what we see below will make this series of tutorials even more worthwhile. There is a lot of “high noise” in the particular tile we picked. We should have noticed the unusually high z range of 406.42 meters in the Google Earth pop-up. Is this high electromagnetic radiation interfering with the sensors? There are a number of high-tech government buildings with all kind of antennas nearby (such as the United Nations Bonn Campus the mouse cursor points at).

Significant amounts of high noise are in the first returns of the DOM tile we picked.

Significant amounts of high noise are in the first returns of the DOM tile we picked.

But the intended area of interest was found. You can see the iconic “triangulated” roof of the building that is across from the World Conference Center Bonn.

The World Conference Center Bonn is across from the building with the "triangulated" roof.

The World Conference Center Bonn is across from the building with the “triangulated” roof.

Please don’t think it is the responsibility of OpenNRW to remove the noise and provide cleaner data. The land survey has already processed this data into whatever products they needed and that is where their job ended. Any additional services – other than sharing the raw data – are not in their job description. We’ll take care of that … (-:

Acknowledgement: The LiDAR data of OpenNRW comes with a very permissible license. It is called “Datenlizenz Deutschland – Namensnennung – Version 2.0” or “dl-de/by-2-0” and allows data and derivative sharing as well as commercial use. It only requires us to name the source. We need to cite the “Land NRW (2017)” with the year of the download in brackets and specify the Universal Resource Identification (URI) for both the DOM and the DGM. Done. So easy. Thank you, OpenNRW … (-:

New ‘laspublish’ creates Web Portals for 3D Viewing and Downloading of LiDAR

PRESS RELEASE (for immediate release)
February 18, 2016
rapidlasso GmbH, Gilching, Germany

Just in time for ILMF 2016, the makers of the open LASzip LiDAR compressor annouce the latest addition to their LiDAR processing software LAStools. The new ‘laspublish‘ from rapidlasso GmbH creates stand-alone Web Portals for interactive 3D viewing of LiDAR points and for selective downloading of LAZ or LAS files. The new tool is based on the cutting-edge streaming point cloud viewing technology of Potree that optimizes large LiDAR point clouds for streaming via the Web such that anyone can visualize, explore, and (optionally) download them with any modern browser.

3D viewer and download portal created with 'laspublish'

3D viewer and download portal created with ‘laspublish’

The interactive 3D viewer streams on demand only the relevant parts of the point clouds. It not only visualizes the LiDAR in many useful and intuitive ways, but is also equipped with measurement tools to calculate distances or areas and profile or clipping tools for close-up inspections. With its integrated LASzip compression, options to color LiDAR by classification, return type, intensity, and point source ID, stunning visuals via Eye Dome Lighting (EDL), and the optional 2D download map, the new ‘laspublish‘ empowers professional and novice users alike to create stand-alone LiDAR Webportals with just a few button clicks.

a few clicks and 'laspublish' creates a professional LiDAR portal

a few clicks and ‘laspublish’ creates a professional LiDAR portal

The new ‘laspublish‘ is now an integral part of the LAStools software and is bundled together with all necessary components of the Potree software. It provides an instant and cost-effective solution for generating a set of Web pages that realize a self-contained LiDAR portal offering interactive online visualization and exploration as well as easy and intuitive distribution of large LiDAR data sets via the optional download map.

About rapidlasso GmbH:
Technology powerhouse rapidlasso GmbH specializes in efficient LiDAR processing tools that are widely known for their high productivity. They combine robust algorithms with efficient I/O and clever memory management to achieve high throughput for data sets containing billions of points. The company’s flagship product – the LAStools software suite – has deep market penetration and is heavily used in industry, government agencies, research labs, and educational institutions. Visit http://rapidlasso.com for more information. As the only Diamond sponsor, rapidlasso GmbH has been the main financial supporter of the open source Potree package by Markus Schütz over the past two years.

About Potree:
Potree is a WebGL based viewer for large point clouds. The project evolved as a Web based viewer from the Scanopy desktop point cloud renderer by TU Wien, Institute of Computer Graphics and Algorithms. It will continue to be free and open source with a FreeBSD license to enable anyone to view, analyze and publicly share their large datasets. Visit http://potree.org for more information.

The dArc Force Awakens: ESRI escalates LiDAR format war

The empire has not changed their evil ways, despite an encouraging email from ESRI’s founder and president Jack Dangermond in response to the Open Letter by the OSGeo that was delivered to ESRI, OGC, and the ASPRS. Facing an incredible backlash by the LiDAR community over the release of their “LAZ clone” there was a new hope that unnecessary format fragmention could be avoided by working together within the Point Cloud Domain Working Group of the OGC. In fact only one thing happened: ESRI went silent on the controversy. They temporarily stopped promoting their “LAZ clone” and focused on locking in more content.

dArc_force_awakens

The message of the rebellion has been consistent and clear like in these two videos from the TC meeting of the OGC in Nottingham and the ASPRS side bar in Reno: a roadmap forward to avoid format fragmentation by exploiting the “natural break” in the format due to LAS 1.4. But there was zero technical contribution from ESRI during the past three PC-DWG meetings of the OGC. The slide sets that bored the audiences in Boulder and in Nottingham were not meant to contribute but merely stalled for time. Recently in Sydney ESRI was awefully quiet, knowing they were doing the exact opposite of what the OGC stands for. And now the empire strikes back.

laztozlas

There is a dArc force awakening that threatens the peace within the LiDAR community. ESRI has just released a new tool (see above) that enslaves point clouds by converting them from the open LAZ format to the near-identical but closed “LAZ clone” that they call “zLAS” or “Optimized LAS”. This comes just a few months after an entire nation‘s LiDAR was enslaved in this proprietary format. We have repeatedly warned about the ramifications of locking up Petabytes of LiDAR data in a closed format that is controlled by a single vendor.

ESRI is one of the largest GIS training organizations. By instructing LiDAR novices to “optimize” their LiDAR files and pushing LiDAR providers to switch from open LAS or open LAZ to closed zLAS, they effectively destroy the current success of our open formats. ESRI’s command of the GIS market can – little by little – turn their own proprietry format into the dominant way in which LiDAR point clouds are stored. Then we loose our open exchange formats. Hence, ESRI’s proprietary format threatens all that we have achieved with LAS (and LAZ) over the past years: compatible LiDAR data exchange and incredible LiDAR software interoperability.

ESRI is now escalating the LiDAR format wars. Join the rebellion, Jedis: download your lazer sabers and liberate some LiDAR.

This is not an anti-ESRI campaign. For the past three years we have been trying to resolve this situation. We have repeatedly reached out to ESRI to prevent format fragmentation. We have repeatedly offered to create a joint compressed format. We have plead, begged, and bargained for the sake of our LiDAR community and the sake of their ArcGIS user community not to promote a near-identical yet incompatible way for storing massive amounts of point cloud data.