new compressed LAS format by ESRI

NEWSFLASH: update on Jan 7th, 12th, 19th, and Feb 7th (see end of article)

Today I got an email from a LAStools user at NOAA pointing out a new entry in the ArcGIS 10.2 documentation of ESRI that mentions a *.zlas format for the first time. This may have been an oversight at ESRI since there was no press release, blog post, etc preceding this documentation update (that happened 11 days ago). A screenshot of the entry can be found below.

I have heard about LAS compression by ESRI since Gene mentioned it in a blog entry after ESRI’s 3D Mapping and LiDAR Forum. Back then I throught they were talking about LAZ and that our 1.5 years of talking about including support for LASzip-compressed LiDAR into ArcGIS were finally getting somewhere. But turns out they have been doing their own thing. Here some rumors I have heard about ESRI’s new *.zlas format:

  • similar compression rates as LAZ
  • includes spatial indexing
  • (maybe) re-orders points during compression
  • performance is like laszip.exe or better
  • will be available in ArcGIS 10.2.1
  • can be used without the LAS Dataset
  • “free” Windows executable will be available soon
  • development libraries with API will follow
  • ESRI has been giving data providers heads up that clients may soon demand this format

My first thought was that this might be a reengineered version of the LizardTech LiDAR CompressorTM but it is not. This seems to be ESRI’s own development. Does anyone have more details on this?

What was their motivation? Is LAZ too slow for them? I would have happily adressed whatever LASzip was lacking as (compatible) extensions to the LAZ format – which has become de-facto standard for LiDAR compression and is open source. But instead they invested serious money and man-power into creating an entirely new format. Anyone want to speculate why …?

screenshot of ArcGIS 10.2 documentation

screenshot of the ArcGIS 10.2 documentation mentioning *.zlas

UPDATE (January 7th): It is official now. Apparently, Gene who has mentioned our rumors on his blog just received heads up from Clayton Crawford at ESRI that the LAS Optimizer is available for download here. The EzLAS.zip file contains a PDF with interesting details. Below a snapshot of the GUI and what the website says:

“This executable is used to optimize and compress LAS format lidar. It creates *.zlas files, an optimized version of LAS that’s useful for archiving, sharing, and direct use. zLAS files are much smaller and more efficient to use, especially on the cloud and over networks, than regular LAS.
  •  The standalone executable does not require an ArcGIS install or license.
  •  The same executable is used for both compression and decompression.
  •  The download zip file contains more information and help in an included pdf document.”
GUI of the LAS Optimizer

GUI of the LAS Optimizer

Apparently, ArcGIS 10.2.1 is available for general release on January 7th and will support direct read of optimized LAS (*.zlas) via the LAS dataset. Now it’s your turn to try out what ESRI has cooked up and comment … (-:

UPDATE (January 9th): I was told that ESRI will be releasing an “official statement” soon explaining why they have developed their own LiDAR compression format. And they really should do so. I have received (and continue to receive) a fair number of “off-the-record” emails from people across the industry expressing feelings that range between “disappointment”, “anger”, and “disgust” over what is seen as an attempt to sabotage our multi-year effort of creating an open, free, and efficient compressed LiDAR exchange format … time for some xkcd humor.

UPDATE (January 12th): This might just be that “official statement“. Seems someone is really trying hard to avoid using the word LASzip … (-;

UPDATE (January 19th): The story has been picked up by a number of blogs like Paul Ramsey’s “LiDAR format wars“, James Fee’s “LAS, LAZ, LASzip, zLAS and You“, and Randal Hale’s “LiDAR and your software“.

UPDATE (February 7th): The front-lines harden as an unlikely coalition of open source knights, laser guardians, imperial agencies, and competing thugs forms a rebel movement against the approaching Desri Star who threatens the free world announcing that the dArcG is going to spawn “parallelized LAZ clones“. Encrypted instructions to Jedis spread like “Point Clouds on the Horizon” “Towards an Open Future” as the insurgency prepares a better future for compression, free of proprietary oppression. The clone wars might be starting soon. Will the FOSS be with LAZ? (-;

Geoinformatics magazine interviews rapidlasso

Eric van Rees, the editor of the Geoinformatics magazine, met with rapidlasso GmbH at INTERGEO 2013 in Essen to have a quick chat about LAStools, LASzip, and PulseWaves as well as our LiDAR processing toolboxes for ArcGIS and QGIS.

Geoinformatics magazine interviews rapidlasso

In the three-page article we talk about the beginnings of rapidlasso’s software, about when and how the company got started, about our various open and closed source products, and about current projects. The interview was published in the most recent October/November edition of the Geoinformatics magazine that was distributed in print at the SPARELMF 2013 conference in Amsterdam this week. If you did not get a copy you can read the online version of the magazine here.

GRAFCAN launches “DSM on steroids”

GRAFCAN has launched a new product based on LiDAR for the Canary Islands: a digital suface model (DSM) that is literally “on steroids”. It is a synthetic view (not image-derived) that lets the user intuitively understand the territory. The product combines standard hillshading with a height-based color-coding enabling the viewer to “see” where the trees are taller and to grasp height differences between buildings. Darker shades of green indicate high vegetation and lighter shades of green indicate low vegetation. For man-made structures, the shade of red is darker for taller buildings and lightest for one-story houses. The new product is available for all of the Canary Islands at a resolution of 2.5 meters/pixel via the GRAFCAN Web viewer and also as a WMS service.

Example of a DSM on steroids: The viewer intuitively grasps the height and spatial distribution of trees in vegetation areas and the density and size of buildings in urban areas.

Example of the new DSM on steroids: The viewer intuitively grasps the height and spatial distribution of trees in vegetation areas and the density and size of buildings in urban areas.

The product is complemented with a layer with the water areas and a layer of the main roads (both extracted from topographic maps). Around the airports the terrain is very flat and mostly empty. Therefore an intensity map was extracted from the LiDAR and fused into the imagery to provide additional details.

Comparison between orthophoto and DSM on steroids: The ridge is practically invisible in the ortho and the dark green colors of the vegetation are quite missleading to the untrained eye.

Comparison between orthophoto and DSM on steroids: The ridge is practically invisible in the ortho and the dark green colors of the vegetation are quite missleading to the untrained eye.

The LiDAR was flown between 2011 and 2012 with a Leica ALS60 with an average density of 1 pulse per square meter. Most of the LiDAR processing was done with LAStools batch scripts:

  • clean-up of header with las2las.exe
  • orthometric height correction with lasheight.exe
  • tiling with lastile.exe
  • bare-earth extraction with lasground.exe
  • noise removal (clouds and low points) with lasheight.exe
  • DSM rasterization at 2.5 meters per pixel with las2dem.exe
  • flattening reservoirs, ponds, swimming pools with las2dem.exe
  • normalize LiDAR to height above ground with lasheight.exe
  • create height above ground map with lasgrid.exe
  • generating intensity rasters for airports using LiDAR with lasgrid.exe

Other processing tasks involved ArcGIS and Photoshop:

  • separate height above ground map into vegetation and building height map with ArcGIS
  • generating hillshaded rasters from the DSM with ArcGIS
  • rasterizing building footprints colored by height with ArcGIS
  • rasterizing roads from topographic maps with ArcGIS
  • rasterizing reservoirs, ponds, swimming pools with ArcGIS
  • extracting water line of Ocean (contour at 1 meter) from DSM with ArcGIS
  • some manual editing on complex areas (classification errors, ocean line…) with ArcGIS
  • fusing airport intensity rasters into hillshade rasters with Photoshop
  • blending all the layers into the final product with Photoshop
Comparison between bare earth DTM and DSM on steroids. The green houses show up nicely as "planar low vegetatation". This is because they are made out of coarse maze fabric (instead of glass) that lets the the laser through and does not deflect it (like glass houses would).

Comparison between bare earth DTM and DSM on steroids. The green houses appear as “low planar vegetatation”. They are made out of coarse maze fabric (instead of glass) that lets the laser through and does not deflect it (like glass would).

GRAFCAN is a company that  produces and publishes the geographic information of Canary Islands. The company has done LiDAR flights since 2010. You can explore the Canary LiDAR directly in 3D via their interactive Web viewer of IDECanarias (press the “Lidar” button).

Another beautiful example of a DSM on steroids.

Another beautiful example of a DSM on steroids.

rapidlasso selects LIDAR Technology Co. as first LAStools reseller

PRESS RELEASE
October 7, 2013
rapidlasso GmbH, Gilching, Germany

Just in time for INTERGEO 2013, the creators of LAStools and LASzip have partnered with LIDAR Technology Co., Ltd. of Zhubei City, Taiwan as their first exclusive reseller for the Taiwanese market. The two companies decided to strengthen their ties after holding a successful LiDAR processing workshop together with over 60 attendees following the International Symposium on Mobile Mapping Technologies (MMT 2013) in Tainan City last May. “We were looking for a partner with a strong technical mindset who is committed to capacity building,” says Dr. Martin Isenburg, founder of rapidlasso GmbH, “and LIDAR Technology Co., Ltd. is investing in R&D – including full waveform LiDAR – and has strong ties to academia. We couldn’t have found a better collaborator.” Dr. “James” Jin-King Liu, the CEO of LIDAR Technology Co., Ltd. concurs “our companies are a great match. LAStools is already widely popular in Taiwan, yet to employ them to their fullest potential can require some guidance. Our company has the technical expertise to provide this kind of support locally.”

About rapidlasso GmbH:

Technology start-up rapidlasso GmbH specializes in efficient LiDAR processing tools that are widely known for their high productivity. They combine robust algorithms with efficient I/O and clever memory management to achieve high throughput for data sets containing billions of points. The company’s flagship product – the LAStools software suite – has deep market penetration and is heavily used in industry, government agencies, research labs, and educational institutions. Visit http://rapidlasso.com for more information.

About LIDAR Technology Co., Ltd.:

LIDAR Technology Co., Ltd. was founded in 2010 out of the airborne LiDAR team of the Industrial Technology Research Institute (ITRI), a National Laboratory in Taiwan. Their primary business is research, development, and application of earth observation, geoinformatics, and location-based services. LIDAR Technology Co., Ltd. performs acquisition, handling, and analysis of space-borne and airborne data with a focus on LiDAR, including full waveform, for which the company has developed own specialized solutions for their customers. Types of applications include mineral resources, geological mapping, geo-hazards studies, land-use and land-cover mapping, coastal zone mapping, forestry mapping, …  visit http://www.lidar.com.tw for more information.

With CEO "James" Jin-King Liu and CFO Ian Hsu in front of the office of LIDAR Technology Co., Ltd.

With CEO “James” Jin-King Liu and General Manager Ian Hsu in front of the office of LIDAR Technology Co., Ltd.

rapidlasso adds LAStools LiDAR processing toolbox to QGIS

PRESS RELEASE
September 30, 2013
rapidlasso GmbH, Gilching, Germany

At the code sprint following the FOSS4G 2013 conference, rapidlasso GmbH completed a toolbox that exposes the extensive LiDAR processing capabilities of the LAStools software suite within QGIS. QGIS (previously known as “Quantum GIS”) is a comprehensive desktop geographic information system that is free and open source software (FOSS). QGIS has seen rapid growth in acceptance as a viable and cost-effective alternative to the commerical ArcGIS products of ESRI. The newly integrated “Processing” framework (previously known as “SEXTANTE“) allows to combine the LiDAR functionality of LAStools with native raster and vector operations of QGIS as well as capabilities of other popular GIS packages such as GDALGRASS, SAGA, TauDEM, PostGIS and the (geo-)statistics package R. Detailed instructions on how to add the new LAStools toolbox to QGIS are available here.

About rapidlasso GmbH:
Technology start-up rapidlasso GmbH specializes in efficient LiDAR processing tools that are widely known for their high productivity. They combine robust algorithms with efficient I/O and clever memory management to achieve high throughput for data sets containing billions of points. The company’s flagship product – the LAStools software suite – has deep market penetration and is heavily used in industry, government agencies, research labs, and educational institutions. Visit http://rapidlasso.com for more information.

how to install LAStools toolbox in QGIS

The more complex installation instructions are only relevant for older QGIS versions. Starting with QGIS 2.10 or newer you can skip directly to step 4 and disregard most of step 6:

We had an ArcGIS LiDAR processing toolbox since April 2012 and now we have one for QGIS as well. It has been tested successfully with QGIS 1.8.0-Lisboa, 2.0.1-Dufour, 2.2.0-Valmiera, and 2.4.0-Chugiak. Download and install QGIS. For version 2.2.0-Valmiera and older follow all the steps. Do not delete or copy any files for the newest version 2.4.0-Chugiak but skip directly to step 4 and disregard most of step 6:

  1. If you already started QGIS then exit the application.
  2. Delete (or rename) the entire folder “C:\Program Files\QGIS Valmiera\apps\qgis\python\plugins\processing\lidar”.
    QGIS install (step 1)
  3. Put the “.\lidar” folder that is inside this ZIP file into its place. This ZIP file only works for QGIS 2.2.0-Valmiera. For QGIS 2.0.1-Dufour you need to use the contents of this ZIP file instead. And for QGIS 1.8.0-Lisboa use this ZIP file and delete the folder “C:\Users\Martin\.qgis\python\plugins\sextante\lidar” instead.
  4. Get the most recent version of LAStools by downloading the 36 MB “LAStools.zip” file.
  5. Drag and drop (or extract) the “.\LAStools” folder from inside the ‘LAStools.zip‘ file such that there is no space in the path (e.g. bad: “C:\Program Files\LAStools”, good: “D:\software\LAStools”).
  6. Start QGIS. If there is a Python script error carefully repeat steps 1 to 3.
  7. Turn on the “Processing Toolbox” as shown below.
    turning on the "Processing Toolbox"
  8. Switch the processing toolbox from “Simplified Interface” to “Advanced Interface” as shown below.
    Switching the processing toolbox from "Simplified Interface" to "Advanced Interface"
  9. Open the “Options and configuration” sub menu of “Processing” as shown below.
    Opening the "Options and configuration" sub menu of "Processing"
  10. Check the “Activate” button for “Providers->Tools for LiDAR data” and fill in the path to your local LAStools folder as shown below. Type or copy & paste the path to LAStools (as the browse popup is broken). Press <strong>ENTER</strong> after entering the path so the field is greyed out again <em>before</em> clicking <strong>OK</strong>.
    Checking the "Activate" button for "Providers->Tools for LiDAR data" and filling in the path to the local LAStools folder.
  11. Now you should see the “Tools for LiDAR data” in the toolbox and all the LAStools as shown below.
    Seeing the "Tools for LiDAR data" in the toolbox and all the LAStools
  12. Start lasground via double click and fill in the settings as shown below.
    Starting lasground via double click and filling in the settings
  13. Look at the file you just created by running lasview from the toolbox. Yay!
  14. Create this with two quick calls to las2dem and las2iso … sooo easy.
    Result after running las2dem and las2iso on fusa.laz sample

Kudos to Victor Olaya for creating this whole framework and providing me with example scripts for a simple LAStools toolbox that I just had to modify … (-:

ArcGIS LAStools toolbox for LiDAR processing

In recent days more and more users are discovering the LAStools toolbox that adds powerful LiDAR processing capabilities to ArcGIS versions 9.3, 10.0, 10.2, and 10.2 allowing you to exploit your LiDAR data in the most sophisticated manner through the familiar toolbox interface (see images at the end). ALso check out the new LAStools Production toolbox and the LAStools Pipelines models built-upon them.

arcmap_model

However – as shown above – the most exciting thing I just learned recently myself: You can easily use the ArcGIS model builder to create entire LiDAR workflows in a very intuitive manner by combining individual LAStools from the toolbox into a pipeline of LiDAR processing tasks that can, for example, turn a set of five individual flight strips into DTM, a DSM, and a CHM at a click of a button by connecting up the individual tools with the proper parameters as outlined above. Please let me know if you have created similar model builder LiDAR workflows on your own using the ArcGIS LAStools toolbox for LiDAR processing …