new “LAStools Production” and “LAStools Pipelines” for ArcGIS

In what some may call a “good LAZ, bad zLAS” routine, rapidlasso GmbH announces better integration of their LiDAR processing technology into ESRI’s ArcGIS platform with new and improved LAStools toolboxes. This comes merely two days after pranking ESRI with an April Fools’ Day press release that was cheered on by a user community that is growing increasingly tired of ESRI bulldozing their own path instead of working together with them.

The new “LAStools Production” toolbox allows to batch automated LiDAR processing tasks across folders of LAS or LAZ files, which is a necessity for production workflows dealing with the Terabytes of LiDAR data involved in typical real-world projects. The new toolbox allows advanced users to further customize processing with an option for providing “additional command line parameters”. This capability has also been added to the good old LAStools toolbox, which is the better choice when operating on only a single LAS or LAZ file at a time.

The new "LAStools Production" toolbox and built-upon the new "LAStools Pipelines" models for batch processing entire folders worth of LiDAR.

The new “LAStools Production” toolbox shown here alongside some example “LAStools Pipelines” models that combine the new production tools to batch process entire folders worth of raw LiDAR flightlines into final products.

Built-upon the “LAStools Production” toolbox are some example “LAStools Pipelines” using the ArcGIS Model Builder that combine the new production tools to batch process entire folders worth of raw LiDAR flightlines into final products. Advanced users may find it more elegant to use Python scripting to combine the batch processing modules from the “LAStools Production” toolbox into more complex workflows. The “LAStools Pipelines” shown below were created in ArcGIS 9.3 and can be downloaded as part of the LAStools distribution (20 MB). Some LAStools Pipelines may not load due to incompatibilities between different versions of ESRI’s ArcGIS Model Builder. Please contribute your own pipelines or share modifications that work for other versions of ArcGIS. Below some screen shots of the six “LAStools Pipelines” that are included as an example in today’s release of LAStools (version 140403). One of them implements the algorithm for constructing “pit-free” CHMs that was presented by Khosravipour et. al at Silvilaser 2013 (find poster and abstract here).

Esri and rapidlasso develop joint LiDAR compressor

PRESS RELEASE (April Fools’ Day)
April 1, 2014
rapidlasso GmbH, Gilching, Germany

In a positive spin of events, Esri and rapidlasso are announcing to join forces and together develop a LiDAR compressor for LAS 1.4 in open source avoiding unnecessary format fragmentation. Their new “LASeasy” tool not only compresses but also optimizes LAS files for efficient area-of-interest queries. LASeasy extends the popular LASzip compressor to handle LAS 1.4 content and includes the tiny spatial indexing *.lax files into the *.laz file via Extended Variable Length Records (EVLRs). More importantly, LASeasy provides new features such as optional spatial sorting and precomputed statistics – motivated by Esri – that allow exploiting LiDAR in the cloud.

To minimize disruption in existing workflows, their joint effort uses a clever strategy that capitalizes on the natural “break” in the ASPRS LAS format from version 1.3 to 1.4. LAS files compressed by Esri will automatically be upgraded to the new point types introduced with LAS 1.4 (and be losslessly downgraded on decompression). LiDAR software already supporting LAZ will instantly be able to read all LiDAR produced by Esri with the same DLL update that will be needed to access future compressed LAS 1.4 content – achieving maximum compatibility with minimal disruption for users of ArcGIS, LASzip, and the larger LiDAR community,

Martin Isenburg, chief scientist and CEO of rapidlasso GmbH, was all smiles during the announement. “Yes, I had some hard feelings when hearing about their ‘LAZ clone‘ because our presumed open dialogue suddenly felt so very one-sided,” he said, “So over Martin Luther King weekend I proposed this LAS 1.4 trick as a joint development quoting MLK’s ‘We must accept finite disappointment, but never lose infinite hope’ and that seemed to resonate with them.” Speaking on the condition of anonymity an executive of Esri’s management added “For a global geospatial player like us it can happen that we do something ‘evil-by-accident’. We occasionally need someone like Martin to poke some good-natured fun at Esri to remind us of our values.”

LASeasy optimizes LAS files by reordering points along an adaptive space-filling curve for efficient LiDAR queries in the cloud. To access the corner of the LiDAR tile only the points shown in blue need to be loaded and decompressed.

LASeasy optimizes LAS files by reordering points along an adaptive space-filling curve for efficient LiDAR queries in the cloud. To access the corner of the LiDAR tile only the points shown in blue need to be loaded and decompressed.

new compressed LAS format by ESRI

NEWSFLASH: update on Jan 7th, 12th, 19th, and Feb 7th (see end of article)

Today I got an email from a LAStools user at NOAA pointing out a new entry in the ArcGIS 10.2 documentation of ESRI that mentions a *.zlas format for the first time. This may have been an oversight at ESRI since there was no press release, blog post, etc preceding this documentation update (that happened 11 days ago). A screenshot of the entry can be found below.

I have heard about LAS compression by ESRI since Gene mentioned it in a blog entry after ESRI’s 3D Mapping and LiDAR Forum. Back then I throught they were talking about LAZ and that our 1.5 years of talking about including support for LASzip-compressed LiDAR into ArcGIS were finally getting somewhere. But turns out they have been doing their own thing. Here some rumors I have heard about ESRI’s new *.zlas format:

  • similar compression rates as LAZ
  • includes spatial indexing
  • (maybe) re-orders points during compression
  • performance is like laszip.exe or better
  • will be available in ArcGIS 10.2.1
  • can be used without the LAS Dataset
  • “free” Windows executable will be available soon
  • development libraries with API will follow
  • ESRI has been giving data providers heads up that clients may soon demand this format

My first thought was that this might be a reengineered version of the LizardTech LiDAR CompressorTM but it is not. This seems to be ESRI’s own development. Does anyone have more details on this?

What was their motivation? Is LAZ too slow for them? I would have happily adressed whatever LASzip was lacking as (compatible) extensions to the LAZ format – which has become de-facto standard for LiDAR compression and is open source. But instead they invested serious money and man-power into creating an entirely new format. Anyone want to speculate why …?

screenshot of ArcGIS 10.2 documentation

screenshot of the ArcGIS 10.2 documentation mentioning *.zlas

UPDATE (January 7th): It is official now. Apparently, Gene who has mentioned our rumors on his blog just received heads up from Clayton Crawford at ESRI that the LAS Optimizer is available for download here. The EzLAS.zip file contains a PDF with interesting details. Below a snapshot of the GUI and what the website says:

“This executable is used to optimize and compress LAS format lidar. It creates *.zlas files, an optimized version of LAS that’s useful for archiving, sharing, and direct use. zLAS files are much smaller and more efficient to use, especially on the cloud and over networks, than regular LAS.
  •  The standalone executable does not require an ArcGIS install or license.
  •  The same executable is used for both compression and decompression.
  •  The download zip file contains more information and help in an included pdf document.”
GUI of the LAS Optimizer

GUI of the LAS Optimizer

Apparently, ArcGIS 10.2.1 is available for general release on January 7th and will support direct read of optimized LAS (*.zlas) via the LAS dataset. Now it’s your turn to try out what ESRI has cooked up and comment … (-:

UPDATE (January 9th): I was told that ESRI will be releasing an “official statement” soon explaining why they have developed their own LiDAR compression format. And they really should do so. I have received (and continue to receive) a fair number of “off-the-record” emails from people across the industry expressing feelings that range between “disappointment”, “anger”, and “disgust” over what is seen as an attempt to sabotage our multi-year effort of creating an open, free, and efficient compressed LiDAR exchange format … time for some xkcd humor.

UPDATE (January 12th): This might just be that “official statement“. Seems someone is really trying hard to avoid using the word LASzip … (-;

UPDATE (January 19th): The story has been picked up by a number of blogs like Paul Ramsey’s “LiDAR format wars“, James Fee’s “LAS, LAZ, LASzip, zLAS and You“, and Randal Hale’s “LiDAR and your software“.

UPDATE (February 7th): The front-lines harden as an unlikely coalition of open source knights, laser guardians, imperial agencies, and competing thugs forms a rebel movement against the approaching Desri Star who threatens the free world announcing that the dArcG is going to spawn “parallelized LAZ clones“. Encrypted instructions to Jedis spread like “Point Clouds on the Horizon” “Towards an Open Future” as the insurgency prepares a better future for compression, free of proprietary oppression. The clone wars might be starting soon. Will the FOSS be with LAZ? (-;

Geoinformatics magazine interviews rapidlasso

Eric van Rees, the editor of the Geoinformatics magazine, met with rapidlasso GmbH at INTERGEO 2013 in Essen to have a quick chat about LAStools, LASzip, and PulseWaves as well as our LiDAR processing toolboxes for ArcGIS and QGIS.

Geoinformatics magazine interviews rapidlasso

In the three-page article we talk about the beginnings of rapidlasso’s software, about when and how the company got started, about our various open and closed source products, and about current projects. The interview was published in the most recent October/November edition of the Geoinformatics magazine that was distributed in print at the SPARELMF 2013 conference in Amsterdam this week. If you did not get a copy you can read the online version of the magazine here.

GRAFCAN launches “DSM on steroids”

GRAFCAN has launched a new product based on LiDAR for the Canary Islands: a digital suface model (DSM) that is literally “on steroids”. It is a synthetic view (not image-derived) that lets the user intuitively understand the territory. The product combines standard hillshading with a height-based color-coding enabling the viewer to “see” where the trees are taller and to grasp height differences between buildings. Darker shades of green indicate high vegetation and lighter shades of green indicate low vegetation. For man-made structures, the shade of red is darker for taller buildings and lightest for one-story houses. The new product is available for all of the Canary Islands at a resolution of 2.5 meters/pixel via the GRAFCAN Web viewer and also as a WMS service.

Example of a DSM on steroids: The viewer intuitively grasps the height and spatial distribution of trees in vegetation areas and the density and size of buildings in urban areas.

Example of the new DSM on steroids: The viewer intuitively grasps the height and spatial distribution of trees in vegetation areas and the density and size of buildings in urban areas.

The product is complemented with a layer with the water areas and a layer of the main roads (both extracted from topographic maps). Around the airports the terrain is very flat and mostly empty. Therefore an intensity map was extracted from the LiDAR and fused into the imagery to provide additional details.

Comparison between orthophoto and DSM on steroids: The ridge is practically invisible in the ortho and the dark green colors of the vegetation are quite missleading to the untrained eye.

Comparison between orthophoto and DSM on steroids: The ridge is practically invisible in the ortho and the dark green colors of the vegetation are quite missleading to the untrained eye.

The LiDAR was flown between 2011 and 2012 with a Leica ALS60 with an average density of 1 pulse per square meter. Most of the LiDAR processing was done with LAStools batch scripts:

  • clean-up of header with las2las.exe
  • orthometric height correction with lasheight.exe
  • tiling with lastile.exe
  • bare-earth extraction with lasground.exe
  • noise removal (clouds and low points) with lasheight.exe
  • DSM rasterization at 2.5 meters per pixel with las2dem.exe
  • flattening reservoirs, ponds, swimming pools with las2dem.exe
  • normalize LiDAR to height above ground with lasheight.exe
  • create height above ground map with lasgrid.exe
  • generating intensity rasters for airports using LiDAR with lasgrid.exe

Other processing tasks involved ArcGIS and Photoshop:

  • separate height above ground map into vegetation and building height map with ArcGIS
  • generating hillshaded rasters from the DSM with ArcGIS
  • rasterizing building footprints colored by height with ArcGIS
  • rasterizing roads from topographic maps with ArcGIS
  • rasterizing reservoirs, ponds, swimming pools with ArcGIS
  • extracting water line of Ocean (contour at 1 meter) from DSM with ArcGIS
  • some manual editing on complex areas (classification errors, ocean line…) with ArcGIS
  • fusing airport intensity rasters into hillshade rasters with Photoshop
  • blending all the layers into the final product with Photoshop
Comparison between bare earth DTM and DSM on steroids. The green houses show up nicely as "planar low vegetatation". This is because they are made out of coarse maze fabric (instead of glass) that lets the the laser through and does not deflect it (like glass houses would).

Comparison between bare earth DTM and DSM on steroids. The green houses appear as “low planar vegetatation”. They are made out of coarse maze fabric (instead of glass) that lets the laser through and does not deflect it (like glass would).

GRAFCAN is a company that  produces and publishes the geographic information of Canary Islands. The company has done LiDAR flights since 2010. You can explore the Canary LiDAR directly in 3D via their interactive Web viewer of IDECanarias (press the “Lidar” button).

Another beautiful example of a DSM on steroids.

Another beautiful example of a DSM on steroids.

rapidlasso selects LIDAR Technology Co. as first LAStools reseller

PRESS RELEASE
October 7, 2013
rapidlasso GmbH, Gilching, Germany

Just in time for INTERGEO 2013, the creators of LAStools and LASzip have partnered with LIDAR Technology Co., Ltd. of Zhubei City, Taiwan as their first exclusive reseller for the Taiwanese market. The two companies decided to strengthen their ties after holding a successful LiDAR processing workshop together with over 60 attendees following the International Symposium on Mobile Mapping Technologies (MMT 2013) in Tainan City last May. “We were looking for a partner with a strong technical mindset who is committed to capacity building,” says Dr. Martin Isenburg, founder of rapidlasso GmbH, “and LIDAR Technology Co., Ltd. is investing in R&D – including full waveform LiDAR – and has strong ties to academia. We couldn’t have found a better collaborator.” Dr. “James” Jin-King Liu, the CEO of LIDAR Technology Co., Ltd. concurs “our companies are a great match. LAStools is already widely popular in Taiwan, yet to employ them to their fullest potential can require some guidance. Our company has the technical expertise to provide this kind of support locally.”

About rapidlasso GmbH:

Technology start-up rapidlasso GmbH specializes in efficient LiDAR processing tools that are widely known for their high productivity. They combine robust algorithms with efficient I/O and clever memory management to achieve high throughput for data sets containing billions of points. The company’s flagship product – the LAStools software suite – has deep market penetration and is heavily used in industry, government agencies, research labs, and educational institutions. Visit http://rapidlasso.com for more information.

About LIDAR Technology Co., Ltd.:

LIDAR Technology Co., Ltd. was founded in 2010 out of the airborne LiDAR team of the Industrial Technology Research Institute (ITRI), a National Laboratory in Taiwan. Their primary business is research, development, and application of earth observation, geoinformatics, and location-based services. LIDAR Technology Co., Ltd. performs acquisition, handling, and analysis of space-borne and airborne data with a focus on LiDAR, including full waveform, for which the company has developed own specialized solutions for their customers. Types of applications include mineral resources, geological mapping, geo-hazards studies, land-use and land-cover mapping, coastal zone mapping, forestry mapping, …  visit http://www.lidar.com.tw for more information.

With CEO "James" Jin-King Liu and CFO Ian Hsu in front of the office of LIDAR Technology Co., Ltd.

With CEO “James” Jin-King Liu and General Manager Ian Hsu in front of the office of LIDAR Technology Co., Ltd.