The dArc Force Awakens: ESRI escalates LiDAR format war

The empire has not changed their evil ways, despite an encouraging email from ESRI’s founder and president Jack Dangermond in response to the Open Letter by the OSGeo that was delivered to ESRI, OGC, and the ASPRS. Facing an incredible backlash by the LiDAR community over the release of their “LAZ clone” there was a new hope that unnecessary format fragmention could be avoided by working together within the Point Cloud Domain Working Group of the OGC. In fact only one thing happened: ESRI went silent on the controversy. They temporarily stopped promoting their “LAZ clone” and focused on locking in more content.

dArc_force_awakens

The message of the rebellion has been consistent and clear like in these two videos from the TC meeting of the OGC in Nottingham and the ASPRS side bar in Reno: a roadmap forward to avoid format fragmentation by exploiting the “natural break” in the format due to LAS 1.4. But there was zero technical contribution from ESRI during the past three PC-DWG meetings of the OGC. The slide sets that bored the audiences in Boulder and in Nottingham were not meant to contribute but merely stalled for time. Recently in Sydney ESRI was awefully quiet, knowing they were doing the exact opposite of what the OGC stands for. And now the empire strikes back.

laztozlas

There is a dArc force awakening that threatens the peace within the LiDAR community. ESRI has just released a new tool (see above) that enslaves point clouds by converting them from the open LAZ format to the near-identical but closed “LAZ clone” that they call “zLAS” or “Optimized LAS”. This comes just a few months after an entire nation‘s LiDAR was enslaved in this proprietary format. We have repeatedly warned about the ramifications of locking up Petabytes of LiDAR data in a closed format that is controlled by a single vendor.

ESRI is one of the largest GIS training organizations. By instructing LiDAR novices to “optimize” their LiDAR files and pushing LiDAR providers to switch from open LAS or open LAZ to closed zLAS, they effectively destroy the current success of our open formats. ESRI’s command of the GIS market can – little by little – turn their own proprietry format into the dominant way in which LiDAR point clouds are stored. Then we loose our open exchange formats. Hence, ESRI’s proprietary format threatens all that we have achieved with LAS (and LAZ) over the past years: compatible LiDAR data exchange and incredible LiDAR software interoperability.

ESRI is now escalating the LiDAR format wars. Join the rebellion, Jedis: download your lazer sabers and liberate some LiDAR.

This is not an anti-ESRI campaign. For the past three years we have been trying to resolve this situation. We have repeatedly reached out to ESRI to prevent format fragmentation. We have repeatedly offered to create a joint compressed format. We have plead, begged, and bargained for the sake of our LiDAR community and the sake of their ArcGIS user community not to promote a near-identical yet incompatible way for storing massive amounts of point cloud data.

Two ASPRS awards for “pit-free” CHM algorithm

PRESS RELEASE (for immediate release)
July 29, 2015
rapidlasso GmbH, Gilching, Germany

The paper “Generating Pit-free Canopy Height Models from Airborne LiDAR” co-authored by rapidlasso GmbH and published in the September 2014 issue of PE&RS (the journal of the ASPRS) was awarded twice at the IGTF 2015 – ASPRS Annual Conference in Tampa, Florida last May. The paper took home the John I. Davidson President’s Award for Practical Papers (2nd Place) as well as the Talbert Abrams Award (2nd Honorable Mention).

The John I. Davidson President’s Award for Practical Papers (2nd Place).

The “pit-free” CHM paper wins the John I. Davidson President’s Award for Practical Papers (2nd Place) and the Talbert Abrams Award (Second Honorable Mention).

The “pit-free” CHM paper is joint work with Anahita Khosravipour, Andrew K. Skidmore, Tiejun Wang, and Yousif A. Hussin of ITC and University of Twente. It describes a technique that can create raster Canopy Height Models (CHMs) without the so called “pits” that tend to hamper subsequent extraction of individual tree attributes such as number, location, height, and crown diameter. The paper uses data measured in the field by ITC researchers to show that “pit-free” CHMs significantly lower the commission and omission errors in single tree detection.

Side-by-side comparison of a "standard" CHM and a "pit-free" CHM.

Visual side-by-side comparison of a “standard” versus a “pit-free” CHM.

The “pit-free” CHM algorithm can easily be implemented with LAStools either by modifying an available batch script or by executing the LAStools Pipelines distributed with the toolboxes for ArcGIS and QGIS. A detailed blog article that compares various different methods for creating CHMs is available via the Web pages of rapidlasso GmbH.

We at rapidlasso GmbH like to especially congratulate the main author, Ms. Anahita Khosravipour, who managed to get two awards with her very first academic publication. Those who like our “pit-free” CHM algorithm will probably also love the new technique that our team will introduce later this year at SilviLaser 2015 in France.

About rapidlasso GmbH:
Technology powerhouse rapidlasso GmbH specializes in efficient LiDAR processing tools that are widely known for their high productivity. They combine robust algorithms with efficient I/O and clever memory management to achieve high throughput for data sets containing billions of points. The company’s flagship product – the LAStools software suite – has deep market penetration and is heavily used in industry, government agencies, research labs, and educational institutions. Visit http://rapidlasso.com for more information.

Five Myths about LAS, LAZ, and “Optimized LAS”

The Open Letter by OSGeo was delivered to ESRI, OGC, and the ASPRS last week and the initial reponses – including an email from ESRI’s founder and president Jack Dangermond – are very encouraging. Attendees of last weeks’ ASPRS conference were discussing how to respond to ESRI’s proprietary “Optimized LAS” that threatens the achievements of the open LiDAR formats LAS and LAZ that the community has been using for many years now. Below five clarifications to five wrong statements overheard at these meetings:

1) Martin’s “LAZ” format is also proprietary.

Wrong. LAZ – just like LAS – is an open format. LAZ is defined by a well commented open reference implementation in C/C++ and described in a PE&RS paper published in February 2013. LAS is defined via a specification document but has no reference implementation. Both can be freely used by anyone and (re-)implemented on any operating system and in any programming language. For example, there is now a javascript version of LAZ that someone else created.

2) We have no argument because ESRI provides a free API for “Optimized LAS”.

Wrong. “Optimized LAS” can only be used via the mechanism, the programming language, and the operating system of ESRI’s choosing. This is the very definition of “proprietary format”. Here is what Wikipedia says:

A proprietary format is a file format of a company, organization, or individual that contains data that is ordered and stored according to a particular encoding-scheme, designed by the company or organization to be secret, such that the decoding and interpretation of this stored data is only easily accomplished with particular software or hardware that the company itself has developed. The specification of the data encoding format is not released, or underlies non-disclosure agreements.

In contrast an open format is a file format that is published and free to be used by everybody.

3) Martin’s “LAZ” format is only used by LAStools.

Wrong. Large parts of the LiDAR industry embrace LAZ and have added read & write support for the LAZ format using the open source code or the DLL. Examples are QT Modeler, Globalmapper, FME, Fugroviewer, ERDAS IMAGINE, ENVI LiDAR, Bentley Pointools, TopoDOT, FUSION, CloudCompare, Gexel R3, Pointfuse, …and many more. Notable exceptions are ArcGIS and the product line offered by Lewis Graham’s GeoCue group. We maintain an (incomplete) list of software with native LAZ support here.

4) ESRI has engineered “Optimized LAS” for the cloud and “LAZ” cannot compete.

Wrong. The extra functionality in “Optimized LAS” is a simple mash-up of LAZ with spatial indexing LAX, an optional spatial sort, and a few extra statistics. This is why ESRI’s format is also known as the “LAZ clone”. We were able to feature-match these minor engineering changes in an afternoon which – a few days later – resulted in this April Fools’ Day prank. In fact, LAZ has been used “in the cloud” for well over 4 years on OpenTopography – the first and probably the premier Web accessible LiDAR cloud service of our industry. It is also used by many other LiDAR download servers. We maintain an (incomplete) list of portals offering compressed LAZ here.

5) ESRI’s “Optimized LAS” does not prevent people from using LAS.

ESRI is one of the largest GIS training organizations. If they teach hundreds of LiDAR novices to “optimize” their “unoptimized LAS” files while simultaneously lobbying large LiDAR providers into switching from LAS or LAZ to zLAS they will effectively destroy the current success of our open formats. ESRI’s command of the GIS market can – little by little – turn their own proprietry format into the dominant way in which LiDAR point clouds are exchanged. Then we loose our open exchange formats. Hence, ESRI’s proprietary “Optimized LAS” format “threatens” what we have achieved with LAS (and LAZ): open LiDAR data exchange and incredible LiDAR software interoperability.

This is not an anti-ESRI campaign. We hope to work with ESRI to resolve this situation. Below an image and a quote from ESRI’s ArcNews Spring 2011 news letter about the importance of open formats, standards, and specifications …

ESRI: "Esri continues to advocate the need for open access to geographic data and functionality through support for widely adopted and practical standards and specifications. Esri follows an open system strategy for accessing and using geographic data and functionality."

“Esri continues to advocate the need for open access to geographic data and functionality through support for widely adopted and practical standards and specifications. Esri follows an open system strategy for accessing and using geographic data and functionality.” — ArcNews, Spring 2011

The LAS format, the ASPRS, and the “LAZ clone” by ESRI

We are concerned about ESRI’s next moves in forcing yet another proprietary format into wide-spread deployment. Forwarded emails, retold conversations, and personal experiences suggest that sneaky tactics are being used to disrupt the harmony in open LiDAR formats that we have enjoyed for many years.

laz_and_lazclone_smallSome time has passed since we broke the news about the proprietary “LAZ clone” by ESRI. We were expecting the ASPRS to eventually comment on the issue. ESRI is promoting their lock-in product by the name of the open LAS specification (for which the ASPRS holds the copyright) calling their closed format “Optimized LAS“. We have been asked (in various forums) about the position of the ASPRS on this issue. ESRI’s use of LAS (*) makes it seem as if their “LAZ clone” was somehow an ASPRS thing (as evidenced by Harold’s comment). Despite ESRI’s media-blah-blah about “open and interoperable” they are – once again – luring the geospatial community to fall for a new proprietary format. So far the ASPRS has not released a statement on ESRI’s closed version of the LAS format.

The LAS Working Group (LWG) is part of the Lidar Division of the ASPRS. It has been maintaining the evolving LAS format from its 1.0 version that was (apparently as early as 1998) created by the LiDAR industry’s pioneers and eventually donated to the ASPRS (more recent LAS history is linked here). The good and early decisions of the LWG have created an incredible successful open data exchange standard for discrete LiDAR points that is nowadays supported by practically every software. “Kudos” to the original members for this achievement.

We did not join the LWG until 2011 to help avoid broken compatibility in LAS 1.4. After weathering the following “laser storm of 2011” the working group has been rather quiet. Its most recent activity was in 2013 for tendering the development of an official ASPRS LAS Validation Suite (LVS) that eventually resulted in ‘lasvalidate‘  – an open source LAS validator.

So who is this LWG? And why are they not commenting on such an important controvery like this “LAZ clone” with the seductive name “Optimized LAS”? The latest document on the Web pages of the LAS Working Group (LWG) lists the following people as members:

asprs_lascontrovery_1_members

This list from 2011 is hopelessly out of date, but it should give you an idea of the composition of the LWG. Most likely rapidlasso is still a member of the LWG but it is hard to tell because there have not been any emails recently and because there are no regular meetings. In the past we had some real bad luck with bringing up issues directly with the LWG, so here we go:

Dear ASPRS and LWG,
we are the guardians of the open LiDAR data exchange
specification, the LAS format. What is our response
to the proprietary format called "Optimized LAS" that
is being agressively promoted by ESRI?

Dear concerned ASPRS member,
how would you like your organization to respond now
that a large geospatial company uses its dominance
to push a closed format into the market, sabotaging
the accomplishments of an open data exchange standard
maintained by the ASPRS.

We are worried that ESRI – beyond lobbying agencies to convert their current holdings to the proprietary “LAZ clone” or to tender future deliveries in the closed zLAS format – may also be trying to form strategic alliances with vendors of popular LiDAR processing packages. Many of these vendors are also members of the LAS Working Group and would be in a conflict-of-interest if they were to “sell out” to ESRI’s lock-in ambitions.

You can imagine the red flag that went up a few days ago when we saw a technical comment on a LinkedIn post by Gene Roe that suggested intimate familiarity with the capabilities of the “LAZ clone” by Lewis Graham who has been leading the LAS effort since 1998 and who is the chair of the LAS Working Group. That Lewis’ comment has since been removed did little to calm our worries. As a side note: Gene’s posts being titled “LAS Data Format” further dilutes the difference between open LAS and closed zLAS.

Please inform us (or comment below) about any lobbying you hear about. Given the agressive moves by ESRI – in face of our repeated attempts to reach out – we do not think we can afford to err on the side of caution any longer … (-;

——————-

(*) It is fair to note that our products such as LAStools, LASlib, and LASzip also use the name “LAS”. This is for historic reasons. That is what we called the simple package for reading, writing, and processing LAS files we created back in 2005 for our own research before releasing them as open source in 2007. During our postdoc years at UC Berkeley we did not anticipate that these tools would become so or that we would start a company a few years later …

Keeping ESRI Honest

Friends of LASzip and LAZ, it has come to my attention (from more than one source) that a certain company East of LA has started to more aggressively promote their proprietary LiDAR format known as the “LAZ clone” (more hereherehere and here but also read the comments) by approaching individual stake holders of the LiDAR community. Trying to convince others to drop support of the open source LASzip compression for LiDAR management, server, and storage in favour of the proprietary “LAZ clone”, said company resorts – if needed – to bad-mouthing the LAZ format with made up arguments that attack the suitability of LASzip for “professional” use.

It appears the LiDAR lock-in PR campaign is now in full swing despite of my repeated attempts to reach out to said company for creating a joint LAS 1.4 compressor that avoids fragmenting the LiDAR data market. And I may just be hearing back about a tiny tip of the drink-the-koolaid iceberg PR. This seems to mark the start of the “clone wars” that we at rapidlasso tried to avoid … (-;

laz_clonesOne way to engage the Empire is to draw them out from the darkness into full technical transparency. After all, the only argument for why the “LAZ clone” had to be created was that LAZ lacks a “particular feature” for efficient employment in the cloud, yet the question what this “particular feature” was has always been dodged (because we would have happily added it to LAZ).

How can you help?

  1. Tell us about any “LAZ clone” campaigs you hear about. Especially those targetting stake holders.
  2. Attend LiDAR training events and Webinars by ESRI that may be “teaching” the unsuspecting newbies to lock their LiDAR into the zLAS format as part of a “smart management strategy” and
    – record any arguments made for the “LAZ clone” or against LAZ so we can publically defute them
    – keep them honest by making your unwillingness to drink the koolaid on zLAS known as early as possible.
    – ask tough questions and disrupt with a strong technical critique on the user-unfriendly and vendor lock-in nature of a closed proprietary format soon as they “teach” the LAS to zLAS conversion
  3. Educate folks about the difference between open LAZ and propietary zLAS.

You may comment below (also anonymously) or email us at ‘honestesri@rapidlasso.com‘ if you have any (juicy?) details about “LAZ clone” campaigns or on video or live training courses where it was “taught” to lock-up the LiDAR into zLAS during an initial “optimization” step …

Realizing Open LiDAR with ArcGIS Online and LASzip

Putting aside for a moment our controversy with ESRI over their “uncool” move to create yet another proprietary format whose only benefit is vendor lock-in (read this, this, or this for more on their “LAZ clone“), they do have some neat Web technology. And maybe not all hope is lost and we can rejoin our efforts for a single “native LAS 1.4 compressor” as suggested a few days ago.

But on to the actual story: The Sonoma County Vegetation Mapping and LiDAR Program is using ESRI’s ArcGIS online Web map technology to implement an interactive download portal for open LiDAR and derived products such as contours, building footprints, and elevation rasters.

The five year SonomaVegMap program – a joint program of Sonoma County Open Space and the Sonoma County Water Agency – aims to survey Sonoma County’s topography, physical and biotic features, and diverse plant communities and habitats. The program has already produced several products including county-wide LiDAR data which is now freely available for download as LASzip compressed LAZ files that are linked in increments of watersheds via an ArcGIS online Web map as shown below.

open LiDAR download portal of SonomaVegMap

download portal of SonomaVegMap project using ArcGIS online

Clicking the download icon opens another window (as shown below) where the user can select either the raw LiDAR or the various derivatives (or all) and start downloading.

selecting the ZIP archive

selecting ZIP file with LASzip-compressed LiDAR in LAZ format for download

After about an hour of downloading and a few more minutes of unzipping the 1.4 GB ‘Bodega Head LAZ Files.zip’ file we finally get to see its contents: 103 LAZ files totalling 1.4 GB that decompress to 296,897,792 LiDAR points.

sonomavegmap_lidar_1

A small personal plea: “Please do not put spaces in file names of folder names but use underscores instead!“. Spaces have traditionally been used to separate commands and having them in file or directory names can break batch processing or python scripting. Using the GUI of lasview we load the 103 files and see how this watershed is tiled:

sonomavegmap_lidar_2

The information bar on the right tells us that the LasMonkey software of Watershed Science Inc. has processed points of type 3 (i.e. with GPS time stamp and RGB color) into California State Plane II coordinates and stored them with a resolution of “centi survey feet” (i.e. scale x y z are all 0.01). We pick a horizontal strip and view the points with different visualization options.

(1) RGB (2) intensity (3) classification (4) elevation (5) flightline (6) user data (7) hillshaded ground

(1) RGB (2) intensity (3) classification (4) elevation (5) flightline (6) user data (7) hillshaded TIN of ground points

Everything looks correct although we are not quite sure which values are stored in the ‘user data’ field of each point. Apparently these values more intense for water returns and maybe someone from Watershed Science Inc. will be able to enlighten us. Below a single tile as well as a close-up of one particular area of this tile followed by a lasinfo report. All 103 tiles pass the lasvalidate test with flying colors, nice job guys!

a closer look at tile 'SOCO_0024_121.laz'

a closer look at tile ‘SOCO_0024_121.laz’ of the Bodega Head watershed

close-up with ground and building points triangulated

close-up with ground and building points triangulated

lasinfo report for tile '   '

lasinfo report for tile ‘SOCO_0024_121.laz’

Let me close this article with a 1 minute and 30 second video that summarizes the SonomaVegMap project.