The dArc Force Awakens: ESRI escalates LiDAR format war

The empire has not changed their evil ways, despite an encouraging email from ESRI’s founder and president Jack Dangermond in response to the Open Letter by the OSGeo that was delivered to ESRI, OGC, and the ASPRS. Facing an incredible backlash by the LiDAR community over the release of their “LAZ clone” there was a new hope that unnecessary format fragmention could be avoided by working together within the Point Cloud Domain Working Group of the OGC. In fact only one thing happened: ESRI went silent on the controversy. They temporarily stopped promoting their “LAZ clone” and focused on locking in more content.

dArc_force_awakens

The message of the rebellion has been consistent and clear like in these two videos from the TC meeting of the OGC in Nottingham and the ASPRS side bar in Reno: a roadmap forward to avoid format fragmentation by exploiting the “natural break” in the format due to LAS 1.4. But there was zero technical contribution from ESRI during the past three PC-DWG meetings of the OGC. The slide sets that bored the audiences in Boulder and in Nottingham were not meant to contribute but merely stalled for time. Recently in Sydney ESRI was awefully quiet, knowing they were doing the exact opposite of what the OGC stands for. And now the empire strikes back.

laztozlas

There is a dArc force awakening that threatens the peace within the LiDAR community. ESRI has just released a new tool (see above) that enslaves point clouds by converting them from the open LAZ format to the near-identical but closed “LAZ clone” that they call “zLAS” or “Optimized LAS”. This comes just a few months after an entire nation‘s LiDAR was enslaved in this proprietary format. We have repeatedly warned about the ramifications of locking up Petabytes of LiDAR data in a closed format that is controlled by a single vendor.

ESRI is one of the largest GIS training organizations. By instructing LiDAR novices to “optimize” their LiDAR files and pushing LiDAR providers to switch from open LAS or open LAZ to closed zLAS, they effectively destroy the current success of our open formats. ESRI’s command of the GIS market can – little by little – turn their own proprietry format into the dominant way in which LiDAR point clouds are stored. Then we loose our open exchange formats. Hence, ESRI’s proprietary format threatens all that we have achieved with LAS (and LAZ) over the past years: compatible LiDAR data exchange and incredible LiDAR software interoperability.

ESRI is now escalating the LiDAR format wars. Join the rebellion, Jedis: download your lazer sabers and liberate some LiDAR.

This is not an anti-ESRI campaign. For the past three years we have been trying to resolve this situation. We have repeatedly reached out to ESRI to prevent format fragmentation. We have repeatedly offered to create a joint compressed format. We have plead, begged, and bargained for the sake of our LiDAR community and the sake of their ArcGIS user community not to promote a near-identical yet incompatible way for storing massive amounts of point cloud data.

Five Myths about LAS, LAZ, and “Optimized LAS”

The Open Letter by OSGeo was delivered to ESRI, OGC, and the ASPRS last week and the initial reponses – including an email from ESRI’s founder and president Jack Dangermond – are very encouraging. Attendees of last weeks’ ASPRS conference were discussing how to respond to ESRI’s proprietary “Optimized LAS” that threatens the achievements of the open LiDAR formats LAS and LAZ that the community has been using for many years now. Below five clarifications to five wrong statements overheard at these meetings:

1) Martin’s “LAZ” format is also proprietary.

Wrong. LAZ – just like LAS – is an open format. LAZ is defined by a well commented open reference implementation in C/C++ and described in a PE&RS paper published in February 2013. LAS is defined via a specification document but has no reference implementation. Both can be freely used by anyone and (re-)implemented on any operating system and in any programming language. For example, there is now a javascript version of LAZ that someone else created.

2) We have no argument because ESRI provides a free API for “Optimized LAS”.

Wrong. “Optimized LAS” can only be used via the mechanism, the programming language, and the operating system of ESRI’s choosing. This is the very definition of “proprietary format”. Here is what Wikipedia says:

A proprietary format is a file format of a company, organization, or individual that contains data that is ordered and stored according to a particular encoding-scheme, designed by the company or organization to be secret, such that the decoding and interpretation of this stored data is only easily accomplished with particular software or hardware that the company itself has developed. The specification of the data encoding format is not released, or underlies non-disclosure agreements.

In contrast an open format is a file format that is published and free to be used by everybody.

3) Martin’s “LAZ” format is only used by LAStools.

Wrong. Large parts of the LiDAR industry embrace LAZ and have added read & write support for the LAZ format using the open source code or the DLL. Examples are QT Modeler, Globalmapper, FME, Fugroviewer, ERDAS IMAGINE, ENVI LiDAR, Bentley Pointools, TopoDOT, FUSION, CloudCompare, Gexel R3, Pointfuse, …and many more. Notable exceptions are ArcGIS and the product line offered by Lewis Graham’s GeoCue group. We maintain an (incomplete) list of software with native LAZ support here.

4) ESRI has engineered “Optimized LAS” for the cloud and “LAZ” cannot compete.

Wrong. The extra functionality in “Optimized LAS” is a simple mash-up of LAZ with spatial indexing LAX, an optional spatial sort, and a few extra statistics. This is why ESRI’s format is also known as the “LAZ clone”. We were able to feature-match these minor engineering changes in an afternoon which – a few days later – resulted in this April Fools’ Day prank. In fact, LAZ has been used “in the cloud” for well over 4 years on OpenTopography – the first and probably the premier Web accessible LiDAR cloud service of our industry. It is also used by many other LiDAR download servers. We maintain an (incomplete) list of portals offering compressed LAZ here.

5) ESRI’s “Optimized LAS” does not prevent people from using LAS.

ESRI is one of the largest GIS training organizations. If they teach hundreds of LiDAR novices to “optimize” their “unoptimized LAS” files while simultaneously lobbying large LiDAR providers into switching from LAS or LAZ to zLAS they will effectively destroy the current success of our open formats. ESRI’s command of the GIS market can – little by little – turn their own proprietry format into the dominant way in which LiDAR point clouds are exchanged. Then we loose our open exchange formats. Hence, ESRI’s proprietary “Optimized LAS” format “threatens” what we have achieved with LAS (and LAZ): open LiDAR data exchange and incredible LiDAR software interoperability.

This is not an anti-ESRI campaign. We hope to work with ESRI to resolve this situation. Below an image and a quote from ESRI’s ArcNews Spring 2011 news letter about the importance of open formats, standards, and specifications …

ESRI: "Esri continues to advocate the need for open access to geographic data and functionality through support for widely adopted and practical standards and specifications. Esri follows an open system strategy for accessing and using geographic data and functionality."

“Esri continues to advocate the need for open access to geographic data and functionality through support for widely adopted and practical standards and specifications. Esri follows an open system strategy for accessing and using geographic data and functionality.” — ArcNews, Spring 2011

New LASliberator “frees” LiDAR from Closed Format

PRESS RELEASE (for immediate release)
April 20, 2015
rapidlasso GmbH, Gilching, Germany

The latest product by rapidlasso GmbH – creators of LAStools and LASzip – is an open source tool aiming to liberate LiDAR points locked-up in proprietary “Optimized LAS” – a highly controversial, closed LiDAR format. The new LASliberator can be downloaded here. It comes as both, a simple command line tool for scripting and with an easy-to-use graphical interface.

The GUI version of the LASliberaor has a simple and easy-to-use interface.

The GUI of the “LASliberator” has a simple, easy-to-use interface.

The LASliberator reads LiDAR points from closed “Optimized LAS” files that use the “.zlas” extension and converts them to open ASPRS LAS files that use the “.las” extension. Alternatively, the points can be stored to compressed LAZ files – using the open source LASzip compressor – that use the “.laz” extension. In addition, the tool creates tiny spatial indexing files that use the “.lax” extension. These can then be exploited for accelerated area-of-interest queries via open source LASindex when using LAStools or the latest version of the LASzip DLL.

Note that the LASliberator cannot entirely be open source as it depends on a particular proprietry library. The closed nature of the “Optimized LAS” format does not allow for a full open source implementation. It is therefore not possible to port the LASliberator to other operating systems or into other programming languages.

Selecing open in the GUI pops up a file selection dialogue allowing the user to find the file that is to be set free.

The user can select a file to liberate by pressing “open” in the GUI.

The new LASliberator comes on the heels of an outcry in the community over the LiDAR format fragmentation “Optimized LAS” is creating. It provides an immediate solution to go from closed zLAS to open LAZ for people whose LiDAR got stuck in yet-another-proprietary-format.

About rapidlasso GmbH:
Technology powerhouse rapidlasso GmbH specializes in efficient LiDAR processing tools that are widely known for their high productivity. They combine robust algorithms with efficient I/O and clever memory management to achieve high throughput for data sets containing billions of points. The company’s flagship product – the LAStools software suite – has deep market penetration and is heavily used in industry, government agencies, research labs, and educational institutions. Visit http://rapidlasso.com for more information.

The LAS format, the ASPRS, and the “LAZ clone” by ESRI

We are concerned about ESRI’s next moves in forcing yet another proprietary format into wide-spread deployment. Forwarded emails, retold conversations, and personal experiences suggest that sneaky tactics are being used to disrupt the harmony in open LiDAR formats that we have enjoyed for many years.

laz_and_lazclone_smallSome time has passed since we broke the news about the proprietary “LAZ clone” by ESRI. We were expecting the ASPRS to eventually comment on the issue. ESRI is promoting their lock-in product by the name of the open LAS specification (for which the ASPRS holds the copyright) calling their closed format “Optimized LAS“. We have been asked (in various forums) about the position of the ASPRS on this issue. ESRI’s use of LAS (*) makes it seem as if their “LAZ clone” was somehow an ASPRS thing (as evidenced by Harold’s comment). Despite ESRI’s media-blah-blah about “open and interoperable” they are – once again – luring the geospatial community to fall for a new proprietary format. So far the ASPRS has not released a statement on ESRI’s closed version of the LAS format.

The LAS Working Group (LWG) is part of the Lidar Division of the ASPRS. It has been maintaining the evolving LAS format from its 1.0 version that was (apparently as early as 1998) created by the LiDAR industry’s pioneers and eventually donated to the ASPRS (more recent LAS history is linked here). The good and early decisions of the LWG have created an incredible successful open data exchange standard for discrete LiDAR points that is nowadays supported by practically every software. “Kudos” to the original members for this achievement.

We did not join the LWG until 2011 to help avoid broken compatibility in LAS 1.4. After weathering the following “laser storm of 2011” the working group has been rather quiet. Its most recent activity was in 2013 for tendering the development of an official ASPRS LAS Validation Suite (LVS) that eventually resulted in ‘lasvalidate‘  – an open source LAS validator.

So who is this LWG? And why are they not commenting on such an important controvery like this “LAZ clone” with the seductive name “Optimized LAS”? The latest document on the Web pages of the LAS Working Group (LWG) lists the following people as members:

asprs_lascontrovery_1_members

This list from 2011 is hopelessly out of date, but it should give you an idea of the composition of the LWG. Most likely rapidlasso is still a member of the LWG but it is hard to tell because there have not been any emails recently and because there are no regular meetings. In the past we had some real bad luck with bringing up issues directly with the LWG, so here we go:

Dear ASPRS and LWG,
we are the guardians of the open LiDAR data exchange
specification, the LAS format. What is our response
to the proprietary format called "Optimized LAS" that
is being agressively promoted by ESRI?

Dear concerned ASPRS member,
how would you like your organization to respond now
that a large geospatial company uses its dominance
to push a closed format into the market, sabotaging
the accomplishments of an open data exchange standard
maintained by the ASPRS.

We are worried that ESRI – beyond lobbying agencies to convert their current holdings to the proprietary “LAZ clone” or to tender future deliveries in the closed zLAS format – may also be trying to form strategic alliances with vendors of popular LiDAR processing packages. Many of these vendors are also members of the LAS Working Group and would be in a conflict-of-interest if they were to “sell out” to ESRI’s lock-in ambitions.

You can imagine the red flag that went up a few days ago when we saw a technical comment on a LinkedIn post by Gene Roe that suggested intimate familiarity with the capabilities of the “LAZ clone” by Lewis Graham who has been leading the LAS effort since 1998 and who is the chair of the LAS Working Group. That Lewis’ comment has since been removed did little to calm our worries. As a side note: Gene’s posts being titled “LAS Data Format” further dilutes the difference between open LAS and closed zLAS.

Please inform us (or comment below) about any lobbying you hear about. Given the agressive moves by ESRI – in face of our repeated attempts to reach out – we do not think we can afford to err on the side of caution any longer … (-;

——————-

(*) It is fair to note that our products such as LAStools, LASlib, and LASzip also use the name “LAS”. This is for historic reasons. That is what we called the simple package for reading, writing, and processing LAS files we created back in 2005 for our own research before releasing them as open source in 2007. During our postdoc years at UC Berkeley we did not anticipate that these tools would become so or that we would start a company a few years later …

Keeping ESRI Honest

Friends of LASzip and LAZ, it has come to my attention (from more than one source) that a certain company East of LA has started to more aggressively promote their proprietary LiDAR format known as the “LAZ clone” (more hereherehere and here but also read the comments) by approaching individual stake holders of the LiDAR community. Trying to convince others to drop support of the open source LASzip compression for LiDAR management, server, and storage in favour of the proprietary “LAZ clone”, said company resorts – if needed – to bad-mouthing the LAZ format with made up arguments that attack the suitability of LASzip for “professional” use.

It appears the LiDAR lock-in PR campaign is now in full swing despite of my repeated attempts to reach out to said company for creating a joint LAS 1.4 compressor that avoids fragmenting the LiDAR data market. And I may just be hearing back about a tiny tip of the drink-the-koolaid iceberg PR. This seems to mark the start of the “clone wars” that we at rapidlasso tried to avoid … (-;

laz_clonesOne way to engage the Empire is to draw them out from the darkness into full technical transparency. After all, the only argument for why the “LAZ clone” had to be created was that LAZ lacks a “particular feature” for efficient employment in the cloud, yet the question what this “particular feature” was has always been dodged (because we would have happily added it to LAZ).

How can you help?

  1. Tell us about any “LAZ clone” campaigs you hear about. Especially those targetting stake holders.
  2. Attend LiDAR training events and Webinars by ESRI that may be “teaching” the unsuspecting newbies to lock their LiDAR into the zLAS format as part of a “smart management strategy” and
    – record any arguments made for the “LAZ clone” or against LAZ so we can publically defute them
    – keep them honest by making your unwillingness to drink the koolaid on zLAS known as early as possible.
    – ask tough questions and disrupt with a strong technical critique on the user-unfriendly and vendor lock-in nature of a closed proprietary format soon as they “teach” the LAS to zLAS conversion
  3. Educate folks about the difference between open LAZ and propietary zLAS.

You may comment below (also anonymously) or email us at ‘honestesri@rapidlasso.com‘ if you have any (juicy?) details about “LAZ clone” campaigns or on video or live training courses where it was “taught” to lock-up the LiDAR into zLAS during an initial “optimization” step …

Realizing Open LiDAR with ArcGIS Online and LASzip

Putting aside for a moment our controversy with ESRI over their “uncool” move to create yet another proprietary format whose only benefit is vendor lock-in (read this, this, or this for more on their “LAZ clone“), they do have some neat Web technology. And maybe not all hope is lost and we can rejoin our efforts for a single “native LAS 1.4 compressor” as suggested a few days ago.

But on to the actual story: The Sonoma County Vegetation Mapping and LiDAR Program is using ESRI’s ArcGIS online Web map technology to implement an interactive download portal for open LiDAR and derived products such as contours, building footprints, and elevation rasters.

The five year SonomaVegMap program – a joint program of Sonoma County Open Space and the Sonoma County Water Agency – aims to survey Sonoma County’s topography, physical and biotic features, and diverse plant communities and habitats. The program has already produced several products including county-wide LiDAR data which is now freely available for download as LASzip compressed LAZ files that are linked in increments of watersheds via an ArcGIS online Web map as shown below.

open LiDAR download portal of SonomaVegMap

download portal of SonomaVegMap project using ArcGIS online

Clicking the download icon opens another window (as shown below) where the user can select either the raw LiDAR or the various derivatives (or all) and start downloading.

selecting the ZIP archive

selecting ZIP file with LASzip-compressed LiDAR in LAZ format for download

After about an hour of downloading and a few more minutes of unzipping the 1.4 GB ‘Bodega Head LAZ Files.zip’ file we finally get to see its contents: 103 LAZ files totalling 1.4 GB that decompress to 296,897,792 LiDAR points.

sonomavegmap_lidar_1

A small personal plea: “Please do not put spaces in file names of folder names but use underscores instead!“. Spaces have traditionally been used to separate commands and having them in file or directory names can break batch processing or python scripting. Using the GUI of lasview we load the 103 files and see how this watershed is tiled:

sonomavegmap_lidar_2

The information bar on the right tells us that the LasMonkey software of Watershed Science Inc. has processed points of type 3 (i.e. with GPS time stamp and RGB color) into California State Plane II coordinates and stored them with a resolution of “centi survey feet” (i.e. scale x y z are all 0.01). We pick a horizontal strip and view the points with different visualization options.

(1) RGB (2) intensity (3) classification (4) elevation (5) flightline (6) user data (7) hillshaded ground

(1) RGB (2) intensity (3) classification (4) elevation (5) flightline (6) user data (7) hillshaded TIN of ground points

Everything looks correct although we are not quite sure which values are stored in the ‘user data’ field of each point. Apparently these values more intense for water returns and maybe someone from Watershed Science Inc. will be able to enlighten us. Below a single tile as well as a close-up of one particular area of this tile followed by a lasinfo report. All 103 tiles pass the lasvalidate test with flying colors, nice job guys!

a closer look at tile 'SOCO_0024_121.laz'

a closer look at tile ‘SOCO_0024_121.laz’ of the Bodega Head watershed

close-up with ground and building points triangulated

close-up with ground and building points triangulated

lasinfo report for tile '   '

lasinfo report for tile ‘SOCO_0024_121.laz’

Let me close this article with a 1 minute and 30 second video that summarizes the SonomaVegMap project.

new “LAStools Production” and “LAStools Pipelines” for ArcGIS

In what some may call a “good LAZ, bad zLAS” routine, rapidlasso GmbH announces better integration of their LiDAR processing technology into ESRI’s ArcGIS platform with new and improved LAStools toolboxes. This comes merely two days after pranking ESRI with an April Fools’ Day press release that was cheered on by a user community that is growing increasingly tired of ESRI bulldozing their own path instead of working together with them.

The new “LAStools Production” toolbox allows to batch automated LiDAR processing tasks across folders of LAS or LAZ files, which is a necessity for production workflows dealing with the Terabytes of LiDAR data involved in typical real-world projects. The new toolbox allows advanced users to further customize processing with an option for providing “additional command line parameters”. This capability has also been added to the good old LAStools toolbox, which is the better choice when operating on only a single LAS or LAZ file at a time.

The new "LAStools Production" toolbox and built-upon the new "LAStools Pipelines" models for batch processing entire folders worth of LiDAR.

The new “LAStools Production” toolbox shown here alongside some example “LAStools Pipelines” models that combine the new production tools to batch process entire folders worth of raw LiDAR flightlines into final products.

Built-upon the “LAStools Production” toolbox are some example “LAStools Pipelines” using the ArcGIS Model Builder that combine the new production tools to batch process entire folders worth of raw LiDAR flightlines into final products. Advanced users may find it more elegant to use Python scripting to combine the batch processing modules from the “LAStools Production” toolbox into more complex workflows. The “LAStools Pipelines” shown below were created in ArcGIS 9.3 and can be downloaded as part of the LAStools distribution (20 MB). Some LAStools Pipelines may not load due to incompatibilities between different versions of ESRI’s ArcGIS Model Builder. Please contribute your own pipelines or share modifications that work for other versions of ArcGIS. Below some screen shots of the six “LAStools Pipelines” that are included as an example in today’s release of LAStools (version 140403). One of them implements the algorithm for constructing “pit-free” CHMs that was presented by Khosravipour et. al at Silvilaser 2013 (find poster and abstract here).