Guidelines for a new ASTM standard defining E57 version 2 file format

There are three parts to this document – housekeeping recommendations and performance improvements, changes to the XML and a review of a number of possible extensions, including compression in a Phase 2 perhaps.

The original standard and software are getting old, and they need to be updated to match the current technology.

1. The first requirement is to support all E57 V1.0 files and E57 V1.0 extensions so as to insure backward compatibility.

2. Changes to E57 Header Section

a. Change the “versionMajor” to “2”. (E2807 7.2)
b. Change the “pageSize” from 1024 to something bigger like 16384 (E2807 6.2.1)
- i. This would increase the performance of E57 with larger chunk sizes.
c. Remove the CRC checksum bytes requirement from all pages. (E2807 6.2.4)
- i. This allows the logical sequences and physical sequences to be the same. This would mean that JPG, PNG, XML and HTML data would be stored in the raw form making it easier to recover and faster to process.
d. Add a duplicate XML section and header block at the end of the file to better support archiving and recovery.
- i. This would help in the recovery of a damaged E57 file.
- ii. The duplicate XML section xmlOffsetDup can be calculated by this formula: xmlOffsetDup = fileLength – xmlLength – sizeof(E57Header);
- iii. Likewise, the E57HeaderDup can be calculated by this formula: E57HeaderDupPosition = fileLength – sizeof(E57Header);

3. Changes to E57 XML Section

a. Add the missing IndexLimits structure for gridded point clouds. This will support restoration of the complete scan.
- i. This extension adds the IndexLimits structure to the data3D structure.
- ii. When the standard defined the IndexBounds structure the standard states that the bounds are defined to be tight to the data given. The standard defines rowMaximum to be "The maximum rowIndex value of any point represented by this IndexBounds object." The problem with this is that it doesn't meet the real world need to know the index limits for structured point clouds. In order to know all the invalid points that occurred outside the indexBounds mim/max bounds, we need to know what the index limits were when it was scanned.
- iii. See missing IndexLimits structure for gridded point clouds under the Issues with ASTM E2807 Standard on libE57.org.
b. Change Cartesian and Spherical Bounds to allow integer and scaled integer types.
- i. The standard only allows double types to be used as the cartesian and spherical bounds coordinates. However, point data can be any type including scaled integer. This mismatch will cause round off errors when comparing the bounds with the point data.
- ii. See Cartesian and Spherical Bounds only allows for type double under the Issues with ASTM E2807 Standard on libE57.org.
c. Change timestamp data precision.
- i. This extension adds a higher precision of time bounds to the data3D structure. The E57 standard defines the timestamp data relative to the GPS “acquisitionStart” field. The problem with using the full GPS time is that it is only accurate to about 1 ms. The common practice is to only use the GPS week time without the GPS week number mixed in order to give a better resolution down to .1 us. This extension gives the minimum and maximum value of the timestamp data in GPS week time using the same accuracy as found in the point cloud timestamp data.
- ii. See the timestamp data doesn’t have enough precision under the Issues with ASTM E2807 Standard on libeE57.org.
d. The standard defined an Image2D::associatedData3DGuid for the Data3D object that was being acquired when the Image2D picture was taken. This allows for a 1 to 1 association between the scan and an image and a 1 to many associations between a single scan and many images. However, the many scans with a single image case is not addressed.

4. Review the E57 extensions

Review the E57 extensions that have been defined so far to see which ones should be part of a new E57 standard. (Note - This has no effect on the standard itself as support for extensions was part of the original standard. However, the E57 committee could add these extensions as part of version 2) See the Public E57 Extensions on libeE57.org.
- a. E57_EXT_surface_normals
- b. E57_LEICA_Camera_Distortion
- c. E57_LEICA_Terrain_Classification
- d. E57_ZF_InfraredTemperatureC
- e. E57_RIEGL_2011
- f. E57_USIBD (integration of PoP (Pictues on Plans))
- g. E57_LIDAR_PRECISION (Not written yet)

5. Add Compression (Phase 2)

a. See the Original ASTM 3D Image Data Format requirements on libE57.org.
b. Change the file extension to Z57 when Phase 2 Compression is used.

6. Changes to the LibE57 reference implementation

a. Fix all outstanding bugs as documented on the libE57.org website.
b. Update to the latest XercesC++ and Visual Studio compiler release.
c. Add the SimpleAPI into the binary release.
d. Remove the BOOST library dependency.
e. Support more platforms like Linux and MacOS

7. Review all of the LIDAR technologies and determine changes to support them.

NOTE: Stan Coleby (stan.coleby@gmail.com), one of the key members of the original team that developed the standard, the open-source software and the reference website (libE57.org) prepared this document with editorial support from Gene Roe (gvr2003@comcast.net) in the fall of 2023. Gene is the LiDAR News Editor and is the chairman of the ASTM E57.11 3D Imaging Data Interoperability subcommittee.

E57Version2Changes_DocumentV5.docx