NASA, Esri speed delivery of cloud-based imagery data
File formats and compression algorithms. Boring stuff, eh?
How about high-res satellite photos delivered to your desktop in a flash?
NASA and Esri have combined their technical expertise in the former to deliver the latter. Even better, they’ve given the new technology to the community. In late December 2015, the two space agency and GIS mapping powerhouse announced they are sharing with the public a jointly developed raster file format and a unique compression program designed to handle large amounts of mission-critical imagery data.
According to Peter Becker, Esri’s imagery product manager, the result is that imagery can be moved to and from the cloud much, much faster. “In comparison to things like JPEG2000, it’s something like 10 to 15 times faster,” he said.
The file format is MRF -- or meta raster format -- which was originally developed at NASA.
“We’ve always had a concern over how we access large volumes of imagery, especially from cloud storage,” Becker said, noting that storage-area networks and network-attached storage don’t scale well. “We realized we had to make use of the elasticity and availability of cloud storage, but we couldn’t connect our servers directly to cloud storage because our servers are expecting file shares. So we were trying to work out how to resolve that.”
Fortunately, Becker related, a former NASA programmer who had worked on the MRF format had recently taken a position at Esri suggested that the architecture of MRF might be just what the team was looking for.
“As we looked more and the MRF format we realized that what they had invented there was actually really simple,” Becker said, adding that “simple scales and complex fails.”
According to Becker, MRF’s special power is that it breaks files into three parts that can be cached separately, making it especially well suited to cloud storage. “Since the files are broken up, we can bring [down] the metadata files and store those locally,” he said. Users can examine data on file contents and download the data-heavy portions only when needed. “We’re minimizing the number of requests that are going back and forth between the servers and storage,” Becker said.
The other challenge the team faced was coming up with a compression strategy appropriate for critical geolocation data. While most “lossy” compression algorithms for imagery throw out data that the human eye won’t miss -- hence, the “lossy” tag -- compressing elevation data presents problems. “The problem with lossy compression is that you end up not knowing the accuracy of your results,” Becker said. “If you’re flying a plane in the mountains and your elevation data is off by 20 meters, that is critical.”
The answer, developed at Esri, was LERC -- limited error raster compression. LERC allows the analyst to define a tolerance for compressing a file. One might, for example, instruct LERC to compress an elevation file as much as it can while ensuring accuracy within 10 centimeters.
As a bonus, Becker said that the team soon realized that LERC, which was patented last year, not only worked well on elevation files, it also worked very well in compressing high-bit-depth satellite imagery.
Using LERC within the MRF file format, users not only get faster performance but also lower storage requirements. Becker estimated that using the technology with cloud storage costs approximately one-third as much as traditional file-based enterprise storage. “This can result in very significant cost reductions for organizations with large volumes of imagery,” Becker said.
In addition to releasing the technologies to the public, Becker says Esri is continuing to work on improving them by developing better application programming interfaces. The team is also looking to incorporate new features in the MRF format, including versioning of imagery and time-tracking stamps.
Editor's note: This article was changed Jan. 27 to correct the definition of acronym LERC.
Posted by Patrick Marshall on Jan 20, 2016 at 3:16 PM