The National Aeronautics and Space Administration’s Center for Climate Simulation (NCCS) processes reams of data that are as dense as music playlists containing 190,000 years’ worth of songs.
To harness its high performance computing (HPC) and data analysis needs, NCCS used its existing servers to create the Advanced Data Analytics Platform (ADAPT), its own private cloud. ADAPT’s capabilities include persistent data services, databases, remote visualization tools, and fast disk nodes.
“What we’ve done is literally taken decommissioned high-performance computing servers and moved them into cloud world. These are 3- and 4-year-old HPC servers, but we have about 5,000 cores,” said Dan Duffy, head of NCCS. “Effectively, the compute is free. It’s already been paid for. It just happens to be 3- and 4-year-old technology.”
Duffy, who spoke at FedInsider’s “Pushing the Boundaries: High Performance Computing in Government” webinar Feb. 8, said ADAPT is helpful for taking large amounts of input data and creating small output data. For example, NCCS used land satellite data to examine water level changes in Hay Lake and Beaver Hill Lake, both located in Canada, from 2001 to 2011. ADAPT processed 20 terabytes of data from the 10-year span to create four maps depicting the water-level changes.
ADAPT is also useful for reducing data friction, according to Duffy. He defined data friction as “how hard it is for you to use data, find data, and move data around.” He used the example of asking a coworker to re-send an email because it has been lost under a mountain of other emails to characterize a big data friction problem.
Another example of data friction, Duffy said, is charging someone with downloading NASA’s moderate resolution imaging spectroradiometer (MODIS) data without use of the science cloud. He said this task would take about nine months. However, with the use of the science cloud, accessing the same data would take one to four days.
“That gives more time to ‘science.’ We created an agile environment within ADAPT,” Duffy said. “This really reduces that steep learning curve for users to get into HPC systems.”
Duffy is stationed at the Goddard Space Flight Center in Greenbelt, Md. His office specializes in weather and climate data, and uses the Goddard Earth Observing System Model to collect information. Using the Goddard model, he and his team can view not only precipitation across the world, but also fires, dust storms, sea salt masses, and aerosol emanating from cities.
He stated one of his ongoing projects is counting all the trees in sub-Saharan Africa.
“When you’re pushing research, you’ve got to push as hard as you can,” Duffy said.
