PASADENA, Calif., Oct. 18 (UPI) -- NASA says new strategies will be needed to manage the ever-increasing flow of large and complex data streams from the agency's many space missions.
Dozens of missions pour in data every day like rushing rivers -- data that need to be stored, indexed and processed so spacecraft engineers, scientists and people across the globe can use the data to understand Earth and the universe beyond, the agency said.
For NASA missions, hundreds of terabytes -- one terabyte is equivalent to the information printed on 50,000 trees worth of paper -- are gathered every hour, creating what the technology community dubs "big data."
"Scientists use big data for everything from predicting weather on Earth to monitoring ice caps on Mars to searching for distant galaxies," Eric De Jong of NASA's Jet Propulsion Laboratory in Pasadena, Calif., said.
De Jong is the principal investigator for NASA's Solar System Visualization project, which converts NASA mission science into visualization products that researchers can use.
"We are the keepers of the data, and the users are the astronomers and scientists who need images, mosaics, maps and movies to find patterns and verify theories," he said.
Scientists face three challenges in dealing with the huge amounts of data from space missions, he said -- storage, processing and access.
Rather than build more hardware for storage, engineers are developing creative software tools to better store the information, such as "cloud computing" techniques and automated programs for extracting data.
For processing, JPL has been increasingly turning to open-source software, creating improved data processing tools for space missions.
"We don't need to reinvent the wheel," said Chris Mattmann, a principal investigator for JPL's big-data initiative. "We can modify open-source computer codes to create faster, cheaper solutions."
Read more:
NASA strives to tame 'big data' flowing in from dozens of missions