Its almost a rite of passage in physics and astronomy. Scientists spend years scrounging up money to build a fantastic new instrument. Then, when the long-awaited device finally approaches completion, the panic begins: How will they handle the torrent of data?Thats the situation now, at least, with the Square Kilometre Array (SKA), a radio telescope planned for Africa and Australia that will have an unprecedented ability to deliver data -- lots of data points, with lots of details -- on the location and properties of stars, galaxies and giant clouds of hydrogen gas.In a study published in The Astronomical Journal, a team of scientists at the University of Wisconsin-Madison has developed a new, faster approach to analyzing all that data.Hydrogen clouds may seem less flashy than other radio telescope targets, like exploding galaxies. But hydrogen is fundamental to understanding the cosmos, as it is the most common substance in existence and also the stuff of stars and galaxies.As astronomers get ready for SKA, which is expected to be fully operational in the mid-2020s, there are all these discussions about what we are going to do with the data, says Robert Lindner, who performed the research as a postdoctoral fellow in astronomy and now works as a data scientist in the private sector. We dont have enough servers to store the data. We dont even have enough electricity to power the servers. And nobody has a clear idea how to process this tidal wave of data so we can make sense out of it.Lindner worked in the lab of Associate Professor Snezana Stanimirovic, who studies how hydrogen clouds form and morph into stars, in turn shaping the evolution of galaxies like our own Milky Way.In many respects, the hydrogen data from SKA will resemble the vastly slower stream coming from existing radio telescopes. The smallest unit, or pixel, will store every bit of information about all hydrogen directly behind a tiny square in the sky. At first, it is not clear if that pixel registers one cloud of hydrogen or many -- but answering that question is the basis for knowing the actual location of all that hydrogen.People are visually oriented and talented in making this interpretation, but interpreting each pixel requires 20 to 30 minutes of concentration using the best existing models and software. So, Lindner asks, how will astronomers interpret hydrogen data from the millions of pixels that SKA will spew? SKA is so much more sensitive than todays radio telescopes, and so we are making it impossible to do what we have done in the past.In the new study, Lindner and colleagues present a computational approach that solves the hydrogen location problem with just a second of computer time.For the study, UW-Madison postdoctoral fellow Carlos Vera-Ciro helped write software that could be trained to interpret the how many clouds behind the pixel? problem. The software ran on a high-capacity computer network at UW-Madison called HTCondor. And graduate student Claire Murray was our human, Lindner says. She provided the hand-analysis for comparison.Those comparisons showed that as the new system swallows SKAs data deluge, it will be accurate enough to replace manual processing.Ultimately, the goal is to explore the formation of stars and galaxies, Lindner says. Were trying to understand the initial conditions of star formation -- how, where, when do they start? How do you know a star is going to form here and not there?To calculate the overall evolution of the universe, cosmologists rely on crude estimates of initial conditions, Lindner says. By correlating data on hydrogen clouds in the Milky Way with ongoing star formation, data from the new radio telescopes will support real numbers that can be entered into the cosmological models.We are looking at the Milky Way, because thats what we can study in the greatest detail, Lindner says, but when astronomers study extremely distant parts of the universe, they need to assume certain things about gas and star formation, and the Milky Way is the only place we can get good numbers on that.With automated data processing, suddenly we are not time-limited, Lindner says. Lets take the whole survey from SKA. Even if each pixel is not quite as precise, maybe, as a human calculation, we can do a thousand or a million times more pixels, and so that averages out in our favor.
Reference:Autonomous Gaussian Decomposition, Robert R. Lindner et al., Astronomical Journal, April 2015, Vol. 149, No. 4 [http://iopscience.iop.org/1538-3881/149/4,http://arxiv.org/abs/1409.2840%5D.
Please follow SpaceRef on Twitter and Like us on Facebook.
Here is the original post:
Automation Offers Big Solution to Big Data in Astronomy
- I Spy With My Little Eye… - November 7th, 2009 [November 7th, 2009]
- A Crack Opens in the Ethiopian Landscape, Preparing the Way for a New Sea | 80beats - November 8th, 2009 [November 8th, 2009]
- The Politics of Addiction | The Intersection - November 8th, 2009 [November 8th, 2009]
- Finally! An iPhone App That Lets You Track Your Bathroom Habits | Discoblog - November 8th, 2009 [November 8th, 2009]
- Don’t Pack Your Bags Yet—New Planet-Finder Hobbled by Electronic Glitch | 80beats - November 8th, 2009 [November 8th, 2009]
- In Controversial Scent Lineups, a Dog’s Nose Picks Out the Perp | 80beats - November 8th, 2009 [November 8th, 2009]
- Are You a Cognitive Miser? | Cosmic Variance - November 8th, 2009 [November 8th, 2009]
- The Secret Lives and Loves of Great White Sharks | 80beats - November 8th, 2009 [November 8th, 2009]
- Britain’s New Protected Minority: Tree-Huggers | Discoblog - November 8th, 2009 [November 8th, 2009]
- Inspired by Maple Seeds, a Robotic Whirligig Takes To The Skies | Discoblog - November 8th, 2009 [November 8th, 2009]
- New Statesman on Accommodationism | The Intersection - November 8th, 2009 [November 8th, 2009]
- Laser-Powered Robot Climbs to Victory in the Space-Elevator Contest | 80beats - November 8th, 2009 [November 8th, 2009]
- Checking Back In With SEAPLEX | The Intersection - November 8th, 2009 [November 8th, 2009]
- Podcast: An Embarrassment of Genomes | The Loom - November 8th, 2009 [November 8th, 2009]
- A Baby Neutron Star, Swaddled in a Carbon Atmosphere | 80beats - November 8th, 2009 [November 8th, 2009]
- Butterfliiiies… iiinnnn… SPPPAAAAACCCCEEEEE! | Bad Astronomy - November 8th, 2009 [November 8th, 2009]
- My Slate Dialogue with Michael Specter Begins | The Intersection - November 8th, 2009 [November 8th, 2009]
- Musical, Fahrvergnügen-Inspired Staircase Makes Commuters Less Lazy | Discoblog - November 8th, 2009 [November 8th, 2009]
- Laser-Etched Fruit Is an Answer in Search of a Problem | Discoblog - November 8th, 2009 [November 8th, 2009]
- Ares and the carnivals | Bad Astronomy - November 8th, 2009 [November 8th, 2009]
- Toddler Gets a Telescoping, Prosthetic Arm Bone That Grows With Him | 80beats - November 8th, 2009 [November 8th, 2009]
- Neutered HIV Virus Delivers Treatment to Fatally Ill Boys | 80beats - November 8th, 2009 [November 8th, 2009]
- Specter’s First Reply: Denialism Kills People | The Intersection - November 8th, 2009 [November 8th, 2009]
- LRO sees a Moonslide | Bad Astronomy - November 8th, 2009 [November 8th, 2009]
- Al Gore’s New Book: A Focus on Solutions | The Intersection - November 8th, 2009 [November 8th, 2009]
- The Universe Has Us in Its Crosshairs | Bad Astronomy - November 8th, 2009 [November 8th, 2009]
- Makers of Universes | Cosmic Variance - November 8th, 2009 [November 8th, 2009]
- Can Your Pet Catch & Spread Swine Flu? Yes, If Your Pet’s a Ferret | 80beats - November 8th, 2009 [November 8th, 2009]
- Droid 2.0 Vs iPhone | The Intersection - November 8th, 2009 [November 8th, 2009]
- Tangled Bank News: An Excerpt and More | The Loom - November 8th, 2009 [November 8th, 2009]
- LHC Shut Down By Wayward Baguette, Dropped by Bird Saboteur | Discoblog - November 8th, 2009 [November 8th, 2009]
- Careidolia | Bad Astronomy - November 8th, 2009 [November 8th, 2009]
- Slate Reply to Specter Up–We Need a National Dialogue on Synthetic Biology | The Intersection - November 8th, 2009 [November 8th, 2009]
- Pray this doesn’t get passed | Bad Astronomy - November 8th, 2009 [November 8th, 2009]
- You Can’t Make This Stuff Up | Cosmic Variance - November 8th, 2009 [November 8th, 2009]
- Mother Tongue, Indeed: Newborn’s Cries Mimic Mama’s Accent | 80beats - November 8th, 2009 [November 8th, 2009]
- Ripped From the Journals: The Biggest Discoveries of the Week | 80beats - November 8th, 2009 [November 8th, 2009]
- Alternative Landscapes | The Loom - November 8th, 2009 [November 8th, 2009]
- Can an iPhone App Decipher Your Baby’s Cries? | Discoblog - November 8th, 2009 [November 8th, 2009]
- Reminder: Carl Sagan Day | Bad Astronomy - November 8th, 2009 [November 8th, 2009]
- Are There Pesticides in Your Soup? Dunk a Pollution Dipstick to Find Out. | 80beats - November 8th, 2009 [November 8th, 2009]
- Log in and Join the Conference - November 8th, 2009 [November 8th, 2009]
- Conference Ends - November 8th, 2009 [November 8th, 2009]
- Conference Archive Opens - November 8th, 2009 [November 8th, 2009]
- Galaxy Zoo - November 8th, 2009 [November 8th, 2009]
- .Astronomy 2009 Dates - November 8th, 2009 [November 8th, 2009]
- .Astronomy 2009: Programme and venue details - November 8th, 2009 [November 8th, 2009]
- .Astronomy Gets Some IYA Love - November 8th, 2009 [November 8th, 2009]
- 2009 Posters and Imagery - November 8th, 2009 [November 8th, 2009]
- 2009 Sponsors - November 8th, 2009 [November 8th, 2009]
- When in Holland… - November 8th, 2009 [November 8th, 2009]
- The WHAT Cloud? - November 8th, 2009 [November 8th, 2009]
- The Jewel Box - November 8th, 2009 [November 8th, 2009]
- Happy Halloween! - November 8th, 2009 [November 8th, 2009]
- Ares 1-X Launch - November 8th, 2009 [November 8th, 2009]
- Confessions of an Alien Hunter - November 8th, 2009 [November 8th, 2009]
- LRO Spies Apollo 17 Site - November 8th, 2009 [November 8th, 2009]
- Mercury in Color - November 8th, 2009 [November 8th, 2009]
- Hubble and M83 - November 8th, 2009 [November 8th, 2009]
- Cassini Flyby of Enceladus - November 8th, 2009 [November 8th, 2009]
- Preserving A Moth [Science Tattoo] | The Loom - December 12th, 2009 [December 12th, 2009]
- Another Russian rocket spiral lights up the sky | Bad Astronomy - December 12th, 2009 [December 12th, 2009]
- A (Very Gentle) Riddle to Complete Your Saturday - December 12th, 2009 [December 12th, 2009]
- Darwin Gets Swine Flu: The YouTube Edition | The Loom - December 12th, 2009 [December 12th, 2009]
- Happy Slothy Holidays | The Loom - December 12th, 2009 [December 12th, 2009]
- Jetting to Copenhagen | The Intersection - December 12th, 2009 [December 12th, 2009]
- Michael Gerson Attempts Thoughtfulness on “ClimateGate,” Then Gives it Up | The Intersection - December 13th, 2009 [December 13th, 2009]
- Incredible VISTA of the cosmos | Bad Astronomy - December 13th, 2009 [December 13th, 2009]
- Bundle up Sunday Night to Watch the Geminid Meteor Shower | 80beats - December 13th, 2009 [December 13th, 2009]
- “ClimateGate” a PR Disaster That Will Be “Taught in University Communications Courses” | The Intersection - December 13th, 2009 [December 13th, 2009]
- Is Google the Guardian Angel of Rainforests? | 80beats - December 13th, 2009 [December 13th, 2009]
- La ciencia es importante. Una vez mas. | Bad Astronomy - December 13th, 2009 [December 13th, 2009]
- Sensenbrenner Pulls an Inhofe, Asserts Global Warming is an “International Conspiracy” | The Intersection - December 13th, 2009 [December 13th, 2009]
- Let Kids Eat Dirt: Over-Cleanliness Linked to Heart Disease | 80beats - December 13th, 2009 [December 13th, 2009]
- What Are The Best Science Papers Of The Past Decade? | The Intersection - December 13th, 2009 [December 13th, 2009]
- Finally! Math Shows How to Cut Evenly Sized Pizza Slices | Discoblog - December 13th, 2009 [December 13th, 2009]
- Half-baked math | Bad Astronomy - December 13th, 2009 [December 13th, 2009]
- Can “Biological Passports” Save Sports From Doping? | 80beats - December 13th, 2009 [December 13th, 2009]
- Aiiiieeee! Slow down! | Bad Astronomy - December 13th, 2009 [December 13th, 2009]
- Weekly News Roundup: Bad Headlines, Martian moons, and Rotating Houses | Discoblog - December 13th, 2009 [December 13th, 2009]