Saturday, July 10, 2004

LINUX: Linux helps make weather forecasts more accurate

NewsForge | Linux helps make weather forecasts more accurate

Title Linux helps make weather forecasts more accurate
Date 2004.07.07 0:45
Author roblimo

Weather is a tricky thing to predict, but advances in Linux-based High Performance Computing (HPC) are helping climatologists and global weather modelers achieve faster, more accurate results.

Thomas Zacharia, associate lab director at Oak Ridge National Laboratory (ORNL), said the Linux movement has had a strong impact on weather and climate research, as well as other scientific areas requiring vast simulations,"because obviously you're harnessing the collective knowledge of large numbers of people around the world."

Zacharia said ORNL's 456-processor Altix system from SGI running Linux delivers six times better performance than the previous generation of systems from a variety of vendors. Now, Zacharia said, most of the major vendors have delivered successful, high-power computing solutions with Linux to model and study the weather -- SGI's Altix, Cray's Red Storm and IBM's BlueGene among the most prominent. Those systems, he explained, are giving researchers better results than they got from the systems they used to use.

"The advantage of Linux is, operating system plays an important role as you build systems of this scale," Zacharia said. "The ability to efficiently scale the operating system along with the machine is crucial."

Referring to the humongous simulations that are used in climate research, Zacharia said scientists are benefiting from a good balance of increased processing power and memory, better interconnect technology, a sturdier OS -- Linux -- and advances in mathematical algorithms.

"Without a doubt, we're in a wonderful era with a number of purposeful systems and machines that are entirely focused on the scientific and technical market," he said. "I believe computing is going to be the great enabler in science and research in climate because there's no other way."

Usual benefits of Linux

That focus has proven fruitful for SGI, according to manager of environment marketing Jill Matzke, who said HPC users in general are shifting from proprietary systems based on RISC processors and Unix variants to more industry standard-based systems with the main drivers being price-performance and system portability. SGI has used its Altix system to help fuel the transition for the weather and climate community as well as the broader HPC market, according to Matzke.

"Researchers are gaining cost savings and interoperability from volume market components like Linux and Itanium 2 while still enjoying the powerful performance of the SGI Altix," Matzke said. "They benefit from the incredible resources of Intel in delivering top-performing 64-bit processors, which combined with Altix architecture offers superior scaling of the very large and complex models that weather and climate researchers must use."

Matzke explained that with terabytes of data and extremely complex models to deal with, researchers also have trimmed HPC budgets and need to cover all of their needs -- computing, data storage, and visualization -- with those limited funds. SGI offers the compute piece with its servers running Linux or a complete end-to-end data management solution on Linux as well, Matzke said.

"By offering all of these system elements on Linux, SGI leverages open source technology on behalf of our customers in weather and we save them money without any compromise on performance," Matzke said.

SGI is looking to extend its Linux weather forecast by providing data management solutions along with climate research compute power, and soon the Mountain View, California-based company will offer a visualization solution as well, Matzke said.

"The challenge is getting all the necessary system components to work together on Linux, and this is SGI's focus," she said.

Modeling marches on

To tackle some of the toughest problems in HPC today -- climate, as well as life sciences, biotech and nanotech -- researchers need systems that can scale to the data and degree for running "so many years of climate in a day," according to ORNL's Zacharia.

Matzke said because weather and climate scientists are looking for as much compute power any vendor or budget can give them, they look to systems such as the Altix to deliver "truly productive compute cycles that scale real-world problems, not just industry benchmarks."

"This makes it possible for them to complete model runs in minutes or hours where it used to take days and even weeks," Matzke said."And when it comes to production weather forecasting, when time-to-solution is paramount in making timely forecasts, the power of these systems allows forecasters to run higher resolution and more complex models in the alloted time, thus increasing forecast accuracy."

John Parks, deputy division chief of the Advanced Supercomputing Facility at NASA's Ames Research Center, said there is indeed a migration to Linux and other technologies, including Cray's X1, in climate and weather research. Parks said researchers anticipate Cray will release a Linux version of the X1, formerly known as SV2.

"So even there, you may end up with a Linux platform," Parks said.

Proprietary clouds clearing

Parks said the main advantage of Linux over proprietary systems is its ability to run on many different hardware platforms. "However, in the past several years, we have built applications that take advantage of proprietary operating systems, so when you move (to Linux), you have to (port applications)," Parks said. "For the most part, that's a trade off that you're willing to make."

Parks said some scaling issues need to be addressed, and that while Linux delivers flexibility, it may not take advantage of hardware nuances that made proprietary solutions better in some circumstances. Again, though, Parks said computer scientists working with climate researchers and the Linux community manage to make Linux superior in enough other ways to make the open source operating system a good choice.

Parks said input/output is another issue with Linux "that seems to be getting rapidly addressed," and indicated that the challenge now is for Linux to keep pace with other advances in HPC, particularly new processors from AMD and Intel.

"Linux has much to do with where the technology is going in general," Parks said. "If we can have an operating system available that can be readily architected for different hardware, it's better for us, and the speed at which these systems are capable of operating will allow us to scale and get (even more) speed."

Future forecast

Domagoj Podnar, assistant research computer scientist at the environmental research nonprofit Desert Research Institute (DRI), said although his organization's use of Linux came with the use of SGI's Altix, he has long seen the advantage of Linux in clusters to run atmospheric modeling.

DRI, which runs a 13-node, dual-Xeon cluster to run atmospheric models in real time, assists firefighters responding to wildfires and produces forecasts for prescribed agricultural burns. Podnar, who gained familiarity with Linux on workstations, said there is a big push in the industry to run climate models on Linux PC-type computers "because of its cost advantage."

Podnar said a look at the most powerful HPC systems in the world and the prevalence of Linux highlights the operating system's suitability for high-performance computing such as climate and weather research.

"I think it's the way of the future," Podnar said.


1. "Oak Ridge National Laboratory" - http://www.ornl.gov/
2. "Ames Research Center" - http://www.arc.nasa.gov/index.cfm?flash5=true
3. "Desert Research Institute" - http://www.dri.edu/


Post a Comment

<< Home

Get Firefox!