Supercomputing, with a few exceptions, is a shared resource that is allocated to users in a particular field or geography to run their simulations and models on systems that are much larger than they might otherwise be able to buy on their own. Call it a conservation of core-hour-dollars that allows a faster time to model in exchange for limited access.
So it is with the Norddeutschem Verbund fr Hoch- und Hchstleistungsrechnen (HLRN) supercomputing alliance in Northern Germany. The HLRN consortium, which provides calculating oomph for the German federal states of Berlin, Brandenburg, Bremen, Hamburg, Mecklenburg-Western Pomerania, Lower Saxony, and Schleswig-Holstein, has used a variety of different architectures from different vendors over the past several decades, and as such is representative of mainstream HPC shops that, as we pointed out recently, comprise the majority of the revenue stream in the HPC sector and account for thousands of HPC facilities worldwide. HLRN in particular has a very large number of university and research institution users, at close to 200, all jockeying for time on the system, so adding capacity makes the lines a bit shorter, at least in theory.
The second phase of the HLRN-IV supercomputer, known by the nickname Lise after Lise Meitner, an Austrian-Swedish physicist who was one of the discoverers of nuclear fission in 1939, has fired up recently, and the machine is noteworthy for a few reasons. First, Atos is the prime contractor on the machine, and second, it is based on the doubled-up Cascade Lake-AP Xeon SP-9200 Platinum processors that Intel launched last April and that are employed in custom enclosures that Intel itself manufactures.
Since its founding in 2001, the HLRN consortium has operated a distributed system across two datacenters; one is usually at the Zuse Institute Berlin and the other has been located at Leibniz University in Hannover or at the University of Gottingen. The initial HLRN-I system, which was called Hanni and Berni across its two halves, was comprised each of a 16 node cluster of IBMs RS/6000 p690 servers based on its dual-core Power4 processors, which debuted that year. The p690 machines had 32 sockets and 64 GB of main memory each and were connected by a proprietary federation interconnect that IBM created for its parallel NUMA systems. This HLRN-I machine had 26 TB of disk capacity and had a peak performance of 2 teraflops at 64-bit double precision. You can get a graphics card with way more floating point performance these days, and it fits in your hand instead of taking up two datacenters.
In 2008, these systems were upgraded wit a pair of Altix ICE supercomputers from Silicon Graphics in Berlin and Hannover, called Bice and Hice naturally. This system had a mix of NUMA and scale-out nodes. The NUMA nodes were comprised of a mix of two-socket Altix XE 250 nodes and two-socket Altix UV 1000 nodes using a mix of Xeon processors from Intel (four-core and eight-core chips with fatter memory) and the NUMAlink5 interconnect to share the memory across the 2,816 cores and 12.5 TB of main memory across the 200 nodes in the machine. The regular, scale-out part of each side of the HLRN-II system had a mix of two generations of Xeon processors across its 10,240 cores in 1,280 nodes and a total of 12.1 TB of main memory. Add it all up and the HLRN-II machine had 124.76 teraflops of double precision floating point calculating capacity; this was balanced out by an 810 TB Lustre parallel file system.
Enter HLRN-III in 2013, which we wrote about five years later. This machine, which cost $39 million and which was built in phases like prior systems using a mix of generations. In this case, by Cray based on its Cascades XC30 and XC40 system designs and their Aries interconnect. The HLRN-III systems were nicknamed Konrad and Gottfried and they each used a mix of Ivy Bridge and Haswell Xeon processors, with the Berlin system having a total of 1,872 nodes with 44,928 cores and 117 TB of memory yielding a peak performance of 1.4 petaflops and the University of Leibniz (which is where the Gottfried name comes from, the mathematician and co-creator of calculus) had a total of 1.24 petaflops of oomph and 105 TB of memory across its 1,680 nodes and 40,320 cores. Each machine had a 3.7 PB Lustre file system and a 500 TB GPFS file system.
With the HLRN-IV system, the two halves are not just a little bit different, but really distinct systems that were installed at different times. The Emmy system at the University of Gottingen, which was operational in October 2018, was named after groundbreaking German mathematician Amalie Emmy Noether, who blazed a trail for women in that field as much as Meitner did in physics. The Emmy system at Gottingen had 449 nodes, with 448 of them having just Skylake Xeon SP-6148 Gold processors and one of them having four Volta Tesla V100 GPU accelerators from Nvidia added. Not counting that GPU-accelerated system, Emmy had 17,920 cores across its 448 nodes and 93 TB of memory. These nodes were interlinked with a 100 Gb/sec Omni-Path interconnect from Intel, and its performance was never divulged. Presumably Emmy will be upgraded at some point to deliver the expected 16 petaflops of aggregate performance
The Lise half of the system in Berlin, which is just coming online, has significantly more computational power than that initial Emmy partition in Gottingen. This system currently has 1,180 nodes with 113,280 cores in total using a pair of the Xeon AP-9242 Platinum chips per node, which themselves put two 24 core Cascade Lake processors into a single socket for a total of four chips and 96 cores per node. These nodes are also interlinked with 100 Gb/sec Omni-Path interconnect. This machine is noteworthy in that it is showcasing Intels multichip Cascade Lake-AP processors, which have not really dented the attack by the AMD Epyc processors and which are not exactly taking the HPC market by storm. (We suspect HLRN got a great deal on these Intel Cascade Lake-AP chips and the servers that sport them, with Atos as the system integrator hopefully making some dough.) Back in November 2019, when the Lise system was tested with 103,680 of its cores on the Linpack benchmark, it was rated at 5.36 petaflops, so there must be some pretty big upgrades on the horizon to get to the 16 petaflops and more than 200,000 cores that the final HLRN-IV system (Emmy plus Lise) will eventually encompass. The completed system with all of those 16 petaflops spread across the Berlin and Gottingen sites will cost 30 million, or about $32.6 million.
The interesting bit as far as we are concerned is that the combined HLRN-IV system will have 6.2X more double precision performance at 16.4 percent lower cost than the HLRN-III system it replaced seven years later. This illustrates the principal that we have talked about before, which is that it is far easier to increase the performance of a supercomputer than it is to lower its price. HPC centers have tended to budget linearly over the decades, but it is getting more expensive to make the flops leaps. Still, a 7.4X improvement in bang for the buck over seven years can get a deal done.
We realize that our bang for the buck comparisons are imprecise because of the lack of publicly available data on supercomputer costs over time, but at around $15,000 per teraflops back in 2013, the HLRN-III cluster was twice as expensive per flops as Tianhe-2 system in China, which used GPU accelerators, but about half the price of the all-CPU and very custom PrimeHPC systems from Fujitsu that were inspired by the K supercomputer at RIKEN lab in Japan. The price of systems, particularly those that used accelerators, dropped significantly between 2013 and 2018, and GPU accelerated machines like Summit and Sierra cost just north of $1,000 per teraflops around the time the all-CPU Emmy portion of the HLRN-IV system was going in, which cost $2,038 per teraflops at current euro to dollar exchange rates. Call it two grand.
So in general, all-CPU machines are, it seems, more expensive, and this stands to reason. The programming is harder for GPU accelerated machines, and that costs money, too. Or, you can as many HPC centers do outside of the largest national labs, stick with all-CPU architectures and pay the premium there. GPU-accelerated exascale machines due to be installed in the United States in 2021 through 2023 will cost on the order of $400 per teraflops, and we suspect that all-CPU systems over that timeframe will cost 2X to 3X that per teraflops. None of that counts the facilities or electricity costs that come with the architecture choices, of course. As best we can figure.
Read the original post:
Sometimes The Road To Petaflops Is Paved With Gold And Platinum - The Next Platform
- New Microsoft Ads Take Aim at Mac Pricing - November 8th, 2009 [November 8th, 2009]
- Adobe Flash Comes to TV - November 8th, 2009 [November 8th, 2009]
- Microsoft Introduces Windows 7 Starter Edition - November 8th, 2009 [November 8th, 2009]
- Mac Viruses and Trojans Becoming More Prevalent - November 8th, 2009 [November 8th, 2009]
- Apple ‘Customer Experience’ Continues to Trounce PCs - November 8th, 2009 [November 8th, 2009]
- Seagate Introduces ‘Replica’ Drive to Backup Entire PC - November 8th, 2009 [November 8th, 2009]
- Still Love XP? Run it on Windows 7! - November 8th, 2009 [November 8th, 2009]
- Is Microsoft Ditching Vista? - November 8th, 2009 [November 8th, 2009]
- The Kindle DX: Not Exactly a Textbook Killer - November 8th, 2009 [November 8th, 2009]
- The Smart Shopper’s Guide to Buying a Wireless Router - May 19th, 2010 [May 19th, 2010]
- iTunes 10: So Long, Ringtone Creator - Thanks for the Memories - October 17th, 2010 [October 17th, 2010]
- iTunes 10: So Long, Ringtone Creator – Thanks for the Memories - February 14th, 2011 [February 14th, 2011]
- How to Make Your Laptop Last Longer - February 14th, 2011 [February 14th, 2011]
- Client Build 5 UPDATE: Personal Super Computer 2011 (SR-2 X5690 OCZ Vertex 3 GTX590 Nvidia Tesla) - Video - March 29th, 2012 [March 29th, 2012]
- Super Micro Computer, Inc. Announces 3rd Quarter 2012 Financial Results - April 25th, 2012 [April 25th, 2012]
- Super Micro Computer Q3 Profit Slips - Quick Facts - April 25th, 2012 [April 25th, 2012]
- Super Computer Maker Cray and Intel strike Partnership - April 25th, 2012 [April 25th, 2012]
- Super Micro Computer Q3 12 Earnings Conference Call At 5:00 PM ET - April 25th, 2012 [April 25th, 2012]
- Herd mentallity and the information super highway - Video - April 25th, 2012 [April 25th, 2012]
- Brain vs. Computer - Video - May 4th, 2012 [May 4th, 2012]
- Minecraft World First - Most wanted redstone device - Video - May 4th, 2012 [May 4th, 2012]
- PS3 Jailbreak Tutorial 4.11 WORKING - Video - May 4th, 2012 [May 4th, 2012]
- China's Tianhe-1 supercomputer begins operations - Video - May 4th, 2012 [May 4th, 2012]
- June 2011 TOP500 Review looks at Japan's K Supercomputer - Video - May 4th, 2012 [May 4th, 2012]
- Super Vision for Soldiers - May 5th, 2012 [May 5th, 2012]
- The Super Sonic Show Episode 0-Computer Help - Video - May 7th, 2012 [May 7th, 2012]
- Why Super Micro Computer's Earnings May Be Less Than Awesome - May 10th, 2012 [May 10th, 2012]
- Magnetic bacteria may help build computer hard drives - May 10th, 2012 [May 10th, 2012]
- SUPER WHY! Around the World Adventure Kicks off PBS KIDS Summer Learning Initiative This June - May 10th, 2012 [May 10th, 2012]
- Tutorial SUPER COMPUTER girl 3750 sylvia Vs fem game 4 (3550) - Video - May 10th, 2012 [May 10th, 2012]
- SUPER COMPUTER Wii best 3750 sylvia Vs learn chess 4 (3550) - Video - May 10th, 2012 [May 10th, 2012]
- SUPER COMPUTER girls city 3750 sylvia Vs RYBKA 4 (3550) - Video - May 10th, 2012 [May 10th, 2012]
- John Laban - Open University Super Computer Room - Video - May 10th, 2012 [May 10th, 2012]
- Can A Super Computer Save Banking? Part 2 of 2 - Video - May 10th, 2012 [May 10th, 2012]
- Supermicro® Launches Widest Range of UP Server Platforms Supporting Intel® Xeon® E3-1200 v2 - May 16th, 2012 [May 16th, 2012]
- Supermicro® Debuts New X9 DP and 4-Way MP Platforms - May 16th, 2012 [May 16th, 2012]
- Supermicro® Launches Widest Range of Server Platforms Supporting Intel® Xeon® E3-1200 v2 - May 16th, 2012 [May 16th, 2012]
- Invention kit for banana pianos, alphabet soup keyboards - May 16th, 2012 [May 16th, 2012]
- A few errors could be key to super-efficient computer chips - May 20th, 2012 [May 20th, 2012]
- Supermicro® Highlights Latest GPU SuperServer®, SuperBlade® and ... - May 20th, 2012 [May 20th, 2012]
- Kontron HPEC Platform Chosen by Military Embedded Systems Magazine for Editor's Choice Award - May 20th, 2012 [May 20th, 2012]
- Raspberry Pi to rebirth an era of Woz-like super creativity? - May 20th, 2012 [May 20th, 2012]
- Taste and tale of success - May 20th, 2012 [May 20th, 2012]
- 1 Reason to Expect Big Things From Super Micro Computer - May 25th, 2012 [May 25th, 2012]
- Bump's Super Popular App Just Got A Million Times Cooler With Its Latest Update - May 25th, 2012 [May 25th, 2012]
- Is The Computer 'Cloud' Compromising You Privacy? - May 26th, 2012 [May 26th, 2012]
- Super MP3 Download 4.8.2.6 - May 28th, 2012 [May 28th, 2012]
- Radiohead's Kid A and OK Computer, Now in 8-Bit - May 29th, 2012 [May 29th, 2012]
- ASUS P6T7 WS Super Computer MoBo - Video - May 29th, 2012 [May 29th, 2012]
- Photonic Super Computer 2012 - Video - May 29th, 2012 [May 29th, 2012]
- Kaspersky discovers super-complex Flame malware - May 30th, 2012 [May 30th, 2012]
- Supermicro® X9 5x GPU SuperWorkstation Delivers Maximum Performance with NVIDIA Maximus Certification - May 30th, 2012 [May 30th, 2012]
- Super-virus Flame raises the cyberwar stakes - May 30th, 2012 [May 30th, 2012]
- Super-stealthy ‘Flame' computer virus spies on Iran - May 31st, 2012 [May 31st, 2012]
- Super-stealthy ‘Flame' computer virus spies on Iranians - May 31st, 2012 [May 31st, 2012]
- Was flame virus written by gamers? Code similar to apps such as Angry Birds - May 31st, 2012 [May 31st, 2012]
- Massive cyber attack on Iran came from U.S., report says - June 2nd, 2012 [June 2nd, 2012]
- Massive cyber attack on Iran came from US, report says - June 2nd, 2012 [June 2nd, 2012]
- Supermicro® Exhibits its Latest X9 Server and Storage Innovations at Computex, Taiwan - June 5th, 2012 [June 5th, 2012]
- Supermicro® Hadoop Solutions Accelerate Innovation with Launch of EMC® ... - June 5th, 2012 [June 5th, 2012]
- Super 57000 Video Game (Family Computer) - Video - June 5th, 2012 [June 5th, 2012]
- Security Cameras Turn into Super-Fast Sleuths - June 7th, 2012 [June 7th, 2012]
- Quantum computers move closer to reality, thanks to highly enriched and highly purified silicon - June 7th, 2012 [June 7th, 2012]
- Research Makes Ultrafast Quantum Computer Concept a Reality - June 9th, 2012 [June 9th, 2012]
- Supermicro's New Compact Embedded Server Appliance Supports 3rd Generation Intel® Core™ i7/i5/i3 Processors - June 11th, 2012 [June 11th, 2012]
- The PC which is truly personal: 'Computer' on a memory stick offers COMPLETE privacy for browsing and documents - June 11th, 2012 [June 11th, 2012]
- 'Purified' silicon nudges quantum computing ahead - June 11th, 2012 [June 11th, 2012]
- Apple serves up 15.4-inch MacBook Pro with Retina Display - June 11th, 2012 [June 11th, 2012]
- Apple debuts next-gen MacBook Pro, iOS 6 - June 11th, 2012 [June 11th, 2012]
- How to Invest Like the Super-Rich - June 13th, 2012 [June 13th, 2012]
- Super Computer for Sale - Video - June 13th, 2012 [June 13th, 2012]
- Supermicro® Launches FatTwin™ Architecture - June 15th, 2012 [June 15th, 2012]
- Computer Workstation utilizes NVIDIA® Maximus(TM) technology. - June 15th, 2012 [June 15th, 2012]
- Supermicro® Launches FatTwinâ„¢ Architecture - June 15th, 2012 [June 15th, 2012]
- Acer: Aspire S5, super-thin Ultrabook, coming to U.S. in late June - June 15th, 2012 [June 15th, 2012]
- Supermicro(R) Launches FatTwin(TM) Architecture - June 15th, 2012 [June 15th, 2012]
- Sheldon Adelson: 7 surprising facts about 2012's biggest donor - June 15th, 2012 [June 15th, 2012]
- lego super computer - Video - June 17th, 2012 [June 17th, 2012]
- Age of Empires: The Conqurors - vsing Duke AI 1.6 - Super computer - Video - June 17th, 2012 [June 17th, 2012]
- Supermicro® FatTwin™ Takes Center Stage at International Supercomputing Conference 2012 - June 18th, 2012 [June 18th, 2012]