Ethernet in the TOP500
For the past 21 years, the TOP500.org has been ranking supercomputers by their performance on the LINPACK Benchmark. Reported two times a year, the release of the list is anticipated by the industry. As with any such ranking, the top of the list often garners the most attention. However, such emphasis on the top of such a list, would limit one’s understanding of the different supercomputers in the TOP500, and especially the interconnect technologies that they are based upon.
Attempting to understand Ethernet’s role in high performance computing by merely looking at the premiere spots on the TOP500 list would be quite misleading. Ethernet’s story is one where “when good is enough” — simply meaning that the performance is sufficient for the cost incurred. That philosophy may be counter to a space where eking out every last nuance of performance is perceived as desirable. Rather than merely accepting that philosophy, an in-depth review of all of the lists of the past 21 years may provide a better picture of this dynamic space.
Figure 1 shows the presence of Ethernet- and Infiniband-based systems in the TOP500, as well as the total sum represented by systems based on these two interconnect technologies. Simple observations include that Ethernet hit the start of its hockey curve in late 2001, while Infiniband had a slower growth until late 2006, when it approached a growth rate similar to Ethernet. From June 2008 on, these two interconnect families together were the interconnect families for 80 percent of systems on the TOP500. The final observation is by mid 2010 — Ethernet and Infiniband enjoyed similar representation in the TOP500, flipping back and forth three times with the lead position.
This information is fairly high-level and does not provide enough basic understanding of the deployment of either interconnect technology in systems on the TOP500 list. Fortunately, in the TOP500, there is a field that characterizes the “segment” of each system, which may prefer further insight. Figure 2 plots the distribution of the various systems, based on segment, over the past 21 years.
From this data, it can be gleamed that since its inception, the TOP500 list has been dominated by three segments: industry, academic and research. Furthermore, since 1999, with the exception of the two TOP500 lists for 2013, the “industry” segment has had more entries in the TOP500 than “academic” and “research” combined, and can be considered the 500-pound gorilla of the TOP500 list. Digging into the data further, Figure 3 compares the total number of Ethernet-based entries versus the number of entries categorized as “industry.” From this chart, it is clear that Ethernet-based entries are targeting this application space, especially for the past two years. This would beg the question — in the industry segment, how do Ethernet and Infiniband compare?
Figure 4 clearly illustrates that, as a percentage of ports in the “industry” segment, Ethernet is the dominant interconnect family. This data clearly illustrates that, as the two dominant interconnect families over the past several years, there is a relationship between these two families, as a rise in one is offset by a decline in the other. While the distribution has been relatively flat over the past five years, in the past year, there has been a slight increase in the number of Infiniband ports, which as in the past, has been offset by a drop in the number of Ethernet ports.
Diving even further into the TOP500 datasets, one can consider the different variants of Ethernet and Infiniband. Figure 5 looks at the Ethernet and Infiniband interconnects (subsets of the interconnect families) in the “Industry” segment. Note that, given the timeframe that Infiniband-based entries are on the list, Fast Ethernet is not shown, as there were no Fast Ethernet based entries — only Gigabit Ethernet and 10 Gigabit Ethernet. For Infiniband there are four entries — Infiniband, Infiniband DDR, Infiniband QDR and Infiniband FDR.
There are a number of insightful trends that can be ascertained from this figure. The fundamental point made at the on start of this article is that Ethernet’s success is one of price for performance.
Figure 5 clearly highlights that Gigabit Ethernet-based systems are the dominant interconnect solution in the “industry” segment, as well as the overall TOP500, as its 127 entries represent 25 percent of the TOP500. The presence of 10 Gigabit Ethernet showing up in late 2010 is interesting, as this is nearly eight years after Gigabit Ethernet-based systems appeared on the TOP500 and the ratification of the 10 Gigabit Ethernet standard. Clearly, it took some time for 10 Gigabit Ethernet to reach a point where the price for performance was deemed acceptable to move forward.
The path traveled by the Infiniband family has been quite different. Since its first appearance in June of 2004, a new variant of Infiniband (DDR, QDR, FDR) has found its way onto the TOP500 list approximately every three years. Thus, in the same way that Ethernet and Infiniband have split up the share of the “Industry” segment, the total Infiniband share has been split between the different variants, with no one variant ever developing a share that was even 50 percent of Ethernet. And despite its late appearance, 10 Gigabit Ethernet has more entries on the TOP500 list than any Infiniband variant. However, it should be noted that Infiniband FDR is currently in the midst of a rising presence, and will need to be monitored going forward.
The Infiniband trend is something that the Ethernet community may wish to consider. The premise of this analysis has been that Ethernet provided an acceptable cost for performance. Currently, the Ethernet community will be considering the formation of a 25GbE Study Group at its July 2014 Plenary session to target Top of Rack (ToR) applications. Thus, there are a lot of questions regarding Ethernet’s future rate map — which is currently, 10GbE to 40GbE to 100GbE to 400GbE. It is speculated that the addition of 25GbE could then be used to justify the addition of 50GbE, as 8x50Gb/s solutions are under consideration for 400GbE. Debate regarding Ethernet’s rate roadmap is fueling a large number of conversations, as all of these different efforts are considered. Many would like to see these debates concluded in order for the industry to move forward. To that end, the Ethernet Alliance will be holding a Technology Exploration Forum — The Rate Debate to gather the Ethernet industry to help drive industry consensus on this topic. More information on this event may be found at http://bit.ly/1nWRlY2.
Ethernet has been a dominant force in the high performance computing market for the past 14 years, as shown by its presence on the TOP500 list. While it may not capture the attention of many by dominating the top of the TOP500 list, its presence is undeniable. As noted, Ethernet is about price for performance — and given that Gigabit Ethernet is the base technology of 25 percent of the overall TOP500 list is a testament to this concept. Ethernet’s success is attributed to its ability to evolve to meet the needs of industry. Given current cost pressures, it will be interesting to see the market’s deployment of Ethernet technologies into the HPC space.
John D’Ambrosia is chairman of the Ethernet Alliance and chief Ethernet evangelist, CTO office at Dell. He may be reached at editor@ScientificComputnig.com.