High speed Copper Interconnects Address Critical HPC Hurdles
ACCA offers alternative to passive copper and optical cables
Datacenter cable management and power consumption are becoming increasingly critical for high performance computing (HPC) systems, enterprise server and storage applications. While data center power efficiency has been improved dramatically with mulitcore processing and open standards, the cabling that connects tens of thousands of servers has not kept pace. Cooling is also a paramount issue, and the industry is in need of low energy, small form factor solutions (SFF) for this escalating problem.
Taking its place alongside traditional copper and optical assembly choices, “active” copper cable is delivering a new high-performance option to the interconnect arena (Table 1). Active copper cable assemblies (ACCA) are making significant advancements in maximizing reach and cable density for next generation data center (DC) protocols, offering alternatives to cumbersome American Wire Gauge (AWG) 24 cable or more expensive optical cabling. By removing cable impairments and attenuation with active low-power analog silicon, thinner, longer cabling can be used. This reduces weight, increases room for airflow and saves power by avoiding the need for optics.
In lengths of 10 to 20 meters, active copper technology can reduce DC power consumption to one-fifth to one-half that of optics, with the added benefits over passive copper of air cooling from small-gauge cable. Lengths up to 20 meters at 10+ Gbs and 10 to 15 bit error ratio (BER) are achieved on copper cables between AWG32 and AWG28, and are available for 8 Gb to 10 Gb Infiniband, Ethernet and fiber channel. Strides in high-bandwidth differential cable constructions, low-power limiting amplification and active equalization chip sets are the leading technologies enabling copper technology to cost-effectively scale the data center footprint over smaller cable and higher data rates.
Table 1: Types of Interconnects
• Traditional passive copper — does not consume power and is an ideal choice for up to 12 meters. However, bulkiness, weight and signal integrity problems start to crop up at distances greater than seven to 10 meters. Limited bend radius and air flow blockage also are restricting factors.
• Optical — the accepted medium for long-distance network communications. More expensive than copper options. Multiple lasers complicate the inherent weaknesses of the optical cables.
• “Active” copper — thinner and lighter than passive copper, “active” copper’s circuits draw power only at each end. Can extend to 15 to 20 meters.
Reducing power consumption
Figure 2: Gore SFP+ and QSFP active copper interconnects are thinner and lighter then passive copper and can reach beyond 10 meters.
Adoption of 10 gigabit Ethernet (10GbE) has been dampened by system costs and power consumption of electronics and modules. Recognizing this, an ad hoc industry group known as the SFF Committee developed two form factors designed to reduce costs and power consumption, improve reliability and reduce thermal footprint:
• SFP+ — small form factor pluggable plus (specified under SFF-8431)
• QSFP — quad small form factor pluggable (specified under SFF-8436)
Originally intended for an optical form factor, these interfaces have grown to include copper interconnect solutions.
In general, the SFP and QSFP form factors are much lower power-consuming modules, as compared to previous generations of optical modules. Lower power (less heat) increases reliability, as electrical components follow a first-order Arrhenius equation, where a 10°C increase in temperature doubles failure rate. Even a 1°C temperature increase matters. As seen in Table 2, SFP+ fiber optic transceiver modules consume about one watt per module, or two watts per node-node interconnect. Taking this a step further, Gore has developed copper assemblies that consume 0, 0.4 or 0.7 watts per node-node interconnect. (Table 3)
Thermal footprint considerations
Looking at the thermal issues which are impacting today’s HPC and networking solutions, we see several drivers that will continue to require better and more effective thermal management in the near future. Some of these factors are relevant to all electronics packaging, such as increased watt density on smaller package footprints, but others are of particular importance to HPC:
• Power density — watts/square inch of printed circuit board (PCB) area, continues to increase.
• Packaging is getting smaller and hotter, inherently increasing watt density.
• Complex waveforms decrease amplifier efficiency, resulting in more energy lost to heat.
• Higher temperatures reduce component reliability.
• Electrical performance of many components varies significantly with temperature.
Table 2: Optical Form Factor Properties
When the heat generated by an active device cannot be dissipated, then the temperature of the device rises. With all the drivers pointing toward more power and higher physical footprint densities, the watt density becomes such that temperatures can easily exceed the qualified temperature for many of the individual components. Figure 1 shows data relating junction temperature to mean time-to-failure. The key to reducing junction temperature is to increase the rate at which heat can be removed from the device and from the working area immediately adjacent to the device.
The greater power consumption of fiber optic assemblies results in greater heat generation, placing greater cooling needs on the system and demanding even further energy consumption. (Approximately two watts of power are required to dissipate one watt of heat due to efficiencies.) These elevated temperatures also result in higher mean time between failure (MTBF) which leads to more e-waste and more consumption of replacement electronics (resulting in further use energy during manufacturing and consumption of mineral resources).
Table 3: SFP+ Power Consumption
Furthermore, optic’s higher thermal environment impacts the junction temperatures of the semiconductors in the optics module and can elevate the nearby PHY semiconductor temperature environment. The temperatures could impact the solder joint reliability of the surrounding environment and drive up warranty costs, as well as impacting vertical-cavity surface-emitting laser (VCSEL) infant mortality and alignment. In contrast, active copper technology is not prone to thermally induced performance or reliability degradation.
Active Copper Cable Assemblies have the potential to fulfill a valuable role in the acceptance of the 10 GbE standard. Increased demands on storage networks and switched fabric I/O are combining to drive demand for 10 GbE, but the current price premium suppresses its adoption rate. As lower cost technologies and
advancements are introduced, the 10 GbE adoption will accelerate. Direct attach copper cable is one such technology that will reduce not only capital costs, but, also operating costs (less power consumption and less cooling required).
The segmentation of cable interconnect solutions appears to fall into three categories:
• Based on October 2008 SFF-8431 preliminary specification work, passive cable assemblies will have value to roughly eight meters based on the most recent loss budget agreements. However, with high-quality PHY devices, bit error free data has been demonstrated up to 12 meters with passive cables.
• In the eight to 25 meter range, active copper cable assemblies provide a lower cost, lower power, higher reliability solution than SFP+ SR optical modules.
• Above 25 meters, SFP+ SR modules are poised to fulfill the needs of the marketplace. It is also important to note that, in the five to 10 meter range, there is an opportunity to utlize active chip technologies to implement much smaller profile copper cables. This smaller profile enables greater air flow and reduces “crowding” with really dense port counts.
Russell Hornung is a new business development manager at W.L. Gore. He may be reached at editor@ScientificComputing.com.
10GbE 10 Gigabit Ethernet | ACCA Active Copper Cable Assemblies | AWG American Wire Gauge | BER Bit Error Ratio | DC Data Center | HPC High Performance Computing | MTBF Mean Time Between Failure | PHY Physical Layer | PCB Printed Circuit Board | QSFP Quad Small Form Factor Pluggable | SFF Small Form Factor | SFP+ Small Form Factor Pluggable Plus | VCSEL Vertical-cavity Surface-emitting Laser | WEEE Waste Electrical and Electronic Equipment