What was the first galaxy you looked at through a telescope? M31, Andromeda? M33, Triangulum? Galaxies are fascinating to look at, especially when you look at one that is somewhat like our own Milky Way, such as Andromeda. It gives you a sense of reference, like ‘I am here’ in the universe. And that’s important, because our galaxy is only one of billions of galaxies that make up the overall structure of the universe. They are all connected on a galactic scale.The more scientists understand how galaxies are formed and evolve, the better we, as residents, understand our Milky Way—the realm of our home, Earth.

Our Milky Way is between 100,000 and 180,000 light years across, according to Wikipedia. Andromeda is about 220,000 light years across. But there are much, much bigger components in a galaxy that astrophysicists, such as Dr. Ben Oppenheimer, a researcher at University of Colorado Boulder (CU Boulder), must consider for their research and models of galaxy formation and evolution.

Ken Strandberg

The Science of Building Galactic Models

“When you look at something like a spiral galaxy through a telescope, like our Milky Way, you just see the tip of the iceberg, the stellar components with its grand-design spiral arms,” said Oppenheimer. Galaxies are made of hundreds of billions of stars, planets, at least one supermassive black hole, and dark matter; all components that are constantly evolving on cosmological timescales. Some galaxies are actively forming stars, while others appear “passive” and have quenched star formation. Much of a galaxy can be seen with an optical telescope. But, much of the constituents can’t be seen in the visible light spectrum. “There is a lot of gas surrounding the galactic disk that is more massive than what is visible, as much as 30 to 100 times the size of the galactic disk itself. But it is very diffuse, low density, and can’t be optically observed.”

Scientists like Oppenheimer want to understand unknown makeups of those gases, among other components, in the circumgalactic medium (CGM) to help build their galactic models. “As astrophysicists, we are actually trying to make realistic-looking galaxy computer simulations to understand the evolution of certain galaxies and the physics involved,” added Oppenheimer. “There are many aspects of galaxy formation and evolution, such as from where does it get its gas, and can we explain its evolution by the physics we know and put into these simulations, which include gravity, hydrodynamics, a prescription for star formation, galactic superwinds, and supermassive black hole growth. Or is there something else that we need to understand? We are taking many aspects and prescriptions of astrophysics and putting them into one computer code to try to create a self-consistent, holistic model that can explain the formation and evolution of our galaxy.” Oppenheimer focuses on simulating the physics and chemistry of CGM gas on a supercomputer.

“There are multiple observations available to scientists to gather data of the makeup of a galaxy outside the visible spectrum,” stated Oppenheimer. “There are X-ray telescopes, like the Chandra X-ray Observatory, that tell us about the X-ray emissions structure of the galaxy. There are also ultra-violet absorption spectroscopy instruments, like the Cosmic Origins Spectrograph (COS) on the Hubble telescope.” Scientists use the COS to measure the light absorption by various gases in space to determine what kind of gas it is. “I work on simulating observations from the COS to study the distribution of gas within the CGM around the galactic disk. Then we compare the simulations to observations and refine our models

The circumgalactic medium is much, much bigger than the visible galaxy. Courtesy Dr. Ben Oppenheimer/CU Boulder

Galactic Gases Help Reveal Their Evolution

Cold gas ions in the CGM, like neutral hydrogen, ionized carbon, and ionized oxygen, have different absorption characteristics that the COS can observe. That is, a cold gas absorbing a wavelength of light indicates the presence of the gas in the medium due to its absence in the overall measured spectrum by the COS. But, an observation takes a background light source. So, as Hubble orbits around the earth, scientists point the telescope at a distant galaxy in such a way that a quasar or other illuminating object beyond the galaxy ‘shines’ through the CGM, illuminating the gas present in the galactic halo. The light absorption by the constituent gases creates an ‘absorption line signature,’ identifying the gases in the medium. “Hundreds of orbits by Hubble allow the COS to gather many absorption lines, which are compiled into an absorption line spectrum, indicating the makeup and density of the gases surrounding the galaxy,” stated Oppenheimer.

Hydrogen and helium are basic elements in the universe. To an astrophysicist, other elements, like carbon, oxygen, and magnesium, are considered heavy elements,

“We are looking at the amount of hydrogen and the amount of metals, that is, the heavier elements of carbon and oxygen, in the dynamic gas flows they trace. Their presence in the CGM indicates energetic processes, like supernovae and/or superwinds from supermassive black hole growth that have moved a galaxy’s mass worth of gas into the CGM. But are these gases and metals inflowing into the galaxy from the CGM or outflowing from the galaxy into the CGM? Is the galaxy in some kind of hydrostatic equilibrium? We are trying to understand the physics and state of the gases, which determine the hydrostatic state or the hydrodynamic state of the galactic halo.”

One of these simulations in particular discovered an interesting phenomenon. “A few years ago, our research team took observations from Hubble of gases around blue star forming galaxies, and also around ‘red and dead’ elliptical galaxies, in which star formation had ceased with a large number of stars remaining in the galaxy. There was a difference in enrichment of one ion, five times ionized oxygen. We correlated our findings with star formation rates in the target galaxies and their red shift.” We observed that there was much more O VI (O5+) (ionized oxygen) around blue star forming galaxies than around red and dead passive galaxies, even though there are more stars and more metals produced by the “quenched” elliptical galaxies. “I simulated both blue star-forming galaxies, and red and dead passive galaxies. We discovered that the blue star forming galaxies were forming stars, but the circumgalactic O VI was enriched cosmic eons ago, by earlier star formation 5-10 billion years earlier.” But they questioned why there was far less O VI around passive elliptical galaxies. They postulate the oxygen could have been more ionized, which the COS cannot observe in the ultraviolet. It will take observations by powerful X-ray telescopes to detect more heavily ionized oxygen, hopefully providing motivation for future instrumentation and studies.

Frame from a galactic gas simulation. Courtesy Dr. Ben Oppenheimer/CU Boulder

Seeing More with Supercomputing

Currently, one of Oppenheimer’s key objectives is to create “zoom” simulations that show more detail than ever before.

“Galaxies are massive objects with billions of solar masses across hundreds of thousands of light years of the galactic disk. But I’m simulating the observed gases across one hundred times that distance. When you run a simulation, you’re not resolving individual stars, but an integrated population of stars. My simulations resolve the stellar population at eight times better resolution than the current suite of runs produced in 2014. On the RMACC Summit supercomputer, we aim to increase the resolution by another eight times for a total of 64 times better resolution. The increased processor speed and high-memory per node on RMACC Summit let us to run the next generation of EAGLE zoom simulations, which will allow us to explore new ionization mechanisms of the CGM and new physical modules for launching galactic superwinds out of the galaxy and into the CGM.”

RMACC Summit, the Rocky Mountain Advanced Computing Consortium (RMACC) cluster supported by University of Colorado Boulder and Colorado State University, was purchased and began installation in mid-2016. Dell and Intel worked together to integrate its components and get it ready for scientists like Oppenheimer. The system has 395 nodes (servers) of Intel® Xeon® processor E5-2680 v3 and Intel® Xeon® processor E7-4830 v3 with a total of 9,600 cores. Another 20 nodes with Intel® Xeon Phi™ processor 7200 series with it Many Integrated Core (MIC) architecture were recently added. “The cluster is mostly built on Dell PowerEdge* C6320 Rack Servers,” said Jonathon Anderson, part of University of Colorado Boulder Research Computing. RMACC Summit’s aggregate measured performance (Rmax) is 375 TFLOPs—about 2.5 times faster than their previous supercomputer, called Janus—only with one-third the node count and almost half the cores.

Intel’s Fabric Critical to Performance

Scientific codes, like those run by Oppenheimer, rely on distributed computing of massive amounts of data spread across the entire system. The time-to-solution for his simulations depends a lot on how fast the individual computations can communicate across the thousands of cores. That requires a fast network, so RMCC Summit was installed with a 100 Gbps Intel® Omni-Path Architecture (Intel® OPA) fabric. Intel’s next-generation fabric of switches and interface cards was introduced only in the last 18 months, yet it is already being installed in many of the world’s fastest supercomputers as listed by the

“With the newest generation of interconnect and processors, the amount of bang you get for your space and electrical costs is just so much greater with newer equipment,” added Peter Ruprecht, Senior HPC Analyst in CU Boulder Research Computing.

CU Boulder chose Intel as the foundation for RMCC Summit, including Intel OPA. “We wanted to be a test bed for other sites that might be interested in deploying Intel OPA,” said Anderson “We thought there might be interesting experiences that could come out of the new technologies.”

“Performance for the system was a key metric,” according to Mark West of Intel. “When we talked with the University, we emphasized that with Intel OPA fabric, they’d spend less on the network than alternatives without sacrificing the fabric’s throughput or latency, which are key to the kind of breakthrough research these systems are used for. With Intel OPA we could use fewer switches and still deliver high bandwidth across more servers.” That allowed CU Boulder to spend more budget on additional compute nodes to improve overall system performance.

This aligned with Anderson’s position on technology. “I’m a kind of standards-oriented person, so I like the idea of going down the Intel HPC path as far as we could, or as wide as we could, with a full-stack solution as much as possible in a single cluster.”

For Dr. Oppenheimer, it means creating simulations that resolve the complex flows of gas through the circumgalactic medium and into galaxies. “It takes many ingredients to form a realistic galaxy: the gravitational force of dark matter, the energetics of supermassive black hole feedback, and of course the fuel itself—gas from the circumgalactic medium that is coming into focus by observations of COS on Hubble” he said. “Can EAGLE zoom simulations run on RMACC Summit build galaxies as we observe them, or is there some other astrophysics, perhaps exotic, we need to add to future simulation runs?  The answers rely on state-of-the-art simulations with ample processing power.”

Ken Strandberg is a technical story teller. He writes articles, white papers, seminars, web-based training, video and animation scripts, and technical marketing and interactive collateral for emerging technology companies, Fortune 100 enterprises, and multi-national corporations. Mr. Strandberg’s technology areas include Software, HPC, Industrial Technologies, Design Automation, Networking, Medical Technologies, Semiconductor, and Telecom. He can be reached at