#: locale=en
## Tour
### Description
### Title
tour.name = HPC
## Skin
### Multiline Text
HTMLText_33711F51_3D9B_3042_41C8_5BBC76E1A7A1.html = Microsystems and Engineering Sciences Applications (MESA)
Welcome to MESA, home of Sandia’s advanced nuclear weapons research, design and development functions, as well as integrated materials research, and the production facilities for microsystem technologies.
Focused primarily on the nuclear weapons mission, the facilities that make up MESA ultimately connect with all of Sandia's mission areas via microsystems research and applications.
After eight years of construction, MESA was completed in 2007 at a cost of $518 million. It was Sandia's largest construction project since the Labs' first permanent buildings were built in the 1940s.
898 Lobby
Building 898 is a three-story, partially pre-fabricated, facility housing offices and light laboratories. Designed as an integrated facility promoting interaction among groups via workspace blocks, the building sports a postmodern, high-tech industrial design. The soaring lobby and high ceilings create an open and empowering environment for the design work housed here.
Gowning Room
Clean rooms environments are kept clean both by limiting the volume of particles introduced to the space and by constantly moving the air to remove particles. Workers wear protective garments to reduce the number of contaminants they introduce to the room. This changing station allows staff to put on appropriate layers prior to entering a clean room; additional changing rooms allow for full changes of clothing.
858 Lobby
Originally built in the late 1980s to house the Manufacturing Development Laboratory and its offices, Building 858 expanded significantly when 858EF (the MicroFab) and 858EL (MicroLab) were added as part of the MESA construction. The building houses wafer manufacturing, compound semiconductor fabrication, microelectromechanical systems (MEMS) production, and electronic circuit manufacturing.
MicroFab
The MESA Microsystems Fabrication facility is one of the most complex buildings at Sandia. It is the first in the world to combine silicon processing with fabrication of compound semiconductors under one roof. This is the heart of microsystem manufacturing, done primarily in cleanrooms.
858 SiliconFab
Building 898 is a three-story, partially pre-fabricated, facility housing offices and light laboratories. Designed as an integrated facility promoting interaction among groups via workspace blocks, the building sports a postmodern, high-tech industrial design. Its polished stone and steel and glass exterior includes curvilinear walled ribbons of windows leading to the northeast entrance, and extended wings with sun-protecting louvered windows. The soaring lobby and high ceilings create an open and empowering environment for the design work housed here.
898 Corridor
State-of-the-art videoconferencing and stunning projection capabilities make the VIEWS Corridor a prime viewing and sharing spot for scientific computing displays. Output from the visualization tools used in Sandia's high performance computing arena can be projected in 3D. The area is a magnet for VIP visits and tours.
858 Photolith Lab
Sandia maintains a clean room outside of the MicroFab area for its photolithography processes. This allows the MESA workers to develop and maintain customized processes without disruption to the manufacturing effort.
You've heard of hot air balloons, but what about a cold air balloon? Well, back in 2005, Sandia built a computer called Thunderbird and at the time, it was the fifth fastest computer in the world, but all that speed also meant it generated a lot of heat. The system was air cooled, but the problem was there was no air containment. Basically, the air that was flowing over the system to keep it cool was actually moving too quickly, and that was heating up nodes near the floor, not keeping them cool. The solution was remarkably simply; put a tarp on it. Yup, a tarp was put on the top of one of the fastest computers in the world. That tarp helped create a column of contained air and that rolled back down to cool the machines. It worked so well that when one of the managers at Sandia heard about it they decided to replace the tarp with the same material used in a hot air balloon. It was put over the entire aisle, and became Sandia's first air containment system. This system was used for the entire life of the Thunderbird system, and was another example of Sandia leading the way for engineering solutions in high performance computing.
This guy with the label maker isn't really doing anything important, he's just posing for the picture.
Keeping systems cool is a vital part of any data center. That make this facility one of the coolest at Sandia; because it actually keeps things cool, and because walking through all those pipes and big pumps is a unique and very cool experience.
These pumps transfer cooled water from this facility to the Data Center to keep the super computers running cool. The pumps push 5000 gallons of water per minute through underground pipes to feed the high performance cooling systems.
Throughout the Data Center tour we talk about the importance of keeping all those super computers cool. That's such an important task that Sandia actually has two buildings dedicated to cooling. This building and its counterpart help pump around 5000 gallons of water per minute through 10,560 feet of piping that feeds the cooling systems of the Data Center across the street.
During your tour through this museum, you'll read a lot about FLOPS, but what exactly are FLOPS? Well, in this case, it has nothing to do with trying to get the other team carded in a soccer game. Rather FLOPS stands for "floating point operations per second" and is basically a measure of computer performance and speed. FLOPS indicate how many multiplications can be performed within one second on the machine. Thus, the CDC 6600, the world's fastest computer in 1964-1969, had a processing speed of about 1 megaFLOPS (one million FLOPS). ASCI Red's peak was 3.2 teraFLOPS (one trillion FLOPS), while its successor, ASCI Red Storm, reached a peak of 284 teraFLOPS. Machines have since surpassed petaFLOPS (one quadrillion FLOPS), and in 2020, Japan's Fugaku supercomputer clocked in at a speed of 415 petaFLOPS.
If you'd like to read more about what takes place here at the Data Center, be sure to check out this series of annual reports that highlights various accomplishments and how HPC is helping with real-world problems.
This door leads into the actual supercomputer annex where many of Sandia's "big machines" are kept. You might hear an ominous hum as you approach it, but don't worry, that just means there's a lot of power in there. Not only is it loud, but it's also kind of warm in there.
This monitor displays who's currently inside the supercomputer annex. It quickly shows who's working, who's visiting, and who's bringing donuts (actually, it doesn't really show that last one).
Sandia and Cray collaborated on the XT3, which was installed at Sandia as Red Storm in 2003. In 2004, Cray made the XT3 available commercially. Red Storm was actually constructed from commercially available off-the-shelf parts. The interconnect chips of Red Storm allowed it to efficiently pass data between its over 10,000 processors even while applications were running. Those interconnects also made it possible for this machine to create three-dimensional meshes to make 3D representations of complex problems. Its 40 teraFLOPS helped it set world records for high performance computing (HPC) visualization and for two of the HPC Challenge benchmarks.
With so much wire, so many parts, and so many computers in the Data Center, one has to wonder; what happens when something breaks or loses connectivity to the network? It's not like you can run out to Radio Shack for replacement parts (mostly because, you know, they're out of business). Fortunately, the Data Center has its own Radio Shack, of sorts. The Data Center keeps plenty of spare parts on hand, like extra nodes. It also has plenty of cable for installing networks when adding new computers. Most of these cables are used to network machines together. On average the Data Center adds about eight to nine new systems a week, and these are the parts that get those systems are up and running.
This term is used a lot when exploring the world of high performance computing. A node is a collection of hardware that consists of just CPUs and RAM. It's your desktop computer can do, but without all the peripherals (no mouse, keyboard, monitors). Nodes unleash the HPC potential in hardware interconnected with no gaps or interruptions.
This monitor shows the electricity usage for the annex, including power provided by solar panels on the roof. It's basically a dashboard that shows the monumental power requirements for the Data Center, and provides a quick way to view how that power is being used while showing how the Data Center adheres to "green" standards of power consumption.
Computing history is often told and celebrated as linear progress. The increases in speed, storage capability, materials development, chip manufacturing, volume, complexity, and did I mention speed? The museum shows all of that, but it also displays the chaos of computer evolution and development. The rate of change and the plethora of problems computing may address generates a lot of technology, new concepts, and new products all at the same time. It was not always obvious what tech would win out in research and in the marketplace. There, amongst the mugs and the tan plastic towers, the museum illustrates the excitement of so many options resulting in so many solutions.
It's the Computing and Communications Museum, after all. The communicating part is represented in two large displays built into the wall of the hallway leading to the main museum and entrance to the supercomputing annex. One display covers the user side - largely the evolution from one black telephone on a main office desk and different tools for receiving multiple callers at one centralized location through different styles of telephone handsets to the very modern, very small phones we all carry. Of particular note are the massive, heavy, black, rotary dial explosion-proof telephones made out of Bakelite. These were in Sandia's explosive-handling and -testing facilities and were made to prevent dust from explosives from getting into the telephone's inner working and experiencing a spark. Because...boom.
Unlike tape drives, on which data must be accessed serially, disk drives allow for random access storage, increasing the speed of access to the data. The phrase "Hard drive" indicates the rigidity of the platters or disks that hold data. Hard drives usually include several platters mounted onto the same spindle and encased in a protective box. The disks are read by the head - usually one per side of each disk. Over time, significant increases in storage density on hard drives have been achieved, largely via the arrangement of the magnetic material on the disk.
In April 2012, Sandia's Computing and Communications Museum opened. This Museum is a series of posters with timelines and display cases holding artifacts that encompass both the history of computing in general and Sandia's own computing history. It largely demonstrates the overall move from the Lab's use of commercially available computing products, its work with industry to refine and specialize some of those products for Sandia's use, and what Sandia itself has created and contributed to the history of computing. It is a living collection; new items are added as they are located and as they are created.
This video walks you through what high performance computing (HPC) is, what it's used for, and its history at Sandia. This is a great place to start for some background on what you'll see throughout the rest of this tour.
In this display case are several models of the supercomputers of the past. Pictured is a model of a Cray X-MP system, the first multiprocessor supercomputer. It debuted in 1982 and was capable of a maximum 940 MFLOPS (or mega FLOPS).
In 1984, Sandians Jim Davis and Diane Holdridge used Sandia's Cray 1S to crack the 69-digit Mersenne number, the longest number ever factored at the time. They broke their own record from the previous year when they factored a 63-digit and 67-digit number. Using the Cray for factoring large numbers was part of the cryptographic work in computer encryption going on at that time; work that contributed directly to the secure transmission of data that allows for all of the electronic transactions we now engage in.
There are a lot of blinking lights as you walk through the Data Center, but they aren't just there for show, they indicate what's happening in the system. Some of the lights are status indicators. They show green, and that means everything is functioning as it should. An amber light, however, indicates that something may be wrong; like a bad connection to the node. As for the blinking, well, that just means the machine is hard at work processing data. There are also network connectors that use lights to indicate speed, so yeah, there are a lot of blinking lights, all serving as at-a-glance indicators of how well (or not) the machines are running.
Sitting off to the side in the Data Center is a piece of hardware that is now obsolete but is also responsible for everything that goes on inside this room. This is a CRAY 1. The first CRAY 1 was deployed at Los Alamos National Laboratory in 1975 and had a top speed of 160 megaFLOPS (MFLOPS). All those boards on a CRAY 1 put together are the equivalent of a single processor core in a modern computer.
As for all those wires you see, they connect the transistors and were pin-to-pin interconnects that helped make the processor work. Each wire was hand-cut to a specific length. An oscilloscope was used to time the signal through the wire, and that determined how long each wire had to be. This was done for each and every wire in the machine. If there was a single failure, new wire had to be cut to the exact timing specifications. To get the wires into the machine, Seymore Cray, inventor of the CRAY, hired seamstresses to do the wiring. Their deft touch and attention to detail in cutting those wires made the CRAY 1 possible, and they eventually were known as the cable ladies. While it may seem obsolete sitting next to the shining machines of the future, the CRAY 1 is actually a bit like an Egyptian pyramid; it's an ancient (in computer age) technological marvel that would be difficult to duplicate even today.
The Data Center uses a lot of power, so what happens in the unlikely event that the power goes out? That's where these unassuming cabinets come into play. They house row upon row of lithium ion batteries, each of which could power your entire house for around eight hours. All critical equipment (anything associated with safety or continuity of business) in the Data Center is backed up by battery and generator. If the power goes out, these batteries kick in and help get the generator started. These batteries and the generators they connect to make sure that even when there's no power, the machines in the Data Center will never experience an unexpected shutdown. Naturally, the Supercomputers draw far too much power to actually run off of batteries (they would last mere minutes), hence the need for the generators.
Well, it's not exactly like that, but when there's an issue with a system in the Data Center, these carts are used to plug in and diagnose the problem. The carts, known as crash carts (more of a medical terminology as they're used to look at the "patient", or to triage what's going on inside one of these machines), allow researchers to check the system and find potential trouble within a node. Once the problem is diagnosed, the proper steps can be taken to get the system back up and running. No need to stand clear, though, because no defibrillators are used in these scenarios.
There's a lot of data stored at this facility, operated by SMSS (Sandia Mass Storage Systems) whose sole responsibility is high performance data archiving. These tape libraries are used by the high performance computers for storage, especially if it's a long-term storage need, thus discs are sent to tape libraries like this. The library holds that information for many years; meaning there are many Petabytes stored here. Whenever that information needs to be retrieved, the robot helps find the tape that needs to be reviewed and plugs it into a system in the storage rack so the data can be reviewed.
Sky Bridge was the first direct-to-chip liquid cooled system at Sandia. This display shows a computer node with a liquid-cooled central processing unit (CPU) and a cutout of the coolant distribution unit (CDU) that distributes water to the CPUs in the rack. Water cooling not only helps keep the temperature down, but also keeps the noise level down.
Sky Bridge first entered service at Sandia in 2015 and at the time was the primary unclassified scientific computing platform at Sandia. It doubled the available computing cycles, and its use of the existing Advanced Simulation and Computing (ASC)/TLCC2 compute cluster meant that existing computer codes could run on the system without any modification. It also played a major role in the transition to liquid cooled machines. It has a maximum power draw of 675 kW (kilowatts) and a max performance of 520 tera FLOPS (TFLOPS). It's liquid cooling system and more efficient power distribution saved $700,000 in construction costs and reduced annual operating costs by $120,000. Why the name "Sky Bridge"? Well, the system used the Sandy Bridge chip, and it took the place of the Red Sky system, so put those together and you have Sky Bridge.
Most of this building is made up of an intricate maze of piping, all of which is used to transfer water to the Data Center in order to keep the super computers cool. There are approximately two miles of piping between this facility and the Data Center, much of it underground. The pipes you see here are really just the tip of the ice berg in the massive effort to keep the machines of the Data Center cool.
You will notice the glass panels on the floor was you walk through the racks of high-powered computers. They serve multiple purposes. First, it just looks kind of cool. Second, this is how the data center gets liquid cooling to the computer racks, so the glass tile has a very practical function; it helps when doing walk throughs to see how the racks are doing. One just has to glance at the temperature gauges on the pipes through the glass to see if everything is functioning as it should. It's also easier to notice any leaks; all without ever having to remove any of the tiles. Finally, since the tiles are moved less often, it helps from a safety perspective as well.
This panel controls the settings for the energy efficient chiller. It's the brains of the unit, and it makes sure the temperature isn't too cold or too warm so the machines in the Data Center don't over heat.
Note: the plant next to the panel isn't real, but provides some nice greenery in a facility that's mostly just pipes.
The Data Center already has plenty of fancy lighting, so all it needs it some music to turn it into a really hip club. Well, of course it will never really be a club, but it does have a great sound system. There are JBL speakers embedded in the ceiling of the data center, so if researchers want, they can practice the Macarena while their data is processed.
This complicated set of controls is a water treatment station. The water quality of the facility is tested on a regular basis to make sure the pipes are staying clean and the chemical makeup of the water is correct.
At first glance, one might assume the label on this piece of machinery refers to how much it weighs. However, that 1,300 lbs. is actually how much refrigerant the chiller can hold. That refrigerant cools the water that cools the computers in the grand circle of cooling at the Data Center.
Tom Brady, love him or hate him, is one of the most successful NFL quarterbacks of all time. Between 2016 and 2018, he appeared in three consecutive Super Bowls and won two of them. Overall, he's made it to ten Super Bowls and won seven (so far). Well, ASCI Red, a collaboration between Sandia and Intel that began in 1995, had a similar string of success during the late 1990s. It was the number one system on the Top 500 list of world's fastest super computers for four consecutive years, from 1997 to 2000. ASCI Red was the first computer to be built under DOE's Accelerated Strategic Computing Initiative (ASCI) program to support the Stockpile Stewardship Program. Running just three quarters of ASCI Red, the machine reached 1.06 TFLOPS, making it the first supercomputer to exceed one teraflops. After memory and processor upgrades in 1999, it achieved 2.38 TFLOPS. Further upgrades to Pentium 11 Xeon processors brought performance to a maximum of 3.2 TFLOPS.
This cylinder looking thing is temporary storage for refrigerant. Whenever the main chiller needs maintenance, the refrigerant is moved into this container while the main unit is worked on. Think of it as a refrigerant IV for a sick chiller unit.
In the Data Center, the computer clusters are all networked together, which naturally takes a lot of wiring. This interface for the system allows the clusters to talk to each other like it's all in one big system. The network topologies are what define what these wires look like, and in this case, they used a topology known as the Fat Tree topology (Fat tree is a topology, or the arrangement of the nodes to communicate with each other, that has a hierarchy resembling the branches of a tree). All of that wiring adds up, and for a machine like Eclipse, there's about 9.86 miles of copper and optic cabling involved.
These rows of machines may all look similar, but they actually serve several different purposes. The first row is a series of virtual machines for the Sandia enterprise. Using these virtual machines saves on the number of actual machines needed to run daily tasks. One rack equals what would have been one full row of computers. The next three rows are used for the common engineering environment to solve engineering problems for Sandia's wide variety of customers. The last three rows are advanced architecture test systems, or test beds. These are used to test and prototype the newest in emerging hardware and software technologies (they may not exist anywhere else in the world), which are tested here to see if they're viable for use in the next generation of supercomputers. They're also used to test new applications to see if their codes will run on Sandia systems. Other test beds are used to assess software for bigger machines at other labs, such as Lawrence Livermore (CA) or Los Alamos (NM). While they're all used for different purposes, these rows of machines have one thing in common, they're cooled by a Laminar flow.
When it's time to install a new enclosure in one of these racks, it's not quite the same as replacing your hard drive on your laptop. In fact, it requires a little more heavy lifting. This forklift like machine is used to install each node. It can lift, replace, or install new servers in the rack.
This room is where everything stays cool. The blue pipes are the main warm water process loop that supplies water directly into the high performance computers on the floor of the data center. Think of it like the arteries and veins in a human body cycling fluids to help keep things running smoothly and cooly.
This is the system console; it allows someone to connect to one of the big machines and drive it from this console. As for the big speakers, well, like in the 880 Data Center, sometimes computer engineers need a little music playing in the background to help keep their productivity high.
The Man: Steve Attaway spent over thirty years at Sandia, and during that time he was a researcher, a mentor, and a pioneer. He broke new ground in parallelizing code so it could run on thousands of cores at once, and he always made time to invest and help develop the following generations of researchers that would continue to use the resources in Sandia's data center to solve the nation's "big" problems and challenges.
The Machine: The Attaway Cluster is a bleeding-edge machine, clocking in at around 1.93 petaflops (PF) while also using a next-gen water cooling system. It has 1,488 nodes, with 192 GB of RAM per node. The cooling system uses a negative pressure liquid cooling solution, which not only keeps the machine cool, but helps keep it leak free. (see video: Negative pressure, positively cool for more).
A lot of the work to keep the machines cool takes place under the floor. Here, some workers are connecting 4" pipes that will feed warm water coolant to the large computers to remove heat. This method is far more efficient than past, traditional methods that removed the heat only with air, which was not very effective or energy efficient for these larger powerful computers.
These large, 4" hoses connect chilled water to the CDU (cooling distribution unit) that in turn feeds the now warmer, medium temperature water in the process loop through a heat exchanger within the CDU. It's just one part of the complex process of keeping the world's fastest computers from overheating in the desert.
What does it take to install a new super computer. Well, as this time lapse video shows, it's takes more than just the few cords you need to plug in for your home desktop. It takes a lot of time, a lot of planning, a lot of power, a lot of cords, and a lot of people.
A negative-pressure liquid cooling system was designed for the highly advanced Attaway Cluster to help keep it cool and leak free. The system keeps the machine's temperature at around 50 deg C with a low power draw. As for the negative pressure, the system essentially works in a vacuum, which means if there is a leak, instead of spraying out, the liquid gets "sucked" back in (more accurately, air is drawn in instead of the liquid leaking out). This advanced cooling technique not only prevent leaks, but means the system can continue to operate even with minor leaks present. It also makes maintenance easier since disconnecting a server automatically evacuates the water, leaving the system dry and ready to be worked on.
In the past, working on supercomputers could be dangerous, and not just because they might turn into Skynet...
The XT3 machine had dual core processors that ran at about 10 GFLOPS/core, making an entire compute cabinet just under 1 TFLOP/Cabinet. While that was better than ASCI-RED, today's tech can do that with a single chip. This board helped upgrade Red Storm to 254 TFLOPS, or XT4 technology. It used a SEASTAR interconnect, a proprietary networked interconnect that provided node-to-node connection using a 3D network topology, known as TORUS, that was developed by Cray, Inc. and Sandia. An algorithm calculated what paths were available (kind of like a Terminator brain) so if there was any disruption in the connections, a redundant path could be used, keeping the machine online. With a little help from XT3 Red Storm, it set world records as the second fastest computer in the world (it never did make it to number one).
This is a CRAY 2, of which there aren't many left in the flesh...or...in the wiring, or whatever... The CRAY 2 debuted in 1983 and was the world's first gigaFLOP super computer; meaning one CRAY 2 was as powerful as 10 CRAY 1s. This was also the first super computer to utilize a modern UNIX operating system. However, what makes this machine truly unique is that the world's supply of CRAY 2s is rather limited; only 24 were ever made. This particular one was found in condemned building, sitting next to a bunch of legacy hardware. It was basically a "barn find"; hidden under a tarp and a thick layer of dust. It required a specialized lift to move it, of which there is only one in the entire world. At the time, the lift was at the British Museum, and it was shipped to Sandia to help move this heavy CRAY from the obscurity of a barn to the prominent place it now holds in Sandia's Data Center.
Considering how fast technology changes, these relics are the fossils of the computing era. Starting on the far right there's a Sony WatchCam, next to it is a vintage Macintosh SE, there's a Compaq "portable" PC in the middle - considered to be the first laptop despite weighing 18lbs - and a TRS 80 Model III (often referred to as the "Trash 80") on the left. Directly below it is a TRL Line Printer, which happens to be from Radio Shack. On the bottom right is an Atari 800XL, the gaming company's attempt at also producing PCs. And yes, that is some vintage, green and white, dot matrix printer paper sitting there in that box.
The Cray XT 3 I/O board provided network and storage system connectivity. There were hundreds of these boards connected to 44 storage cabinets that held over 40,000 hard drives. To connect these I/O boards, around 60 km of optical cable (basically the distance from Albuquerque to Santa Fe) was used to connect the system together. These boards provided accessibility to the supercomputer, meaning users would log into these nodes to access the full potential of a supercomputer.
Most data centers don't put windows in the building, but this one is unique. Not only does it have windows, but they help save energy. The windows face north, so sun never directly shines into the Data Center, but they still allow in plenty of natural light. That means lights in the data center aren't always needed, and that helps save energy. Plus, it's always nice to know if it's raining outside.
In 2018, Astra was the world's fastest Arm-based supercomputer (Arm is a type of computing processor) according to the TOP500 List. It has a speed of around 1.529 petaflops (PF). Not only was Astra the fastest, but it was among the first supercomputers to processors based on Arm technology, which ironically, was originally used mostly for low-power mobile computers, including cell phones and tablets. Arm processors in each of Astra's nodes, however, are about one hundred times faster than ones in a cell phone, and Astra has 2,592 nodes. Feel the power!
This building was designed with expansion in mind. The wall can come down to divide the building, or to open it up. The other side contains the National Security Computing Center.
This computers in this row are from Lawrence Livermore National Laboratory and were about to be decommissioned. Since Sandia has a similar system running in the Data Center and since the manufacturer of the system is no longer producing support components, they brought here to the data center for spare parts. The name of the computer was Cab (after a wine of some sort, maybe a nice cabernet) because Livermore is in wine country, not chile country.
These three boards are from Sandia's nCube machines, originally installed in 1987. The original nCube/ten processors were 200 KFLOPS (that's kiloFLOPS) processors with 512 Kbytes (kilo bytes) of memory each, giving 200 MFLOPS (megaFLOPS) peak and 512 MBytes (megabytes) of memory for the entire 1024 processor machine. One of the boards served as an I/O board, with 16 processors filling one quarter of the board, and a pool of associated memory. The other is an empty 64-processor nCube/ten board that proved defective but critical to the reliability of the machine when the original 1024-processor configuration demonstrated cooling problems. The empty board was inserted next to the bank of processor boards to promote the desired airflow within the system. The third board is an nCube 2 board that dates from 1990 and has a full 64 processing modules on it; that is, 64 daughter cards each with a CPU and six memory chips. Each processor on this board was a 2 MFLOPS processor with 4 Mbytes of local memory. Each board cost $250 when purchased.
Many of the test systems in the Data Center use unique names. For instance, the Yacumama system is a liquid cooling technology test system being evaluated, and is named for a mythical sea monster. See, makes sense, right? A liquid cooled machine, sea monsters are in the water...and it just sounds cool. The naming of the computers is a fun event as many names and themes are passed back and forth through various personnel that have or will do work on the large computers. One of the other recent themes is based on chile peppers (being from New Mexico and our great chile) and which ones are hotter, which are then used for a computer that also produces a large amount of heat while running at very high speeds.
High-Performance Computing (HPC) systems consume substantial amounts of energy to perform the large-scale computations required. A biproduct is substantial amounts of heat, requiring stringent cooling regimens to keep the computers running. Although a typical building is designed to meet heating and cooling needs for the comfort of its human occupants, HPC data centers must provide massive cooling power for its banks of servers. This usually results in high water and energy usage. Therefore, employing energy efficiency and conservation techniques in building data centers is crucial for operation and reducing ongoing costs. Many green building strategies and new, innovative systems were implemented in the 725E Data Center to get it to LEED Gold.
The 725E LEED Data Center is also equipped with a Thermosyphon Cooling System (TCS), which uses passive heat transfer to make the building more efficient. The TCS saved more than half a million gallons of water within a six-month period. One of the unique features of the TCS is that its cooling unit intercedes passively. Its refrigerant rests in a shell that surrounds an outgoing pipe like a glove on a hand, absorbing heat until the liquid evaporates into a gas, like how boiling water becomes steam. The gas rises in vertical pipes until it reaches the upper limits of the device. There, it gives up its acquired heat to the atmosphere, coalesces back into liquid and sinks down, ready to cool again.
The Airside Economizer brings fresh air in and cycles it to the floor for cooling. Heat generated by the computers rises and some of it escapes outside via exhaust, while a portion of that risen heat remains at the 25-foot-tall celling, allowing maintenance of a more constant indoor temperature of around 78 degrees. The roof of this Data Center also has a cell deck (like a swamp cooler on hyper steroids!) that is used only about 17% of the year. During other times, it just cycles the cooler air into the data center.
Eclipse, so named because it's the name of a New Mexico chile and it was purchased in 2017 or the Year of the Great American Eclipse, came online at Sandia in 2018. It was part of the Commodity Technology Systems (CTS) project that added over almost 300,000 compute cores to the high performance computing capacity at Sandia. At 1.2 petaFLOPS (PFLOPS), just one Eclipse has the same compute capability as two Sky Bridge clusters. Eclipse also incorporated innovative, direct water-cooled processors that help lower operations costs and increase energy efficiency.
The CRAC, or the Computer Room Air Conditioner unit, is one of the many systems in the Data Center that keeps these machines cool. The red lights on the top are where the filters are, and this is where hot air comes into the system. The hot comes in from on top, goes across the filters and some cooling coils that go all the way back to a big chiller, and is then pushed under the floor to help cool the computers. To increase efficiency, this system adjusts based on the temperature in the room; if the computers are working hard, its uses more air to keep things cooler, but when things are slower, it pulls less air to save more energy. Average air temp from the floor is between 72 and 78 degrees. This Data Center is one of the top twelve in the world when it comes to energy efficiency, and it's because of systems like this. This system saves around 26 million gallons of water a year, which is vitally important in a desert environment like New Mexico.
3M introduced the first magnetic tape in 1947 for audio recording, but it can be traced back to 1951 for data storage when it appeared with UNIVAC I. Initially, tapes were large (10.5 in), open reels, slowly moving to cassettes and cartridges beginning in the 1970s. Although it is being replaced, magnetic tape is still in use for backing up infrastructure computing systems.
Sandians attend computer conferences, vendors give out branded items, and...yeah...what else are you going to do with them; drink coffee out of a different mug every day? Better to just put them in the display case.
Sandia is continually looking for ways to reduce energy use in high performance computing (HPC), which includes not only the efficiencies of the compute platform, but the effectiveness of the infrastructure and facilities. As Sandia strives to make its HPC centers the most energy efficient in the world, it's addressing the energy problem with a system view; from the processors to the water cooling and distribution systems. New Mexico's climate makes it possible to use non-mechanical cooling (using water pumps as opposed to previous systems that were turbine air-cooled and pushed air through the cabinets), which when combined with warm water ( ~ 75 F to ~ 85 F), saves considerable energy. Sandia will soon be able to capture energy from an elevated return water (~ 95 F to ~ 115 F) to support indirect energy needs of the facilities such as process water, domestic hot water, and adsorption cooling processes.