Immersion Cooling in the Data Center – the next big thing?

By: Barry Elliott, Director, Capitoline
CEng MIET MCIBSE MBA BSc(Hons) DCE ATD

 

For the highest density rack cooling requirements, that is from 10 kW to beyond 100 kW per rack, the most efficient method of cooling IT equipment is to immerse the whole server/computer in a bath of non-conductive dielectric fluid which will rapidly take away the heat produced by the device.

As the fluid is non-conductive it has no effect on the electrical circuits. The fluid used can be oil-based or a fluorocarbon, similar to the material used in gas-based fire suppression systems. Manufacturers’ warranties will have to be consulted to ensure that they are happy to see their standard equipment submerged and that hydrocarbon-based liquids won’t dissolve hydrocarbon-based components. One should anticipate seeing IT equipment manufacturers soon stating ‘immersion-cooling ready’ or ‘immersion-cooling compatible.’

The fluid must be connected to a heat exchanger to lose its heat energy and the heat exchanger will be linked to a more conventional chilled-water cooling system. The chilled water system itself may be connected to some form of ‘free-cooling’ device, such as an evaporative cooling tower, to further enhance the efficiency.

As usual there are many variations such as single and two-phase cooling and ‘enclosed chassis’ and ‘open bath’

In general, there are two main liquid categories, hydrocarbons (i.e., mineral, synthetic or bio-oils) and fluorocarbons (i.e. fully engineered liquids). Dielectric liquids are divided into single- and two-phase applications.

In single-phase immersion cooling, fluid remains in its liquid phase. Electronic components are directly immersed in dielectric liquid in a sealed but readily accessible enclosure where heat from electronic components is transferred to the fluid. Pumps are often used to push the heated fluid to a heat exchanger, where it is cooled and cycled back into the enclosure.

In two-phase immersion cooling, fluid is boiled and condensed, which claims to increase heat transfer efficiency. Electronic components are directly immersed in dielectric liquid in a sealed but readily-accessible enclosure where heat from electronic components causes the fluid to boil, producing vapour that rises from the liquid. The vapour condenses on a heat exchanger (condenser) within the tank, transferring heat to facility water that flows outside of the data center.

There is the “Enclosed chassis” construction where the chassis are usually based on traditional rack style implementations. There is usually a small, closed circuit cooling loop with a coolant to protect the flow integrity through relatively small pipes and connectors. The closed circuit is connected to a CDU or Coolant Distribution Unit, which usually can handle multiple racks at once.

The “Open bath” construction refers to the “open” liquid-air interface. Open bath systems are usually tanks which contain a larger body of dielectric liquid where electronics are immersed into the bath. Multiple electronic assemblies will be sharing the same liquid. This liquid can be based on single or two-phase technology. Regardless of the term, open bath systems can be fully sealed, but are always opened from the top to service IT equipment. The water circuit for open bath immersion systems is either connected to a CDU which circulates the dielectric liquid, or to an integrated heat exchanging device which is part of the tank. For a facility interface, CDUs are usually designed for 100 kW or more, whereas an integrated heat exchanging device is usually designed for 10-100 kW cooling capacity. (Definitions from GRCcooling.com)

The main advantage of immersion cooling is the efficiency it has in removing heat from IT equipment. It is 100% ‘efficient’ in that all of the heat is removed from the IT and transferred directly to the cooling liquid. In normal air-cooled environments, which is 99% of data centres, between 30% and 90% of the cold air produced returns to the air conditioning unit without ever passing through any IT equipment and thus doing any useful work. Immersion cooling will drive the PUE down towards ‘one’ with its concomitant benefits of lower electricity costs and lower carbon footprint.

Other advantages are a lower noise environment and a lower floorspace requirement because of the greater equipment density.

The added efficiency allows the method to take away much more heat, hence the claim to achieve greater than 100 kW heat density per rack. In contrast, air-cooled racks seldom get past about 4 kW per rack of actual cooling, and an enclosed hot or cold aisle system may achieve 10 kW per rack, averaged across the enclosure. Immersion cooling, therefore, seems ideal for applications that include heavy-weight blade server style intensive computing.

A downside of immersion cooling is the initial capital cost! All the additional hardware and the heat exchangers and the chilled water system it has to connect to will come at a substantial cost. There will be some offset however, from the reduction in air-cooling equipment normally required.

An additional disadvantage, which doesn’t often seem to be addressed, is how to manage the concurrent maintainability and removal of single points of failure needed for Rating 3 and 4 installations?

Rating 3 or 4 will need two heat exchangers in the liquid bath, two sets of pipework connections and two sets of chilled water systems to drain the heat to. Even then, if the liquid bath fails and loses its fluid this is a single point of failure. The complexity and cost of this will be substantial. There are similar issues in water-cooled racks where concurrent maintainability is very difficult to achieve.

In summary: immersion cooling is very effective and very efficient. There are high initial capital costs, but the efficiency may lead to a lower cost of ownership. Introducing concurrent maintainability and removing single points of failure from the design as required in a Rating 3 and 4 environment is difficult and must be addressed early in the design stage.

 

 

The ideas and views expressed in this guest blog article are those of the author and not necessarily those of TIA or its members companies.