Tips For Picking a Data Center Facility Part 3: Cooling, Fire Suppression, and Network

So in the last two videos we covered building your own vs. leasing colo space, different tier levels, redundancy factors, frequently used terms, facility considerations, company financials, MEP, PUE and power systems. So if you haven’t seen the first two videos, you can check them out here.

Cooling

Let’s talk about cooling. This is one of the most important and challenging features of a facility. Cooling capabilities are usually what limits the amount of power consumption and density of your facility. Cooling is dependent on the amount of energy you put into the room, the volume of the room and the amount of airflow.

When I was building facilities, we looked at the amount of total cooling capacity we had for a volume rather than the kWh per rack. So sometimes, in order to deal with higher power demands per rack, we would limit the amount of total racks in a given space. Facilities often talk about max power per rack but, often times it is really more of an aggregate power discussion. You will definitely encounter some facilities that put a cap on power per rack though, you typically see these lower power limits in older facilities.

Let’s talk about some terms. There are CRACs or Computer room Air Conditioners. These are large units that have fans and pumps in them. They take hot air in through the top and push cool air out the bottom. Typically these systems have a refrigerant in them. Then there are CRAHs Computer Room Air Handlers. These do much of the same thing, however they generally don’t use refrigerant. Instead, they typically use chilled water systems to produce the cooling.

It is important to know how these systems are powered. Even though you have redundant cooling systems they also need to be supplied by diverse power systems. Often, these systems are not on UPSs and rely on generator power. They will go down when the power switches to UPS, but will return to service when the generator kicks in. 

Some facilities make use of in-cabinet cooling systems. I am not a fan of these typically. They can be good in very high power cabinets that need it, but for most applications, they lack the resiliency most systems need. Some facilities can support liquid cooling to the systems for HPC types of environments, but those tend to be challenging and expensive and you don’t see too much of it in the wild.

There are also in row units. These are cooling systems that take up the footprint of a rack or half a rack and deliver cooling right into the row. This can be a good option for a completely diverse cooling system, but you rarely see this type of setup due to the costs and complexities involved. There are also completely self contained, “data center in a box” solutions where everything is preconfigured in a prebuilt container that provides cooling, power and connectivity. These tend to be seen more on factory floors and places where you need a reliable data center, but don’t have all of the infrastructure in place to support it.

Some facilities lack sufficient load for the cooling systems to work properly. I have seen several facilities over the years that are overcooled. These are typically privately owned facilities that were built out to support a much higher rack density, but due to more powerful systems and virtualization, the actual utilized footprint becomes very small. Sometimes when there is too much cooling going on, the return air is so cold it can threaten to freeze the pumps. This is why many of these CRAHs and CRACs have the ability to provide heat as well as humidity control. This is why you will sometimes see some CRACs in heating mode while others are in cooling mode. In fact in new data centers that aren’t at capacity, you will see them bring in load banks, which are basically giant space heaters to put enough load on the systems to work. They also use these for test and commissioning facilities. You can also see a similar problem with humidity control and air handlers fighting over humidity set points if things aren’t configured correctly.

There are temperature and humidity sensors throughout the cooling systems, but there should also be temperature and humidity sensors on the cabinets so that they can be properly monitored for hot spots. The sensors can be fed into monitoring systems to provide data to improve the operation of the facility. In some cases it is fed back to the cooling systems so that they can properly regulate themselves.

Cooling Delivery Systems

Let’s talk about the delivery systems. In most data centers, cooled air is delivered through a plenum under the floor, and hot air is returned through a ceiling plenum. The height of the raised floor is important, because it can determine how much cooling capacity the facility can provide. Data Centers are essentially closed systems, so what you put in is what you have to take out. Cooling systems need to be designed and calibrated correctly to maintain the proper static pressure to provide the appropriate amount of air flow through the facility.

Perfecting the airflow is a tricky job. There are many different types of floor tiles that you can use to route the cooling. There are high air flow tiles for higher powered racks and lower air flow tiles for lower powered racks. How you configure the tiles can effect how well your systems are cooled and how efficient the systems are. 

Not all facilities have raised floor. In some cases the rack are bolted directly to the concrete floor. In these cases, you typically see ducted systems. Often these have damper systems that can attenuate airflow based on the room temperature. They can be very efficient at putting cool air where it needs to be. 

Containment

Another thing to look for is hot and cold aisle containment. These amount to everything from plastic butcher curtains to high end glass containment systems. They are both effective for containment.  It is really important for containment that you have all of the fans on your equipment blowing the correct direction and that you make good use of the blanking panels to really create separation.

Whether to do hot aisle or cold aisle containment is a religious battle, and honestly, I have found  little difference to each approach. The best is both hot and cold aisle containment. In facilities with proper containment, you will often find that they have a higher set point for the rooms. When you are walking through the non-equipment aisles, they can be warm. This is because they are focusing on cooling a smaller volume of space. 

There has been a lot of study around the proper temperature to run systems at. When temperatures are elevated, there can be a high equipment failure rate, so understanding the limits is important. All this is to say that HVAC can be a complicated process that involves a lot of tweaking to optimize, so make sure you are comfortable with the facilities staff capabilities.

Fire Suppression

Lets talk about fire suppression systems. Most facilities have a mix of things. And there is a lot of complexity to designing these systems, and if you don’t get things right, the fire marshall will shut you down. First off, everything needs to have the right fire sealing. All the wall penetrations have to be sealed properly. Ask what the fire rating of the walls is. This is a measure of how long it will take a fire to burn through a wall so you know how much protection you will have if there is a fire in another part of the facility. You can usually identify the fire systems as they are often colored red by code.

There are usually multiple detection systems. You will find more traditional smoke detection systems throughout the facility, in the air handling systems, under the floor and in the ceilings. Then there are the VESDA systems (Very Early Smoke Detection Apparatus). These systems are extremely sensitive to particulates in the air. They pump air into the system through orange tubes around the facility. This is one of the reasons why having cardboard boxes is discouraged in a facility. Beside the fact that they are a fire accelerant, when you tear them, the particulates can set off the VESDA systems. Sometimes even moving dusty floor tiles can set off the under floor systems as well.

Most facilities are required by law to have a sprinkler system. It is usually a bad idea to have a large volume of water sitting in pipes above sensitive equipment, so you have dry pipe pre-action systems. In these systems, the pipes are pressurize with air so they stay dry, but if a sprinkler head breaks off the air will escape and water will fill the pipes. Facilities also often have wet pipe systems in non-critical areas like hallways, so don’t be surprised by that.

Most facilities also have chemical suppression systems. These can put out a fire without doing damage to the equipment. In the old days it was Halon gas, but now there are safer and more modern gases like Novec 1230 and others. Usually these are triggered by the VESDA systems or a combination of VESDA and smoke detectors. It usually requires more than one sensor the trigger a dump, because dumps can be dangerous and expensive. 

There is usually a button by the door to manually prevent a dump to give you time to resolve any issues or evacuate staff. If the chemical suppression systems fail to do the job, the heat from the fire will cause the sprinkler head to release and water will dump on your equipment, most likely destroying it. These systems are typically zoned, so when they go off not everything is impacted.

Another type of system that you will occasionally see is mist systems. Highly purified water is non-conductive and can be used to suppress fires without destroying equipment. These systems were traditionally used in submarines, where you have a very contained environment and not only is putting out the fire critical, not suffocating is also really important. You can’t just open the door underwater.

These systems are expensive and difficult to work with. They often use pressurized nitrogen to produce the mist. They emit a fine mist of purified water that lowers the temperature of the fire until it goes out. There was a time when these were more popular, but I don’t see too many of these as they don’t always work as advertised.

Another fire code requirement is to have EPOs or Emergency Power Off tied into the fire suppression system. When the fire suppression systems go off, the power to the area impacted should shut off. You should understand how these systems work and what the scope of the impact is.

Network

Let’s talk a about network. The facility should have diverse entrances for the telco you need. Make sure your provider has the building “lit” otherwise you may be forced to pay access fees to other carriers and it may mean that your diverse carriers traverse the same path into the facility. I have seen companies that thought they had diversity lose all their network and phone connections because everything was coming in over the same access equipment.

Telco typically comes into several different meet-me-rooms throughout the facility. Sometimes carriers terminate in those rooms, and sometimes they will extend directly to the customer. Usually it gets dropped off as single mode fiber of different flavors depending on the distance from the telco CO to the facility. Some carriers will drop circuits off to equipment they own in the meet me rooms and extend multimode or copper to the customer. Check your orders thoroughly to make sure you have the correct handoff and port speed. This is one of the biggest problems people have in data centers. It is very typical to get circuits delivered in a way that your equipment can’t use it. Always confirm the handoff.

Most data centers offer an Internet connection product that is usually a blend of two or more carriers over a blended BGP offering. You can usually peer with the data center routers and receive and publish routes for these connections if you have your own IPs and ASNs. Make sure their offering meets your needs. These offerings can be very affordable, but they are often shared connections, so keep that in mind when choosing. You may encounter a noisy neighbor scenario.

Another service to investigate is whether they have inexpensive connectivity to another facility or even to the NAP. This can save you a lot of money as well. Many of these operators will have multiple facilities around the world, interconnected by their own fiber, and they will sell you slices of it for a good price.

All of this will come to you over cross connects. This can be a real profit center for some facilities. The pricing for cross connections can vary widely. Horizontal cabling is usually handled by the facility, but in multi-story buildings there is vertical cabling that a riser management company, separate from the facility, maintains and there can be charges for that as well.

Last I want to mention again, if you are looking at utilizing cloud, make sure the facility you are using has direct connect capabilities. This can save you money, hassle, and time, and provide you with low latency connectivity to your favorite cloud providers. Investigate your options.

I’m going to wrap it up there. I will be back for the final installment where I will cover security, add on services and tips for negotiating your contract.

Thanks for watching. Like subscribe and hit that bell for updates, and I will see you in the next video.