Good Evening to My Valued Partners,
I would like to ask for your time in reading this important blog post thoroughly to update you on the updated measures we are taking moving forward, what the escalation plan looks like, and receive comments or feedback from you if you would like.
About 3 weeks ago we released a public notice of enhanced precautionary measures related to COVID19 and reviewed/affirmed our DR plans should they be called for. We took a precautionary stance, per CDC guidelines, to cover many common sense, yet overlooked measures in an office environment via increased sanitization stations as well as actively encouraging our employees/ regular visitors to the facility to stay at home if sick.
During my career in data centers, and prior to that while working for MIT Lincoln Laboratory in Group 46 – Homeland Protection Systems, I have specialized on risk assessment as it relates to distributed disaster response, scenario analysis, and mitigation. I would surmise the reason many of you are on this list as a customer is appreciation for the careful thought and attention to detail that we demonstrated to you in our design and operations, as well as the commitment to the mission of serving you.
When dealing with mission critical infrastructure, it becomes necessary to think well ahead of the curve not just in design, but also staffing. The Uptime Institute’s most recent survey states that 70% of outages are due to human error. I like to think this human element is the greatest frontier in reliability upon which we invest the most time and effort.
Current Threat Analysis:
Based on current growth trajectory in the United States, and per CDC guidance, we anticipate that the virus will take foothold in Houston. There are many unknown’s related to a novel coronavirus such as how long it will take to vaccinate, will seasonality be enough to dampen the spread with respect to the virulence and long periods of shedding of the virus, and the extent and lethality to which the virus may impact our team or our customers.
Early data from other countries shows that with social distancing measures and closures, the spread of the local endemic foothold may be tapered off. This is very promising and gives us guidance to expected posturing, but I can assure you we also plan for the worst.
Based on these data, we are currently posturing for/expecting 2-3 months of varying/progressing degrees of social distancing measures to ensure the continuity of the facility for our clients that will likely ramp up in as early as the next two weeks. To be prepared, we have made provisions for sustained operations of the facility indefinitely which I will discuss in more depth later.
Here I will summarize some different threat vectors and defensive postures that we will be tracking actively going forward:
- For disruptions to public utilities we view the risk as normal.
- For disruptions to our supply chain for sustained operations we view the risk as heightened but mitigated. We have already identified any foreign, manufactured parts and ensured redundancy or stock, as well as any diesel supply chain should a disruption to public utility occur. We have on site water retention for fire water as well as an on-site well and septic system and have the capability to operate entirely off the grid.
- We believe that the risk of public unrest that could affect the facility is low.
- We believe that the risk of general emergencies is low.
- At this time and based on our anticipated timeline that we are currently posturing for; we believe that the risk of hurricanes or severe weather is low. In the event of a prolonged duration that is compounded later by severe weather, our staffing strategy (discussed later) will remain the same but we will place additional preventative postures in place to mitigate compounded risk. Based on the design of the facility, as always, our risk to those types of natural disasters is low.
Immediate Measures and Future Escalation Levels:
In a datacenter, we must consider not only our own staff required for critical operations, but also our customers and their equipment. We have reviewed the DR plans of ourselves, but also peer-reviewed other data centers in industry to make sure no stone is unturned. Most data centers today are reflecting a risk posture that are commensurate to infection numbers TODAY. I would like to take a moment to share why I think this is wrong and encourage you to think about these things for your own staff planning. In a briefing today, Tony Fauci put it beautifully in the words of Wayne Gretzky – “Skate to where the puck is going, not where it has been”. We have been and will continue to heed that advice to stay many steps ahead of fulfilling our mission to you.
Our original defense posture 3 weeks ago was a level 1 – heightened awareness. We are escalating that to level 2. If the CDC or local government officials recommends any type of city-wide shutdown or travel, we will increase to our 3rd level of escalation which we will discuss.
Level 1- Increased awareness – active encouragement of sick people to stay home. Wash your hands. Encourage customers to utilize our free remote hands on instead of trips to the datacenter.
Level 2- Active mitigation (Defense Posture Today)– active encouragement of sick people to stay home, sick people including customers may be asked to leave the facility, increased sanitization, equipment handling and receiving modification, encourage customers to utilize our free remote hands on instead of trips to the datacenter, non-essential personnel and REDUNDANT critical personnel staged at home.
Level 3- Emergency Response – facility will be staffed indefinitely with critical on site team members 24/7 with little to no entry or exit, customer access strongly discouraged but may be evaluated and must be approved on a case by case basis with appropriate PPE procedures taken, enhanced operational support with remote hands on. Non-essential personnel and REDUNDANT critical personnel staged at home.
Our Staffing Strategy, and Achieving Fault Tolerance of the Human Element in a Pandemic:
When contemplating fault tolerance, it is important to think of our people. Today, we have two teams staged in Houston for primary operations of the facility including redundant executive members of the team, along with a backup third party contractor resource that may also be used if necessary. One team as of today has been encouraged to work from home and to minimize travel, especially to the datacenter to prevent risk of breaking physical isolation required to maintain redundant local resources in the event of an infection.
In addition to that, we have executive officers that are trained and masters in the art of data center operations staged in two different localities around the United States with easy point to point travel via owned private plane. Both executive officers are pilots and the plane is owned to ensure ease of access and travel in a timely fashion. Car travel would be the backup in the event that both local teams were compromised and requiring a remote executive member to travel to the facility if private air travel is not possible.
We would encourage you to think about physical isolation between critical staff on your team as an immediate measure, and consider the risk when making trips to the shared datacenter environment, and the ethical burdens we have as professionals in supporting our community and other members of the facilities which include Healthcare here at TRG today. We view our service in times of need as a public service, a passion, and ask you to do your part in ensuring the continuity of our city, your business, and your community.
If you would like to discuss the plan in further detail or discuss contingencies, please do not hesitate to reach out and we can set a time to meet formally or informally. We are open to considerations and want to stress that this is OUR – meaning yours as well as TRG’s – plan that we have provided as part of the services you purchase from us, and we welcome feedback and will do everything we can do to accommodate any needs or questions that you may have.
Thank you for your continued partnership.