Increasing dependence upon IT systems to deliver healthcare services to 140,000 people means availability and uptime are mission-critical at Birmingham Women and Children’s NHS Foundation Trust. Schneider Electric explains how EcoStruxure IT Expert has helped the Trust take a major step forward in the assured delivery of services.
The Trust was founded in 2017 as a merger between two existing hospitals and is a specialist provider of healthcare services to women, children and families living in the Midlands.
Operations are supported by an extensive IT infrastructure whose functions were combined and continue to be integrated on the campuses of both hospitals. Following the merger, most IT equipment is centralised in what Jas Purewal, IT infrastructure manager, describes as a “shiny new data centre” at the larger of the two sites, formerly the Birmingham Children’s Hospital.
Another older data centre is located at the site of the former Birmingham Women’s Hospital and there are also various hub rooms – distributed IT facilities – throughout both campuses which house servers running applications needed to support healthcare, switchgear and networking equipment. Some hub rooms also host physical servers running SQL-based applications, but there is an ongoing programme to virtualise the majority of the Trust’s applications on NetApp servers housed in the main data centres.
The necessity for always-on power for IT and patient health
Continuous uptime is the top priority. Any impairment to the IT systems running patient-management or clinical applications inevitably impacts the delivery of treatment to patients. To ensure continuous operation in the event of a computer system’s failure, the Trust maintains a detailed disaster-recovery (DR) plan. This sees the data centres in each of the constituent hospitals acting as failover resources to one another; in the event of a systems outage in one location, the servers in the other take up the load immediately.
Of more fundamental concern is the mains electricity supply itself. To guard against any service disruption, the Trust operates nearly 100 uninterruptible power supply (UPS) systems which provide battery back-up power to essential systems in the event of a utility power interruption or blackout. These are designed to provide temporary ride-through power either until mains power is restored, or in extremis, until secondary back-up power generation can be brought online. Many of the UPSs are installed outside the main data centres, in remote edge locations, on both campuses.
A runtime challenge for the Hospital Trust
Maintenance of UPS systems is a vitally important routine. Because UPS batteries have a finite operating lifetime, they must be checked and replaced at regular intervals to ensure that they are fit for purpose should they ever be required to respond to the loss of mains power.
Traditionally, an inspection of the UPS battery systems has been a manual process conducted annually. The disadvantage of this method was that any degradation of the batteries that occurred between scheduled maintenance operations was not visible. This presented the risk that in the event of an outage or blackout, the UPS systems would be unable to power the IT infrastructure until the mains power was restored.
Just such a problem occurred during a routine generator test: a UPS that was specified to provide more than 20 minutes runtime to the load failed after only a few seconds, causing the main data centre to crash.
According to Purewal, “If our systems shut down gracefully, they will come back up gracefully. But in this case, because the power was cut abruptly, it took us several hours to recover all our systems. The most important ones were returned to full operational status within an hour, however, others took three to four hours to recover. We also had to replace some disks over the next few days, so it was a very busy time with all hands on deck.”
Whilst the loss of IT services presented no risk to the health of people in the hospital’s care, the incident caused a headache for the IT department, with the realisation that an unplanned mains outage might adversely affect the hospitals’ ability to deliver critical IT services. As a result, the issue was discussed at the board level and a decision taken to improve the resilience of the Trust’s IT systems to withstand any similar instance in the future.
Automating inspections; using data to make decisions about availability and uptime
The Trust decided that it needed a better system for monitoring key infrastructure, such as UPS systems, and it engaged the support of Advanced Power Technology (APT), an Elite Partner to Schneider Electric and specialist provider of energy-efficient critical power and cooling systems. APT had previous experience at the site, having installed Schneider UPS systems for the Trust.
APT looked at what was needed and recommended the installation of EcoStruxure IT Expert, Schneider Electric’s next-generation data centre infrastructure management (DCIM) software to remotely monitor and manage the UPS and battery systems. This cloud-based application allows connected hardware assets in data centres, as well as distributed IT and edge locations, to be monitored continuously from a central console. Status updates and alarms can be routed to any remote access device, such as a notebook, tablet or smartphone.
“The main thing we like about EcoStruxure is its dashboard facility,” says Purewal. “You can very quickly see if you have a critical problem that needs immediate attention. It also has very useful features like power forecasting, which allow us to estimate how long a battery is likely to last, even if it’s not in critical condition so that we can plan for timely replacement.”
Another useful function is the ability to assess the effects of increasing the load on a UPS before additional equipment is installed. “This is where EcoStruxure is very good,” says Purewal. “We can assess the impact of additional load and determine whether we need to upgrade the UPS at the same time. Using data from the application, we can decide whether to scale up or scale down the UPSs to right-size them for the IT and optimise the system for efficient operations.”
Evaluating EcoStruxure IT Expert
EcoStruxure IT’s capability extends beyond accurate monitoring of UPS systems and batteries, as it can also be used to monitor equipment throughout the physical and IT layers, as well as environmental conditions such as temperature and humidity. The dashboard feature means that information can be easily absorbed by management who are not necessarily IT specialists. This is particularly useful in the case of the Trust where responsibility for the environmental management of the smaller IT hub rooms is shared between Purewal’s IT department and general estate management.
“The data centres are my team’s responsibility, but a lot of the networking gear is in small rooms – remote IT installations where the environment can be dusty and the ventilation poor. Overheating in these crowded facilities could easily become a problem,” he says.
“With the dashboard and graphical functions available in EcoStruxure IT Expert, I can make the case to management in other departments, or higher up the chain, that some small investment in air conditioning, or perhaps moving the equipment to a more suitable room, could reduce operating costs in the long run, as well as improving availability and reducing the total cost of ownership.
“We’ve got this great tool now from a leading vendor and that gives us great peace of mind as we try to get the most benefit out of it.
“APT’s people have extensive knowledge that we can tap into, and they have been very supportive when we have any questions.”
Moving forward, the use of EcoStruxure IT Expert is being extended to monitor environmental conditions in the IT rooms including temperature and humidity. This will enable the IT department, in cooperation with estates management, to provide detailed information to senior management to drive investments that will not only guarantee maximum uptime but also help the Trust to manage its hardware assets in a more sustainable and efficient way.