To get an idea of how massive a healthcare network infrastructure is, imagine you’re the manager for a healthcare network that has two corporate data centers, 1,300 servers (1,000 of which are virtual), 120 active directory domain controllers across the country, and three VM environments and their 600 more servers.
iTunes|Stitcher|Google Play|TuneIn Radio|SoundCloud
On the third episode of the Data Transfer Show, host Greg Mooney and Steven Rogacki, IT Manager at Universal Health Services, talk about how network monitoring is actually a life-or-death matter.
You’re talking about thousands of devices to monitor on a given basis. Then you take all that and you add the complexity of monitoring about 300 Bedside Management Devices (BMDI).
BMDI devices are IV pumps, heartbeat monitors, and a lot of other essential care biomedical devices. “We do lots of basic monitoring, up and down,” Rogacki explained. “It gets a little complex on the alerting side.”
Depending on the device, if it goes down, you really want everybody—the whole world—to know about it immediately. “And then other devices aren’t as critical. After maybe an hour or so then we want to know if that device is down.”
Out With the Legacy, In With the New
The most complex and strangest issue Rogacki has ever had to troubleshoot on his network from a technical standpoint is slowness. “You know, going through an active directory and diagnosing why this log in is slow,” he said.
“Which one of the nine million factors that can be thrown in to a slow log in could be causing that issue? Is it really PC related? Is it a network related issue? Is it a LAN/WAN issue? That’s one of the probably one of the toughest issues my guys come across.”
One thing that makes all this network monitoring easier is phasing out legacy systems promptly.
“We’re pretty good at keeping stuff up to date,” Rogacki said, especially in comparison to other healthcare networks. “A lot of the legacy, like 2003 servers, are still out in our environment. But from our corporate enterprise standpoint, we’ve actually tilted out a good majority of them at this point.”
UHS had a big initiative in all its facilities across the country when Windows 2003 was coming out of support. “Everybody really pulled together and really did a good job getting rid of a lot of that stuff.”
That kind of concerted effort made it a lot easier to budget and plan for updates, too. Now they’re moving away from 2008. “Our standard is 2012,” Rogacki said. “Don’t talk to us about 2008. You need to go on 2012 or 2016 server at this point. It helps you stay ahead of that curve.”
Four years at UHS has trained Rogacki and his team to plan ahead. “We’ve become so proactive with things that we worry about the future and where we’re going with things, not just having to worry about constantly battling fires. Knowing what’s going on in your environment to be able to stay ahead of it helps big time.”
Healthcare IT Is About Humans
Service-oriented architecture (SOA) in healthcare really shows why forward thinking is so essential for IT.
“When you get into healthcare IT, it’s a little different because on the other side of the technology is someone in a bed, a patient,” Rogacki said. “That could be a child, it could be an adult, it could somebody’s mom, it could be somebody’s dad, it could be a brother or sister. And you always think of that human aspect to the technology all the time. It’s not just the server down, it’s what the server does.”
Rogacki urges his team to think about the humans on the other side of the machine. When they are architechting an enterprise solution for a dietary product, they had to really think about the implications of diet when tiering the disaster recovery aspect of it.
In healthcare, diet isn’t just about who’s eating how many calories a week. “If this app is down, a dietician might not have access to someone’s allergy record. And if they give them some kind of food that contains traces of tree nuts or peanuts in there, it could actually kill that patient.”
Rogacki credits his human perspective to having his son in a hospital where he worked in IT. Seeing how things he did in the background affected his son’s care flipped a switch for him to see how his work relates directly to the patient.
“So we always look at it as here’s the human aspect. It’s not just, here’s our financial app, or here’s our coder app. It’s what’s on the other side of that. How does that affect patient care, and life and death?”
It’s not even background stuff that affects patient care anymore. From when a person first walks in, care has changed. If you’re having a same-day procedure, a nurse can do all your triage with a wireless on wheels right at your bedside so you can relax and get ready.
“It’s just amazing how technology has changed throughout the years,” Rogacki said. “There are so many different things, even privacy and security concerns—so many aspects come in to it when you look at IT in healthcare. There’s a lot of different realms and considerations to every kind of architecture and project that you look at.”
Managing a Remote Network
Rogacki’s biggest concern in dealing with remote modern networks is latency. “We try and tackle false positives about whether a server is down in a couple different ways.” Using ICMP ping, putting in a delay, changing the packet sizes, to name a few.
It’s one thing to walk down to the hall to look at a server, but it’s another when it’s across the country at someone else’s data center. “You have that latency issue across the wind. Is there an issue in that person’s data center or at that hospital? Are they having a power outage? Or did they reboot one of their virtual hosts?”
A thick skin is required in network monitoring this vast. “That’s probably one of the biggest tackles, saying what’s really up and what’s really down,” Rogacki said.
It’s about knowing your network. Behavioral Health runs their Windows updates to just workstations at random times during the week. So you don’t jump instantly to worrying that the world is going to end.
Tackling the big issues is a delicate thing. And sometimes, you want to build down a little bit of a delay. “Can I sacrifice five minutes to say if this is real or not? In five minutes, am I going have a hundred phone calls or a hundred tickets? I could still be proactive toward an issue five minutes later.”
The next time you’re in a hospital, remember that it’s not just the doctors and nurses looking after you. It’s people like Steven Rogacki who are actually behind the scenes working on all these networks that are keeping these devices up for that situation.
Did you think it was eye-opening to know that IT people are critical to the operations of the healthcare system? Remember you can follow the Data Transfer Show @ipswitch on Twitter or at blog.ipswitch.com.