What can real-time data do in an emergency?

DAN LOPEZ

I think it was February 11th when it just dropped below freezing outside and the temperature had been consistently plunging for a good number of days. And, starting on the 11th, it just stayed below freezing.

REBECCA AHRENS

What was it like trying to take care of two young children while you're going through all of this and not really knowing what's going to happen next? You know, your water goes out. You're experiencing these dips in electricity. Maybe it's going to go out entirely the next day. You don't know. I mean—just what was going through your mind?

DAN

I mean, I'm not going to lie. It did scare us a little bit. There was a point where we still had power out and we were just watching the temperature in the house get lower and lower and lower to the point where—since we have all this data we know what our house temperature normally is—we can just say it's colder in our house than it's ever been. As in: the kids do not know what it's like to be in a house this cold.

REBECCA

From AVEVA Studios, this is Radio Pi. The podcast that investigates how data and technology are shaping the future of the industrial economy. So, let's get into it.

Hey, I'm Rebecca Ahrens. This is Radio PI. And today we've got a special episode for you. We're going to be talking to one of AVEVA’s own engineers about how he used a home version of AVEVA’s industrial data management platform, the PI System, to protect his home and family and even do his small part to help support the power grid during the polar vortex in Texas last winter. We'll also hear about how power and water, utilities, research facilities and other industrial organizations can use these same tools to make their operations more reliable and energy efficient.

At its heart though, this is a story about personal ingenuity and the incredible power of operational information—even when we use it on a very small scale. We’ll talk about how data can make us more resilient and give us a sense of control in the face of a catastrophe.

The polar vortex meets data

DAN

My name’s Dan Lopez and I’m a senior pre-sales engineer in Austin, Texas—so good ole center of the US.

REBECCA

As you might remember, this past February, Texas was hit with one of the worst winter storms in its recorded history.¹ Millions of people went for days without water or power. Some of them even died just trying to keep warm by any means necessary.

So I'd love it if you could just walk me through the story of what happened to you during the storm. When did you first kind of realize there was going to be a problem?

DAN

Really, the part that started, started making us all worried because we just saw any standing water, even just things are just barely damp, just start getting ice crystals all over them. And the part amongst all this that really got us worried was on Valentine's Day, actually. We have our home PI System set up to monitor not just all this really exciting data from our house, but also the status of these sensors.

REBECCA

Just for a little background, the PI System is a data management platform that allows users to access real-time sensor-based data from virtually any asset.² These assets are typically pieces of equipment like machines or devices or even just parts of machines that are used in industrial settings. Once you've collected your asset data into the PI System, the software allows you to structure, visualize and track it in an accessible and intuitive way.

This makes it easy for industrial operators and managers to keep track of how well their equipment is working day by day, hour by hour, or sometimes even millisecond by millisecond. Like I said, the PI System is designed for industrial use, but because Dan works for AVEVA and is generally an inquisitive and enterprising kind of guy, he has access to his own home PI System, which he uses to run fun data collection experiments and explore all kinds of interesting data-driven questions. But more on that later. For now, back to what happened with Dan and his PI System during the storm.

DAN

And on the 14th in the afternoon, we got a message saying that the sensor that we have installed on our water meter, like a below-ground water flow sensor, it went offline—because, basically, it froze. At that point we were kind of getting a little worried because if a sensor that's below ground, right next to our water pipes, is freezing, then, I mean, it's feasible that the pipe itself could also freeze. And that's, that's really what got us a little afraid.

REBECCA

And this was before the services had been shut off?

DAN

Yeah. So, we still had water flowing, so the water was still going. So, this is the point where they always tell you to leave your taps dripping a little bit. And we were already doing that. But now it's like, okay, let's be really vigilant about it. Let's do every single possible thing we can to keep water going. And then, of course, three days later, then suddenly we start hearing weird like popping and crackling sounds and then water service just stops.

REBECCA

Yikes. Wow—

DAN

Yeah.

REBECCA

And what about the power? When did you first lose power?

DAN

When the power went out at, I think it was, like, before 5:00 A.M.. I noticed the clicking sound from all these things, stopping the fan going. For some reason that woke me up. But then the first time I was like, “wow, it's going to start getting colder. Francisco and Amelia are going to start getting colder.” And then suddenly I'm awake, 100%. No need for coffee. It’s like, okay, we're up.

So, Amelia hits the point where she has to take her nap. So I say, okay, it's too cold for her to nap by herself anymore. So, there was a morning where I just had to sit there back on the bed and lie down and just to be a warm bed for her and not move, not check my phone, just kind of lie there. And that's the point that really—that's a real eye-opener for you when you're like, “wow, I'm just I'm having to be a warm bed for her,” because I have no idea how long the power is going to be out. And I've got blankets ready to just keep piling them on top of her because if it gets colder, we're just, we're just going to have to make sure that she doesn't get colder than she's ever gotten in her life.

REBECCA

Wow. Such a powerful image to think about. So, I'm curious about the system that you have in your house for monitoring the water and the power. So, if you could just, you know, on a very basic level, talk us through what that system is and then talk a little bit about what first motivated you to put the system in place—because, obviously, you've had it in place since before the storm, it just happened to come in extremely handy during the storm.

DAN

The nice thing is that the way we tackled our house was very similar to the way our customers do it. And that was actually the original inspiration. My background was in mechanical engineering, not in software, and, so, I've used a lot of software through school. It was just so helpful to be able to go through that in-person experience of, like, what our daily users do, who use this kind of software. So, the whole reason it started was the same way that people always use software like this. We had data-driven questions and we wanted to answer them. So, the first one was, I think, we started with looking at our HVAC, with temperature sensors. So, we're able to pull data from the house thermostat and then also from a bunch of other little IoT sensors we had. And it just really kind of, like, spiraled out of there.

So we've done everything from putting sensors on our refrigerator door, which is a project about doing interrupt-driven data collection—so not pulling or scheduled, but as soon as on a hardware level, the door gets open, it fires a message to a PI server—to putting something on my dog's collar because we were looking at geolocation and having him be a little, little geolocation beacon. And so for the water and electricity, that was, actually, really, really straightforward because that's very well understood from an industrial standpoint. So, for the water meter, we put in a non-contact water meter that detects the spinning metal inside a water meter because every spinning piece of metal technically makes a bit of a field and the field cycles at the same rate as your impeller.

So we say, oh, well, you can use that to back out how much water usage you're using in real time. That's exactly how our meter works. And then for electricity, just right inside our breaker box, we had an electrician install some clamp-on current transducers, which just directly let us see exactly how much electricity we're using, like, in as real time as humanly possible.

REBECCA

So, it sounds like, you know, this is primarily motivated by your own curiosity and ingenuity.

DAN

Yeah.

REBECCA

Did you have a thought—you know, I hear you when you're saying this is a lot about learning and getting that data from your own house—did you have a thought at the time like, oh, well, maybe we could use this to regulate how much power we're using, maybe save the family a little money or that kind of thing? Was that in your mind at all?

DAN

Oh, for sure. I mean, for one thing, it was really interesting and really fun to see this data in mind. But, but I mean, just working where I do, I get to see—and in particular in my job role, which is really fun—I get to see people put these kind of tools in action.

So we get to see people say, “Oh, we prevented this piece of equipment from melting down,” or “we shaved 10% off of our annual usage,” and this translates into however many thousands of dollars. So, making any little changes really can make a difference. And same thing with electricity. Both my wife and I have some pretty massive home office setups and all sorts of little sensors and gadgets around the house and knowing, "hey, wait, we can use this data to let us be smarter about this," pretty quickly, we can just back out how many dollars and cents we'll save per year. And it's just it's super clear. Wow, this makes a huge difference.

REBECCA

And so how did it feel having these tools available to you when the storm hit, knowing that you could collect that kind of data?

DAN

And that's a really interesting question because I think I kind of contrast that with just also being a parent. So I remember, for instance, like when a kid gets sick, the first thing you think of—or the first thing that I thought of is like, "What do I know?" Like, "what do I know? What do I do?" And it's almost as if like you're, like, my mind’s just reaching out, straining for—Like what things I do, what medicine I know, what are her symptoms, what have I read, what do I remember?

So you reach out for things to grab, like the things that you can do to change this. But in this case, it was really interesting because we weren’t in a situation where there was nothing we can do. We knew how to melt water for the toilets and ration out the water. Like, we knew those things, but knowing that those terrible scenarios out there, like a water leak, something breaking, something bursting and ruining one of our walls or ceilings, it was, it was just amazing knowing, wow, that's not something where we have zero control. We actually have a tool that can make a difference here. Like, it was, it's not necessarily something that I can say it was going through my mind in the moment. But when I look back and see how we were able to stay, stay so calm through it, I can say that that has to be a factor.

REBECCA

Sounds like it must have felt a little empowering, right? Like you were saying, there's a thing I can do, there's something I can hold on to, to feel like I have some control.

DAN

Definitely, like empowering in the purest sense of the word, for sure.

Applications for utility companies

REBECCA

So you already talked about this a little bit, but as you mentioned, you know, for your job, you often talk to power utilities, for example. Can you explain how a system like the one you have set up would benefit a larger utility?³ Can you explain what are some of the benefits of these technologies?

DAN

I think if you ask anyone who works in TND or in power, “hey, if you could park your desk next to your substation or next to your transformer or right next to the main stator, would you do that?” They’d say, “oh yeah, in a heartbeat. So then I can just walk over, I can make all the readings.” But there's just so many pieces of equipment out there, so many devices that that's, that's not feasible in a system like this means they have all that data immediately at their fingertips whenever they need it, and in exactly the right format so they can get back to doing their normal day to day work without having to just be scurrying everywhere.

But that's actually one of the really nice things of my role was that, actually, even currently I work with folks who operate some of these assets here in Texas and help to maintain them. And it's just, it's really clear from that role how the tool helps buy people time. It helps them apply their expertise at a greater scale than a single person could.⁴

They can do more with a few clicks than they would have if they had to do it all manually. Even in our house, when power started flickering and we saw some notices from different entities saying, "hey, any effort that folks could do whatsoever to reduce their usage would, can make a vast impact." We actually used the locked data from our current transducers to do that exact same thing.

REBECCA

Because that just means if each individual household is using less power, that means there's more power available on the grid for other people.

DAN

Oh, yes. For me, it's a little hard to realize "oh, how much of a difference can we make?” But when electricity came back on for us and the thermostat kicks back on, we start warming up and we read that on our phones. We immediately are thinking, “Okay, let's see what we can do.” We're looking at a live dashboard of our electricity usage and we see how many watts we're using. We just look at this and you think, "Okay, well, I think we could make a difference." And there's a certain baseline level and you can perhaps even see little patterns from different devices. That is just massive motivation to keep searching and just keep finding things. And ultimately, I think we shaved about 100 watts off of our typical minimum usage.

REBECCA

And so what is your typical usage? (Just to put that in context.)

DAN

Our usual minimum usage, like, our baseline, hovers at around 277 watts. And we're saying, okay, that's about the electricity that we use, say, for instance, during the night hours when no one's using anything active—and, yes, our AC and HVAC, like that kicks on and off. But usually around like 277. And we were able to get it down to 111 flat.

REBECCA

Wow. So quite a dramatic drop.

DAN

Yeah. And that's just what's what was staggering because we're thinking, well, all we're doing is going through and finding all these tiny little changes and realizing that, wait a second—

REBECCA

Unfortunately, at this moment in our conversation, the connection dropped out briefly. But what Dan was going to say is that all of these little changes and incremental improvements do over time add up. And if you think the impact that one family in Texas can have using these tools to reduce their own energy usage is impressive, just imagine the possibilities of using these kinds of data collection, visualization and analytics tools on an industrial scale.

For example, Thames Water is the UK's largest private water and sewage utility provider. The utility handles six times the volume of water in Sydney Harbor, each year, delivering 1 trillion liters of water to customers and processing 2 trillion liters of storm and wastewater. As we discussed in the last episode, water treatment and distribution is extremely energy intensive. The total amount of energy Thames Water uses in a year translates to about 1% of the total amount of energy used in the entire UK. To help reduce this energy burden, the utility uses real-time sensor data from its SCATA system, along with some of the same tools Dan and his family use to gain greater visibility into its operations. In one year alone, Thames Water was able to reduce its energy usage by 10%.⁵

Applications beyond power utilities

America's National Institute of Health has used the same tools to help improve equipment reliability by 50%, leading to a 10% to 17% annual utility savings. And a team of Harvard Medical School researchers has used the same real-time data collection and visualization tools to monitor and control campus electric steam and chilled water assets to prevent expensive overages during peak times. As a result, they cut energy consumption by 15%. Even NASA has used these tools at its Langley Research Center to collect data from over 400 metering devices and saved over 3000 kilowatts per year in energy consumption.⁶

And of course, all these efficiencies don't just save power, water and money for industrial facilities and the taxpayers who support them. They're also reducing our overall carbon footprint. For more details, you can check out the links in the resources section for this episode at OSIsoft.com/Radio-Pi-Podcast. And now back to my conversation with Dan.

So what would be kind of like a comparable example on the grid level, like, in transmission and distribution throughout a state? What would some of those small changes and adjustments be that would give savings to a grid operator?

DAN

Yeah. So this is what's really interesting, because in my case, I'm able to look at a steady state—like the baseline usage while we're asleep—and make that my target. When you get things to a system like that, with so many variables, so many independent loads on the grid, the problem's just magnified immensely. Long story short, there are a lot of different events or scenarios that occur on a regular basis.

REBECCA

These events or scenarios Dan is talking about can be a sudden curtailment event that a power generator has to respond to, or conversely, a sudden increase in demand caused by weather or even something like a festival or big event that draws a lot more people into a city or town than are normally there. All of them plugging things in and putting a larger-than-average strain on the grid. As we'll learn in an upcoming episode, balancing supply and demand on the power grid is an extremely delicate, high-stakes, real-time response of Dan's, in which thousands of people's lives may be on the line if an accident or misstep creates a cascading outage.⁷

DAN

And the tactic I'm talking about is—a single person, of course, can only hold so many memories of such a complicated event in their head where they're doing comparisons. And so one of the things that I see people doing is saying, "hey, we've got the expertise to define what some of these events are. We know of distant process variables, different megawatt readings from single assets or collections of assets that we can all plot in combinations with flags from the ISO."

REBECCA

ISO stands for independent system operator, an entity that coordinates the entire power grid within a state or group of states in the U.S..

Possibilities in predictive maintenance

DAN

Let's make sure that we never forget those and also that we can very easily draw comparisons between those events and what's going on day to day. I mean, the old adage of people who don't know their history are doomed to repeat it. What we're seeing people do is use basically, like, event-tracking or, like, bookmarking sections of their history to make sure that those events don't go unnoticed. And that doing those kind of comparisons isn't something that's limited by an individual's capacity to remember data. Things like, “hey, let me just take a look at these last five or six or seven or 30 days that all meet this certain criteria that we want to avoid, or that we want to replicate.”

On one hand, of course, the data gives you that real-time visibility. It gives you history so you can see how things have performed in the past. You can build up, like, predictive algorithms, say what you see, what we expect generation to be, given a set of current conditions over the past day, two days in that we are seeing people do that using predictive analytics to try to guess a little bit, and make very, very informed, very educated guesses—and shockingly reliable ones—about how these assets are going to perform. And that is the interesting thing because, at the end of the day, there's still going to be a degree of control that's not there. And so, while that predictive and advanced analytics and pattern matching, all those efforts are incredibly valuable, just in terms of like, “hey, how can I make best use of my time right now?” Well, there's also a pretty staggering amount that can be done just to make sure that the rest of the infrastructure is working as reliable as possible. And so, using these same tools to say, “Hey, well, if we know that there's going to be a degree of unreliability, let's see like what we can reduce in the overall chain like condition-based maintenance, like predictive maintenance, making sure that we can guarantee that everything else is as at tip-top shape as we can.”

Does that make sense in like making sure that we focus on that side of things, too?

REBECCA

Yeah, I guess my understanding of what you're saying is that there's going to be unreliability inherent in any system.

DAN

Yes.

REBECCA

Some of that we can mitigate with technology. Some of that would be harder to mitigate with technology, like when the wind is blowing, for example. But for those things that we can mitigate with technology, we should, and we can improve the systems if we do.

DAN

Yes, for sure.

REBECCA

And just as my last question, if you could sort of give one piece of advice for engineers who are looking to improve specifically power distribution systems—you know, maybe making their grid more resilient—what would be your one piece of advice?

Tapping into an ecosystem of knowledge

DAN

Hands down, the folks who keep the lights on in America right now and across the world, they're ultimately the experts in every aspect of this. One of the exciting things here is it's less of like a single capability of the software but more in fact that, how connected everyone is and how connected we are to stories from other folks in the industry.⁸ Whether that's interest missing distribution, whether it's like food and beverages, manufacturing, or metals and mining, folks are having amazing successes.

I mentioned with things like predictive maintenance or with creating predictive algorithms that let them say, “hey, is this current flow level going to rise above a certain amount, or are we expected to have a drop in generation in two or 3 hours?” I think one of the really exciting things is I think we're starting to move into an era when people are doing incredible things and it's not a success that stays within a certain region or a certain city and that isn't able to propagate throughout the world. Thanks to the Internet, thanks to computers, things like reductions in storage, all these different things, these stories are getting captured and they're being shared so folks can learn from each other. I think one of the amazing things is when people use software to make their lives easier, like, when they're working one of these critical roles, the software in and of itself helps capture some of the workflow and some of the steps that people do. The software is able to remember that, to, like, package it all up and allow it to be something that's actually just shared with someone else.

So I guess my big piece of advice is, I mean, the folks who run these assets again, like, hands down, they're the experts in them. But when it comes to what are our capabilities that our software can do for letting me, like, extend my knowledge further, for letting me do that faster, regardless of which piece of software you're using, maybe see if you can spare a moment to reach out to the folks who are actually maybe the experts in your tool and say, “Hey, how can I make better use of this, this fantastic tool that I already have?”

REBECCA

So, it kind of sounds like what you're saying is don't be afraid to learn from other people, even if they're not in your particular industry. There are others in other parts of the industrial economy that are doing things with technology and software that might be applicable to you.

DAN

Oh, completely. And I totally get how when it's your job to keep a power plant or to keep a portion of the grid running, you might think, “oh, my gosh, having to do that research to figure out what are all these case studies? Who can I draw from? Like, who's a good peer that I could reference?” I can understand that it seems, oh my gosh, it just seems like a lot of work. The convenient side of it is, there's folks like me and there's other folks out there in similar roles. That is our day-to-day work. We are the resources who know of all of those things and it's just sitting down and finding time in people's schedules to have those conversations and letting us share those stories from others.

REBECCA

Learning from stories about the essential industries is what we're all about here on Radio PI. In future episodes, we're going to dive deeper into the ways people, companies and industries are growing more connected and are starting to share more data, insights and stories that can help everyone learn and improve. It's a phenomenon that some have started calling the connected industrial economy and it's likely to shape the future of, well, everything.

But for now, that's our show. Special thanks to AVEVA engineer, Dan Lopez, for sharing his remarkable story. If you like hearing stories like this, you can subscribe on Apple Podcasts, Google or Spotify. Or you can find us on YouTube or on our website.OSIsoft/radio-Pi-podcast, where you'll find transcripts, show notes, links to other episodes and resources. Until next time. This is Radio Pi.