The future of the power grid: Aging infrastructure

REBECCA

In the Spring of 2005, a transformer caught fire and exploded at an electricity substation in the southeast of Moscow.¹ The failure sent the city into a total blackout, which quickly cascaded into nearby provinces. Houses, office buildings, factories—everything went dark. Pumping stations went offline, taking out much of the city’s water supply.

The power outage trapped tens of thousands of commuters underground in stranded subway trains. Surgeons in dark hospitals did what they could by flashlight. In all, an estimated 2 million people were without power for nearly a day.

As for the cause, experts point to two main contributing factors. At the time of the blackout, Moscow was in the middle of a prolonged heatwave, which had residents throughout the city running their AC units on high. That alone would have put dangerous strain on any power grid. But Moscow’s power grid hadn’t been modernized since 1966, leading to contributing factor number two: decades of equipment wear and tear and inattention had left the entire system vulnerable.¹

While the Moscow blackout was a particularly dramatic example, this issue isn’t just a Russian issue. The problems that can arise from aging infrastructure are something grid operators everywhere have to worry about.

From AVEVA studios this is Our Industrial Life, the podcast that brings you stories from the essential industries and investigates how data and technology are shaping the future of the connected industrial economy.

I’m your host Rebecca Ahrens and today we are continuing our discussion of the future of the power grid. In this episode, we’re talking about aging infrastructure: what it means for the grid today, how it might impact efforts to decarbonize the grid in the future, and what data and technology can do to mitigate some of the risk factors.

Aging assets and transformer explosions

PAT KENNEDY

You ask about aging infrastructure. That’s a good question, because our power grid—a lot of it is relatively old. And it runs pretty much the way it did 100 years ago.

REBECCA

This is Dr. J. Patrick Kennedy, who often just went by, “Pat.” I just want to note here that we first recorded this interview with Pat a couple of years ago. Pat recently passed away, so I want to take a moment to acknowledge his immense contribution as a pioneer in industrial software. Over 40 years ago, Pat and his team launched a plant information software program known as AVEVA PI System. Decades later, the PI System is still an essential tool for over 1200 power utilities and transmission and distribution grid operators.²

Pat was regularly consulted by members of the critical infrastructure industry about how to maintain grid stability and reliability as we transition to new sources of electricity. You’ll hear excerpts from our interview in this episode and can hear more from Pat in the other two episodes in our three-part series on the future of the power grid. Here’s part of our interview:

So Pat, can you say a little bit more about these aging assets that make up the power infrastructure here in the U.S.? What kinds of things are we talking about specifically?

PAT

Well, some of the assets are monster assets. If you think, for example, that transmission voltages can be in the hundreds of thousands of volts, but distribution voltages, which go around the city, those are in the tens of thousands of volts. And the way you bring that down is you have these very large transformers that accept power at these hundreds of thousands of volts. And then they take them off at tens of thousands of volts. Those transformers are critical because they generally are not spared.

REBECCA

Right, and I’ve heard they’re also really expensive. So how do operators ensure that transformers live a long, productive life? I mean, what do they do to build that resilience, I guess you could call it?

PAT

You have to be careful because the word resilient has very definite meaning to a grid operator.

REBECCA

OK.

PAT

You have reliability, which means it stays at a certain percentage—and reliability is measured in the minutes per year. Resiliency means that you have enough flexibility left in the grid to manage an emergency. So if you're looking at a hurricane moving in, or you're looking at an ice storm or an earthquake, managing that grid during these stressful times is called resiliency.

REBECCA

I see. OK, that’s a helpful distinction.

PAT

But a good story of that: we were working with one of our power utility customers that were doing some asset management with our software. And what they had to do is that—based on what the asset was—they had to identify what these variables meant. This is the input voltage, this is the output voltage, this is the temperature, this is the current, etc. And then put them together into a calculation to see if there was a problem.

So they had all the variables in place, and at their central facility, they put the calculations together. And the instant they did that, the transformer turned bright red, indicating it was in incipient failure stress mode. They immediately sent people out—and the transformer was hissing, which is a very bad thing to happen in a transformer, because you‘re dissociating the acids inside there, and you could basically have an explosion.

REBECCA

Wow.

PAT

So they were able to take that transformer out of service, which was a fairly complex procedure, schedule a routine, and then go ahead and fix it, and bring it back up.

REBECCA

And, without that asset management software—and the advance notice that it enables, I imagine this story would have a very different ending?

PAT

That would be called a transformer explosion. If you go on YouTube and say “transformer explosion,” there are several of them that were caught on film. And these are dramatic things. These—the one in New York was so energetic it actually set up a wave pattern across the river when it blew up.

REBECCA

Wow.

PAT

And so that's what happens if these things are—if these things get out of control.

But can you really put spares in? These transformers are millions of dollars. One of the transformers in that same utility—I believe it was 85 years old.

Decarbonizing the power grid

REBECCA

A quick aside: I did look up the transformer explosion Pat mentioned. It happened in New York in 2018. You can go see for yourself.³ The night sky goes wild with this brilliant cyan light. It caused such a panic, in fact, that LaGuardia briefly grounded flights on a very busy holiday-travel weekend. You can find footage online of people filming the sky with their phones, wondering out loud if we’re under attack by a foreign enemy or being invaded by aliens. That’s how intense it looks.

Unfortunately, the possibility of exploding transformers is not the only problem caused by aging infrastructure. We’re living in an increasingly populous and electrified world, and much of it runs on equipment that might be, as Pat said, over 85 years old.

To decarbonize the power grid, we’ll need infrastructure that isn’t just less vulnerable to failures—we’ll need infrastructure that’s more efficient in order to meet the rising demand for power.⁴

The challenge is this: how do we get the most out of those existing assets and prevent them from failing in a catastrophic way that could have ripple effects across the entire grid? That’s where industrial software and data come in. But what does that actually look like—to use data to address these issues?

Using data to address aging infrastructure

DAVID BARTOLO

That is a fantastic question.

REBECCA

This is David Bartolo. He’s the Head of Asset Intelligence at AGL, where he leads a small team that concentrates on operational technologies supporting power generation assets, especially distributed, renewable assets, like solar and wind.

And before we get into the interview, I just want to note that David’s audio is a little noisy. We did our best to clean it up, but apologies about any bumps or scrapes, or email chimes or passing cars you might hear in the background.

David, I know your team has a lot of experience on this point. Could you help us understand how data and software can help operators improve the efficiency of your existing infrastructure?

DAVID

Just with wind farms—large, centralized wind farms—before we set up diagnostics, and we're monitoring using some fancy software that helped us understand anomalies—you can risk hidden losses. Wind generation is extremely noisy because the wind goes up and down.⁵

REBECCA

In other words, if you graph a data stream that represents the power output from a particular wind turbine over time, you will see that line move around a lot. It won’t show a steady output like you would expect to see from a traditional gas-powered turbine.

This variance makes it hard to determine what is normal variation in output due to changes in wind and what’s abnormal variation that might point to problems with the equipment. It’s a tricky problem to solve.

DAVID

So it’s very easy to have hidden, massive losses that you can’t see easily and manage easily. And that gets actually much more profound and of a much higher risk level when you have thousands of small assets. So you could have 1,000 homes, with their solar systems shaded by trees that have grown over the last six months through the spring in the summer period, and they’re now showing that they're not producing as much as they could. You could have batteries that are in a state of distress, that are not working properly. And you could easily not see this and not manage it.

So you may have billions of dollars’ worth of assets in the field that could provide you massive capacity. But that capacity is effectively eroded because you're not properly monitoring and understanding the health of your assets—and the degradation of those assets. So that's why it's such a big risk to us. It makes sure that we understand what we can do with the asset in real time, and where we need to take action to avoid further loss.

REBECCA

So it sounds like these are kinds of problems that you're describing that are particularly insidious, because they might, like, at the small scale, it might not be that big of an issue. But if you take that problem and compound it across hundreds or thousands of assets, then you're talking about a huge problem that’s maybe costing thousands or millions of dollars.⁶

DAVID

Absolutely. You know, you've got enough customers, and you've got enough assets out there to store 100 megawatt-hours of energy. But that's been degraded to only 70 megawatt-hours of energy, and you don't even know it. So you're incorrectly in the market, you're not correctly positioned. When you do need that energy, it's not there for you. You're paying thousands of dollars because you weren't aware of these—of the condition of these—thousands of assets that you're trying to support a national electricity network with.

REBECCA

What about, you know, bigger, sort-of more immediately show-stopping events? I mean, I hear what you're saying about these small inefficiencies that add up to a larger impact, but things like, you know, maybe a whole power generation system goes down: what role does data play in helping you make sure those kinds of catastrophic events are avoided?

Virtual power plants

DAVID

As we move towards virtual power plants becoming more predominant, the risks rise exponentially.

REBECCA

If you haven’t already listened to our first episode in The Future of the Power Grid series, a virtual power plant might not be a familiar phrase. A virtual power plant, often abbreviated to “VPP,” is a cloud-based network that aggregates the capacities of many decentralized, distributed energy resources, like rooftop solar panels, small-scale wind farms, battery storage systems, and other small- to medium-scale power-generating assets. In the aggregate, all these distributed resources form a kind of gigantic battery that can be used to support the grid.

DAVID

So in the future—maybe in ten years’ time—the VPP may be our largest, high-speed, dispatchable energy source into a large region—maybe a state, etc. If you don't understand the exact capacity of what you can do at an instant, you could place the entire network at risk.

If we think we can discharge 300 megawatts, instantly, of energy into that network during an emergency—and that is a frequency-control ancillary service that we are bidding legally into that network —and we can’t provide that instant power, that state’s network could collapse instantly. That is of extreme risk to this business. So that's…that's the nightmare scenario: where we thought we could do 300 megawatts, we actually turned it on during an emergency, we only got 100 megawatts out there, and that was too short. It left the network short and the network collapsed. That would be a nightmare scenario. Data—accurate data, well-analyzed data, well-understood data—is our only defense to stop that from happening.

REBECCA

Yeah, OK, that makes sense. So you’re saying it’s easier to keep an eye on the availability of, say, a single peaker plant, but when you have a lot of generation assets out in the field, suddenly it can get a lot more complicated to keep an accurate measure of your available capacity.

So, what are you doing to try and address that?

Reliable real-time data

DAVID

In the past, all we did with our wind plants is try to manage availability. That only gets you part of the way there. We looked at what was available in the market. And we found that with our real-time data infrastructure, many of the features that have helped to increase the yield from a wind farm—not just the availability—could be programmed ourselves.

REBECCA

Another quick aside here. When David refers to AGL’s infrastructure—or system—he’s talking about software that allows the company to collect, organize, tag, visualize and analyze real-time data coming in from the different power-producing assets.

You can find more information on how AGL is using this software on the episode page.⁷

So what kind of changes would you be making on the wind farm to improve yield? What are you looking at specifically—what kind of data streams?

DAVID

We try to gather as much time-series data from the turbines as possible. And we centralize that through our infrastructure.

And then we start to visualize different aspects of how the farm is performing. But also, we can try to produce future data forecasts.

REBECCA

Can I ask what you’re visualizing on these turbines? What is the data showing you, exactly?

DAVID

So first of all: just visualizing the output of each turbine in a way that can be really easily visualized over a month, day-by-day—on a single page or just for the day. And you can start to identify the turbines that are lazier than the others. Then you can do other correlations, because they might be— for certain directions and wind—they may be at a disadvantage, where you know that you're going to be a bit weaker in this direction, but you should have been struggling in that direction. What we're trying to do is see through the noise of the wind to see those turbines that are not performing.

REBECCA

And once you can visualize all of this—you use software to create displays—and you can start pinpointing which turbines are under performing, and so on, then what do you do?

DAVID

We then build more sophisticated tools that help us understand yaw error, where the turbines are not pointing 90 degrees into the wind. We produced some really, really nice tools to help us identify those turbines—we were losing big money on those—to the point it was so accurate that they could do—they’re starting to do—just digital realignment without having to climb the tire and do a full recalibration of the wind vane. We actually changed to a better-quality wind vane as well across a farm because we found it was so bad.

And now we’re doing much more accurate wind forecasting for the next five minutes. Again, it optimizes the way we bid that farm into the market. And we avoid fines for being inaccurate.

So there are just a few examples of what we’ve built from our data. Gather as much data as you can store it centrally and efficiently, and then produce more and more sophisticated and more and more mature models to get you more and more value over time. And that’s gone really well.

REBECCA

So it sounds like, you know, with these turbines, being able to compare, say, two turbines that are in a similar location right next to each other, in some configuration, you can start to dig into the causality of one that might be underperforming.

Because if you don’t have that data, you don’t know that it’s underperforming relative to, you know, its neighbors. Then you would never be able to figure out what might be causing that. Is it a material issue? Is it an orientation issue? Is it, you know, some other mechanical failure? Is it just a, you know, bad data stream? You have to have all of that information and be able to do that visual comparison to even start to address the issue of what might be the root problem.

DAVID

Absolutely. And there’s a number of vectors of doing that. There’s neighbor-comparison, as you said, where we compare them to each other. But we also do advanced pattern recognition monitoring of each turbine as an individual, where we learn the history of how each point data point correlates with each other data point. And we can pick up anomalies that way as well. So neighbor comparison, fleet comparison, advanced pattern recognition. And also, even with the forecasting, it's not just, you know, a few wind data points, and try to understand what the output will be. What the team has produced is where they understand that the wind is propagating across the farm, like waves on a beach. And we're able to program that in and forecast based on an understanding of how those wind waves are propagating across the farm. It's actually quite fascinating.

Historical data and advanced modeling

REBECCA

Hmm. So this seems like this is where it becomes important not just to have the real-time data that’s informing you about what's happening right now, but to also have a rich history of data about what's happened before, so that you can compare the current situation with prior situations. And like you said, compare, maybe, the state of a turbine to its past performance.

Or start to understand these, you know, this movement of wind as waves as you described, just to understand—if you have that historical context, you can see more clearly what your current situation is telling you.

DAVID

Absolutely. And the history—you touched on the history, which I didn't touch on hard enough. The more rich history you have at the higher resolution, the better understanding where you are. You can understand where your asset is right now.

And advanced pattern recognition is extremely powerful. We're running three-and-a-half thousand models across the 11 gigawatts of generation we have—including large solar, large wind, nine wind farms, two big solar farms, hydro, fossil fuel, gas turbines and steam turbines. We've got three-and-a-half thousand models predicting what 52,000 data points should be doing every five minutes. And then when you can see those deviations from normal behavior, you're picking up failure modes earlier than any other technology can provide.⁸ That’s rich.

REBECCA

How much value can advanced modeling technology like this add to your operations?

DAVID

We try to keep a log of the value-per-year of that system. There’s three engineers full-time on that system. And we're delivering between six and seven Australian million dollars per year value recovery from just that one system.

REBECCA

Wow. Yeah, that’s a lot of savings from a single system. And I think it helps put into perspective, you know, what a massive savings there is to be had just from collecting and analyzing operational data.

Thank you so much, David, for walking us through everything you’ve been doing at AGL. We really appreciate it.

DAVID

Rebecca, it’s been my absolute pleasure. Thank you so much for being patient with me as well.

Physical infrastructure

REBECCA

There’s another important piece of the infrastructure puzzle we still need to consider, here. Yes, we need to optimize a lot of the infrastructure that exists today. But if the goal is to eventually decarbonize the power grid, there are also pieces of equipment that are going to have to be phased out and replaced altogether. And as we discussed in our previous episode, we’re going to have to do that at the same time that we increase our overall generation capacity to meet rising power demands.

To dive a bit deeper into that conundrum, we have Joshua Rhodes. Joshua, do you want to introduce yourself?

JOSHUA RHODES

Sure. My name is Joshua Rhodes. I'm a research associate at the Webber Energy Group, the University of Texas at Austin.⁹

REBECCA

So, we’ve already talked about optimizing existing power infrastructure and preventing disastrous failures of aging assets. I want to ask you about how—if we’re going to decarbonize by 2035—how will we need to expand infrastructure around the world?

JOSHUA

I mean, you can look at it to parallels with other infrastructure, like the interstate highway system, or the, you know, the railroad system. The story is that Dwight D. Eisenhower, as, you know, as a young man in the army, it took him two weeks to get from the east coast to the west coast. And then when became President decided to build, you know, an interstate highway system so that you could do it in just a couple days. And that really opened up the ability of us to move goods and people and services, you know, across the vast country. But we're still kind of locked in terms of transmission. Our electric transmission doesn’t go that far. And so, if we can unlock those areas that have those good resources—particularly, you know, the cheapest ones being solar and wind—and move that to the, to the places where you know, people want to consume it, I think that'll be the cheapest way to move forward and also, you know, help us match supply and demand.

REBECCA

I know it’s tough to think about things aging. Most of the time we might rather not think about how much work and repair these critical systems need. It’s kind of like when the check engine light pops on on the dashboard, and—if you’re me—you might think: oh man, I wish I hadn’t just seen that. But here, instead of a ride to work or the grocery store, we’re talking about a system that makes every part of our daily lives possible. 

But while the prospect of expanding and modernizing an old system can be daunting, it can also be exciting to think about the changes ahead. The power grid has been around all our lives. And we’re talking about a massive change to how it’s structured. We still have a lot to do, and to figure out, and to invent, but it really feels like we’re arriving at one of humanity’s big technological milestones: a modernized, decarbonized power grid, and an energy-secure future. And that thought, at least, leaves me feeling pretty—forgive me for the terrible pun, but…energized.

OK. That’s our show for today. If you haven’t already, go check out the first episode in our series which is all about what it will take to decarbonize the grid.

You can find more information about our guests and links to follow the interesting work they’re doing on the episode page. And special thanks again to Pat Kennedy and his family. We’re incredibly grateful for all the wisdom and innovation Pat brought to the industrial sector. Our hearts go out to you.