The Failure Files: San Bruno, Part 1

Rhett Dodson On today's edition of Pipeline Things, the first of a two-part series, we cover what is arguably the most, or one of the most defining events, in pipeline integrity. A 600-foot crater, 108 affected homes, eight people are killed, ten serious injuries, 48 sustained, minor injuries. It was unsure of whether it is a plane crash or a pipeline explosion that has happened in the middle of a heavily populated suburb. You probably already know what event we're talking about. So, join us to see what the impacts are of this major event, both on the community and in the regulations that follow. Boom, all right. Welcome to today's episode of Pipeline Things. I am your host, Rhett Dotson. My co-host, Mr. Christopher De Leon. And we are excited to be with you on this journey as we go through, I'd say the origins of what you call modern regulation. I believe, as you guys know, this is … Christopher De Leon It’s like the origin story. Rhett Dodson Yes, it is a good origin story. This is going to be the third episode today. And I'll say, you know, like today's episode, I’d say there are two challenges with today's episode, Chris. One is the content itself, because it's so when you take on something as significant as San Bruno, which is our topic today, you feel the weight of it. You're like, Oh God, I can't halfway do this. Like I need to give this the attention, the justice that it deserves. The second one is it's so much material, so I kind of feel like sometimes maybe I understand why people don't read NTSB reports. Christopher De Leon I mean, it’s where you hone in, right? Where do you hone in? Rhett Dodson Yeah, so I took a 137 pages of NTSB material and in my first stab I boiled it down to 16 pages and in my second stab I got it down to ten. So, audience, you're with us for ten pages of Rhett's condensed notes of the NTSB failure on San Bruno. Christopher De Leon Yeah. Rhett Dodson You don’t seem excited about that. Christopher De Leon It's because I've seen you present, if any of you have ever been in a presentation with Rhett, he has a style, right? It's like a ton of content. And he goes, really, really, really, really fast. And it's like, that was fantastic. I heard and learned a lot, but I'm not sure if I can apply. Rhett Dodson So, I'm not going to listen to this on 1.1, I’m gonna listen to this on 0.9 just to slow Rhett down. Christopher De Leon Yeah, so I'll try to pull out the stuff that maybe we should we should talk about a little bit more, but go ahead. Rhett Dodson Yes. This event, San Bruno, I'll say, you know, what's really interesting about it, Chris, for me is I've had I'm going to say maybe, I don't wanna call it fortune because I don't think it's fortune, but I've had the unique position of being a part of two industries and seeing events that define those two industries. So for me, San Bruno is to the pipeline industry what the Deepwater Horizon was to the offshore. I think the difference between, because I worked at both. So I came from, I was actually in the process of transition. Christopher De Leon It feels like the modern definition and maybe I do want to qualify that because I mean, the Olympic failure also had a pretty significant impact. 00;03;39;05 - 00;03;43;08 Rhett Dodson But I wasn't. I mean, look, what was I, I mean I was in school when that was going on. 00;03;43;08 - 00;03;53;12 Christopher De Leon Yeah, yeah, my point is still it's like unfortunately, we've had these milestones, these milestone events that have also kind of refined or further shaped us if that makes sense. Right. Rhett Dodson But this is different. The difference is both you and I were working, well I was, we were both working professionals at this time. Yeah, I had actually been working already for around six years. Christopher De Leon Yeah. Rhett Dodson And I do remember when this happened, but I wasn't as invested in the natural gas industry at the time. Christopher De Leon I was a natural gas operator learning a whole lot about line inspection and pipeline integrity. Rhett Dodson And but again, you know, nobody working in the offshore industry is unfamiliar with what happened in Deepwater Horizon. I think it's safe to say that we want to make sure all of our listeners, certainly any engineer who comes in to gas pipeline integrity, should know about San Bruno. Christopher De Leon I'll go another way. I'll actually challenge to say anybody who is in pipelines should know about Marshall, Michigan and San Bruno as a minimum. Rhett Dodson Yes. Christopher De Leon Because of the impact it had on modern regulations. Right. I mean, we see this time and time again, we're working with a liquid pipeline and we'll see gas regulations as a reference. I mean, they're really cross talking a lot now, right? That that boundary of well this is a liquid rig versus this is a gas rig. We're kind of seeing how some of it maybe the advancements of control or more strict regulation has happened first on the gas side. Obviously, we're going to talk about how San Bruno has influenced that. But, you know, we're seeing a lot of that that cross talk. So, if you're in pipeline, specifically in pipeline integrity or pipeline integrity is somehow influencing your day to day. Yeah. These Marshall, Michigan and San Bruno are absolutely impacting how you operate one way or another. Rhett Dodson And hopefully, you know, again, one of the things we’ve already said is that the only thing we learn from our failures is that we don't learn from our failures. Hopefully, any of our listeners, you know, we know a lot of them out there are involved in pipeline integrity. You hear this and it's just a good chance to be reminded of like, Oh, wow. It's kind of like when you're driving on the interstate and you pass that that speed sign, you're like, Oh, yeah, that's the speed limit here. I need to be reminded. I need to slow down. That’s for you. Christopher De Leon No, I have a better one when I'm driving. Mine is like when I get the low fuel signs that like my truck tells me, oh, 60 miles to E, 59 miles to E, and then it says low fuel, right. Some people would just go get fuel. Right. I'm like, no, like I've got at least 32 miles left at my average driving velocity. I'm like, this is okay. I don't need to I don't need to feel pressure to go get gas all of a sudden. Rhett Dodson That worked well for you until you drove a different truck, right? Christopher De Leon That is 100% true. And we're going to see some of that in, I think that's a great segue actually into this, right. Is it's whether it's recency bias or we've always done it that way. When you try to apply that a different way, it might not work out the way you want. Rhett Dodson So without further ado, we're going to jump in. So I want to take the audience back. September 9th, 2010, wherever you were at that point in time, I want you to remember. It's 6:11 p.m. That's when this the story, the incident occurs. The, we're actually going to start a little bit before that. And we're going to talk a little bit about where the pipeline in question comes from and what it is. So we're talking about what's known as line 132. Line 132 as part of a bigger system and we're going to be talking about segment 180 within line 132. Okay? So this is a 30-inch diameter segment operated by PG&E that runs in San Bruno. So what I want you to know is, again, the segment we're talking about is 180, but the report refers almost exclusively to line 132, and the segment is within line 132. Line 132 originates in the Milpitas terminal and flows to the Martin Station. So for our audience out there, that's going from south to north in a peninsula in California, right? So gas is flowing from south to north. The line 132 is actually part of a three-line looped system. So there are actually three tangential lines that run next to each other, I don’t know if tangential was actually the right word there. Maybe side by side, three side by side lines. 101, 109, and 132. And they have six cross ties in between them. So these are heavily interconnected lines, which is one of the issues that comes into play here, we’ll say, that allows gas to flow in between all of these lines. But they all leave from the Milpitas terminal and flow to the Martin Station, which what's interesting about that is that line 132, again, is composed of multiple diameter segments. So it's got 24, 30, 34 and 36 inch diameter segments. So all of those segments exist within the system that is line 132. Christopher De Leon Yeah. Rhett Dodson Okay. Christopher De Leon And between Milpitas and Martin. Rhett Dodson Martin. Yes. Christopher De Leon And so let's just say if you wanted to put an ILI tool in there, you would have to try to navigate all those nominal pipe sizes. Rhett Dodson Oh yeah right. And that's obviously I mean... Christopher De Leon That's another discussion. But we're just trying to frame this for an integrity engineer, right? Is it's like are those multiple segments, is it one? And just keep that in mind, right? You've got all these diameter changes. Rhett Dodson So again, this was a, the difference between those two is a little over 40 miles. And so it's interesting that this would traditionally be a pickable segment, but this is not a pickable system. Christopher De Leon Yeah, it could be... Rhett Dodson It could be in terms of length, but it's definitely not a pickable segment. And this is the 30-inch segment that we're talking about here, but even that segment has multiple weld types, and it's really important that one section of it was relocated in 1956 to accommodate residential construction. That section that was relocated within line 132 was about 1800 feet long. Right. And PG&E actually directly performed that work. They didn't contract that out. So what PG&E believed was in the ground if you had asked them at the time was 30-inch diameter, seamless pipe material API 5LX42 nominal wall thickness 375 installed in 1956. So PG&E, according to their records, believed that it had been part of the relocation project. They did not know the manufacturer and the line operated, all this is important information, the line operated at 400 PSI, that was the MAOP. They had a self-imposed maximum operating pressure of 375 due to some other piping in the system. So just to set the stage, that means that we're operating in a stress of about 16,000 PSI, which is about 38% of the listed spice. For most of us that are out there, this is not what we would consider to be... When we think about transmission pipelines, always thinking about things that are operating north of 50% SMIS, maybe as high as 72%. Right. But in a very heavily populated area. All right. Any questions about the system from you, Chris? Christopher De Leon So far, so good. I think we understand what's going on. You’ve done a great job. So let's go. Actually the only thing I would add is it's if you've ever flown into San Francisco, you’ve basically seen San Bruno because it is right adjacent to the SFO airport. I mean, so it's densely populated. San Bruno is not rural by, it's not rural by any means. It is very urban, densely congested, lots of family neighborhoods, all that fun stuff. So as part of this, if you're listening, if you’re at your computer, I would say, you know, Google and take a look at the incident. I mean a picture, it really sets the stage for what we're going to talk about today. Rhett Dodson Yep. And so, again, as I mentioned, now, I want to work backwards from the failure. So the failure itself occurred at 6:11, but we're not there yet. The story begins closer to around 4:30 that afternoon. And like all the failures we've talked about, the control center and the SCATA systems come into play, maybe not as much as some of the other ones, but we're going to see a lot of things occurring around that. So around 4:30, they have some electrical work that's taking place on the uninterruptible power supply. Right. So they are actually transferring loads to incorporate a new UPS system. And during that time, the report goes into detail, they're having to change valves over from automatic to manual. None of that initial work is actually pertinent to the failure, other than that it wasn't documented properly and it does again create this dialog that's happening between the people that are doing the work and the SCADA Center, but they do know that prior to the failure they had put the valves back into the automatic position. The next stage of work. What happens is that the workers removed power from an unidentified breaker, so it was an unidentified breaker. They didn't know what it controlled. They shut off the loads from the breaker and then they immediately noticed that some of the displays in this local panel go blank and they start getting erroneous low-pressure signals to the valve controllers. So they're getting erroneous signals to the valves in order to prevent anything from going crazy. The valves actually will lock in a fully open position, you know, is what happens. So the valves, because they shut off this breaker, go to a fully open position. And that means that the only means of accidental overpressure becomes monitor valves that are further down the line. Again, the report goes in a lot of detail, and it's not going to be too critical to our discussion, but it is important because when they shut off that breaker, what happens is at around 5:22 p.m., so we're still a little less than an hour before the failure, I'd say that the SCADA system alarm panel goes nuts. Within a few seconds we see 60 alarms happening. There are controller error alarms, high differential pressure and high high pressure alarms on several of the lines coming out of the Milpitas terminal, including line 132. At 5:25, the skaSCADA ter operator actually calls the Milpitas technician and says, “Hey, these signals look real. What's going on?” Right. And but they realized that that this time that that whole panel had gone black. And so they realize, hey, it's something related to what we've done. And they're also noticing that the SCADA data is inconsistent. So they're getting all these high pressure and high pressure alarms, but they're seeing constant pressures on the downstream lines. So they're not sure what's going on. What's interesting, again, is we see confusion in the SCADA center just prior to an incident right. All of it related to some electrical power work. So this begins a conversation between SCADA and the Milpitas technicians that are performing the work. They begin to realize that the SCADA data, the SCADA center, is not receiving valid information from the Milpitas terminal, which is why all the alarms are going off. The monitor valves, which remember I told you were that last line of defense, were set at 386 PSI. But there's a delay between when the monitor valves record time and when it gets sent to the SCADA system that creates more confusion. And so basically they get concerned that they're over pressuring the system at around 5:30. And the Milpitas technician says, hey, can I reduce the local set point on the monitor valves from 386 PSI to 370 because I want to bring the line pressures down. Christopher De Leon Yeah. Rhett Dodson And again, they're still trying to figure out what's going on. So between 5:28 and 5:52, the conversation continues between the technician and the SCADA data operator. They, he basically asked the Milpitas technician to place a physical gauge on line 132 so that he can sort out what's going on. And he gets a reading of around 396 PSI, which again is higher than they had been operating. It actually leads the operator at the SCADA center, to make the statement, “We have a major problem at Milpitas and we've over pressured the whole peninsula.” So at this point, they're honestly really not sure what's going on, but they get a few more readings on line 132 and they see that it brings it down to around 386. So at about 6:00, right around the time the failure happens is when SCADA is coming off of their high craziness thinking that, okay, all of this work that was taking place has upset the SCADA system, but it looks like things are normalizing. Problems under control. We're good. Unfortunately, no, that's not the case. Shortly after the 6:11 line 132 ruptures, this is only about 10 minutes after those other events that occurred to you. So, again, keep in mind in the last 30 minutes is when all of these conversations have been going back and forth, they're trying to figure out what's going on. And then that's when the very first low-pressure alarm for line 132 goes off, at 6:15, which is after the failure. And it happened at the Martin station, which is downstream. 20 seconds later, they get another low-pressure alarm and the system has come down to 150 PSI. By 6:36, only 20 minutes later, the pressure at the Martin station was 50 PSI. So things have changed quickly. But between that time of 6:15 and 6:30, this data says SCADA operators are still unaware that they've got a failure rate. So let's take boots on the ground. What's actually happened? We know that at 6:11, a failure happened. On the ground, what takes place is the fire fighters see the explosion happen from like almost across the street. They end up immediately on scene and taking care of things at 6:13 p.m., right. Within 2 minutes, you have firefighters on scene. And I'll say, the NTSB report says that the emergency response is on point throughout all of this. So kudos to emergency response. Christopher De Leon And just to give you perspective, when you say the emergency response saw this almost immediately. Rhett Dodson Local fire. Christopher De Leon Reports are that the flame reached approximately 300 feet in the air. I mean, so this is significant. And again, you are in a very rural, urban populated location. So you've got the attention of people now. So what, so what do you do? Rhett Dodson So this is when I think a lot of, or remember, the failure happened at 6:11. I'm going to take you to around 6:27 where 16 minutes in or so, SCADA and operations are still not aware of what's going on. I think if you took all the incidents related to the electrical stuff out, they probably would have recognized they had a failure incident immediately. But they're confused because of all the other stuff that was happening. Around this time, somehow because emergency response was on scene and they're not sure what happened. They actually come to the conclusion that a jet has fallen or has crashed from the local San Francisco airport and it begins to circulate that the fire is actually due to a jet plane that has crashed in the area rather than a lot. And that rumor actually makes its way into the SCADA system or the SCADA control center. Eventually, at 6:27, a dispatcher calls SCADA to see if they’ve seen a pressure drop. And the operator recall says, hey, we haven't seen any calls about this incident. But again, keep in mind, you know, emergency response is already on scene right now. There were reports of a plane crash or a gas station explosion or some combination of the two. That's again, I've already told you, going through the terminal right now. But three minute, within 3 minutes, by 6:31, within 4 minutes of receiving that call, SCADA recognizes that they've had a failure. Right. And in fact, the operator reports, the SCADA operator, reports there's been a major pressure drop at a station near San Bruno, of course, referring to the Martin Station. So they're putting the pieces together by 6:30 that they have a failure. This is where, unfortunately, between 6:30, it takes another hour before the gas is actually shut off. Right. And that gas shut off is kind of a cool side story. An amazing side story, I'd say in here. The way it happens is again, 6:30 p.m., one of the local PG&E measurement and control mechanics. He's actually off of work, but was qualified to operate the main line valves. He sees the media reports and suspects a transmission break. So he notifies the dispatch center and then immediately goes to what's known as the Coleman Yard. This mechanic arrives there at around 6:50 and he waits for the second mechanic to arrive. These two guys get together with a map of the system. They're looking at where the incident has occurred. They're looking at the system. They come to the conclusion before anyone else that line 132 has failed. Before operations confirms it to them, before SCADA confirms it to them, they are convinced that it's line 132. They call their supervisor and say, “Hey, we're going oscillate the valve. Will you give us permission?” The supervisor permits them to do it. They go out there and then manually shut the upstream valve, meet the supervisor, go to the downstream valve and shut off that valve, effectively isolating the gas flow. And then they actually have to shut off a few, they have to pinch off a few distribution mains to fully contain the flow. But that takes them a while because they have to leave from 6:50, go to the first station, shut off the valve, drive at the second station, shut off the valve. What's amazing is that the NTSB report says that mechanics one and two were unnecessarily held at the yard and the response could have been delayed even longer if the two mechanics had awaited from official orders from PG&E. Further, the SCADA center staff could have reduced the flow sooner by shutting the remote valves at the Martin Station, but they did not. These delays needlessly prolonged the release of gas and prevented emergency response from requesting or from accessing the area. Say it's cool simply because you have these two guys that kind of took matters somewhat into their own hands. Christopher De Leon And that's always a complicated situation. Right. How many, even in some very recent catastrophic events, right, that you and I have talked about at length in our personal time. Right. Is it’s where if you are competent, and there is an opportunity, at what point do you wait for instruction and follow a process, right? So, but we'll leave that maybe for a different podcast. Rhett Dodson Yeah, it's a whole other, again, a lot of the NTSB report, if you read it, goes into detail of the lack of unified command at SCADA. We don't want to get into that. I mean, let’s just say it's outside the scope of this podcast. Christopher De Leon Part of our goal here is for you guys to understand what happened and then part, you know, the second phase of this, obviously, is to talk about now, how does this impact us and what we do? Rhett Dodson But suffice it to say, all the way up until 7:00, even after the mechanics have left to go close the valve, the operators within the control center are still under responding to the rumor that it's a plane crash. And one of them says, “It's easy to believe it's a plane crash. We still have an indication that it is a gas line break and we're going to stay with that. If you talk to the fire department, I would inform them of that.” So there were still people trying to say even after SCADA had the data, they believe that it was a break. They were still trying to say, oh, no, this is potentially a plane crash. Right. And again, you see that that it unfortunately delayed some of the response. Right. Ultimately, the gas isn't fully stopped until close to 7:57 p.m. A whole 90 minutes and the significance of that, again, remember that I told you is by 7:57 p.m., they stopped gas flow well after 90 minutes, but by 6:30 they had indications that it was with a line pressure going down, that it was that a line break had happened. But unfortunately there were no automatic shut off valves. There were no remote shut off valves. The only way to shut this off was due to the actions of the two mechanics that were going around doing that. Christopher De Leon The qualified personnel on site. Rhett Dodson Yeah, and this is one of the major findings. Again, we're not going to talk about it at length, Chris, but the NTSB says that the use of either automatic shut off valves or remote control valves would have reduced the amount of time taken to stop the flow of gas by at least one hour. The outcome is the NTSB recommends to amend Title 49 CFR 192 to direct to directly require that automatic shut off valves or remote control valves in high consequence areas be installed and spaced at intervals that consider the factors listed in the regulation. We know that conversation has been ongoing even until today. That conversation continues, but it was a direct outcome of its I'd say it's the first of our direct outcomes, but again, not one that we're going to spend a lot of time talking about. So what happened? What were the outcome of this? Again, you mentioned looking at the pictures, I'll give you a visual picture. Most of us live in subdivisions of closely spaced homes. This is no different, right? So maybe if you live in a standard master planned community, walk out in your backyard and I want you to imagine a crater 72 feet wide, 72 feet long and 26 feet wide. That's the crater that blew out of the ground. Imagine a section of pipe 28 feet long and weighs 3000 feet blown 100 feet into your neighbor's yard. Christopher De Leon Weighs 3,000 pounds, continue. Rhett Dodson Weighs 3,000 pounds. Look around you and find the nearest 38 homes and level all of them. Because this fire destroyed 38 homes and damaged 70 and ended up killing eight people. They had ten serious injuries and 48 minor injuries. They ended up excavating 300 homes, evacuating sorry, evacuating 300 homes. And the fire damage extended for a 600-foot radius affecting 108 homes. Christopher De Leon Yeah. Rhett Dodson That's the extent of the damage, right. It’s catastrophic and occurring within a heavily populated area. The repair cost at the time, and we know it went way beyond this, the line at the time actually wasn't going to be repaired, it would have been $13,500,000. We know that the actual cost far exceeded that. So again, I think tragedy like I'd say this, it's astonishing, right, when you see that that 700-foot crater and what happened there. And it's a memory of what we don't ever want to happen again. Christopher De Leon Yeah. So it's a bit of a social license, right? Because you could argue out of all those homes, how many of those people were aware of the risks? And I don't even mean the risk of what the NTSB uncovers, just the fact that there is a high-pressure pipeline, even if it's only even if it's operating under 50% SMIS. The fact that it's there, there's there is a duty of care, right? There is a social license that we all take in being in this industry. Right. And I know, you know, there's we do a lot better with public awareness and call before you dig. But, you know, when you have over 300,000 miles of transmission pipe, I mean, it's just there's a lot of responsibility that we take. Rhett Dodson I think this particular incident, Chris, definitely highlights that when one of us fails, all of us fails. I don't think the people there cared who the operator was. You see the public noticeably upset because and I think it's because it's so relatable. The reason I tell everybody to walk outside and imagine a 600-foot crater is many of you may not realize where these pipelines are. They did fine, again, the report goes into great detail about public awareness. It's a great read. But the majority of the public remains largely unaware. Fortunately, I will say largely unaware of the gas assets that exist under their homes. And that's how we would like them to be. We don't want them to certainly be aware of this. So when we come back, because we’re going to take a break, but when we come back, we're going to pick up with the cause and the fallout. And again, I think similar to our last episode, this is probably going to be a long one, but it's going to be a good one. I think you're going to now begin to get a full understanding of maybe why some of these contentious conversations happened so hang with us and we'll be right back. Rhett Dodson All right. Coming back from the break, you probably thought you were going to get the conclusion of the San Bruno episode. Christopher De Leon Nope. Rhett Dodson I came, we came, to the realization together that, honestly, this is too much to handle in one episode. So you're going to have to come back next week or in two weeks to hear us really close out the full impacts. So, for instance, we haven't even talked about the NTSB’s opinion on stable manufacturing defects. I was significantly surprised at how much content there is around in-line inspection in this document, even though line 132 wasn't inspectable. So come back in two weeks because we're going to deep dive into that and how that is reflected in current regulation. We hope you enjoyed this episode and we look forward to seeing you again in two weeks.