The Failure Files: Chaos on the Kalamazoo

00;00;00;00 - 00;00;29;16 Rhett: 192 712. Purely a result of San Bruno or not? Most of you would say yes. If I told you FEMSA expects all cracks to be excavated and that a failure that resulted from insufficient safety margins, inappropriate wall thicknesses, lack of tool tolerances and ineffective crack sizing contributed perhaps more to current regulation than San Bruno, would you agree or disagree? Today we dive into, maybe if you're on the gas side, a not as well-known failure in Marshall, Michigan, that's going to help you understand how and why we got some of the regulation in 192 that we see today. Join us on today's episode of Pipeline Things. 00;01;01;11 - 00;01;10;05 Rhett: All right. Welcome to today's edition of Pipeline Things. I am your host, Rhett Dotson, my co-host, Christopher de Leon, and we are sitting in the big comfy chairs here recording a couple of changes today. Miss Producer, God bless her soul. Chris: No, no, we wish you the best. Rhett: We wish her the best. She's moved on to bigger and better things. We will miss her greatly. But true underdog story comes through. Second-String-Sarah is now First-String-Sarah and has taken on the role of production. So. Chris: What was my first response to that? Rhett: What was your first response? Chris: Yeah, I felt like it was kind of like the Green Bay Packers. Rhett: Oh, Aaron Rodgers Chris: Yeah, when it went from like Brett Favre and everyone was like, “Yes, the Green Bay Packers are gonna suck!” and then here comes Aaron Rodgers and everyone was like “No, another decade of dominance.” Rhett: Yeah, and Sarah had no idea. But yes, she's going to do great for us. We were excited. Chris: She is doing great for us. Rhett: She is doing great for us, thank you so much. So, Chris, today's episode, we're continuing in our examination of notable pipeline failures that led to significant changes in regulation, right? Chris: Yes because that's what NTSB reports do, right? Rhett: It is right. So last time we set the stage with really Bellingham, Washington, and that one had a lot to do with how the original part one regs were defined. Let's say what you call which pipeline integrity 1.0. Chris: Yeah, and not part one of regs per se related to the mega rule, but rather the inception of pipeline integrity as a whole. Rhett: I think 192 and 195 as most people the last ten years have known it and lived it. Chris: Yeah, and that would be, well, let's call that integrity 1.0. Rhett: Yep. So today we are going to flash forward and we're going to take one another significant one. But I think the interesting thing about this one is as somebody who's worked mostly in the gas industry in my life, I don't feel like I had a really good handle on it. Reading this report, and I want to I want to set the hook for our audience out there. If you're one of our gas operators, stay on through the whole episode because we're going to be talking about another liquid rupture today. But I was really astounded by how significant the findings from this NTSB report were with regards to recent gas pipeline regulations that I'm talking about. Obviously, the updated gas rule. Chris: Yeah and just a little bit of grounding. Right. Our intent with this with this arc is understanding the role of these NTSB reports in our current regulations. And sometimes what you'll find is that we can sometimes operate in silos, so like it's like “Oh I work on the gas assets” and, you know, “So I read 192” or “That incident was on a liquid line, so maybe we're not as familiar with it.” So what we're trying to do here is it's just to shine some light on some of these failures, even if they happened a long time ago, ten plus years ago, and the impact that we still have on them. And I think we're going to find a little bit kind of if you know the end, you'll maybe stick along with us is it's oh, obviously we recently have the the new gas rule that was in three parts. And surprisingly, this NTSB report that's older than ten years is cited right in the preamble of Ren 2 etc., on how gas pipelines are going to be regulated. So that's why we think it's super relevant. Rhett: Absolutely. And so without further ado, what we're talking about today, Chris: Is it story time, Rhett? Rhett: It is story time. Chris: Is that why you got all dressed up? I thought it was casual bro. Rhett: No, I'm sorry. Chris: We're usually here. Hey, are we losing it? Are we losing the edge? Rhett: No, we're not. I have another engagement and I have to shoot. I'm going to be talking all day. Chris: And you dressed up? Rhett: I'm shooting for the Clarion. Chris: And you dressed up? Rhett: Of course, I dressed up for those guys, not for you. That's what you should take away that buddy. Okay, so thanks for, like, the free plug for the Clarion course on geohazards. Chris: Of course, amen. Rhett: So we are talking about Marshall, Michigan today. And I do like the set the stage with the story. But this story, again, is a little different from Bellingham. So in Bellingham, the events that surround it and unfortunately the fatalities became a central component. The real story here, Chris: So consequence, right? The consequence there was a loss of lives. Rhett: Yes, lots of loss of life and honestly it was shocking in terms of the number of missed opportunities to engage it. Yeah, this one is very similar in terms of missed opportunities. The consequences far greater on the environmental side. I mean, the environmental destruction wreaked from this is, is is significant, but the story happens in the control room this time. And so I want to take you there, Chris, and as I take you there, most of us and most of our audience doesn't probably exist in the control room. Chris: And true to our brand, there is not an ILI component to this, right? Rhett: Oh, there is. Chris: We have to stay on brand. Rhett: But I am going to brace the audience. The ILI component is going to come in later. But, I want to set the stage for the story in the control room. Chris: All right. Rhett: But since we don't live our lives, you and I, in the control room, I've been in a few of them. They are impressive. Really cool. Yeah, but since we don't live in them, I think we face very similar decisions to these guys in the control room, day to day, in our integrity decisions. Chris: I think we do this all the time. Rhett: We do. Chris: Do you not make decisions all the time that are outside of our profession? I mean, I just invited you to my house yesterday. What did we do? Rhett: Oh my gosh. Chris: What did we do? Rhett: You take me off point. Chris: What were we doing: Rhett: I'm not falling for it. 00;06;26;01 - 00;06;42;15 Chris: What did we do? Rhett: We played poker. Chris: We played poker, right. And what do you do in poker sometimes? Rhett: In poker you make decisions with limited information. I only play in hands to win. I'll be very clear. I'm not a good bluffer. So when I am playing in a hand, I am playing to win. Yeah, every moment I am assessing. 00;06;42;15 - 00;06;56;26 Chris: Okay, so let me ask you this, if you have pocket pair, but it's not aces, are you bluffing if you raise? Rhett: No, I want to chase you out of the pot. I'm trying to chase you out. Chris: So is that bluffing? Look, if you don't know, what everyone else has is that bluffing? Rhett: No. But you get more information. And if the flop turns out, if you're not familiar with Texas Hold'em, there's a few rounds. Everybody gets two cards. You start with. You don't, you only know your cards. Then you get a round of cards called the flop, which is three additional cards and which gives you more information, which you can bet on. Then you get another additional card called the turn. And then you get another additional card called the River. At any point in time, your hand can go from strongest to weakest. I stay in as long as I think I have the strongest hand. So in your scenario, if I have pocket pair, Chris: So let me get this straight. So you play to win, I play to get people out of the game. Rhett: That didn't work well for you because I took you out in the first hand. If you recall how that poker tournament, Chris: I had a pocket pair. What am I supposed to do? Rhett: And I had two pair on the flop and you lost. So, Chris: Should’ve raised more on the flop. Rhett: Maybe so. Chris: But is that bluffing? I don't. I don't know. I think the definition of bluffing, I mean, and we're going to find that here, right? Is how do you define certain things, right? So at what point do you define bluff? Rhett: I think integrity in making a concept of bluffing doesn't work with integrity. And that's, we make, we make decisions and we expect to win. Chris: Based on assessments. So we are not correlating poker to management. Rhett: No, let's jump in. So the story begins on July 25th, 2010. Right. Enbridge is doing a planned shutdown of line six B, right. To give you some idea of line six B 30-inch system. So this is a big oil system, right, that actually originates in Edmonton, Alberta, Canada, and actually terminates in Ontario, Canada. But a portion of it comes down through the United States, and this portion runs through several pump stations. Marshall, Michigan is the most notable one. So this line six B system exists in the United States, although it's a pipeline that starts and terminates in Canada. So they're shutting down the pipeline, forced them to change batch operations right, as they begin shutting down the pumps. And again, July 25th, 5:55 p.m., that's when this starts. They start shutting down the pumps. They immediately get multiple alarms that happen right here, at least three, including one, which is a material balance system, MBS. And that MBS alarm indicates that the amount of oil that was being sent into the pipeline isn't the same as what's being received. But since they're in the middle of shutdown operations, they all attribute these alarms as just due to shutdown, right? So nobody suspects anything on Sunday, July 25th has actually occurred. They're thinking everything's fine. And so the pipeline shuts down. There are some calls that happen, interestingly enough, to 911, of reports of an odor. The fire department goes out, they actually get numerous calls. They investigate, but they can't find the source. So they actually go out for an hour and a half. They're poking around. They can't find the source of the odor. They return back to the fire station. They get a second call later that night at around 11:33. But the 911 operator dismisses it and says, like, we've already tried to find it. There's nothing out there right now. None of this information makes its way back to Enbridge or the control room. So they're unaware that any 911 calls have happened. Right. So the planned shutdown was for 10 hours. Right. At this point, I want to give you, you as an engineer, all you know is you tried to shut down the pipeline and you got a bunch of alarms. 10 hours later, there's a shift change. They get new people in the control room. The information on the alarms isn't conveyed. Yeah, the new control group comes in and they start trying to start up the pipeline at 4:00 am in the morning. Okay. 10 hours later, they immediately start getting alarms. They're going off. They attribute it to what's known as column separation. Right? Not a term you and I are familiar with but column separation. Basically, if you drain the oil out of the pipeline, you get a whole bunch of air. The air compressors. Yep. Right. So they basically have a reason why they're not getting as much flow out of the pipeline as they're putting in yet because they're going to fill it up basically, right Yeah, but they have the procedure says if you try to start the pipeline for more than 10 minutes and it doesn't start, you need to stop. Something is wrong. 10 minutes in the person who started this pipeline says, “Hey guys, we have a problem.” The NBS engineer comes and says, “No, no, no, this is just because of column separation.” And he runs some quick calculations and says “You need to run for at least 20 minutes in order to fill up the pipeline.” So unbeknownst to them, they are pumping oil through a rupture that has happened. Chris: This sounds a whole lot like our first hand in poker dude. I had a high pocket pair. The flop happens. I still have the highest pair, but at that point you had two pairs. So what do I do? I double down. Rhett: You double down. Chris: I put more in to it. Rhett: But you're not considering why I am playing the way that I am. Chris: Why? Why are you still in the game? I don't understand why this guy's still in the game. I have the highest pair, so I put more into it. I double down. Rhett: You are running on limited information. Chris: Yeah I doubled down. I should have asked you what you had. Rhett: You should have stopped. Chris: I should’ve stopped and asked you and said, “What do you have?” and seen your emotional reaction. Rhett: You should have considered the strength of your pair relative to the cards that were on the table. Chris: What was the highest pair on the table? I didn't think you had two pairs. Rhett: So yeah, I did. So they obviously they add the basis of the NBS engineer comes back and says, “no, you should run at least 20 minutes.” So they continue running for 30 minutes. And after 30 minutes multiple NBS alarms continue. They have not increased the pressure downstream of the pump station. So downstream of the pump station, they can't increase the pressure. The NBS alarms are going off and they can't figure out what's going on. So this is going on for 30 minutes. Chris: Again, the analogy works, right? I raise increasing the pressure on you, but I didn’t see it. I didn't see it at all. So now I'm a little bit confused so I know what my next step is. I know what my next step is because I’m gonna win this. Rhett: But let me ask you this. At this point, this is the first point where they stop to think and listen. He says they either really, this is the quote from the operations. “They either really drain the line,” which I think they did, “or else,” and then there's three more dots, “else I'm leaking.” So at this point is the first time worry enters into his mind, something doesn't, something doesn't feel right. Chris: Which is how I felt because I raised and you stayed in. Rhett: So you escalated. They also escalated. Chris: Yeah. Rhett: They called a meeting with the shift supervisor and the control center supervisor. They get everybody on board and they say, “This is what's going on.” The NBS guy comes in and says, “Column separation.” Chris: I did the same thing. I looked at the rest of the people at the table. They all folded except you. Rhett: And so what happens is the control center supervisor says, “Sounds like you need to try again.” The NBS guy says “You need to try and restart.” Chris: And that's exactly what I did. So what did I do? I pushed all my chips in and said “All in.” Rhett: And you hadn't even got the turn. Now we're getting to the turn. The fourth card drops, so it’s 7:09, 3 hours later they try and restart again and they're actually having trouble restarting the pump. The pumps won't kick on because of the NBS alarms and because they can't get pressure in the line and they keep trying. At this point, the control center person calls one of the downstream pump stations and the guy says, one of the operators at the downstream pump station, he says, it's a Sarnia terminal operator, “I have never seen this. And it looks to me like a leak. I have never heard of where you can't get through. I can pump as hard as I want and I'd never overpressure the loan. I don't know, something about this feels wrong.” So at this point with this information, other people are starting to get concerned, right? Chris: Yeah. Rhett: Guess what happens? They change shifts again. A new shift comes in. The new shift operator is far more concerned. He has more experience and he says look I, I am started up the system with less pumps than us, Right? Something doesn't feel right. And he basically, the shift manager at this point tries to engage the regional operations manager in the Chicago office and wants to walk the line. The Chicago manager shoots him down and says no, because we haven't gotten any 911 calls. Yeah, they obviously had gotten 911 calls. Right. The they try to restart again. The Sarnia operator is completely incredulous that they're restarting, but they restart again. Needless to say, at 11:17 they are finally notified when a Consumers Energy employee, a gas person who had gotten numerous odor calls, is on the line, sees the oil leaking into the river, calls the control center and says, “Hey, I'm pretty sure this is your line, pretty sure that you're leaking oil.” Obviously, at this point, the situation begins to dawn. But the struggle here, Chris, is that this is after an hour and a half of continuous operation. So I expand, I gave you a timeline between 4 a.m. and 11 a.m. In that time, they spent an hour and a half operating the pumps, pushing oil into the line. And it's estimated that they pushed 81%. It ended up with 843,000 gallons of oil on the ground. 81% of that happened after the initial shutdown during this time where they were trying to figure out what was going on. Chris: Just like my hand. Right, 81% of my pot went in after pressure wasn't doing what it was supposed to do. Rhett: Yeah and so obviously I'll focus a little bit because I think it's important. The cleanup costs in this were astronomical. $767 million. They fortunately, unlike Bellingham, there was no loss of life. Chris: But that's not why we do it right. I mean we do it, I mean the highlight here is not that they hadn't had, you know, greater than $500 million or $700 million in cleanup. Right. That's not the cost that we care about. 00;17;03;16 - 00;17;21;04 Rhett: Yeah, right. Chris: What we care about is no loss of life. No damage to the environment. Rhett: Well, the environmental damage and the public outcry were massive, right? This poured into a river. The cleanup was a bit of a debacle. I'm not actually going to get into it. It's outside of the scope of this. Chris: But it's a big takeaway, right? Because, I mean, our audience here are generally people who are plugged into the pipeline integrity space. And that's what we do. And our objective is to protect the environment, protect the public and have sound operations. Rhett: So the gas folks are probably wondering, why did you tell me to stay online, we’ve been talking about liquids up until this point. Chris: It wasn't a poker lesson. 00;17;41;19 - 00;18;01;27 Rhett: It wasn't a poker lesson. But if you do play poker with Christopher, I want you to know if you challenge him up front, he's generally going to push all of his hand down. So if you got him beat, go all out. No, but really, the significance here is I'll get into the cause. So this, I told you, was a 30-inch line diesel pipe installed in 1970, and it was tape wrapped. And we know tape wrap gives you a high susceptibility to SCC. Chris: Yeah. Rhett: And that was not necessarily what was attributed here, but what they ended up finding was they had a rupture. The rupture occurred at the base due to a crack which interacted with shallow corrosion adjacent to the diesel long seam. And the report, it's kind of the NTSB report is almost, I would say, inconclusive as to whether or not the cause was corrosion fatigue or near neutral pH FCC. And they actually spent time trying to divulge both of them. Chris: Which is not our scope. Rhett: Definitely not our scope. Chris: Yeah, we're going to talk about how this incident has impacted regulation that we see today. Rhett: And I want to set the stage for the ILI, folks, because, you know, Enbridge, similar to Bellingham, wasn't unaware of this. There was inline inspection data that had identified the feature. It was known to be interacting with shallow corrosion. There were assessments that had been performed stating that the reassessed interval was greater than five years. They had planned future inspections coming up in 2010 using crack technology. So you have to ask yourself, Chris, what went wrong here? Why did we even get into the story in the control room? How and what failures happened that led to this crack being missed? And what I want to talk to you about when we come back is how did that influence regulation? And I really would encourage you hang in there because our gas operators and all of our engineers, you're going to be pretty surprised at how much of this what happens here today turns up or what happened in 2010 turns up in regulation that we see today. Coming right back on pipeline things. 00;20;03;09 - 00;20;21;00 Unknown All right. Welcome back to this episode of Pipeline Things where we are diving into the NTSB report for Marshall, Michigan. And our first part of the segment laid out everything that happened, the failure, the release. So now, Chris, what I want to do is I'm going to take you through the four causal factors that the NTSB did. And as I do, I'm going to reveal why they identified that as a causal factor. I think what I want to encourage you and the audience to do is think about us today, because I am being honest with you, a lot of this still feels very relevant today. Some of it in terms of the prescriptive regulation we got, some of it just in terms of the challenges that we face. 00;20;41;16 - 00;21;03;12 Chris: Right. Rhett: So the NTSB report also references, it's worthwhile for me to reference that the NTSB report identifies two other Enbridge releases that tie into this, both of them related to cracks. One of them was actually a Transportation Safety Board investigation of line three in Saskatchewan in 2007. And another one was Cohasset, Minnesota, on line four in 2004, both of them related to cracking. And so I think it's a little bit of why the NTSB maybe hammers some of what they do in here is they don't feel like this was the first incident. They feel like this is, I almost want to use the words repetitive or systematic at this point. So let's go with the first one, and we're only going to talk about two of these in detail. The first one I thought was interesting, is it points to the failure of Enbridge's Control Center staff to recognize the abnormal conditions. And most of us aren't control center staff. But as I mentioned to you, Chris, I feel like those guys were making decisions, which you related to poker, they're making decisions based on limited information, with the best they have, you know. But this really ties into the confirmation bias. So, listen, what they wrote was “At the time of the incident, the NBC analysts became the de facto team leader because his conclusions provided an explanation for the line six B situation that affected the team's perceptions and actions regarding the line.” And nobody questioned him. So he basically said, “Guys, this is why,” everybody accepted it, and they just ran under his premise. 00;22;09;14 - 00;22;29;23 Chris: So I already want to try to correlate this to our audience and thinking, why does this matter to me? I'm not a control room. If I was to point to a rig, we see that in 192 712, which is gas, you're allowed to do ECAs, or not an ECA per se, but I guess the way I want to say it is, you're supposed to do an analysis for cracking mechanisms. And what we see in there now, which had not been typical for regulation integrity 1.0 was and it says, “and it needs to be reviewed by an SME.” And what we also later will find, we’ll probably bring this up is it's in now in round two, again for gas guys, hey also state that they want you to start documenting your qualifications and the people and the tasks for your SMEs. And so, again, this is control room management. But I mean, there's a lot of lessons learned here. And that's what we mentioned in the last episode, right, is it's from these failures when we look back at things that happened, how are we bringing that stuff forward so that we show as an industry, we are moving towards zero failures, right? We're learning from our mistakes. It's not just statistics where we have the silver tsunami happening and all this pain and damage and lessons learned isn't carried over. But some of us that are now responsible for making these decisions, we're learning from those and I think that's a big one, right? Is it's you said de facto team lead. Why are they the team lead? If you manage a pipeline and you're in charge of making a decision, Rhett: It's your decision. Chris: You should, you should be conscious about that, about your own bias of pride, but also competence, right. In saying, “I am responsible for this. Am I the right person for this?” And if I'm not, maybe raise a hand or find an SME. 00;23;42;13 - 00;23;59;29 Rhett: And when you get that SME’s opinion, being on the side of the SME, don't be afraid to dig into your SMEs. If you don't understand what they're telling you, ask more questions and you should not be afraid to write. At the end of the day, if the explanation they're giving you doesn't make sense, you need to get enough of an explanation for it to make sense to you. 00;24;00;04 - 00;24;19;24 Chris: We actually talked about this in our week two episode, right during the World Cup, if you remember. Right. And we talked to our. Okay, hey are consultants biased? Rhett: Yes. Chris: You think so? Rhett: Yes, we are. And the problem is we have the ability to bias other people. Chris: Let me ask you another question. Do you think pipeline operators are biased? Rhett: Yeah. I mean, look, we saw it in the last one, right? Remember, the guy thought that he's like, look, I've done the right thing. Chris: Yeah, I mean, we’ve got it right here. I mean it’s control management, but it's the whole point of it's “I've done this enough” and then one person will say, “I've done this enough.” But you have someone else that says, “Well, I also have a different experience.” Rhett: Yeah, I mean, if the Sarnia operator had been in there, his opinion was totally different. He was like, “Look, guys, this is, I've never seen anything like this,” right? Chris: Yeah. Rhett: So, I don't want to dig in any more than that. But I think it's interesting how much bias plays into this and how much the roles and opinions of somebody who presents himself as an expert in this case, the NBS gets adopted by the team. Right. So, the second one, and we're not going to discuss this one, is the inadequacies of Enbridge's response plan. Again, I would really encourage all of you out there to read the report. Probably at least a third of it is dedicated to the failures in the response plan, and they're numerous, but it's really kind of outside the scope of what we do. We don't usually get into emergency response, but it was a really good read even for me, not being familiar with it. We could spend a whole separate episode on that. So, the third one is really Chris, where it starts to get in the meat of it, and I'm just going to read to you verbatim what comes out of the NTSB report. Chris: Let’s go. I’m sure they mention FIMS in there somewhere. Rhett: The inadequacies of Enbridge's Integrity Management program to accurately assess and remediate crack defects. Enbridge’s crack management program relied on a single in-line inspection technology to identify and estimate crack sizes. Enbridge used the resulting inspection reports to perform engineering assessments without accounting for uncertainties associated with the data, the tool or interactions between cracks and corrosion. So they just faulted Enbridge for a lot. You're probably like, “Rhett, you didn't give me this information.” Chris: No, that's not it. I mean, I'm good to go, do you have a question? Rhett: Let me explain to the audience where all that comes from. So, when they fought them with uncertainty associated with the data, this was a crack interacting with and, let's just take each one of them individually, because that'll make it more interesting for the audience. I don't have to talk. All right. So, they faulted them first. What I want the audience to understand is Enbridge was aware of these features, had performed critical assessments and deemed that they did not need a response. Chris: Yes. Rhett: And how they did that was based on certain information that proved to be faulty. So the first one associated with the data, what the NTSB is getting at is that this was a crack in corrosion. Enbridge’s teams were separated, corrosion handled one set, crack handled another, and they didn't talk to each other. In addition, Enbridge was unaware of the performance implications on the ILI tool for a crack in corrosion. Read it to you again. Specifically, they were unaware of the impact that corrosion would have on the crack tool’s ability to detect and size that appropriately. They ended up finding a crack that was 89% deep and about 10% corrosion crest. Chris: We could have a whole episode just on this topic. Rhett: We could. Let me read P.I.I. states. So P.I.I. stated that information regarding the impacts on corrosion on crack sizing was not mentioned in its brochures and had not explicitly been given to Enbridge. The following impacts may occur when an ILI tool is detecting a crack in shallow corrosion. POD may be impacted, POI may be impacted, and depth estimation may be impacted. Reactions? Chris: API Standard 1163, and you know that's one of my go-tos, right? So I'll make two points of reference at the time. API 1163 is not regulation, it's not incorporated by reference. Therefore, the process in which an operator performs an in-line inspection in this scenario is ungoverned by a standard through regulation. What about now? Now we have 1163 in corporate observers in both liquid code and gas code. So you'd say, “So what? If I read the title of API 1163, that doesn't tell me that this solve this.” Well, here's where it fits. In 1163 you have a section called ILI System Selection. And in that effort, you were supposed to identify what the goals and objectives of the inspection are as it relates to an integrity assessment. And effectively, you're qualifying the ELI technology. So for our audience, what I would ask is it's when you pick a tool, was it based on commercials? Was it based on a brochure? I heard you say brochure or is it based on the performance specification? And if it's based on a performance specification, if you know you have a specific type of threat, it's not just looking at something as simple as well, they're vetted. You then have to say, okay, if I think this tool is qualified to perform my integrity assessment for this threat, there's layers there. 1163 also gives you a process where it requires you to qualify that system for use. So then the next question I'd have to ask is it's, why do they think this tool was qualified for that specific mechanism of cracking? And you would say, “Chris, this was 12 years ago. I don't know if we were there.” Then I would say back to what I mentioned earlier, how are we learning from that? Rhett: I would say, how do you know? And I'm glad you bring that up. How do you know we're there now? And I'm not saying we’re not, but even today, if you get an interacting crack with corrosion as an operator, how do you ask the questions you need to make sure that that crack or that corrosion sized appropriately? Chris: And so again, trying to stay true to our our theme here is it's you know, what do we learn from these incidents and what does that mean for regulation? Right. And so that's kind of the point that we're trying to do here. And so that's the plug for if you haven't truly read cover to cover 1163, I think that's really important because it's more than just qualifying an ILI system. It really gives you a roadmap to defining terms on how what, what words do we use to define things or describe things. It talks about workflows from how do you have nomenclature around what's an anomaly, what's a feature, what's a defect? And again, what I highlighted here is this concept of you need to understand what you're looking for so that you can set it as an objective and you can engage with the ILI operator and say, “I'm looking for this. Does your tool fit the bill?” And if it does, why does it fit the bill so that you can create lower and upper bounds for what that tool capability is. Rhett: And I'd say from a threat perspective, that's good. Even if what you find in the field is not what you expected, if you find a new threat, you need to ask whether or not that ILI tool can address that threat that you found once you have that information. Chris: And can do that. And if we were to speak to this a little bit more, right. I think sometimes what you'll find is it's, and I know if I'm not mistaken, this was an ultrasonic-based ILI inspection, right? Rhett: It was it was a U2 crack tool. Yeah. Chris: And so what you find is it's these ILI systems will often have more than one technology on it, right. So in this case, we could, I'm going to use my terminology, head straight B, mute or zero degree, which means is your wall thickness. And it's complemented by some kind of shear angled, right. So where it's not it's not zero degrees, but it has some angle to it, some incident angle, and they use those often because you want the straight beam to measure the local wall thickness so that they can better calculate time of flight for the shear wave. And so somebody can interpret that differently. Rhett: Yeah. Chris: Somebody could say, you know, hey, that means you should know what the depth of corrosion is. But that doesn't mean that the algorithm by which they're sizing, the sizing the crack is incorporating that risk. Rhett: It's funny that you mention that because that actually that happened here. They reported a local wall thickness or a wall thickness in the pipe, which is what they used to size the cracks. That wall thickness was higher than the nominal wall thickness. And Enbridge was actually using it in the failure calculations unknowingly. Right. But the ILI vendor understated that the measurement was not intended to be an accurate representation of the local wall thickness adjacent to the crack. But again you had an operator using information they obtained in a way that it wasn't intended to be used. And it's, to me it speaks to the conversation between the ILI vendor and the operator. Chris: And so I'm going to close this or maybe move on to the next one is it's my recommendation to our audience is it's look at your ILI program and if there is not an explicit process that does two things. One is qualify an ILI technology for use in an assessment that needs to be a formal process. And the reason for that is it's you should truly vet what that tool can do and how the data is collected and how the analyst is using that data to make decisions so that you understand how they're using the data and how they're not using the data. And that allows you as an engineer to say, where are my uncertainties and how am I going to account for them in my assessment, that's kind of point one. So I'm qualifying ILI technology for the threats, for the morphology, for my pipeline attributes. That should be a formal process. The other thing is understanding what and how that performance specification was developed because 1163 allows for self-governance on the ILI community. So an ILI company gets to say, “This is how I establish my performance specification under these essential variables.” So when you get one specification from one vendor and another specification from another ILI surge provider, they're not always under the same conditions and same assumptions. So these guys could have taken assumptions A and these guys over here could have taken assumptions B. And if those don't overlap and you're using one data set in one year and another data set for reassessment, there can be some uncertainties there. Rhett: Yep. 00;33;53;28 - 00;34;15;07 Chris: So you really need to understand what are the essential variables and what is the process in which the performance spec was established and understanding those two things. Then you intentionally select an ILI system and describe what data you want so that you can do your engineering assessment for pipeline integrity. Rhett: And so for those of you who are wondering where this appears in regulation, Chris gave you some parts of I would also say this is the whole bit. You see so much focus on interacting threats in ren one. Chris: That is one that's such a good point too right. So you would say if you've ever seen or heard some of these discussions around interacting threats, this is a big part of it, right. And we do see that either in gas code where it talks about that, where you see things like dents with metal loss, that's another example where if you have more than one coincident at a location, and there's a lot of back and forth that happens in our industry around what's difference between interacting and co-location and the impact of one or another, right. Rhett: So the next two are fun only because, I think these two have potential. I think they are really still hot topics today. Chris: All right. Rhett: Enbridge did not include the tool tolerance in the assessment of its defects. Which is why they called out uncertainties in the analysis. However, PII stated the tool tolerance should be incorporated in the report it cracked up. Chris: You got to be careful with that. Rhett: No look, Chris, you're on 1176. I know this is an ongoing conversation. Chris: Yeah. Rhett: They specifically pointed and later on they go through and they say, hey, if Enbridge had accounted for any one of these, if they accounted for the correct wall thickness, if they had run the counter for the crack depth in the tool tolerance, they'd accounted for the corrosion interaction, it would have been flagged as a dig. So go, they're saying right here, they point to it as not using tool tolerance in the calculations. Chris: Okay, so, Rhett: Where do we see it in regulation? Should I ask you that first? Chris: I should have seen a workflow to kind of keep all this organized right? Rhett: It's so much man. I'm warning, I'm warning the audience now, this may be a record length, record length for a podcast episode. Chris: So there's a couple of things you have to dissect here, and I'm going to try to be very left brained on this and logical. So the first thing that I would try to tackle and advise as it relates to this scenario is it's the key is to consider tool tolerance. You need to account for it right. By default, I guess you could add the tool tolerance to it based on the performance specification, right? So if we were to talk about cracking, if they're like, hey, our depth sizing tolerance is plus or minus 40 thousandths or approximately one millimeter in depth, I should just add it. I would say I guess that's a default, but that's not maybe the best practice, right? I think best practice is doing what 1163 says, which is you need to understand the motivation of the level of validation that you that you do to understand how the tool is performing, right? So you can use historical information. So as an operative, you've used this ILI system a ton and you've done a bunch of digs on it and you're very intimate with the technology and how it behaves under certain environments, you probably don't need to go do more because you know how it works and you have a robust system that describes it, right? And there are some efforts in industry. I think the report even cites PRCI as to what they need to do. PRCI does a ton of work on crack sizing and probabilities of detection, probabilities of identification and sizing tolerances. 00;37;27;00 - 00;37;51;17 Rhett: So I'll let you breathe for a moment and I’ll read, the NTSB recommends that the PRCI conduct a review of various online tools and technologies, including but not limited to tool tolerance, POD, POI, and provide a model with detailed step by step procedures to pipeline operators for evaluating the effects of interacting, corrosion and crack threats. That’s all of the NDE work in the projects that we see going on today. What is this, 14 years later at the PRCI TDC in Houston. Chris: Absolutely. And what I would say is it's, that's helpful, but it's still incumbent on the operator to understand how you're considering these tolerances. So is that a level one where you're looking at previous efforts? Is it a level two where I'm looking at previous efforts and there's some uncertainties that I'm going to go fill the gaps on by doing a couple of excavations and direct assessments? Or is it truly I don't have enough information about the ILI system, my pipeline, or how the tool is performing, and therefore I need to maybe look at a more substantial set of digs and even potentially a level three validation. So that said, I'm going to try to tackle this. So that's the ILI component. The reality is there's also this component of the threat assessment, right? The engineering assessment of cracks don't just fail based on their depth, right. There's two other things that we need to think about right here. And one is material properties, right? So what are the material properties that we're using that's kind of point one. And then we're also thinking the other input is usually the load or what type of stresses it’s having to manage. So we'll kind of bucket that into my next statement and that's what's your method for doing the assessment, right? So what type of burst pressure calculations, whether it's empirically based like modified toxicant or is it fracture mechanics based like something like API 579. Right. And so you also have to think about that. So it's not just the ILI tool, it's also how am I using that ILI data in my engineering assessment. And based on that, those two sets, those two processes that can't be fully independent, they kind of have to be embedded and integrated. You then say, “How do I now account for the tool’s uncertainty?” And if you can't or it's too complex, I guess defaults would be apply the tool tolerance and have these relatively arbitrary response conditions which is, you know what? You have a crack, add the tool tolerance, default your metal properties to 607 and anything with a burst pressure less than 1.25 times MOP, you need to go dig it. So you kind of get to pick your poison a bit. So that's my reaction. Rhett: I love it. I thought it'd be a little bit fiery and you did exactly what I wanted you to do. Oh, and again, I think that that discussion of where tool tolerance fits in is still an ongoing discussion. Chris: Absolutely. Rhett: And, but I do think regulatory is has made it appear that they're on the side that it needs to be included. But again, there's that is an ongoing conversation. Chris: But I want to address that right and I'm going to use this word “standard,” right? By standard, I mean like the minimum bar. I think what the regulators are trying to achieve is if you cannot clearly describe what you're doing and it has a sound technical basis, we are responsible for setting a minimum bar. And that default, therefore becomes whatever the ILI vendors are saying the tool limitation is. 00;40;45;21 - 00;41;15;03 Rhett: All right, you ready for the last one? Chris: Let's try it. Rhett: This is one of my favorite ones over. It is, I called it “nonsensical safety factors” because that's what they have. The NTSB labeled it as an insufficiency safety margin on the cracks. And here's why I say nonsensical. They, the corrosion program required excavations below 1.39 times MOP and corrosion tools at this time were described as being fairly well understood. Corrosion growth rates fairly well understood. Therefore, my safety factors 1.39. Enbridge admitted in here that the crack tools had less reliability, there was more uncertainty with crack growth rates than there were with corrosion growth rate, less certainty in the tool’s performance. Chris: So all of these are saying, I understand it less and I have less confidence in it. That's what it sounds like to me. Rhett: But their dig program didn't require them to do anything to 1.25. And so basically FEMSA says neither of these safety factors reflected it a lower safety margin used when assessing cracks. A larger margin of safety would have resulted in a larger number of cracks being eligible for excavation and examination and basically out of this, they conclude in the point where we're about to go to, that prescriptive regulation is needed to govern response criteria for cracks. Chris: And that is where you begin to see it, right? So specific to the gas guys, that's why REN 2 is present. Right. And it's pretty clear as to what it is and the basis for it. I think in the preamble of REN 2, I think it's specifically states right. It says any significant cracking will be an immediate and if it's not then it'll be a one year. So so it's a one year they don't even say scheduled. Right. So if we if we if we look at historically what it did is it's and I'm sure there's people in ASME committees that get fired up about this but for corrosion is based on figure four. It's traditional figure four because I think it's the 2008 version of ASME incorporated by reference, not the latest, and so that has an assumed growth rate in it for a figure four, it's very linear. And so you have a scheduled response. If you have a 1.1 verse pressure for metal loss, that's an immediate but a 1.25 is roughly five years. Here they're saying, nope, four cracks, right? If it's if it's not an immediate it's a one year until you better understand it and then you can approach me and tell you why you think you can have a different response. There's no trust. Rhett: We are going to take the first ever second break because I want to lay the groundwork. I don't know when a dog gets in a fight with a hair stands up on its back. Is your hair stand up on your back, are you? 00;43;42;25 - 00;44;05;20 Chris: That doesn't happen to me. I'm generally kind of even keel. I mean, I mean, I think you never get a natural response. We flare up a little bit, but I don't know if the hair thing is real. Rhett: Well, listen. The NTSB concludes that Enbridge's Integrity Management program was inadequate because it did not consider the following a sufficient margin of safety, appropriate wall thickness, tool tolerances, use of a continuous reassessment approach to incorporate lessons learned. We didn't even talk about that. We didn't have enough time and the effects of corrosion on crack depth sizing. In response inadequate regulator requirements and oversight of defects and pipelines. This is the fourth point that, as these mentioned, Title 49 of the CFR fails to provide clear requirements for performing an engineering and assessment and remediation of crack like defects on the pipeline. In the absence of prescriptive regulatory requirements, Enbridge applied its own methodology and margins of safety. Enbridge chose to use a lower margin of safety for cracks than for corrosion when assessing cracked defects. FEMSA expects pipeline operators to excavate all crack features. We're going to pause on that point and we're going to come right back to pick up this conversation. 00;45;21;01 - 00;45;30;14 Rhett: All right. So we are picking back up after the break. And I left you on that cliffhanger there, Chris, because I do think, you know, that statement is shocking and I'm going to read it again. Inadequate, inadequate regulatory requirements and oversight of crack defects and pipeline. FEMSA was cited for that. The NTSB support specifically cites inadequate regulatory oversight, which is makes it very easy for us to tie this to the updated gas rule. But the second note, the shocking part, FEMSA expects pipeline operators to excavate all crack features. Right? And then later on, they actually specifically quote FEMSA's director of engineering that made that statement. In this line, I’m gonna read it again, because I didn't give you the background. In their crack tool, and I'm reading verbatim, Chris: Go ahead. Rhett: The 2005 crack tool in-line inspections showed over 15,000 defects on line six B. Chris: That one, that won't happen again. I'll tell you why. Rhett: The worksheet listed 929 crack-like features identified by the in-line inspection tool. If FEMSA’s expectation is that operators are going to excavate all crack-like features, I’m guessing line six B becomes non operable. 00;46;46;20 - 00;47;18;17 Chris: Yeah or the operating pressure becomes very challenged. So I'll highlight a couple of things guys. So when I went on my API standard 1163 plug earlier in the show, I mentioned definitions. What does that mean for us? So when, when FEMSA incorporates by reference the whole document of 1163, that means you're also incorporating the definitions and a defect is defined by API standard 1163 as a flaw that is no longer fit for service. So we heavily encourage you guys, when you are talking about anomalous conditions reported by an ILI, you call it a defect when it's not fit for service. We need to get better at that because the way I interpret that is, is they have an exorbitant number of required responses now because those are features that are not fit for service. So that's kind of my first initial reaction Rhett: And that was their language. I read it verbatim. Chris: And keep in mind this was also before Standard 1163 was incorporated by reference. But now that it's incorporated, that is the new standard, right? That is the new bar and how we're supposed to communicate. So that's kind of one of my first takeaways. And I would say that they held true to their word, right? So if we look at the liquid rule, if I paraphrase, basically it's a scheduled response. I think it's a scheduled response for any potential crack that, when excavated, is determined to be a crack. Rhett: It is a 180 day condition, you are correct. Chris: I mean, it's any potential crack. And so if you have a linear indication, there is no room and I'm going to say it. So look at me. FEMSA didn't leave any room for us to do engineering. There's no engineering involved because if a tool reports it, it's a potential crack. You have to go dig it up to determine if it's a crack. And what I would say is this is it's that that does not give credit to the advances in ILI technology now. Right. POD rates are high. We know where there's crack-like flaws and POIs are improving in data integration and years of excavating and years of understanding how pipelines operate, help us understand what we see in API, RP recommended practice 1176. Rhett: So let me... Chris: Hold on. 1176 says you have a you have a way to segregate between likely cracks, possible cracks, and unlikely cracks. And so what I would say is it's really important for pipeline operators to have these documented processes. If you remember Brian Jimenez, define and defend, right, define and defend your practice. Write out your procedures on how you're addressing things that are called by an ILI tool and how you're classifying them as an operator for response and then work with FEMSA or your state regulators to bring some confidence to that process. That's for the liquid guys, for the gas guys. I mean, they define it. Rhett: Let me show you how it gets tied in, because I actually really like how this part of the report reads. When I say like, I like it for the purpose of the podcast. Chris: Yeah. Rhett: So you already mentioned that 452, which was 195 452 at the time, was the portion that governed liquid lines required a potential crack that when excavated is determined to be a crack was a 180-day condition. 00;50;05;03 - 00;50;48;17 Chris: Yep. Rhett: FEMSA’s Director of Engineering stated they expected that all cracks to be excavated. 49 CFR 194 452 does not address the size, depth, location, or suitable engineering assessment methods associated with the predicted failure, pressure of prioritization of crack defects the same way it does with corrosion defects. The regulation addresses cracks as potential cracks that, when excavated, are determined to be cracks, but does not address what constitutes potential cracks or whether excavation is required of all cracks and expectation expressed by FEMSA’s Director of Engineering and Research. Because the regulation is less explicit regarding the assessment of cracked features, it does not clearly state that this or the safety margin that should be applied to predicted failure pressures as it does with corrosion. It goes on to say, listen, based on its findings, and I think this is where you will now begin seeing a very clear tie to today. The NTSB recommends that FEMSA revise 49 CFR 195 to clearly state when an engineering assessment of crack defects, including environmentally assisted cracks, must be performed. The acceptable methods for performing this engineering assessments, including the assessment of cracks coincident with corrosion. Three, the criteria for determining when a probable crack defect in a pipeline segment must be excavated and the time limits for completing those excavations. The pressure restriction limits for crack defects that are not excavated, and five, acceptable methods for determining crack growth rates and any cracks to allowed to remain in the pipe. Chris: So all of that. So here we go. Rhett: So where do you see it? Chris: So all of that is in 195, right, guys? I mean, how old is this report? Rhett: 2012, I think. I think is when it came out. Chris: So all of that is what’s in liquid regulation. Right. If we had that big red X thing, where do we find it? On the gas side. Rhett: I turn it to you. Chris: Yeah, we find it on the gas side. And where do we end up? Rhett: I teed up the softball Chris, this is where you come in. 00;52;27;23 - 00;52;46;15 Chris: Yeah. And where do we see it? Right? We see it on the gas side. And we see it in two parts. Right. So the first one is we see it in subpart O, which if you're intimate with the gas regulation. So subpart O is the part of 192 where they issue the regulation for pipeline integrity management specifically and it targets HCH. So if we if we hone in just on that part, there is now criteria for crack response. Rhett: Absolutely Chris: Right? And guess what they address? Actually, I want to read one of these out, ready? A crack or crack like anomaly, meaning any of the following conditions, is an immediate, crack depth plus any metal loss greater than 50% wallows. There you go. Rhett: So again, what is so amazing to me is these are things that have gotten feedback in the industry. Chris: Yep. Rhett: But this was FEMSA’s response to the NTSB report that pointed to you, just pointed to tool tolerance there. Inadequate, like in light of non-prescriptive regulation, which existed at the time that Marshall happened, it might have taken what do you want to say, seven, eight years, no, nine years is when the first one came out? I'm getting my dates mixed up now, written in 2001 maybe it’s 11 years, point is 11 years later we now get the prescriptive regulatory fallout, Chris: From this report. Rhett: From this report, but not in the liquid rule, in the gas rule. Chris: Right. And that's exactly my point. Rhett: And that's why we told the gas operators to hang on, Chris: The liquid operators. Rhett: Well, we told the gas operators in the very beginning they need to listen. 00;54;04;06 - 00;54;34;25 Chris: Oh yeah, earlier in the episode, yep. Rhett: If you're wondering as a gas operator where 712 where the new 933 why this stuff, it's because of what happened here. This speaks to that a failure of one of us is a failure of all liquids and gas. When a fireball happens, the liquid guys aren't exempt. And when the environment's destroyed, the gas guys aren't exempt. Like FEMSA and the NTSB see them comprehensively together. 00;54;34;25 - 00;54;52;26 Chris: Yeah. And around the engineering assessments. So then we see you would say, Chris, well that's, that's just depth based. What about all the other stuff they mentioned about establishing appropriate methods or qualifications. So that's where 192 712, comes in, right? Where they say it's basically crack analysis and that's where they begin to offer some of those details, right? Like you have to use an approved method for establishing birth pressure calculations. You have to have this approved by an SME. And so, and if you don't have material data, they default to you. They say then you have to use these material data as if you've had an in-service failure or not. Rhett: And now you're putting in the plug for material data, which is really where the next episode, Chris: Which is in the next episode, exactly right. And so the point is and that's kind of the segway almost for the next episode is it's you're seeing the last 15 years of incidents manifest now that aren't just gas related or liquid related. It's clear that there's an opportunity through the gas incidents that have happened where we've had some congressional mandates that say, hey, our expectation is that you're using more ILI. Our expectation is that safety metrics start going up. And so we're starting to see a lot of that, and it's manifested itself first in the gas rigs. Rhett: Absolutely. And I think for me, you know, as we look towards closing out this episode, the thing that was surprising was I don't think I realized the impact that Marshall Michigan had on REN two and three until I went back and saw it. And as you read some of those lines, you're like, now I understand why even though it doesn't, it feels punitive sometimes. You understand why that regulation got written that way. That doesn't make it good or bad. It doesn't mean, I think that, you know, the role of GPAC and the grit in pushing back on the way some of the initial regulation was written has been extremely important. 00;56;25;16 - 00;56;43;26 Chris: Yeah. Rhett: And what I would encourage people to do is, you know, again, I would encourage you out there, if you take some time, pick up the NTSB report on Marshall, Michigan. It is good. If you're also curious where a lot of the regulation came from, pick up the FEMSA ERW study, which also talked about a lot of these methods. It's really important to understand the basis of where this came from. I'd read the ERW study, I had never read Marshall, Michigan. Chris: And what I would also say is it's we have to find competence in what we do. And what we do is not simple systems. It's complex that requires sophistication, right? We need to be good at what we do. And to do that, we have to learn from our mistakes and we need to read. We cannot, competence is formal training with appropriate experience. And then obviously you want some governance and some kind of qualification there. But the idea is we've got to study this stuff. And so those are good reads. Always read API 1163 if you have it, that's always a recommendation. And the other one is it's read the preamble REN 2, it's really important. If you read the preamble for REN 2, you'll see where they bring in Marshall, Michigan, and you'll start getting an understanding for the posture for some of the regulations, which then you start potentially seeing clarity in things like FAQs. Right? And we usually don't talk a lot about FAQs, but FAQs are frequently asked questions, and how FEMSA’s responses establishes their position on how to interpret certain things. So it's really important to grab those other two documents, right? So read the preamble for REN 2 and also check out the FAQs associated with the new gas regulation. Rhett: So I want to thank all you guys for hanging in there. As we close out this episode, I'll just put a plug. We will be tackling San Bruno next. I think something that I think is important to every pipeline operator and anywhere within the United States, regardless of the product that you're running. I think San Bruno, even more than Marshall, Michigan, defined the state of pipeline integrity as we live and do it today. Chris: But what you're seeing is it's you'll see how this was kind of step one. And when you see the proximity of the timing between these two incidents and how the NTSB was dealing with both of these at the same time, how both recommendations. So here's another thing, right? As a reminder, when you read these NTSB reports inches, the NTSB usually comes out with recommendations and they're basically pointed out at FEMSA right. And so the idea is when you see the proximity of both Marshall, Michigan and San Bruno, you begin to understand the shape and the posture and the position that FEMSA had to address the NTSB and to address Congress and to address are our social license as a pipeline industry in the gas rule first. But I think we're all we all can bet what's going to be happening after the gas rule. Rhett: It is, we didn't bring it up, the San Bruno incident was actually mentioned multiple times in the report. But anyway, that's what we're going next. I do want to thank you second string, Sarah, great first episode. We look forward to being back with you in two weeks and as we continue our journey through the NTSB reports. Thank you again. I am your host, Rhett Dodson, my co-host Christopher DeLeon, signing off of Pipeline Things.