Sewage plant failure: design error coupled with electrical configuration problems

Friday, at 4:30pm, Halifax Water finally released the forensic audit of the January 14, 2009 failure of the Halifax sewage plant.

The forensic audit confirms almost everything I wrote in my August, 2009 investigation of the plant failure, “How the sewage plant broke.”

Here’s what I wrote two years ago, with minor adjustments, corrections and new information learned from the forensic audit added in italics where necessary:

Heavy rain alone wasn’t enough to break the plant. In the early hours of that January morning, a series of cascading mechanical and electrical problems occurred—a perfect storm of errors. The disaster started with a Nova Scotia Power failure throughout the north end of Halifax, including the plant, at 2:25am.

The treatment plant is built around an 85-foot-deep “wet well.” A large tunnel carries Halifax’s sewage—toilet flushes, rainwater, anything running through the city’s ancient sewer pipes—to the bottom of the well. Four submersible pumps (and a fifth backup) at the bottom of the well lift the sewage up to the treatment equipment on the main floor. When the plant lost its power and the pumps stopped working, a large iron gate automatically closed over the tunnel, stopping the flow of sewage into the wet well.

Twenty Fifteen minutes later, an on-call technician arrived to fire up two backup generators, labelled 2A and 1B. With the generators online, the technician opened the gate and the four pumps began operating. So far, the plant had responded to the power outage as it was designed to.

[Actually, the sequence was much more complicated than this. The operator opened the gate only in degrees, with one percent at a time, and the automatic SCADA operating system turned on the pumps individually as water levels rose. By 3:5pam, the operator had opened the gate 14 percent, and three pumps—#s1, 2 and 3—were pumping. Only when the operator, at 3:59:50am, opened the gate to 15 percent, did the fourth pump—#5—turn on.]

But the electrical load from the pumps was not evenly shared by the two backup generators. One generator, 1B, carried the load for three pumps (#s 1, 3 and 5), while the second, 2A, carried the load for only one pump (#2– #4, which was also on this circuit, was automatically kept off as the “backup” pump because it had logged more operating hours than pumps #3 and 5). The generator carrying the three pumps (1a) overloaded and shut itself down, leaving just one pump to handle all be rainwater and other sewage coming into the plant.

The plant couldn’t last long with just one of its four pumps working off a backup generator.

[This is incorrect. It turns out that while the generator 2A was still operating, pump #2, which it powered, was not operating. This is because the control and monitoring equipment attached to all the pumps was also connected solely to generator 1B, that overloaded; without the control equipment, pump #2 also automatically shut down.]

To stop sewage from coming into the wet well, the gate should have closed over the tunnel, but the mechanism for lowering the gate was also powered by overloaded generator, 1B, and so didn’t work. Anticipating just such an emergency, the plant has a switch designed to shift the gate’s load to the second generator, but this morning it failed to operate properly. (This is incorrect; the audit mentions that there should have been such a switch, but none existed.) The gate was therefore left slightly open, and sewage continued to flow into the wet well.

The pump mechanisms and motors are in watertight casings, so they weren’t in any danger from the rising sewage. However, they are powered by cables leading from electrical junction boxes placed just 10 feet above the pumps. The boxes are not watertight. When the rising sewage reached the junction boxes in slightly more than three minutes, it flowed into them, down the electrical conduits and into the pump casings, shorting out each of the pump motors, including the motor running the lone working pump. At this point, even if the power came back on, all the pumps would be useless.

With no working pumps and the tunnel gate open, sewage continued to flow into the wet well, and in about 20 minutes rising all the way to “hydraulic grade”—the level where water pressure evens itself out, in this case sea level, which is roughly at the ceiling of the plant’s basement. Much of the plant’s equipment, including the electrical control room and boilers, were in the basement, below hydraulic grade, and therefore completely immersed in sewage.

But, in addition to the above narrative, the forensic audit contains two more previously unknown details, as follows:

• Degremont, Inc.’s design protocol for the two back-up generators in the plant say that the electrical load on the generators would automatically be stepped down by the SCADA operating system when the load on either generator exceeds 80 percent. But that never happened. At 3:56am, when the operator opened the gate to 14 percent, generator 1B had a load that exceeded 93 percent of its capacity, and yet there was no step down in load, as called for. The forensic audit does not explain why this never happened. Fifty seconds later, when the operator opened the gate to 15 percent, the load on generator 1B was 123 percent, and it failed. “From the above sequence,” reads the audit, “it is evident that on January 14, 2009, there was no automatic program to shed loads as indicated in Electrical Power Management in Degrémont’s SCADA documentation (located in Book 2, Tab Q) as additional pumps and the Odour Control Fan started overloading and shutting down Generator1B.”

• The audit states that the operating manual for the sluice gate was modified in June of 2008, but “does not reflect the most recent information on the electrical connections drawing.” Later in the audit, it is noted that the circuit for the sluice gate was designed to automatically close the gate should the situation faced January 14 happen: “The sluice gate should have closed immediately, because generator 2A was still running and the ATS should have sent 600V power to the sluice gate actuator. No record or document showing the testing of these operations were found or reported. From this, it is concluded that although modification to the sluice gate actuator control circuit was done, it was never tested to confirm its proper operation.” Again, the audit does not attempt to place blame for this failure.

My understanding of the failure is now this: There were fundamental design errors for the plant, but additionally the electrical circuits connected to the generators and sluice gate were not configured as to design.

The fundamental design error was that the electrical junction boxes, the sluice gate control, the wet well electronics and, indeed, the entire electrical control room were placed below hydraulic grade. Had they not been, it wouldn’t have mattered what other problems were introduced into the plant—the plant might go off line for a bit, and even flood, but it would’ve have been simple to restart it and get things working like they should. As it was, however, some relatively minor configuration problems were amplified into a disaster, because designers did not take seriously the possibility that there could be systematic failure.

As for the configuration problems, secondary power panels, which ran monitoring and control equipment, were connected only to generator 1B, instead of being connected to both generators via a “tie breaker.” In the case of the sluice gate, there was a modification, but that modification was never tested, and subsequently failed.

Related Stories

Peter Kelly wears the sewage disaster

Because the mayor won’t release the new forensic audit and break the wall of secrecy around Halifax’s broken sewage treatment plant, he’ll end up owning the whole mess.

This article appears in Nov 24-30, 2011.