click to enlarge

Halifax's new sewage treatment plant was turned on in February 2008, and it seemed to fulfill its promised intentions immediately. All you had to know was that parts of the harbour were swimmable again, and the most expensive infrastructure project in Halifax history seemed worth it. By last January, the plant had been operating almost a year, facing every weather condition without a hitch. There was no reason to think the heavy rain forecast for Wednesday, January 14 would prove to be the plant's undoing.

Until recently, we had no clear idea what exactly went wrong---mayor Peter Kelly has hidden information on the plant failure behind a wall of secrecy. He has refused to release a $100,000 forensic audit investigating the failure, and Tuesday he scheduled yet another closed-door, no-paperwork council meeting to discuss the sewage situation.

But over the past several weeks I've interviewed engineers from cities across North America who have explained how their sewage systems work, as well as many local sources who are familiar with the design and construction of Halifax's system. Additionally, officials with the Halifax Water Commission, who are responsible for fixing the plant, have recently become more talkative, and are now answering direct questions. Together, those conversations have led to a deeper understanding of the plant failure.

What went wrong

Heavy rain alone wasn't enough to break the plant. In the early hours of that January morning, a series of cascading mechanical and electrical problems occurred---a perfect storm of errors. The disaster started with a Nova Scotia Power failure throughout the north end of Halifax, including the plant.

The treatment plant is built around an 85-foot-deep "wet well." A large tunnel carries Halifax's sewage---toilet flushes, rainwater, anything running through the city's ancient sewer pipes---to the bottom of the well. Four submersible pumps (and a fifth backup) at the bottom of the well lift the sewage up to the treatment equipment on the main floor. When the plant lost its power and the pumps stopped working, a large iron gate automatically closed over the tunnel, stopping the flow of sewage into the wet well.

Twenty minutes later, an on-call technician arrived to fire up two backup generators. With the generators online, the technician opened the gate and the four pumps began operating. So far, the plant had responded to the power outage as it was designed to.

But the electrical load from the pumps was not evenly shared by the two backup generators. One generator carried the load for three pumps, while the second carried the load for only one pump. The generator carrying the three pumps overloaded and shut itself down, leaving just one pump to handle all be rainwater and other sewage coming into the plant.

The plant couldn't last long with just one of its four pumps working off a backup generator.

To stop sewage from coming into the wet well, the gate should have closed over the tunnel, but the mechanism for lowering the gate was also powered by the overloaded generator and so didn't work. Anticipating just such an emergency, the plant has a switch designed to shift the gate's load to the second generator, but this morning it failed to operate properly. The gate was therefore left slightly open, and sewage continued to flow into the wet well.

The pump mechanisms and motors are in watertight casings, so they weren't in any danger from the rising sewage. However, they are powered by cables leading from electrical junction boxes placed just 10 feet above the pumps. The boxes are not watertight. When the rising sewage reached the junction boxes, it flowed into them, down the electrical conduits and into the pump casings, shorting out each of the pump motors, including the motor running the lone working pump. At this point, even if the power came back on, all the pumps would be useless.

With no working pumps and the tunnel gate open, sewage continued to flow into the wet well, rising all the way to "hydraulic grade"---the level where water pressure evens itself out, in this case sea level, which is roughly at the ceiling of the plant's basement. Much of the plant's equipment, including the electrical control room and boilers, were in the basement, below hydraulic grade, and therefore completely immersed in sewage.

Raw sewage has been flowing into the harbour ever since. City officials say they can put the plant back in working order by spring of 2010.

How to fix it

Placing the pumps' junction boxes and the electrical control room below hydraulic grade was a tremendous design error.

Carl Yates, manager of the Water Commission, tells me that he'll have to completely rebuild the electrical control room. There isn't space to move the new equipment up to the main floor of the plant, but he's confident he can isolate the room from the wet well---that is, he'll seal off the room, have its drainage system lead away from the wet well and lift the electrical cables up over the walls, above hydraulic grade, before descending again into the wet well.

But Susheel Arora, who heads the wastewater division at Halifax Water, defends the placement of electrical equipment below hydraulic grade. Even the quite low placement of the junction boxes---just 10 feet above raging sewage flows---was an acceptable design, says Arora.

"Lots of operations are set up like this---look at all the plants around harbours," he says. Pressed for a specific example, he points to Toronto, which like Halifax has deep pipes feeding its sewage plants.

Indeed, many cities, including Toronto, Boston and Portland, Oregon, have similar "big pipe" sewage systems, but none of those cities favour using submersible pumps or placing electrical equipment beneath hydraulic grade. Instead, each uses a "wet well/dry well" design---the sewage flows into the wet well, but the pumps and electrical equipment are kept safely isolated in an adjacent dry well.

Toronto's system does have some electrical equipment below hydraulic grade, explains Frank Burford, senior engineer with the city of Toronto, but those are in plants that were designed at least 50 years ago.

"In new designs all the electric is at grade or above, so you keep your electrical stuff out of that environment," he says. Moreover, with those older inherited designs, Toronto is moving to rebuild the plants to move equipment above hydraulic grade.

Would Burford design a new system with electrical equipment below hydraulic grade?

"No," he says unequivocally.

Yates, for his part, says the junction boxes in the Halifax plant will be moved above hydraulic grade.

"Those will absolutely come up," he says.

I had that conversation with Yates on Friday afternoon. Tuesday, after the closed-door council meeting, the city issued a press release stating that "all electrical junction boxes will be moving up to the street level area of the plant with the delivery of longer cables required to do this expected by the end of August."

In addition to the design error of placing electrical equipment below hydraulic grade, there were also at least three problems associated with the backup generators in the plant: the generators' powering to the pumps was improperly sequenced, the gate mechanism switch between the generators failed and the generator overloaded. It's unclear at this point if these were strictly design errors or if the equipment was incorrectly assembled, or both.

One question that needs to be addressed: Why was an emergency generator that shuts itself down when overloaded put in the plant? Many emergency generators---for example, those that power fire fighting equipment---are designed specifically not to turn themselves off when they overheat; it's judged better to risk losing a generator than to lose power to the equipment the generator is attached to.

Regardless, it should be a simple matter to right these generator-related problems.

Lastly, there's the tunnel gate, which evidently could not be closed manually, another clear design error. A weighted, manually operated gear assembly will need to be installed, so that a technician can close off the tunnel, even when there's no power in the plant.

Again, after discussing the gate issue with Yates last Friday, the city's press release on Tuesday reflected the gist of our conversation.

It reads, in engineering lingo: "The sluice gate actuator has arrived and is being assembled with the gear box."

Lessons not Learned

It's tempting to treat the sewage plant failure as merely a narrow technical problem---discover the mechanical issues involved, fix them and be done with it.

That's certainly the view of mayor Peter Kelly, who has repeatedly said we should "look forward" and "move on" to next summer, when the plant will again be operating correctly and the harbour once again clean enough to swim in.

That approach has the added benefit of taking the public focus off assigning fault for the failure, a politically messy matter of contention that will no doubt end up in a multi-million-dollar court battle.

But the sewage plant failure is not just a narrow technical problem. Rather, the technical problems at the plant were the result of political and bureaucratic decisions made at city hall years ago---process issues.

Ignoring those process issues, or hiding them behind a wall of secrecy, will mean that they won't be properly addressed, and we'll have other catastrophic or costly events in the future.

At its root, the sewage plant failure is a failure in how government gets stuff built.

Traditionally, when city governments undertook large projects, they'd hire an engineer to design the project, and then contract with a construction company to actually build it. The engineer, employed by the city, represented the city and closely watched to make sure the contractor built the best possible project. After construction, the city would take ownership of the project, and city employees would operate it.

But in recent decades cities have increasingly contracted large projects out entirely to private businesses---that is, the design and construction (called "design/build") and sometimes additionally the actual ownership and operation (called "design/build/operate") of the project is undertaken by private companies.

Halifax's sewage system was built through a design/build contract with a consortium of two private firms---Dexter Construction and Degremont Limited.

"To properly do a design/build---to actually make it work---you also have to give the operation component of the system to the private firm," says Frank Burford, senior engineer for the city of Toronto, who worked most of his career in the private sector. "Because otherwise the private companies will do a design/build that is most economical for them---because that's what it's all about, in money---and so redundancy and extra features that will make it more reliable to operate long-term can be sacrificed, because their costs end when they turn the keys over to the owner.

"I found that design/build in Ontario didn't really work," he continues. "You really do need the fingerprint of the owner on the thing, because then they can review it. It takes maybe a little bit longer to get something done, and it might cost more, but there's more checks and balances."

Burford points out that whatever cost and time savings Halifax gained by going the design/build route for the sewage plant were lost with dealing with the plant failure.

He's ambivalent on whether cities should stick with the traditional method of hiring an engineer to represent them in a building contract or go the opposite direction and contract out in design/build/operate fashion. Either way, he insists, "operations guys" must be involved in the design process.

"You get together with the designer of the facility, your operations people and your engineers in the city," says Burford, "and you look at 'what if'---what if this goes wrong, what if this fails, what kind of contingencies do we have? Then you look at all the Murphy's Law stuff---what if, what if, what if, and are we covered for this? So that at least you can get the system back running."

Obviously, if such a process happened at all in Halifax, it happened badly, as the multiple design failures---which one local engineer calls "boneheaded"---demonstrate.

It's clear that such a process didn't work because the city placed the design of the plant entirely in the hands of the private companies whose interests laid in cutting costs, and not in providing a fail-safe plant operational into the future.

A series of mechanical failures didn't doom the Halifax sewage plant---the design/build contracting system did.

And yet, the city continues to use design/build contracting for other large building projects, including the $40 million four-pad hockey arena slated for Bedford [see correction below], the largest city project to be undertaken since the sewage system was built.

It's unlikely that a poorly constructed hockey arena will result in an environmental catastrophe, but it could very well lead to unexpected future costs to taxpayers related to shoddy construction.

And if not the hockey arena, then so long as the city relies on design/build contracting, some other project will fail.

It's only a matter of time.

However, for the record, I also don't favour design/build/operate projects. I think that either d/b/o OR the traditional method of contracting achieves the desired goal with regard to getting the best project built. BUT, there are, in my opinion, other non-construction-specific considerations; specifically, wage and salary issues. I think that once these sort of projects are completed, they should be operated by well-paid public employees, both because they'll do a better job than lower paid private employees, and because it's the right thing to do--- government facilities should have well-paid, unionized employees. That's ultimately a political argument, however, so I left it out of the article. —TB

Animation: How it Broke

Infographic: how it broke

Fig. 1

The plant operating properly: sewage from a deep tunnel enters a wet well 85 feet below the plant. Four pumps lift the sewage to the main floor of the plant, where the sewage is treated.

Fig. 2

A winter storm knocked out power to all the north end, including the Nova Scotia Power feed to the sewage plant. At this point everything worked as it was supposed to: the tunnel was closed off by a gate, and the plant was completely shut down.

Fig. 3

A technician turns on two backup generators and raises the gate.

Sewage flows into the wet well. But the load from the pumps is unevenly distributed between the generators. One overloads and turns itself off. The remaining working generator powers only one pump.

Fig 4

The single working pump can't handle the volume of sewage pouring into the plant. Sewage rises to the electrical junction boxes feeding the pumps, flows into the junction boxes, down the electrical conduits to the inside of the pump casings, shorting out the motors.