One community makes the leap into MQTT-based utility management

Formally organized in 1834, Waterford Township is located geographically in the center of Oakland County, Michigan, USA, and is home to over 72,000 residents. It is known regionally for its 34 lakes, from which it earns its name.

However, in the public utilities industry, Waterford is known for its leadership and persistent innovation in water/wastewater management. The Department of Public Works has been featured previously in an Opto 22 case study, and a generation later, DPW is working on another landmark project.

With 360 miles of water main and 355 miles of sanitary sewer, water management in Waterford is no small task. The department operates and maintains 19 production wells, 3 storage tanks, 11 treatment plants, and 63 sewer lift stations. To run all this, they invested years ago in integrating core applications, including geographic information systems (GIS), asset management systems (AMS), enterprise content management (ECM), and supervisory control and data acquisition (SCADA), all of them sharing data to enable seamless operations. That system has delivered a lot of value over the years, but nothing lasts forever.

In the last few years, Russell Williams, director of public works, and Frank Fisher, engineering superintendent, started on a project to upgrade core infrastructure. It was an opportunity not just to avoid obsolescence but to improve scalability and fault tolerance. Little did they know that three years into the project Opto 22 would release groov EPIC and the scope of their plans would change.

As Russ tells it, “We attended [a conference] in Chicago about the release of EPIC [in 2018]. We were talking about it on the ride back and said, ‘If this does what it’s supposed to, then it changes the whole layout of everything.'” They were particularly intrigued by MQTT and the way its report-by-exception behavior promised to improve the efficiency and stability of their infrastructure.

Waterford DPW’s legacy communication infrastructure

At the time, Waterford’s system used a polling program to request updates from a network of Opto 22 M4 RTUs (remote telemetry units) that communicated through licensed radio frequency (RF) transmitters. DPW had already begun replacing these with SNAP-PAC-S2 controllers and industrial cellular modems that communicated through a private internet tunnel. But with MQTT on groov EPIC, they saw the potential to reduce the complexity of the system even more. “We have many lift stations that will spend most of their time sitting,” Russ explains, “[So] why transfer data all the time?” And with no dependence on a central polling program, they could eliminate a potential point of systemic failure.

“It just looks too simple. You’ve got to question it,” recalls Russ. But, willing to test the premise, Russ and Frank purchased three EPICs to play with over the summer, and soon they had the evidence in hand.

“We disconnected a controller and within a millisecond the system reported the failure. It really is that easy: change a variable and it shows up in the broker, then on your mobile phone,” says Frank.

“It is that simple!” confirms Russ.

A lift station panel showing the old M4 RTU layout and new groov EPIC layout

The new system they have begun rolling out leverages groov EPIC’s connectivity features to improve the fault tolerance of their system even more:

• Since groov EPIC can communicate directly with cloud-based systems, Waterford chose to host their MQTT broker (Ignition from Inductive Automation) on Amazon Web Services (AWS).
• Each EPIC uses a CA-signed client SSL certificate to establish a secure broker connection using a cell modem configured to block any traffic not originating from a trusted IP address.
• Additionally, with MQTT’s device-originating connections, the groov EPIC's device firewall in each lift station can be closed entirely to outside connections.

With that foundation, Waterford DPW made other infrastructure changes that have put them at the forefront of municipal utilities management. For instance, their Ignition MQTT server is hosted in an Ohio, USA data center, but if there are any issues, in 30 minutes, Frank can have the entire system up and running on a snapshot of the same server hosted in an Oregon data center. And in time, they will likely set up full server redundancy.

Meanwhile, they continue to run the legacy SCADA system (on upgraded hardware) in parallel. It still manages about a third of the system and will stay up until they have fully migrated to Ignition.

Waterford DPW’s modernized communication infrastructure

Russ recognizes that Waterford is leading digital transformation in the public sector. “I was at a FEMA training not that long ago, and they were adamant about not having an internet connection on your SCADA system,” he says. “But everything we are looking at will be more secure than we could do from [the office because then] you make a building a single point of failure.” 

With data coming across much faster, and groov EPIC providing spare data processing capacity, Russ and Frank expect to dramatically increase the connectivity of the system. By integrating residential meter data, for instance, they could help the system stay balanced against demand. And if they can talk other agencies and neighboring counties into sharing data, they see the potential to build an advanced warning system that would improve their reaction time to system disturbances.

As of February 2021, Waterford DPW had converted 36 lift stations and 4 treatment plants to the new system, with more to come throughout 2021. We’ve kept checking back to watch their progress, and here are Waterford's latest developments. And if you'd like to connect with the Waterford team, you can reach Frank Fisher at ffisher@waterfordmi.gov.

Want to have your amazing project featured on the OptoBlog? Drop us a line any time, and let us know what you are working on.

----

UPDATE 10/13/21: Read part two of this story here.