ICC 2021 Discover Gallery finalist, Perceptive Controls, talks about their continued success with the Waterford, MI Department of Public Works

In February of this year, we looked at a pioneering MQTT Sparkplug project at the Department of Public Works in Waterford Township, Michigan. Thanks to the work of system integrator Perceptive Controls, Waterford’s project went on to become a finalist in the 2021 Ignition Community Conference Discover Gallery and was showcased in the latest Sparkplug Working Group virtual meetup. Recently, I followed up with Kevin Finkler, software engineer at Perceptive, to talk about using MQTT in the real world.

Aging infrastructure

You might remember that what started as scheduled maintenance in Waterford turned into a much more ambitious overhaul. Frank Fisher, engineering superintendent, and Russell Williams, director of public works, became excited by the idea that MQTT could help them eliminate long-standing systemic limitations.

With over 90 controllers in their network, the serial polling mechanism they used at the time, combined with the limited bandwidth of their radio network, meant that data from each site would update only every 3-5 minutes. Sometimes a lift station would run briefly in between polling cycles, creating gaps in their reporting and inhibiting operators’ ability to accurately detect issues until alarms eventually made their way through. And for each I/O point they added to the system, this latency only grew worse.

It seemed clear that MQTT’s report-by-exception behavior could both reduce bandwidth usage and ensure delivery of important system actions.

A close-up of Waterford’s new system overview screen in Ignition.

Wrapping your head around MQTT

But jumping into an MQTT project for the first time came with a learning curve.

“This was the first time I had done something that wasn’t strict client-server,” said Kevin Finkler. “The topic system and how you can subscribe to a particular topic is pretty different…. When you first jump into MQTT you understand that clients connect to brokers, but how do you actually send data? … You can browse through the broker and see it there, but understanding how it’s functioning is hard.”

MQTT’s publish-subscribe communication is definitely a departure from the way traditional industrial protocols behave:

1. Each field device connects only to the central server (broker), regardless of where its data needs to go.
2. The broker does not actually request data from any device. Instead, field devices are responsible for sending data to the broker on their own.
3. And report-by-exception means that devices send data only on change, so there is no need for a “send data” command.

Fortunately, both the Ignition SCADA system and groov EPIC provide robust support for MQTT and Sparkplug, so even though it hurt Kevin’s head a bit at first, actually establishing communication was straightforward.

“It kind of happens automagically,” he said. “You basically define a few parameters [in Ignition] to set up the broker. And each of the EPIC devices was pretty simple. You just point it at the broker and it starts sending tags.”

“I love that both of these sides have embraced MQTT,” added Waterford’s Frank Fisher. “It makes the connection seamless.”

With Sparkplug, Waterford added tens of thousands of tags to their system with almost no configuration.

Building defense in depth

With that done, Kevin and Frank built out the mechanisms to secure the new infrastructure.

Waterford has complete control over the AWS server that hosts Ignition, and Frank configured the firewall on that end to permit traffic only from his groov EPIC controllers and specific Ignition clients in Waterford’s and Perceptive’s offices.

Then Frank installed a client SSL certificate on each EPIC so Ignition could authenticate and encrypt the connection, protecting against man-in-the-middle attacks that could expose data or permit unauthorized control.

Of course, every authorized user is required to create strong passwords to access any groov EPIC controller or Ignition client in the system, but over and above this, every user login is tracked and reported throughout the system.

They’ve even integrated physical site security into Ignition. Each lift station is secured with an outer door under lock and key, and a physical switch on the door is connected to the local EPIC. Ignition monitors the switch state to detect when someone enters, and if a user login is not registered within a specific time with access privileges for that specific room, then Ignition generates a global alarm.

A close-up of one of Waterford’s new lift station control screens. The station security panel is shown at center-left.

Return on investment

Since February, Waterford has completed upgrades on all 63 of its sewage lift stations and 6 of its 12 clean water sites. The new groov EPIC-Ignition MQTT infrastructure has reduced field updates from multi-minute cycles to sub-second event-driven publications, with a significant reduction in actual data transmission because of report-by-exception.

In Kevin’s opinion, “Ultra-low latency is probably the biggest benefit. The latency between the controller and the Ignition gateway is less than 200 ms. That’s across the cellular network with all of [the EPICs] communicating to a server in the cloud.” And in fact, for most sites, it’s closer to 50 ms.

That kind of efficiency means Waterford never misses a system action or an alarm notification anymore, and they can publish more data than before. Now, they also have access to communications and controller diagnostics, like update latency, connection timestamps, message size, firmware version, and more, which simply wasn’t possible before.

Waterford’s new infrastructure increased the speed of data updates from minutes to less than a second.

Ignition takes advantage of all this data with a more user-friendly look and feel, highlighting critical elements like wetwell level, run time, and pump flow totals in each lift station, so operators can quickly spot problem indicators. Some of this data existed before, but it often wasn't accurate because of the inherent update lag.

Importantly, with a huge increase in bandwidth and the low administrative requirements of Sparkplug, Waterford can keep adding data to their system for a very long time. Each new endpoint only needs a connection to the MQTT broker in Ignition in order to produce or consume data for/from the whole system.

Their cloud-based infrastructure also enables greater flexibility and reliability. They are able to publish controller updates over the air, which has reduced travel time and allowed them to continue project development unabated throughout the pandemic. In fact, a recent internet outage at the Department of Public Works offices provided an unexpected test of their new system, which kept on working without interruption.

“We only lost the old system,” said Frank Fisher. “Our internal stuff couldn't reach out, of course, but our iPads could connect through Verizon... and I was able to get back in touch. In a situation like this, the old system couldn't send out alarms because it depended on a local connection. The new system didn't even notice or care because it's not running anything local.”

One of Waterford's new water tank control screens.

More to come...

It’s been great to follow along with Waterford’s progress this year, and we hope to see other teams following their lead.

When asked what he thought other engineers needed to understand most about MQTT, Kevin Finkler pointed out that “the client-server communication is essentially handled for them. Before there was [a lot of code] that handled all the communication. It was a lot of work to maintain. Now you just mark a tag as public [in groov EPIC] and that’s all handled for you. It’s less work, so it’s less money spent on engineering time.” And less money spent on engineering time means teams can tackle bigger challenges, moving critical infrastructure closer to a true digital transformation.

In his presentation to the Sparkplug Working Group, Frank said, “We are still trying to figure out what else we can do with this. We have a lot of other instrumentation that we want to be able to pull data from out in the field that wasn’t really feasible before… not just at our lift stations and our treatment plants but throughout the organization. Where can we use it with flowmeters? Where can we use it throughout all of our assets to give us a better overview? We’re just beginning that journey.”

___

Thanks to Kevin and Frank for sharing their story with us. To learn more, visit www.perceptivecontrols.com.

Read the complete Waterford/Perceptive Controls case study here.