At its heart, MQTT was designed for unreliable networks. Back in the ’90s, ConocoPhillips (now Phillips 66) was looking for a way to improve telemetry reporting over its low-bandwidth dial-up and costly VSAT (small satellite dish) SCADA network. IBM partnered with system integrator Arcom Control Systems (now Cirrus Link Solutions) to develop a minimalist communication protocol that could gracefully handle intermittent network outages and high latency among many distributed devices over TCP/IP. 

Jumping ahead 20 years, MQTT had caught on in the IT world as a flexible, efficient transport for IoT data, but these applications didn’t necessarily take advantage of MQTT’s rigorous state management and data integrity features. Cirrus Link knew that for MQTT to thrive in an industrial environment, reliability was critical, and it developed the Sparkplug B specification, in part, to address that need.

If you’re just starting this series, I suggest you go back to parts 1 and 2 and familiarize yourself with groov EPIC’s MQTT clients and the advantages of the Sparkplug B spec. In this post, I will go deeper into the security, history, and failover features that the groov Manage and Ignition Edge clients provide and how they work with Sparkplug B payloads to make MQTT networks even more resilient.

Bonus: The groov Manage client on groov RIO provides all the same MQTT functions described in this series. Read on, groov RIO users!

Security

Cyber threats pose a risk to otherwise stable systems, but fortunately, groov EPIC and MQTT make it easier to protect your data.

Rather than complicating the protocol with its own security mechanisms, MQTT takes advantage of the SSL/TLS encryption built into the TCP/IP stack. Assuming your broker supports it, you can enable encrypted communications in either groov Manage or Ignition Edge by first replacing the protocol and port designations in your broker address. For my broker, rather than using 

tcp://mqtt.groov.com:1883

I would use 

ssl://mqtt.groov.com:8883

In groov Manage, under the MQTT > Configuration > Broker sub-menu, you would also enable the “SSL” option and select the appropriate “CA certificate.” Here is what those changes look like:

If you haven’t dealt with certificates of trust before, they’re an important part in generating encryption keys and in verifying that your EPIC is communicating with the right endpoint. The certificate is created by the MQTT server and distributed by the network administrator for use on trusted edge devices, like EPIC. For more information, check out this post from our series on groov EPIC security.

To upload a security certificate to your groov EPIC, go to the Security > Client SSL menu. Once done, the certificate file appears as an option in the MQTT Broker > Select Certificate dialog box. The groov EPIC User’s Guide has more information on certificate creation and management.

If you’re using groov EPIC’s Ignition Edge MQTT client, here’s what those same changes would look like in the MQTT Transmission module settings:

If you’re concerned about how SSL/TLS affects MQTT’s performance, take a look at this benchmark report from Hive MQ. It concludes that while encryption impacts performance on connection, it makes little difference during sustained use.

Failover connections

Another basic fault tolerance measure is to set up multiple MQTT brokers and configure your groov EPIC or RIO with connections to each.

To set this up in groov Manage, navigate to MQTT > Configuration > Add Broker, and create as many connections as you want.

In the image above, along with my connection to Opto 22’s on-site Mosquitto broker, I’ve added a second connection to our Chariot broker hosted on AWS.

If you’re using the Ignition Edge MQTT client, setting up redundancy is just as straightforward as with groov Manage. Add new server connections from the MQTT Transmission > Settings > Servers tab. Any broker connections that share the same “Server Set” property become part of an automatic failover group.

Ignition Edge allows you to create additional failover groups under the “Sets” tab. For advanced scenarios where you might group MQTT clients around specific brokers, you can create additional MQTT clients, each with its own server set and history store, by defining “Custom Transmitter Settings” under the “Transmitters” tab.

Primary Host

If you’ve configured multiple broker connections for failover in groov Manage or Ignition Edge, you can also set a Primary Host ID. Normally, when using failover connections, an MQTT client will connect to the first available broker and switch round-robin if it loses connection. In the case that you’re sending data to a back-end client like a SCADA, historian, or IoT system, an SpB-compatible client can improve performance by allowing you to designate that system as your primary application.

Then, if either groov EPIC or the back-end client loses its connection to the current broker, EPIC will rotate through available connections until it finds the back-end client again. However, be aware that MQTT will prefer a connection to the back-end client over a connection to the broker. If none of the available broker connections are publishing an ONLINE status for the primary host, the MQTT client will not stay connected until it finds one that does.

To designate a primary host application in groov Manage, go to the main MQTT configuration screen and set the “Primary Host ID” to the MQTT client ID of your target application.

For example, the Ignition SCADA server running on my laptop (not Ignition Edge, but full Ignition), has a client ID of laptop-ignition-scada, so I would enter that as the “Primary Host ID” in groov Manage:

In Ignition Edge, a redundant broker failover group, called a Server Set, can designate a “Primary Host ID” as well. This is found under the “Sets” tab.

History

In the event of a disconnection, plain-vanilla MQTT offers quality-of-service levels that can guarantee message delivery at the expense of the additional traffic used for handshaking. Typically, however, only the most recent message is delivered on reconnect. Sparkplug B provides a different mechanism that minimizes data loss.

Since Sparkplug B decorates MQTT messages with metadata, SpB clients are able to flag certain messages as historical records. With a primary host ID set, groov Manage can store MQTT records temporarily in RAM when EPIC loses connection to the primary application, and then forward these records when the connection is restored.

In groov Manage you’ll enable this feature in the MQTT > Configuration > Device sub-menu by plugging a number into the “Historic Queue” field. We recommend a queue size of 3,600 to strike a balance between storage and performance, but you could go much higher depending on the rate of data production:

Store-and-forward is even better with the Ignition Edge client, which offers larger capacity and a non-volatile storage option. If you select the disk-backed option, you get the additional benefit of groov EPIC’s power-fail-safe file system and industrial SSD.

To enable historization in Ignition Edge, go to MQTT Transmission > History and edit the “Default In-Memory Store.” Select “Enable this History Store” and change “Type” to “Disk-Backed.” (I’m also going to change the name of the history store to reflect these changes.)

You can also adjust the history capacity here if you want. The default setting will buffer up to 100,000 records for 1 week. Wow.

Once that part is done, go back to the MQTT Transmission > Settings menu and select the “Transmitters” tab. Scroll down and set “History Store” to the name of the store you just enabled (“Default Store” in my case).

Note: To get the full advantage of store-and-forward history, you’ll need to be communicating with an in-network MQTT subscriber that is watching for these historical records. Currently, the Ignition MQTT Engine module and the Canary Labs MQTT Sparkplug B Data Collector both do. 

As the popularity of MQTT continues to grow, we expect to see support for this and other creative uses of Sparkplug B added to other SCADA packages and historians as well.

Trade-offs and considerations

Sparkplug B provides an efficient standard for reliable, interoperable industrial communications. groov EPIC and groov RIO provide a secure OT platform for building fault-tolerant MQTT Sparkplug B data systems. At no additional cost, groov Manage offers essential features for security, store-and-forward historization, and failover. Free to try, and only a little more to license, Ignition Edge takes these features up a notch with disk-backed storage and more options for scaling up your infrastructure.

As you consider which option is right for you, remember that groov Manage is better suited for tag counts under 1000, so if you have a larger system, Ignition Edge will be the best option. With groov EPIC, you can take advantage of either option to design systems that deliver high uptime, data integrity, and resistance to cyberattack.

* GROOV-LIC-EDGE current list pricing as of 6/10/20; ** special pricing when purchased with Ignition Edge License

Next time we bring it all together…

Now that we have a secure, reliable foundation for data sharing, we can look at the big-picture architecture that lets MQTT networks scale to industrial levels, including legacy system integration. For a sneak preview, check out our recent webinar with Inductive Automation and Cirrus Link: Ignition Community Live with Cirrus Link: MQTT Workshop.

All posts in this series

Part 1 - MQTT: Basic publishing with groov EPIC
Part 2 - MQTT: Faster, better with Sparkplug B
Part 3 - MQTT: Mission-critical fault tolerance
Part 4 - MQTT: Bringing it all together