New plant uses unique edge architecture to run at the pace of modern business

With demand for hand sanitizer more than doubling across the US, and continuing annual growth of more than 20% expected in coming years, 2020 presented the cleaning industry with a big financial opportunity. Emerald 66 Enterprises (E66) mobilized resources to meet this need, setting up shop in an empty denim processing plant in Seminole, Oklahoma, U.S.A.

In only three months, E66 had automated packaging lines producing up to 1 million bottles of hand sanitizer a week in a cGMP-compliant facility, and they continue to expand their core process capabilities at a rapid clip.

“When we say we do stuff quick,” says E66 chief of operations, Robert Bodnar, “we're talking days, not weeks or months.”

In this blog series, we’ll follow E66’s growth, examining the technologies and techniques they and long-time OptoPartner Northeast Automation Company (NACI) used to achieve competitive advantage in a challenging market environment.

Getting down to business

When E66 hired NACI to automate their bottling and packaging process, they understood they needed to move fast. They were competing against low-paid, high-volume workforces operating manually and believed they could use technology to do more with a smaller, better-paid workforce.

The quickest way to build a new packaging process was by acquiring a variety of equipment at auction, leaving NACI the task of integrating it all into a cohesive system. Tom Coombs, principal engineer at NACI, decided to make Opto 22’s groov devices the focus of their strategy so they could use edge computing to build an information management system as they scaled up.

“Each piece of equipment needs to be intelligent,” Tom explains. “We’re going to make every conveyor and every device smart.” It’s a bold aspiration, for sure, but buying secondhand equipment meant that quality varied widely. NACI had to get creative in order to design a cohesive system at the speed that Emerald 66 needed.

Layered distributed control

To address the different circumstances his team needed to deal with, Tom proposed a unique architecture that enabled separate control systems to function together and also laid a foundation for E66’s data acquisition goals.

Partial Emerald 66 architecture

At the top level, Tom used a groov EPIC edge programmable industrial controller to supervise the process lines and connect disparate devices through REST APIs.

Equipment that arrived with defunct control systems was integrated into this network as remote I/O through SNAP-PAC-R1 controllers, which also provided specialty I/O options not yet available on EPIC.

NACI then used groov RIO edge I/O modules to loosely integrate any functional control systems. As each new unit came online, NACI dropped in one of these PoE-powered modules and used its software-configurable channels to identify the kinds of signals the unit provided. NACI mirrored these I/O signals in parallel with the existing PLC I/O connections and integrated them into the groov EPIC network.

Tom notes, “The ease with which you can do this, you know, you’re talking about a half-hour of wiring. Your biggest problem is finding the documentation from the original manufacturer.”

While these modules appear as remote I/O in the groov EPIC’s control strategy, Tom also engineered an additional layer of control independent of EPIC by allowing each groov RIO to execute limited local control through Node-RED. Living up to their ambition to “make.. every device smart,” NACI added motors, photo eyes, load cells, and other instrumentation to many pieces of semi-automated and dumb equipment, then sprinkled in a little logic to make them work together and report process data up to the coordinating level.

NACI manufactured several of these VFD control panels for local conveyor control using groov RIO

A veteran engineer himself, COO Robert Bodnar, has been involved in much of this design. He explains, “The top-level process [in EPIC] is turning on two lines or three lines. If you have a line coming in and you have a line going out, they may not be running at the same speed… You’ve got cappers that are running at different speeds. You've got label printers and all these different components all kind of running at different speeds, so it’s kinda neat to be able to say, okay, what if we use the groov RIOs to control just the lines and the belt and case packers and things like that…? We’re doing just little simple stuff: turning on and off motors to run a line and trying to match up to how fast a filling machine is spinning. They're kind of little islands of point automation.”

This loosely coupled, distributed architecture allowed NACI to assemble their production line without modifying any of the existing control systems that came as part of their purchased equipment, which saved them loads of development time. In three months, Tom and his team had 15 pieces of equipment up and running, but this wouldn’t be their last challenge.

A rotary filler/capper unit at Emerald 66

The big pivot

Emerald 66’s end-to-end process includes conveyance, filling, capping, printing, labeling, packing, and palletizing operations. Because they had modeled their business around packaging and distribution for a single large purchaser, this process was originally designed to maximize throughput. But the situation changed significantly when that customer suffered a financial setback and had to close production. Then, the whole business had to pivot to allow E66 to become a multi-product facility.

Automation grew from processing a high volume of one-gallon, single-formulation containers to working with a variety of sanitizer chemistries in different batch sizes and packaging form factors: from small two-, four-, six-, and eight-ounce containers, hand pumps, and spray bottles, to large jugs in excess of one gallon.

Emerald 66’s operations grew to accommodate many packaging sizes and product formulations

Fortunately, NACI’s decision to use a loosely coupled production line made it easy to modify individual segments without interrupting their data collection and process integration. In combination with on-site panel building and 3D printing capabilities, their investment in edge-oriented automation allowed them to retool very quickly. Emerald 66 was able to break even on their initial investment within six months.

To that, Mr. Bodnar says, “The groov RIO / PAC control / groov EPIC / business system stack, as you move up, lets you move things around and be pretty modular and respond to business… There are a lot of good things that you can do in the industrial world that people might [comment on], you know, ‘You’ve got good quality, you've got good throughput,’ and whatever, but they usually don't say that you're nimble. It’s not usually the term people think of... But it is fun to be nimble, and on the industrial side, I think that’s one of the things [Opto 22] has really helped us with.”

NACI’s Tom Coombs adds, “The fluidity and dynamics of modern manufacturing requires extremely fast response to changing market demands... With groov EPIC and groov RIO, Opto 22 puts dynamic manufacturing data at the edge of the production line and into enterprise systems simultaneously in real-time.”

More to come

As demand grows, Emerald 66 continues to diversify its business and expand its production capabilities. Recently, they have spun up a lucrative technology division that is using groov devices to develop filling equipment for overseas export. Their automation is also adapting to accommodate new functions like container molding, product blending, and quality control procedures. And as the process expands, so do E66’s ambitions for their data infrastructure and information management systems. We’ll be looking at all of this in the future as this series continues.

For now, we’d like to give a big thank you to Emerald 66 Enterprises and Northeast Automation Company, Inc. for allowing us to share their work with the community. Reach out to Tom and Robert, respectively, at tdcoombs@gmail.com and robert@emerald-66.com.