AI Can be Misused in DevOps Standardization

Background

Rapid increases in technology tempt us to apply AI wherever possible. Many times, that's a good impulse that can drive innovation. But misunderstanding the AI value proposition or applying it in the wrong way leads to degradation.

When standardizing the deployment pipeline of enterprise software there's a right way and a wrong way for AI to help.

Problem

Enterprises frequently need nearly identical deployments of their software to multiple environments. For example the following software deployments should mirror one another as closely as possible:

1. Separate development, test, and production environments facilitate development while avoiding bugs and regression
2. Geographically diverse deployments provide better service closer to customers
3. Redundant deployments provide better disaster recovery and continuity of operations

But when DevOps engineers each have their own "style" of deployment, enterprises complain about lack of standardization. Each style "works" and would be acceptable if uniformly adopted, but lack of uniformity:

1. Masks bugs in test environments later first exposed to customers in production
2. Varies user experience by geographic location
3. Complicates disaster recovery

That leads some to ask, "How can we use AI to improve this?"

Ineffective (But Tempting) Solution Using AI

There's a way to get this wrong which will lead to less standardization, not more.

Consider an enterprise that provides environment permissions to an AI agent. It then prompts the agent to "deploy the software in the same way across environments." In all likelihood, the following results.

AI does its best. Software is deployed to multiple locations. When variations in environment occur, AI makes reasonable assumptions about minimal changes to make to deployment method. At times, it guesses wrong, and human engineers intervene by either 1) telling the agent what's wrong and prompting it to fix it or 2) just manually fixing it themselves. In such a case, the software is deployed to all environments, it's as standardized as AI can get it, and less human touch required means more speed and less expense. Not bad, it would seem.

But understanding what AI is actually doing reveals what problems we haven't solved with this method. AI is inherently nondeterministic. You prompt it today, you get an answer. You prompt it the same way tomorrow, you get a different answer. And that's a good thing. Several features of AI (introspection, continuous ingest of additional data) mean that tomorrow's answer is likely to be better. But tomorrow's answer isn't standardized. And standardization was the whole problem we were trying to solve.

Using this method, we're stuck in the same boat. All environments now run the best attempt AI could make, and they're still all different.

Effective Solutions Avoiding AI

Fortunately, there are a wide range of technologies that can help us solve this problem:

1. Capture environment design in code (rather than ad hoc configuration) using technologies like Helm, Terraform, or Ansible. Then source control that code so that each change is attributable to the person (or AI agent; see below) who changed it and is easy to roll back to known good state
2. Commit those portions of the environment that software must be aware of to source code control so that each developer has a standardized environment. Use Reproducible Build techniques to reduce variation and cyberattack surface.
3. Use infrastructure automation solutions like Puppet, Chef, or Microsoft Desired State Configuration

A Note on Discipline

Organizations will face friction when deciding not use AI. As AI capabilities expand and vendors increasingly push AI driven solutions, engineers and managers will face increasing pressure to apply AI to everything and criticism for saying some AI solution is misplaced.

Organizations with the discipline to develop trusted human capital and demand data-driven reasons for all solutions (AI or otherwise) will avoid such pitfalls.

Effective Solution Using AI Agent

Of course, that doesn't mean AI has no place.

Put all of your DevOps engineers in the same place and tell them it's their job to come up with a standardized solution for deployment. You don't care who's right--you only care that it works and it's the same.

And then give them tools to work with. They can, for example:

1. Write simple policies they all pledge to follow
2. Recommend jettisoning old policies that don't help (and given good reason, you'll support them even if someone objects)
3. Use non-AI traditional tools like those listed above and/or
4. (Enlist AI agents to write policies and infrastructure as code (IaC) to speed up their standardization work.

Conclusion

Excitement around AI will drive unwary executives to apply it in unwise ways. A true technical grounding in how AI works and what other solutions avail allows executives to realize cost and performance benefits rather than incorporating AI without solving anything.