Editor's Note: This essay was originally published on LinkedIn in April 2016.

When I wrote this, cloud computing, virtualization, hyper-convergence, and software-defined infrastructure were reshaping enterprise IT. Looking back today, I realize the deeper question wasn’t about infrastructure at all. It was about complexity. Every new technological wave promised simplification, yet often shifted complexity somewhere else. That question continues to influence how I think about enterprise systems today. I’ve lightly edited this essay for clarity and readability while preserving the original ideas.

In the beginning…

There were JBODs and tower servers.

Actually, let’s take a simpler approach.

I want to break this discussion into a few sections because the evolution of the modern data center isn’t simply a story of technology. It’s a story of changing economics, changing responsibilities, and changing complexity.

The Evolution of the Data Center

I entered the infrastructure software industry in 2004.

One of my earliest memories is walking into a laboratory filled with rows of tower servers connected to JBODs (Just a Bunch of Disks). You could even share a JBOD between two servers.

At the time, zoning storage between servers felt like cutting-edge technology.

The infrastructure landscape was constantly evolving.

Not just between generations of hardware, but between software and hardware, and more importantly, between Capital Expenditure (CapEx) and Operational Expenditure (OpEx).

Every few years the industry reinvented itself.

JBODs gave way to intelligent storage arrays.

Virtualization followed.

Cloud computing emerged.

Software-Defined Everything (or SDx, as I like to call it) became the new industry mantra.

Storage.

Networking.

Compute.

Almost everything eventually became software defined.

Each wave promised a simpler future.

Each wave also introduced a different layer of complexity.

The Pendulum Swings

Originally, mission-critical applications relied heavily on host-based software.

High Availability.

Disaster Recovery.

Storage Management.

Everything lived inside the server.

Then the industry asked an important question:

Why should expensive compute resources spend time running infrastructure software instead of business applications?

The answer was simple.

Move those services to the storage layer.

Applications should run applications.

Storage should manage storage.

Infrastructure responsibilities migrated away from servers.

A few years later another question emerged.

Why dedicate physical servers to individual workloads?

Virtualization became the answer.

Instead of dozens of underutilized servers, organizations could consolidate workloads onto fewer, larger systems.

Resource utilization improved dramatically.

But something else happened.

Infrastructure responsibilities didn’t disappear.

They multiplied.

Applications now depended on:

• Physical infrastructure
• Virtual infrastructure
• Storage infrastructure

Each layer introduced another management plane.

Another operational team.

Another potential point of failure.

Cloud computing accelerated this trend even further.

Everything suddenly seemed to belong in “the cloud.”

Yet physical infrastructure never truly disappeared.

Someone still owned it.

Someone still operated it.

Someone still paid for it.

Cloud computing also changed the people who managed enterprise infrastructure.

Years ago, organizations largely depended on System Administrators.

Today, we have specialists for almost everything.

• Data Center Administrators
• Virtualization Administrators
• Disaster Recovery Specialists
• Cloud Engineers
• Platform Engineers
• Application Administrators

Each role brought deeper expertise.

It also introduced greater coordination.

When everything works, specialization is powerful.

When something breaks, complexity often becomes the biggest challenge.

Then Came Microservices

While infrastructure continued to evolve, software architecture was changing just as rapidly.

We moved from monolithic applications to Service-Oriented Architecture (SOA).

Eventually, microservices became the preferred architectural model.

Ironically, from my perspective at the time, microservices almost made virtualization feel like a temporary stop in the industry’s evolution.

Containers offered many of the benefits organizations wanted from virtualization while consuming fewer resources and allowing applications to be deployed far more efficiently.

Technology, much like fashion, often moves in cycles.

Bell-bottoms disappear.

Then they return with a new name.

Infrastructure evolution sometimes feels remarkably similar.

Today we talk about:

• Virtual Machines
• Containers
• Microservices
• Cloud
• Colocation
• Hybrid Cloud
• Hyper-Converged Infrastructure

Each innovation solves a genuine problem.

Each also introduces a new layer that must eventually be managed.

So… Did We Actually Reduce Costs?

Let’s go back to where we started.

Originally, organizations ran mission-critical applications on physical servers supported by host-based software for High Availability, Disaster Recovery, and Storage Management.

Then we decided infrastructure software should move into storage appliances.

Compute became cheaper.

Storage became more sophisticated and significantly more expensive.

The cost shifted.

Not necessarily down.

Just somewhere else.

Virtualization followed.

Organizations purchased larger, more powerful servers capable of hosting hundreds of virtual machines.

At the same time they invested in:

• Virtualization platforms
• Specialist administrators
• Training
• New operational processes

Infrastructure became more efficient.

Operations became more complicated.

Then cloud arrived.

Infrastructure spending shifted again.

Some capital expenditure became operational expenditure.

Hardware ownership decreased.

Cloud subscriptions increased.

The accounting changed.

The complexity didn’t necessarily disappear.

The Real Question

For years the industry has celebrated each technological wave as the answer to reducing Total Cost of Ownership (TCO).

I’m not entirely convinced.

Have we truly reduced the cost of operating enterprise infrastructure?

Or have we simply redistributed that cost across different technologies, vendors, contracts, and specialized teams?

Perhaps the more important question is this:

Has technology simplified infrastructure, or has it simply moved complexity somewhere else?

I suspect the answer is often the latter.

Looking Beyond Products

This isn’t a criticism of innovation.

Innovation is essential.

But innovation alone does not reduce complexity.

Reducing complexity requires coordination.

It requires interoperability.

It requires organizations to stop optimizing individual products and start optimizing complete solutions.

That means:

• Building open ecosystems rather than isolated products.
• Creating partnerships instead of competing in every layer.
• Designing platforms instead of disconnected point solutions.
• Solving customer problems rather than showcasing technology.

Over time, I have become convinced that long-term competition will not be between individual vendors.

It will be between ecosystems.

Between coalitions of companies capable of delivering integrated solutions that genuinely reduce operational complexity.

Technology evolves quickly.

Complexity evolves with it.

The real winners will be those who reduce both.

Originally published on LinkedIn in April 2016. Republished on abhisekchaudhuri.com as part of the Collected Essays.