What happens when your infrastructure configuration management system becomes a bottleneck? When deploying a single continuous integration (CI) server takes more than an hour and only one team member (barely) understands how it works, it’s time for a change.

This blog post explores our journey from Chef to Ansible for managing CI build servers — critical infrastructure supporting daily developer operations. It covers how we improved team maintainability, cut our configuration codebase by more than 99 percent, greatly improved scaling capabilities, and transformed our deployment process.

Why now?

Ever since I joined Nutrient in late 2023, I wanted to modernize our CI build servers (running as self-hosted Buildkite agents(opens in a new tab)) and the Chef cookbooks behind them responsible for their configuration. Working with them daily was challenging; dead code, minimal documentation, and my own limited Ruby and Chef expertise made maintenance and any new development difficult.

To be frank, we let our Chef ecosystem rot. Documentation was minimal or non-existent, key team members who understood the system had left, and previous upgrade attempts had failed. The infrastructure worked, but it had become a black box we could barely maintain and that we found impossible to develop new features with.

And while our fleet of bare-metal Linux servers on Debian 10 Buster from Hetzner Dedicated Server was functioning adequately, with Debian 10 reaching end-of-life (EOL) on 30 June 2024, we faced growing security and compliance risks — making the impending deadline a clear catalyst for change.

This deadline created the perfect opportunity to reassess our configuration management strategy and transition away from Chef, which had become a maintenance bottleneck.

Core issues identified

We identified several critical problems that made migration necessary:

• Security risk — Running an unsupported OS created compliance issues and security vulnerabilities.
• Knowledge gap — Chef became a black box for our Platform team, with failed previous upgrade attempts.
• Complexity — Critical environments and variables were intertwined in unknown ways, with nested cookbooks referencing decommissioned components written more than a decade ago.
• Outdated technology — Chef lagged behind modern DevOps tools, while Ansible offered greater flexibility and ease of use.
• Scaling difficulties — Our Chef setup required significant time to deploy individual servers or make configuration changes.

Chef vs. Ansible: A practical comparison

When evaluating configuration management tools, we compared Chef and Ansible across several dimensions relevant to our infrastructure needs.

Why move away from Chef?

Chef was an industry standard for configuration management, but our experience revealed significant limitations beyond the core issues we faced:

• High learning curve — Requires Ruby knowledge and understanding of Chef’s architecture, which the current Platform team does not have
• Infrastructure overhead — Maintaining the Chef server, database, and associated toolset created ongoing costs and complexity
• Declining ecosystem — Community and industry support declined relative to modern tools like Ansible
• Technical debt — The time required to refactor and document our legacy cookbooks would be enormous

These factors, combined with our immediate challenges, made migrating to a more modern, approachable configuration management tool a faster and safer long-term strategy than maintaining our Chef infrastructure.

Requirements for CI server migration

We established clear criteria for our migration, outlined below, to ensure success.

Must-have requirements

• Minimal disruption — Existing pipelines remain functional, with developer teams unable to notice any transition
• Current OS — Stay up to date to meet security practices and compliance requirements
• Team maintainability — New setup must be understood and managed by current team members
• Clear documentation — Eliminate undocumented knowledge with easy-to-follow runbooks
• Bare-metal servers — Support specialized workloads like Android emulation requiring low-level hardware access
• Rapid, automated deployment — Reduce time to deploy new servers and eliminate as many manual steps as possible

Nice-to-have features

• Single infrastructure provider — Simplified management with consolidated hosting
• Vendor-neutral tooling — Avoid lock-in with CloudFormation or ARM templates to enable provider flexibility
• Cost optimization — Maximize value and minimize infrastructure expenses
• Elastic scaling — Support easy scaling up or down as requirements change

Solution options explored

We evaluated multiple approaches to find the best fit for our requirements.

Configuration management tools

Packer — Excellent for image-based scaling in cloud environments like AWS EC2, but not suitable for our bare-metal requirements.

Ansible — Simple, readable playbooks with no dedicated server or database requirements. Offered better flexibility for our specific infrastructure needs.

Hosting considerations

We then compared three hosting options for our specialized requirements.

Cost comparison analysis

*Pricing at time of evaluation in USD, hosted in European data centers. AWS costs exclude EBS storage, data transfer, and other fees.

Note: For bare-metal comparisons, Hetzner Dedicated Server (AX52) and AWS EC2 (c5.metal) represent the closest comparable server specifications available from each provider.

Each hosting option has tradeoffs:

• AWS offers comprehensive auto-scaling and managed services albeit at significantly higher costs. Scaling and replacing individual servers is automatic and painless.
• Hetzner Cloud provides good flexibility and IaC support but has vertical scalability constraints.
• Hetzner Dedicated Server offers great value for bare-metal servers, but lacks any autoscaling capabilities and requires fairly meticulous scripts to configure. Contacting Hetzner support is often required to debug any hardware issues.

Decision: Hetzner Dedicated Server offered the best value for our bare-metal requirements, aligning with our existing infrastructure expertise. The cost savings and proven reliability outweighed the lack of autoscaling capabilities.

The path forward: Ansible on Hetzner Dedicated Server

We chose to modernize our configuration management with Ansible while maintaining our proven Hetzner Dedicated Server infrastructure. Our strategy focused on three key principles: replacing Chef cookbooks with Ansible playbooks, executing a gradual transition to minimize risk, and documenting processes for team maintainability.

Implementation plan

Once we decided what to do, we structured our migration as a four-phase process to minimize risk and ensure success.

Phase 1: Proof-of-concept development

Develop initial Ansible playbooks to configure Linux servers with essential components:

• Buildkite Agent — Complete installation and configuration setup
• Android CI support — Testing capabilities for specialized build requirements
• Essential tooling — Docker, Git, Vim, curl, TLS certificates, and development dependencies
• Authentication systems — Repository access and HyperDX Agent integration for observability
• Hetzner-specific features — Rescue mode access and disk encryption configuration

Phase 2: Validation and testing

Comprehensive testing to ensure reliability before production deployment:

• Pipeline validation — Execute major pipelines, including monorepo and website builds, on test nodes
• Health monitoring — Verify server status through Buildkite, SSH access, and essential mount points for disk drives
• Performance benchmarking — Compare deployment times and resource utilization against Chef baseline

Phase 3: Gradual production rollout

Risk-minimized migration strategy executed over a two-week period:

• Sequential migration — Take agents offline individually, remove from Chef state, add to Ansible inventory
• Continuous monitoring — Apply configurations, monitor stability, and iterate based on findings
• Rollback preparation — Maintain Chef configurations as backup during transition period

Phase 4: Infrastructure cleanup

Finalize migration and establish sustainable practices:

• Legacy removal — Eliminate obsolete Chef artifacts, deprecated automation, and outdated runbooks
• Documentation creation — Develop comprehensive playbook documentation and maintainable runbooks
• Knowledge transfer — Train team members on new Ansible workflows and troubleshooting procedures

Migration results and lessons learned

Our Chef-to-Ansible migration delivered significant improvements across multiple dimensions.

Quantifiable improvements

The migration delivered measurable results that transformed our infrastructure operations:

Key takeaways for infrastructure modernization

Our migration taught us several valuable lessons:

• Tool selection isn’t everything — Building systems your team understands and can maintain matters more than choosing the “best” tool
• Documentation is critical — Clear documentation and maintainability equal raw technical capabilities in importance
• Alignment matters — Choose configuration management and hosting solutions that match your team’s skills, operational needs, and budget constraints
• Iterative approach reduces risk — Well-documented, phased migrations provide better foundations for scalability and resilience
• Deadlines can be catalysts — The Debian 10 EOL deadline forced us to address technical debt we might have otherwise deferred

What’s next?

Our migration to Ansible represents more than a tool change — it’s a shift toward simplicity and maintainability over legacy complexity. This foundation enables:

• Improved deployment speed — Deploy new CI agents in minutes, not hours
• Improved reliability — Reduced single points of failure in our infrastructure
• Enhanced collaboration — Multiple team members can contribute to infrastructure management
• Future flexibility — Vendor-neutral approach enables easier provider migrations if needed

By prioritizing team understanding and operational simplicity, we’ve established a sustainable platform for our growing development needs.

If you want to learn more about our historical approach to CI, be sure to check out these posts:

• Continuous integration for small iOS/macOS teams
• Chef on macOS
• Managing macOS hardware: Virtualization or bare metal?
• Automating macOS provisioning with Chef
• Continuous integration: From Jenkins to Buildkite
• Managing CI pipelines with Terraform
• Deflaking CI tests with xcresults

Explore related topics