Every Serverspec suite I inherit looks like a different language

I've been using Serverspec for years and I still reach for it when I want to verify a host actually looks the way I asked it to. The DSL is great. The way it falls apart over time is not.

The pattern goes like this. The suite starts small. Someone wants to share checks across roles, so a helper module appears. Someone else needs a custom matcher, then a define_method to generate similar matchers. The team decides the inventory should live "in the specs themselves" so a case node_role when ... shows up. A year later spec_helper.rb is hundreds lines, has its own unit tests, and a single it { ... } line means tracing two helpers and a metaprogrammed method to figure out what is actually being asserted.

Nothing in there is malicious. The problem is structural: Serverspec specs are Ruby files. describe and it are just methods. Once you've put a DSL on top of a general-purpose language, declarative checks and arbitrary procedural code become syntactically indistinguishable, and the procedural side compounds faster on a long timeline. After a couple of years the suite is the original author's private dialect, and nobody else fully knows what it covers.

I built PanInfraSpec to keep my hands off that loop. It's agenerator that takes Dhall inputs and emits Serverspec *_spec.rb files. The Ruby still does the actual checking, but I no longer write it. There is no spec_helper.rb to grow custom matchers in, because spec_helper.rb is generated and I don't edit it.

Ruby won't even tell you when the spec is wrong

A related, smaller failure mode. Serverspec's resource API is dynamically dispatched, which means this is a perfectly valid Ruby file:

describe service('nginx') do
  it { should be_runnning }   # spot the typo
end

Three n's. RSpec doesn't care — be_runnning resolves through method_missing into something that doesn't fail loudly enough, the test runs, and the suite stays green. Same story for asserting a running state on a package resource (packages don't have one).

Same root cause: a Ruby DSL on top of Ruby has no opinion about which calls were meant to be assertions. The spec-helper sprawl is the visible symptom; this is the quiet one that ships green tests for things that aren't actually running.

The Dhall pitch in 30 seconds

If you haven't touched Dhall: imagine JSON with types and functions, but no I/O and no unbounded loops. Every program terminates, every program type-checks before it runs. You can dhall freeze imports to pin them by SHA-256. There's no eval, no exceptions, no way to accidentally hit the network from a config file.

Things you'd normally bolt onto YAML — Helm templates, JSON Schema validation, Jinja2 — are just language features here. So if your problem is "I want a typed, reusable description of my infrastructure", Dhall fits without dragging Ruby (or Python, or Go templates) along for the ride.

What the tool does

PanInfraSpec takes two Dhall files in:

• an inventory: [{ hostname, ip, role, tags, customAttributes }, ...]
• a plan: a list of (selector, [assertions]) pairs

…and emits Serverspec Ruby out the other side. You then run bundle exec rake spec like you always did.

The IR is deliberately backend-agnostic — Serverspec is the first emitter, InSpec is the next one. The architecture is borrowed from Pandoc; that's also where the "Pan" comes from.

A five-minute tour

brew install ikaro1192/tap/paninfraspec        # macOS
nix run github:ikaro1192/PanInfraSpec -- --help  # anywhere with Nix

.deb / .rpm / tarball / Docker / cabal build paths in docs/install.md.

Inventory

let I = https://raw.githubusercontent.com/ikaro1192/PanInfraSpec/main/dhall/Inventory.dhall
let none = [] : List I.CustomAttribute

in  [ { hostname = "web01", ip = Some "10.0.1.10", role = "Web",       tags = [ "frontend", "metrics" ], customAttributes = none }
    , { hostname = "db01",  ip = None Text,        role = "DBPrimary", tags = [ "metrics" ],             customAttributes = none }
    ] : List I.Node

If you're on Terraform, --from-terraform-state turns aws_instance resources into this list directly.

Plan

let Spec = https://raw.githubusercontent.com/ikaro1192/PanInfraSpec/main/dhall/Serverspec.dhall
let Plan = https://raw.githubusercontent.com/ikaro1192/PanInfraSpec/main/dhall/Plan.dhall

let nginxSpec =
      [ Spec.package "nginx" Spec.PackageState.Installed
      , Spec.service "nginx" Spec.ServiceState.Running
      , Spec.port 80 (Spec.PortState.WithProtocol "tcp")
      ]

in  Plan.make Spec.targetBackend
      [ Plan.onAll              [ Spec.command "uname -a" (Spec.CommandState.ExitCode 0) ]
      , Plan.forRole "Web"      nginxSpec
      , Plan.forTag  "metrics"  [ Spec.port 9090 Spec.PortState.Listening ]
      ]

There are exactly four selectors: onAll, forRole, forTag, forHost. They stack — a node matching multiple selectors gets the union of their assertions. There's no if, no for, no "matching loop" you have to write.

Generate

paninfraspec-gen --inventory examples/inventory.dhall --plan examples/plan.dhall \
                 --target serverspec --out /tmp/out
cd /tmp/out && bundle exec rake spec

For web01 (Web + metrics), the emitter produces:

require 'spec_helper'

# src: examples/plan.dhall — Spec.package "nginx" Spec.PackageState.Installed
describe package('nginx') do
  it { should be_installed }
end

# src: examples/plan.dhall — Spec.service "nginx" Spec.ServiceState.Running
describe service('nginx') do
  it { should be_running }
end

# src: examples/plan.dhall — Spec.port 80 (Spec.PortState.WithProtocol "tcp")
describe port(80) do
  it { should be_listening.with('tcp') }
end

# src: examples/plan.dhall — Spec.port 9090 Spec.PortState.Listening
describe port(9090) do
  it { should be_listening }
end

Assertions sharing the same (kind, primaryKey) get merged into a single describe block. Each block carries a # src: comment pointing back at the Dhall expression that produced it, so you can navigate from generated Ruby back to the source.

What I actually get for the trouble

There is nowhere for custom Ruby to live. This is the main one. The generated <host>_spec.rb files are a mechanical projection of the plan. They contain no helpers, no matcher definitions, no inventory logic. The spec_helper.rb is also generated, and the only thing it does is flip between the :exec and :ssh backends. There is no place where "we needed something custom for this role" can quietly land.

If a teammate genuinely needs to extend behaviour, they do it as a typed constructor in the IR, or reach for one explicit named escape hatch (more below). Both routes are visible in code review.

The typo from the intro is impossible too. Spec.service "nginx" Spec.PackageState.Installed is a Dhall type error — Spec.service wants a ServiceState. The file won't pass dhall type-check, so it never gets as far as generating Ruby. State values are constructors of closed unions, not strings.

Contradictions are caught at generation time. If two assertions fight — command "uname -a" asserted to exit 0 in one place and 1 in another — paninfraspec-gen exits non-zero in CI rather than producing Ruby that's guaranteed to fail.

The obvious objection to all of this is the case where the expected value depends on the host — innodb_buffer_pool_size should be 70% of physical RAM, that kind of thing. Goss falls back to Go templates for this; classic Serverspec falls back to arbitrary Ruby in let(:something) { ... }. Both quietly reintroduce the procedural-leakage problem. PanInfraSpec has a typed sub-IR for arithmetic instead: facts are declared per host (a shell command whose stdout becomes the value) and referenced by name from typed expression constructors, no Ruby strings involved. For cases the IR hasn't grown a constructor for yet, there's a single named escape hatch where raw Ruby is allowed in exactly one field — visible in code review, can't metastasize the way a spec_helper.rb does.

What you give up

You can't drop into Ruby in the middle of a spec file. If your workflow leans on that — a custom matcher per role, a let that reads /etc/whatever and parses it inline — that goes away. PanInfraSpec is opinionated that the "I'll just write Ruby here" door is exactly the door long-term breakage walks in through.

That's a real tradeoff. It depends on how often the Ruby flexibility is paying off versus how often you're getting bitten by the kind of bug I started this post with.

Wrap

If you've ever inherited a Serverspec suite and spent the first afternoon trying to figure out what the spec_helper.rb was doing, this might be interesting to you. If your team has the discipline to keep that file boring forever without a tool forcing it, you may not need this.

The repo is at ikaro1192/PanInfraSpec, MIT-licensed. There's a working set of examples in examples/ — copy one, point it at a real host, see how it feels.

brew install ikaro1192/tap/paninfraspec
nix run github:ikaro1192/PanInfraSpec -- --help

Issues, stars, and "actually I think you got X wrong" comments all welcome.