Clover’s platform handles more than just payments: inventory, employee management, online sales, and customer loyalty programs are all running on a single monolith called the Clover Operating System (COS). Releasing updates to that platform reliably and without disrupting merchants is one of the hardest operational problems a platform team can face.
For a decade, Clover ran HAProxy at the center of its infrastructure. At HAProxyConf, engineers Dilpreet Singh and Anirudh Ramesh explained how they pushed that relationship further with the HAProxy One platform, using HAProxy Enterprise and HAProxy Fusion Control Plane to build a traffic routing model they call “rainbow deployments.”
Watch Clover’s full presentation and read the transcript.
Blue-green deployments provide teams a safe way to release software. You run two environments in parallel and shift traffic from the stable version to the new one when it is ready. If something goes wrong, you can reroute the traffic back.
But Clover’s business demands more than a clean two-environment switch can provide. The company serves a wide range of merchants, including traditional banks and financial institutions that are cautious about version changes (and the risk they can pose to their business). Some customers need to stay pinned to specific versions, while others require hotfixes and experimental features without affecting the broader merchant base.
Blue-green deployments cannot provide that level of control. Once you switch traffic, everyone moves together. What Clover needed was a way to run multiple versions simultaneously and route each customer to the right one.
Instead of two deployment colors, Clover runs three or more at any given time. Blue might carry the current stable release. Green holds the next version. Red runs a version pinned for a specific set of customers, or a build carrying a hotfix.
HAProxy Enterprise sits in the middle of all this, deciding which color handles each request, but how is this accomplished?
The routing uses a weighted map file with number ranges corresponding to different backends. HAProxy Enterprise generates a random number between 1 and 100, looks up its position in the map, and sends the request to the corresponding backend. A split of 1 to 33 might route traffic to green. 34 to 67 to blue. 68 to 100 to red. This would give each color roughly equal traffic, but the team has the freedom to adjust those ranges at any time to shift traffic incrementally toward a new release or to pin a specific customer group to a single backend, entirely.
This approach lets Clover release version changes during normal business hours. Since the new version initially receives a controlled fraction of traffic, the platform team can monitor for problems and expand the rollout gradually. If something breaks, they adjust the map file. No all-hands rollback, no off-hours deployment windows.
In their Hashicorp Nomad cluster, Clover runs three versions of their COS monolith, each represented by a color. Three blue nodes run the blue version of COS, three green nodes run the green version, and three red nodes run the red version. Each node also runs an HAProxy Enterprise instance as a sidecar container.
Traffic from the outside world arrives and is distributed across all nine nodes on port 8080. At that point, HAProxy Enterprise load balancer takes over. The lb-haproxy frontend receives the request, consults the weighted map file, and forwards it to the appropriate color backend. The request then travels to the HAProxy Enterprise instance running on a node of that color, which terminates SSL and passes the decrypted request to COS running locally on port 8020.
HAProxy Enterprise also handles egress from COS. When the monolith needs to talk to a microservice running in Kubernetes, it makes a request to a local port. HAProxy Enterprise picks that up and forwards it to the Kubernetes Istio ingress, which routes it internally. This turns HAProxy Enterprise into a service mesh for COS, without requiring the monolith itself to know where downstream services live.
Getting nine HAProxy Enterprise instances configured consistently and keeping them in sync would be cumbersome to do manually. Clover built a bootstrapping container (a Python script that fires a sequence of REST API calls against HAProxy Fusion Control Plane), to handle the initial setup automatically.
The bootstrapper creates the cluster, configures Consul integration, and registers backend resource templates for each color. HAProxy Fusion then uses Consul service discovery to dynamically populate the backend server pools. When Clover registers a Nomad node in Consul with a blue, green, or red tag, HAProxy Fusion picks it up and adds it to the matching backend. Scaling the cluster up or down does not require manual backend configuration.
Once HAProxy Fusion has the configuration, it pushes it to all nine HAProxy Enterprise instances simultaneously. Changes that previously required touching individual load balancer configs now happen in one place and propagate automatically.
The shift to rainbow deployments changed how Clover thinks about releases. The team can now release during the day because a bad deployment affects only a slice of traffic, not every merchant at once. They can test a new version under real load by giving it 10 or 20 percent of traffic before committing fully. They can pin conservative customers to a stable version indefinitely while still moving the rest of the platform forward.
Singh put it plainly during the talk: zero on-call pages and zero drama. HAProxy has operated as a silent workhorse in Clover's stack for ten years, and the move to HAProxy Fusion extends that reliability into a more complex multi-version deployment model.
The map file and configuration changes were still applied manually through the HAProxy Fusion UI. The team's next goal is to automate that fully through the REST API, so developers can trigger traffic shifts and version pins without touching the HAProxy configuration directly.
Clover also plans to build an application model that abstracts away the deployment target entirely. Application teams would define what they want to deploy, and the platform team's tooling would handle whether it goes to Nomad, Kubernetes, or a cloud function, with HAProxy routing configured automatically as part of the process.
Every cluster, backend, frontend, and Consul integration is configurable through the API, which means the entire deployment pipeline can eventually run without a human touching the load balancer configuration at all.
Subscribe to our blog. Get the latest release updates, tutorials, and deep-dives from HAProxy experts.
此内容由惯性聚合(RSS阅读器)自动聚合整理，仅供阅读参考。 原文来自 — 版权归原作者所有。