A system that decides paychecks
The client calculates commissions and bonuses for its customers' sales teams. In practice, the system decides how much money goes into each salesperson's pocket at the end of the month.
The platform was a Ruby application, with a web app and several processing workers: commission calculation, data cleanup and migration, system routines. There were no standardized environments and no safe way to update. In a system like that, a half-finished deploy, with some services on the new version and some on the old, isn't just instability. It can mean commissions calculated wrong.
Everything in code
I started by containerizing the applications, creating the image repositories in ECR and adjusting the Dockerfile so builds were reproducible. I wrote the infrastructure in Terraform and imported the resources that already existed into it, so nothing was left outside the code.
I migrated six workers to ECS, including the three commission calculation engines, and adapted secrets management to the new model. The web app and the shared environment were migrated by another engineer on the team. I also created the Demo and Beta environments, with the cluster provisioned by a Terraform module and a capacity provider to scale instances on its own.
For deploys, I built a main pipeline in GitHub Actions that triggers each service's pipeline. Since GitHub Actions doesn't support workflow subfolders, I organized the files by prefix.
Either they all go out, or they all roll back
The most important part was the blue/green deploy with CodeDeploy, designed so that four services go out as one transactional batch. Before installing, migrations and the image's dependency validation run, to fail as early as possible. Before shifting traffic, an HTTP smoke test runs on the web app and a synthetic job on the workers. If any of the four fails, whether on a health check, a hook or an alarm, all four roll back automatically. The previous version stays live until the new one is promoted, and migrations must work with both versions, so a rollback doesn't lose data.
The cluster that "wouldn't shrink"
In the Beta environment, the cluster seemed not to reduce instances when load dropped. I investigated in two steps. First, I found the capacity provider in an inconsistent state. Since it can't reuse an old Auto Scaling Group, I recreated both, recreated the web service with CodeDeploy and the load balancer listeners, and imported everything back into Terraform without losing state.
Then I understood the behavior that had caught our attention: it wasn't a defect. The capacity provider's managed scaling uses a conservative window of about 15 minutes before removing instances. I confirmed through the CLI that, with only the web app running, the group drops to one instance after that window.
What changed
Blue/green is in production. The client deploys without taking the platform down, and a failure in any of the four services rolls them all back together, without leaving commission calculation in a mixed state. Workers run in containers with versioned images, and new environments are created through Terraform, not by hand.
What I take from it
In systems that handle money, deploys have to be atomic across services: "half updated" is worse than "nothing updated". And not every odd behavior is a bug. Before fixing the scale-down, it was worth understanding how the capacity provider was designed to work.