Skip to content
Engineering2 min read

Contract Testing Across 40 Services: Killing the Integration Environment

Shared integration environments become the bottleneck every distributed system eventually hits. Consumer-driven contract testing, and the migration that got us there.

Abdul Basit, Senior Software Engineer

Principal Architect

Circuit board close-up representing service-to-service contracts across a distributed architecture

The shared staging environment is a piece of infrastructure that every team depends on, no team owns, and which is broken roughly a third of the time. At forty services it had become the single largest source of delay in the client’s delivery pipeline: a median of two and a half days between a pull request being approved and reaching production, almost all of it queueing for a green environment.

What contract testing actually replaces

Consumer-driven contract testing does not replace unit tests or production monitoring. It replaces one specific and very expensive thing: the end-to-end test that exists only to check that service A and service B still agree on the shape of a message.

The mechanics in one paragraph

The consumer writes a test against a mock provider, and that test emits a contract describing exactly the requests it makes and the responses it expects. The provider then replays every registered consumer contract against its real implementation in its own pipeline. If the provider breaks a consumer, the provider’s build fails - before the change ever reaches a shared environment.

Snippet
consumer: checkout-web  provider: pricing-api  status: verified  provider_version: 4.19.2

The migration, in the order we did it

  • Pick the two services with the most integration-test churn; contract-test only that pair.

  • Add a contract broker and make verification results visible on both teams’ dashboards.

  • Gate the provider’s deployment on consumer verification, with a documented break-glass.

  • Delete the end-to-end tests those contracts now cover, on the same day, or nobody will believe the migration is real.

  • Repeat pair by pair; do not attempt a big-bang rollout across forty services.

Deleting the superseded tests in the same change is the step teams skip. Left in place, the old suite continues to fail intermittently, engineers keep waiting for it, and the contract tests become extra work rather than replacement work.

Where contracts do not help

Contracts verify shape and semantics between two parties. They say nothing about emergent behaviour, performance under load, or whether the business process spanning six services produces the right outcome. Keep a small number of genuine end-to-end journey tests - we kept eleven, down from four hundred - and run them against production with synthetic data rather than against a shared pre-production environment.

Eleven end-to-end tests that always pass are worth more than four hundred that fail for unrelated reasons twice a week.

The result

Median pull-request-to-production went from two and a half days to under four hours across the estate. The staging environment still exists, but it is now used for exploratory testing by people, which is the only thing shared environments are genuinely good at.

Share

Working on something like this?

We embed dedicated dev teams and senior architects into enterprise programmes: AI & cloud solutions, legacy modernization and full-stack web engineering.