Microservices, the Expulsion From E2E Testing Paradise, and the Path to Salvation: Contract Testing

Recently, I was back on the job hunt. At one particular interview, my potential future colleagues and I got into a discussion about testing, specifically the term end-to-end testing. I was in the awkward position that I didn’t have a word to describe a type of test where you run your web application in a real browser and test it from a user’s perspective, but instead of performing a real end-to-end test from the client-side application all the way to the different microservices, use some form of mocking or stubbing to prevent requests to external microservices. To my surprise, my interview counterparts told me they struggled with the same, and that’s why they coined a term for it: end-to-mock tests.

What many call end-to-end tests are often end-to-mock tests.

I was excited not only because it’s a great name for those kinds of tests but also because I found validation that I’m not the only one struggling with the lack of a universal terminology for certain kinds of tests. Moreover, I was relieved that it’s not just me; there seems to be a general confusion about how exactly we should test large-scale, loosely coupled, service-based software systems. In this article, we will explore this topic further and find out why end-to-end testing no longer cuts it. Furthermore, we’ll try to find a better vocabulary to make future testing discussions easier.

A Brief Overview of Test Types

When it comes to testing, there are many different names for different types of tests. Some common types of tests referenced around the web are:

1. End-to-end (E2E) tests: Simulate real user scenarios by testing the entire system from one end all the way to the other end.
2. Integration tests: Ensure that several units (e.g., components or services) work together in harmony.
3. Component tests: These tests isolate individual UI components to ensure they behave correctly. They are specific to frontend frameworks like React, Vue.js, or Angular.
4. Unit tests: Testing the smallest testable units of an application, such as individual modules, classes, or functions.

Those are the types of tests you’ll typically encounter in numerous blog articles on testing web applications. Because this article will focus primarily on testing web applications within a microservices architecture, I won’t go into further detail about what a “component test” might encompass in a different context.

What Are End-to-End Tests, Really?

But before we dig in, let’s take a step back: What do we even mean when we talk about end-to-end tests? Especially in the frontend world, some people call every test that runs in a real browser environment an end-to-end test. Others tend to tie the terminology they use for their tests to the tools we typically use to run them. For E2E tests, the most popular ones are Cypress and Playwright, and for Unit Test, Jest and Vitest. Yet, this is not a valid criterion. All those tools can run various types of tests, and running a test primarily in Cypress or Playwright doesn’t automatically make it an end-to-end test.

Ultimately, there is no authority or final verdict about what is and is not an end-to-end test, but I go with the Wikipedia definition:

System testing, a.k.a. end-to-end (E2E) testing, is testing conducted on a complete software system. – https://en.wikipedia.org/wiki/System_testing

A software system is a system of intercommunicating components based on software forming part of a computer system (a combination of hardware and software). It “consists of a number of separate programs, configuration files, which are used to set up these programs, system documentation, which describes the structure of the system, and user documentation, which explains how to use the system”. – https://en.wikipedia.org/wiki/Software_system

Applied to web applications, this means an end-to-end test is a form of testing performed from the perspective of a user interacting with a whole software system via a browser. To accomplish this, we don’t mock or stub any pieces of the system.

To avoid ambiguity, I prefer to talk about end-to-end system tests so there is no confusion whatsoever to the fact that we’re talking about tests that cover a whole software system from end-to-end.

So, end-to-end system testing means testing a software system from one end (a user interacting with the system, typically via a browser) all the way to the other end (e.g., data that gets written to or read from a database or some other action our system performs in response to a particular trigger) across all parts of the system.

Microservices-based software system.

Performing real end-to-end system tests is relatively straightforward when building applications following the good old monolithic approach, where a single application reads directly from and writes to a single database. But in recent years, things have gotten a lot more complicated.

In the Beginning was the Monolith

With classic, monolithic architectures, E2E system testing is a walk in the park. We have control over the whole system end-to-end, which is crucial for effective testing. Whatever we want to test, we can seed our database with exactly the data we need, and we’re good to go. We can simulate every scenario and every state that our system can possibly be in. As long as we make sure our test environment mimics our production environment as closely as possible, we can have confidence in our tests.

Monolithic (software) system.

For example, let’s say we have a monolithic e-commerce application. We can easily set up a test database with sample products, users, and orders. Then, we can write end-to-end system tests that simulate users browsing products, adding items to their cart, and completing the checkout process. Because we control the entire system, we can ensure precisely which products and how many are available when testing a particular scenario.

Eating From the Tree of the Knowledge of Loose Coupling and Independent Deployability

The end-to-end testing world was good, but then mankind got seduced by the allure of loose coupling and independent deployability. We ate from the tree of knowledge and started to build software systems based on loosely coupled components and services. Microservices and even microfrontends started to proliferate.

System consisting of multiple native applications and web based microfrontends.

The Fall: Why End-to-End Testing Doesn’t Work for Loosely Coupled Software Systems

In a microservices-based architecture, end-to-end system testing becomes increasingly difficult and often counterproductive. The very nature of microservices, with their independent deployability and loose coupling, makes it challenging to control the entire system from end-to-end. Yet, having complete control is a prerequisite for writing reliable and meaningful tests.

End-to-end system testing is a flawed approach for testing loosely coupled software systems. Because of the lack of centralized control over the whole system, it is too hard to get it right. Attempting to perform end-to-end system testing in a loosely coupled system may negate the benefits we seek from microservices, such as the ability to evolve services independently.

If at all, we should use end-to-end system tests only sparingly when our software system consists of loosely coupled microservices and applications. Instead, we need to adapt our testing strategy to align with the principles of microservices.

The Path to Salvation: End-to-Mock, and End-to-Contract Tests

As a better way forward, I propose the addition of end-to-mock (E2M) and end-to-contract (E2C) tests to the testing canon and precisely define end-to-end system tests as follows:

1. End-to-end system tests: These tests are a practical approach to testing monolithic applications only if we have complete control over the entire stack, from browser to database. They are best used in such scenarios and should be used sparingly or not at all for testing loosely coupled systems.
2. End-to-mock tests: These tests shine for client-only single-page applications (SPAs) getting their data from microservices, where we lack control over the service layer but can easily mock requests made from the browser to microservices.
3. End-to-contract tests: We use this type of test for server-side applications (e.g., Next.js, Nuxt, Rails, et al.) that get their data from microservices. We don’t have control over either the service or network layer; therefore, we test against contracts established between the application and the microservices it communicates with.

End-to-mock and end-to-contract tests help us address the challenges of testing applications in a microservices-based architecture. However, although end-to-mock tests are a valid solution for testing SPAs, end-to-contract tests are preferable in most cases, as they nudge us toward thinking API and contract-first. Still, remember that end-to-mock tests are more straightforward to set up as they don’t require us to define contracts and run a stub server. So, depending on the use case, we might nonetheless prefer end-to-mock tests over end-to-contract tests if we can get away with it.

Personally, even considering the simplicity of the end-to-mock approach, I’m convinced that in a microservices architecture, thinking API and contract-first and going the end-to-contract route is worth it. By defining clear contracts between services, teams can work independently and in parallel while confident that their changes won’t break other parts of the system. End-to-mock testing can’t compete with this level of confidence.

Contract Testing

I fully believe end-to-contract testing is best for testing SPAs and server-side rendered applications talking to microservices. What makes this approach so effective is that it considers the whole system while allowing us to test different modules of the system in isolation.

Contract testing allows us to test the interactions between consumers (apps or microservices) and providers (microservices) without the need for end-to-end or integration tests. The idea is to define a contract between consumers and providers that specifies the shape of a provider’s expected inputs and outputs. Then, each team can write tests that verify their service adheres to the contract without setting up and maintaining a replica of the whole (loosely coupled) system.

For example, suppose we have a microservices-based e-commerce application with separate services for products, orders, and payments. We can define contracts between these services and their consumers that specify the API endpoints and request/response formats. Each team can then write tests that verify their service or web app implements the contract correctly, using tools like Pact or Specmatic.

openapi: 3.0.0
info:
  title: Chocolate Bar API
  version: 1.0.0
  description: API for chocolate bars
paths:
  /chocolate-bars:
    get:
      summary: Get list of chocolate bars
      responses:
        '200':
          description: A list of chocolate bars
          content:
            application/json:
              schema:
                type: object
                properties:
                  data:
                    type: array
                    items:
                      $ref: '#/components/schemas/ChocolateBarResource'
components:
  schemas:
    ChocolateBarResource:
      type: object
      properties:
        id:
          type: string
        type:
          type: string
        attributes:
          type: object
          properties:
            title:
              type: string
            sugar:
              type: string
            taste:
              type: string

The benefits of contract testing in a microservices architecture are numerous:

1. Teams can work independently and in parallel.
2. Tests are faster and more reliable, independent of the availability or state of other services.
3. Issues are caught early in development, before integration or deployment.
4. The contracts serve as living documentation of the system, making it easier for new team members to understand how the services interact.
5. Stubbing the requests to services gives us full control over the data we send and receive.

I want to emphasize the last point: Control. It is crucial to have complete control over the testing environment to write meaningful tests. Not only is it much quicker to bring our system into a particular state needed for a certain test scenario, only when we can control what responses we get from our microservices, can we test how our application reacts to errors.

Contract testing provides a way to regain some of the control we had with monolithic architectures while still enjoying the benefits of microservices. By defining clear contracts and testing against them, we can ensure that our loosely coupled services work together seamlessly and reliably without depending on other services during testing.

End-to-Contract Testing

With contract testing, we typically only validate whether a service adheres to a given contract. A side effect of this is that when practicing contract testing, we end up with comprehensive contracts for all of the services of our system. When we have specifications as a basis for the contracts of all of our services, we can easily repurpose those contracts to drive our end-to-endcontract tests, too.

Testing an application end-to-contract; the application gets its data from a stub server.

Based on those contracts, popular contract testing tools like Pact and Specmatic enable us to quickly spin up stub endpoints for various services on which our application depends. And we have complete control over those stubbed services. When we want to test how our application or service deals with a cart filled with plenty of products, we can tell the stub server to return a cart with hundreds of products. If we want to test what happens when the cart service returns an error because it is unavailable, we can tell it to be unavailable for our next test.

When using the contracts specified for contract testing while testing our frontend applications, too, we end up with a perfectly tested yet truly loosely coupled system. Furthermore, we drastically shorten feedback loops. If a team maintaining a particular service were to introduce a change that breaks some frontend application, its build pipeline would fail long before the change goes out into production. If a breaking change is required, first, we immediately know which projects and teams are affected; second, we can quickly communicate the change to them; and last but not least, we can effortlessly run our contract and end-to-contract tests to validate whether all consumers implemented the breaking change correctly.

Wrapping It Up

Just recently, I wrote a very similar article on this topic (https://markus.oberlehner.net/blog/no-more-mocking-write-better-tests-for-microservices-powered-server-side-rendered-applications-with-contract-tests/). So similar that while writing this article I sometimes questioned if I shouldn’t instead make some updates to the older article. However, first of all, the other article leans a little bit more on the practical side, and even more importantly, I don’t primarily write because I want to teach but as a way to clarify my thoughts (still, publishing what I write is very much aligned with my mission to share what I know).

The fact that this is my second article of this kind in only a couple of weeks, therefore, a testament to the topic’s ripening in my mind. There were still some wrinkles I needed to iron out, and this article got me one step closer to this.

I feel like loosely coupled, microservices-based architectures have evolved more quickly than the processes and methods necessary to maintain them successfully in the long run. Tooling and infrastructure are still evolving. We are also still exploring how to test those systems effectively. Of course, there are companies out there who have figured this out, but it hasn’t become canon yet. I hope this article can serve as a small step along the way.