OpenAPI for Contract Testing

OpenAPI, former known as Swagger, is a specification, usually expressed in JSON and YML, that describes how your API looks and behaves like.

It’s a very powerful tool which I don’t believe is used as much as it should. Most people I have talked to think that it’s just a way to generate API documentation. I also remember talking to people who are not in love with OpenAPI because it gets outdated over time. People may change the API and forget to modify or re-generate the openapi spec. And voila! you have an OpenAPI spec which doesn’t even represent how your API actually behaves. Your API clients find that the OpenAPI spec you have is merely a representation of how your API looked at some point in the past and you are back to the old-fashioned way of sharing the details about your API via slack messages.

I am here to tell you that you can ensure your OpenAPI is never out of sync with how your API actually behaves. You could even take it few steps further and use it for contract testing.

Ensuring OpenAPI spec is always in sync with your API

Once you are ready to go down this route, there are a lot of ways to achieve this. A straighforward approach is integrating an OpenAPI validator in a language of your choice and validating requests and responses against OpenAPI spec.

I will show you a simple example for a node’s express based API using express-openapi-validator. Before I show you the code, I will list down some slightly opinionated choices that I have made about the validator.

If the request doesn’t pass the OpenAPI spec, it should be 4xx response. This will reduce burden on request validation layer and reduce validation logic you have to write. I will touch more on this later on.
If the response generated by the the API server doesn’t pass the OpenAPI spec, it will be a hard 5xx response. Some people may frown at this idea, but I will share my reasoning later on.


app.use(
  OpenApiValidator.middleware({
    apiSpec: path.join(__dirname, '../wallet-openapi.yml'),
    validateRequests: true,
    validateResponses: true,
  }),
);

//error handling middleware for when req/response validation fails
app.use((err, req, res, next) => {
  if (
    err instanceof OpenApiValidator.error.BadRequest
    || err instanceof OpenApiValidator.error.InternalServerError
  ) {
    return res.status(err.status).json({
      message: err.message,
      errors: err.errors,
    });
  }

  // pass to default error handle middleware if not related to req/response validation failure
  next(err);
});

Check out the full example here.

You could implement a similiar thing in a PHP API using ThePHPLeague’s openapi-psr7-validator. Here’s an example implementation of a PSR-15 middleware:

namespace App\Http\Middleware;

use League\OpenAPIValidation\PSR7\Exception\Validation\AddressValidationFailed;
use League\OpenAPIValidation\PSR7\Exception\ValidationFailed;
use League\OpenAPIValidation\PSR7\SchemaFactory\JsonFileFactory;
use League\OpenAPIValidation\PSR7\ValidatorBuilder;
use Psr\Http\Message\ResponseInterface;
use Psr\Http\Message\ServerRequestInterface;
use Psr\Http\Server\MiddlewareInterface;
use Psr\Http\Server\RequestHandlerInterface;
use Nyholm\Psr7\Response;

final readonly class OpenApiSpecValidatingPsrMiddleware implements MiddlewareInterface
{
    public function process(ServerRequestInterface $request, RequestHandlerInterface $handler): ResponseInterface
    {
        $schema = (new JsonFileFactory('wallet-openapi.json'))->createSchema();

        $requestValidator = (new ValidatorBuilder())
            ->fromSchema($schema)
            ->getServerRequestValidator();

        try {
            $matchedOperation = $requestValidator->validate($request);
        } catch (ValidationFailed $e) {
            return new Response(400, [], json_encode([
                'error_code' => 'openapi_spec_request_parse_failed',
                'error_message' => $e instanceof AddressValidationFailed ? $e->getVerboseMessage() : $e->getMessage(),
            ]));
        }

        $response = $handler->handle($request);

        $responseValidator = (new ValidatorBuilder())
            ->fromSchema($schema)
            ->getResponseValidator();

        try {
            $responseValidator->validate(
                $matchedOperation,
                $response,
            );

            return $response;
        } catch (ValidationFailed $e) {
            return new Response(500, [], json_encode([
                'error_code' => 'openapi_spec_request_parse_failed',
                'error_message' => $e instanceof AddressValidationFailed ? $e->getVerboseMessage() : $e->getMessage(),
            ]));
        }
    }
}

You can also run an OpenAPI proxy like prism, so you can do the request and response validation in the proxy server outside your application.

npx prism proxy ./openapi-prod.json http:/localhost:3000 --errors

Why 400 for request matching the OpenAPI spec

If you are really trying to use OpenAPI to it’s full capability, your clients should ideally use OpenAPI spec as the first class citizen for integrating to your API. You may not able to control this if you have a public API, but you may have a say on this if you are exposing API to consumers at least within your company. I think it’s totally fair to respond with 4xx because the client didn’t meet the contract when sending the request. 400 Bad Request means the server received a request which it couldn’t really understand or parse.

Using OpenAPI generated API client

When I say “OpenAPI as first class citizen for integration”, it doesn’t mean just using OpenAPI spec generated Swagger documentation. Your API consumers can use the OpenAPI for API client generation using tools like openapi-generator for most languages. Here’s an example way to generate API client for php and JS.

curl https://raw.githubusercontent.com/stripe/openapi/master/openapi/spec3.json > stripe-openapi.json

openapi-generator generate -i ./stripe-openapi.json -g php-nextgen
openapi-generator generate -i ./stripe-openapi.json -g js

Using a generated API client would shift the responsibility of validation from runtime to static time. If your API clients use some form of a good pre-production static code analyzer and code linters in CI, they could detect request not being valid well before deploying/releasing to production.

Another option is to send the raw requests manually but you could validate the requests at runtime using openapi request validation libraries like openapi-enforcer.

Here’s an example axios interceptor to validate all outgoing requests:

const openapi = await Enforcer('../wallet-openapi.yml');

axios.interceptors.request.use(function (config) {
  const url = new URL(config.url);
  
  const [req, error] = openapi.request({
    method: config.method,
    path: url.pathname,
    body: config.data,
  });

  if (error) {
    return Promise.reject(error);
  }

  return config;
}, function (error) {
  return Promise.reject(error);
});

With the above configured axios interceptor, you can validate requests on the client too. Here’s an example:

const response = await axios.post('http://localhost:3000/wallets', {
  name: 'test',
  type: 'Event',
  colour_code: 'Green',
});
console.log(response.status); // 201

try {
  await axios.post('http://localhost:3000/wallets', {
    name: 'test',
    body: 'test',
    description: 'Yellow',
  });
} catch (error) {
  // [ EnforcerException: Request has one or more errors
  //   In body
  //     For Content-Type application/json
  //       Invalid value
  //         One or more required properties missing: type, colour_code ]
  console.error(error);
}

You can checkout the full setup on js-client/axios-interceptor.js.

When not using the openapi sdk, this is pretty crucial, specially if the API clients have enough automated tests where tests would fail if the request doesn’t match the contract defined in OpenAPI spec and the mocked response are also validated against the OpenAPI spec.

Achieving Contract testing with OpenAPI spec

If you have come this far, I imagine you are doing all of the things:

In the API server, you have some form of OpenAPI validation implemented in the server for both the incoming requests and the outgoing response.
- You can take it a step further if you have good coverage of your API tests which responds with 500 and fails tests when response doesn’t meet the OpenAPI spec. This will enable you to detect the issue in CI before the API is deployed.
In the API clients, you are either using the generated clients or using the openapi validator libraries for both request validation and response mock validation with enough automated test coverage.

This enables you to release your clients and backend to deploy seperately without running a full range of E2E tests which are much slower and unpredictable (flaky) in nature. That’s basically contract testing. As long as both the server and the client abide to agree the contract in OpenAPI spec, they can deploy and release independantly.

Detecting breaking API changes

You can use openapi-diff to detect breaking change and potentially fail your build pipeline .

openapi-diff --fail-on-incompatible  <old> <new>

Strict(ify) your schema

Make the schema in your OpenAPI spec as strict and explicit as you can. Here’s an example of a loosely defined schema that leaves a lot of loose ends.

# ...... rest of your spec
components:
  schemas:
    Wallet:
      type: object
      properties:
        id:
          type: number
        name:
          type: string
        description:
          type: string
        type:
          type: string
        colour_code:
          type: string

There are few problems with the above spec:

There’s not enough information on what are the required fields. By default, all the fields are optional and will pass validation against the schema without any of the fields.
Looks like type and colour_code are enum types, but there’s not enough information on what are the allowed values of both of those fields.
Min and max value for fields like name aren’t specified? Can the API client send a full 500 words essay as wallet name? I would imagine that it should not be allowed.

The above OpenAPI spec may be good enough just for documentation purpose but it’s not enough for contract testing. Let me show you how you can make the above schema more explicit:

# ...... rest of your spec
components:
  schemas:
    Wallet:
      type: object
      required: # required fields are mentioned here
      - id
      - name
      - type
      - colour_code
      properties:
        id:
          type: number
        name:
          type: string
          maxLength: 50 # min and max length
          minLength: 1
        description:
          type: string
          maxLength: 255
        type:
          enum: # enum can be used for indicating supported values
          - Travel
          - HouseholdExpenses
          - Event
          - Other
          type: string
        colour_code:
          enum:
          - Red
          - Blue
          - Green
          type: string

You may ask what’s the point of making it more strict. Imagine you are receiving wallet under response body while calling GET /wallets/:id. You can be certain that what fields will surely be there and what not. You can have more strict typing in client side if you are using a programming language supporting static types.

const response = await fetch('http://api.example.com/wallets/10')

if (!response.ok) {
  throw new Error(`Error Response with status: ${response.status}`);
}

const body = await response.json();

const title: string = body.colour_code;
const description: string | null = body.description;

enum ColourCode {
  RED = 'Red',
  BLUE = 'Blue',
  GREEN = 'Green',
}

const colourCode: ColourCode = body.colour_code;

This is just a rough idea of how strict schema can be a game changer to the client with better IDE experience with code suggestions and failing build when type check fail. In real world, there are tons of tools like openapi-zod-client which can generate the API client as well as zod schemas for the requests and responses.

Final Thoughts

The ecosystem of OpenAPI seems to have grown a lot in the last few years. There are more widespread amount of tools to help with a variety of use-cases from client code generation, pre-production checks, mock servers, request/response validating proxies etc in a variety of languages. This space is definitely worth looking into and see where it can help your organization.