Health checks with Terminus and Datadog

We should be aware of whether our application is healthy. One way to do that would be to make various API requests to confirm it manually. It might not be the most elegant solution, though. The answer can be health checks. With them, we can verify if various aspects of our system work properly. Creating a health check boils down to exposing an endpoint that tests our application and throws an error if it is unhealthy. Thanks to tools such as Datadog, we can periodically call the health check endpoint to make sure that everything works as expected.

Introducing Terminus

NestJS comes with a tool called Terminus that can help us implement our health check.

1npm install @nestjs/terminus

To use it, we should create a new controller.

1import { Controller, Get } from '@nestjs/common';
2import { HealthCheckService, HealthCheck } from '@nestjs/terminus';
3 
4@Controller('health')
5class HealthController {
6  constructor(
7    private health: HealthCheckService,
8  ) {}
9 
10  @Get()
11  @HealthCheck()
12  check() {
13    return this.health.check([]);
14  }
15}
16 
17export default HealthController;

We also need to put it into a module that imports the TerminusModule.

1import { Module } from '@nestjs/common';
2import HealthController from './health.controller';
3import { TerminusModule } from '@nestjs/terminus';
4 
5@Module({
6  imports: [TerminusModule],
7  controllers: [HealthController],
8  providers: [],
9})
10export default class HealthModule {}

By doing the above, we achieve a straightforward health check.

With the above endpoint so far, we can know whether our API started or not. However, it is not much information, so let’s add additional checks.

Using built-in health indicators

Terminus comes with a set of health indicators that can check if a particular service is healthy or not. A good example is the TypeOrmHealthIndicator.

1import { Controller, Get } from '@nestjs/common';
2import { HealthCheckService, HealthCheck, TypeOrmHealthIndicator } from '@nestjs/terminus';
3 
4@Controller('health')
5class HealthController {
6  constructor(
7    private healthCheckService: HealthCheckService,
8    private typeOrmHealthIndicator: TypeOrmHealthIndicator,
9  ) {}
10 
11  @Get()
12  @HealthCheck()
13  check() {
14    return this.healthCheckService.check([
15      () => this.typeOrmHealthIndicator.pingCheck('database')
16    ]);
17  }
18}
19 
20export default HealthController;

Under the hood, the TypeOrmHealthIndicator performs a simple  SELECT 1 SQL query to our database to verify that it is up and running and we’ve established a connection. If any of our health indicators fail, the endpoint responds with 503 Service Unavailable instead of 200 OK.

Our /health endpoint responds with a few properties:

• status – if any of our health indicators fail, the status is set to 'error'. If our application is currently shutting down, the status is 'shutting_down'.
• info – contains information about each health indicator that is healthy and has the status 'up',
• error – has information about every health indicator that is unhealthy and has the status 'down',
• details – contains the information about every health indicator that we have.

A list of other built-in health indicators

Terminus has more health indicators that we can use, such as:

• HttpHealthIndicator allows us to perform health checks related to HTTP requests,
• MongooseHealthIndicator with it, we can check if MongoDB responds within 1000ms if we use the Mongoose library (the timeout value can be changed),
• SequelizeHealthIndicator with the Sequelize health indicator, we can execute checks related to Sequelize,
• MicroserviceHealthIndicator allows us to check if a given microservice is up. If you want to know more about microservices, check out API with NestJS #18. Exploring the idea of microservices,
• GRPCHealthIndicator checks if a service is up using the standard health check specification of GRPC. If you want to know more about GRPC, check out API with NestJS #20. Communicating with microservices using the gRPC framework,
• MemoryHealthIndicator performs checks related to memory. Can verify the resident set size (RSS) and the heap space,
• DiskHealthIndicator verifies the disk storage of the machine our application runs on.

Having the above in mind, let’s use more of the built-in health indicators.

1import { Controller, Get } from '@nestjs/common';
2import {
3  HealthCheckService,
4  HealthCheck,
5  TypeOrmHealthIndicator,
6  MemoryHealthIndicator,
7  DiskHealthIndicator,
8} from '@nestjs/terminus';
9 
10 
11@Controller('health')
12class HealthController {
13  constructor(
14    private healthCheckService: HealthCheckService,
15    private typeOrmHealthIndicator: TypeOrmHealthIndicator,
16    private memoryHealthIndicator: MemoryHealthIndicator,
17    private diskHealthIndicator: DiskHealthIndicator
18  ) {}
19 
20  @Get()
21  @HealthCheck()
22  check() {
23    return this.healthCheckService.check([
24      () => this.typeOrmHealthIndicator.pingCheck('database'),
25      // the process should not use more than 300MB memory
26      () => this.memoryHealthIndicator.checkHeap('memory heap', 300 * 1024 * 1024),
27      // The process should not have more than 300MB RSS memory allocated
28      () => this.memoryHealthIndicator.checkRSS('memory RSS', 300 * 1024 * 1024),
29      // the used disk storage should not exceed the 50% of the available space
30      () => this.diskHealthIndicator.checkStorage('disk health', {
31        thresholdPercent: 0.5, path: '/'
32      })
33    ]);
34  }
35}
36 
37export default HealthController;

Custom health indicators

While there are quite a few of the built-in indicators, they are far from covering every case. Thankfully, we can set up a custom health indicator. To do that, we need to extend the HealthIndicator class.

For example, in the 12th part of this series, we’ve used Elasticsearch. Let’s write a health indicator that checks if our instance of Elasticsearch is up and running.

1import { Injectable } from '@nestjs/common';
2import { HealthIndicator, HealthIndicatorResult, HealthCheckError } from '@nestjs/terminus';
3import { ElasticsearchService } from '@nestjs/elasticsearch';
4 
5@Injectable()
6export class ElasticsearchHealthIndicator extends HealthIndicator {
7  constructor(
8    private readonly elasticsearchService: ElasticsearchService
9  ) {
10    super();
11  }
12 
13  async isHealthy(key: string): Promise<HealthIndicatorResult> {
14    try {
15      await this.elasticsearchService.ping();
16      return this.getStatus(key, true);
17    } catch (error) {
18      throw new HealthCheckError(
19        'ElasticsearchHealthIndicator failed',
20        this.getStatus(key, false)
21      );
22    }
23  }
24}

We need to register the ElasticsearchHealthIndicator as a provider before using it.

1import { Module } from '@nestjs/common';
2import HealthController from './health.controller';
3import { TerminusModule } from '@nestjs/terminus';
4import { ElasticsearchHealthIndicator } from './elasticsearchHealthIndicator';
5import { SearchModule } from '../search/search.module';
6 
7@Module({
8  imports: [TerminusModule, SearchModule],
9  controllers: [HealthController],
10  providers: [ElasticsearchHealthIndicator],
11})
12export default class HealthModule {}

Please notice that I’ve imported the SearchModule that we’ve created in the 12th part of this series.

The last step is to use the ElasticsearchHealthIndicator in our health check:

1import { Controller, Get } from '@nestjs/common';
2import {
3  HealthCheckService,
4  HealthCheck,
5  TypeOrmHealthIndicator,
6  MemoryHealthIndicator,
7  DiskHealthIndicator,
8} from '@nestjs/terminus';
9import { ElasticsearchHealthIndicator } from './elasticsearchHealthIndicator';
10 
11 
12@Controller('health')
13class HealthController {
14  constructor(
15    private healthCheckService: HealthCheckService,
16    private typeOrmHealthIndicator: TypeOrmHealthIndicator,
17    private memoryHealthIndicator: MemoryHealthIndicator,
18    private diskHealthIndicator: DiskHealthIndicator,
19    private elasticsearchHealthIndicator: ElasticsearchHealthIndicator
20  ) {}
21 
22  @Get()
23  @HealthCheck()
24  check() {
25    return this.healthCheckService.check([
26      () => this.typeOrmHealthIndicator.pingCheck('database'),
27      // the process should not use more than 300MB memory
28      () => this.memoryHealthIndicator.checkHeap('memory heap', 300 * 1024 * 1024),
29      // The process should not have more than 300MB RSS memory allocated
30      () => this.memoryHealthIndicator.checkRSS('memory RSS', 300 * 1024 * 1024),
31      // the used disk storage should not exceed the 50% of the available space
32      () => this.diskHealthIndicator.checkStorage('disk health', {
33        thresholdPercent: 0.5, path: '/'
34      }),
35      () => this.elasticsearchHealthIndicator.isHealthy('elasticsearch')
36    ]);
37  }
38}
39 
40export default HealthController;

Our health check will now check Elasticsearch and verify if it is up and running.

Performing health checks with Datadog

Although the /health endpoint that we’ve created looks useful, we’ve been using it manually so far. Fortunately, there are services such as Datadog that can call this endpoint for us periodically.

To do that, we need to set up an account and create a new synthetic test.

When we’re there, we need to create a new API test.

If your API isn’t deployed yet, but you want to try the above solution, you can use ngrok to share your localhost API with the world.

When defining our test, we first need to specify the URL of our health check:

You might want to avoid exposing the health of our application to the outside world and require some kind of authentication to access the /health endpoint for security reasons. If you do that, click Advanced Options where you can provide additional request options.

We also need to specify when our test is perceived as successful and what locations we want Datadog to request our API from. Since if any of our health indicators fail, the endpoint responds with 503 Service Unavailable, let’s check if our endpoint responds with 200 OK.

It is also important to specify the frequency of the test and define alert conditions. If something goes wrong, the team can be notified.

Thanks to doing all of the above, Datadog will monitor our health check endpoint and alert us if something goes wrong. We can also view the history of the tests for more details.

Summary

In this article, we’ve created a health check endpoint with a tool called Terminus. While doing so, we’ve used various health indicators built into NestJS and created a custom one. We’ve set up Datadog to periodically request our health check endpoint and alert us about downtime to benefit from it more.

Health checks are more than just a way to check if our API is up. We can validate the connection to our database or even test its performance to notice potential issues. We should create health checks based on the environment of our application and include services our app depends on. Feel free to write more custom health indicators and fiddle more with Datadog to help you keep your API more reliable and stable.