Skip to main content

Prerequisites

Contents​

Understanding the data you want in Central Monitoring​

The first step to utilising your data in Central Monitoring is understanding the type of data you need to use.

The most common type of data required to monitor an application are logs & metrics. Containerised and serverless applications will have different methods of sending log & metric data to Central Monitoring.

Custom payloads can also be sent to Central Monitoring.

Logs & Metrics from Containerised Applications​

Logs must be in a JSON (structured) format. You can read more about structured logs here.

To send Logs & Metrics from a containerised application, a set of Central Monitoring forwarders can be used (built on top of Filebeat, Metricbeat, and Fluentbit).

A custom image of the Central Monitoring forwarders will be developed for the new application. Access to these images will then be provided.

The team responsible for the application is also responsible for the deployment and management of these forwarders. Deployment instructions can be found here.

The Fluentbit Forwarder will export the logs externally off the host to a Central Monitoring pipeline.

Logs & Metrics from Serverless Applications​

Logs must be in a JSON (structured) format. You can read more about structured logs here.

To send Logs & Metrics from a serverless application, a Central Monitoring Event Bus is used.

This requires the team responsible for the application to send their logs cross account to the Event Bus.

Common patterns recommended by Central Monitoring are:

  • CloudWatch logs → Subscription Filter → Lambda → Send cross account to Central Monitoring EventBus
  • DynamoDB Streams → Lambda → Send cross account to Central Monitoring EventBus

These payloads must be in a JSON format. Here is an example of the required payload structure example of payload structure

The event bus endpoint to send logs to will be provided by Central Monitoring.

Custom Payloads​

If you need to send data to Central Monitoring that is not part of your applications standard logs, you can do so by sending the logs to an event bus.

These payloads must be in JSON format

When your application is initially being onboarded, the Central Monitoring team will provide you with details for you event bus.

Tracking provisioning and decommissioning of deployments​

Many monitoring and analytics requirements need tracking of application instance deployments.

Examples of why this may be required are:

  • Detecting if an instance is down
  • Reporting on how many instances exist

The application or application support team is responsible for maintaining this record in Central Monitoring.

This is done through an event bus. Whenever a new instance is deployed, the details should be sent to Central Monitoring via a payload to the event bus.

When your application is initially being onboarded, the Central Monitoring team will provide you with details for you event bus.

Summary​

  • Know what data you want to send to Central Monitoring. It can be log data, or payloads send to eventbus
  • Application must be containerised or serverless
  • Logs need to be Structured and log levels must be used correctly (debug logs won't be ingested)
  • Have an estimate of the volume of logs your application is generating
  • Need to be able to track your deployments and send to Central monitoring (provisioning and decommissioning)
  • Provide permissions to Central Monitoring
  • All fields with same names need to have the same data type