Manual Deployment of Central Monitoring Forwarders

This guide explains how to manually deploy Central Monitoring forwarders (Fluent Bit, Filebeat, and Metricbeat) to your containerized applications. These forwarders collect logs and metrics from your application and send them to Central Monitoring.

---

Contents

• Prerequisites
• Step 1: System Preparation
• Step 2: Authentication Setup
• Step 3: Network Setup
• Step 4: Deploy Forwarders
• Step 5: Verification
• Troubleshooting
• Maintenance
• Support

---

Automation Recommendation

While this guide provides bash commands for manual deployment, we recommend building and using an Ansible playbook if you're comfortable with automation tools. Ansible playbooks provide better repeatability, error handling, and configuration management for deploying the forwarders across multiple environments. Additionally, Ansible excels at working on multiple servers in parallel, making it ideal for pushing updates to all your servers at once rather than manually updating each server individually.

---

Prerequisites

Before deploying the forwarders, ensure your system meets these requirements:

System Requirements

• Docker installed and running
• Python 3 installed (/usr/bin/python3)
• Root access or sudo privileges
• Network connectivity to AWS ECR and Central Monitoring endpoints

Required Information

Provided by Central Monitoring team:

Variable	Description	Example
AWS_ACCOUNT_ID	AWS Account ID for ECR registry	123456789012
VERSION	Forwarder image version	always use latest

Required from your environment:

Variable	Description	Example
DOMAIN_NAME	Your application domain	my-client.my-app.com
CLIENT	Client (company) identifier	Coles
AWS_ENV	Environment (dev/staging/prod)	prod
REGION	AWS region	us-east-1

Pipeline Endpoints

Use the pipeline endpoints that were provided by the Central Monitoring team when your application was originally onboarded to Central Monitoring.

Example endpoint types:

• LOGS_PIPELINE - Primary logs endpoint
• METRICS_PIPELINE - Primary metrics endpoint
• LOGS_PIPELINE_EU - EU logs endpoint
• METRICS_PIPELINE_EU - EU metrics endpoint

EU Endpoints for GDPR Compliance

EU equivalent endpoints are always provided alongside the primary endpoints. The forwarders will automatically route data to the EU endpoints when it is identified as EU-related data for GDPR compliance purposes.

Finding Your Endpoints: If you don't have these endpoints readily available, contact the Central Monitoring team with your application details to retrieve the specific endpoints assigned during onboarding.

---

Step 1: System Preparation

Install Dependencies

# Install Python 3 (if not already installed)
sudo yum install python3 -y

# Install AWS CLI
python3 -m pip install awscli

# Install Docker Python library
python3 -m pip install docker

Verify Docker Installation

# Check Docker is running
sudo systemctl status docker

# Start Docker if not running
sudo systemctl start docker
sudo systemctl enable docker

---

Step 2: Authentication Setup

ECR Authentication

# Login to AWS ECR (replace with the AWS_ACCOUNT_ID provided by Central Monitoring)
aws ecr get-login-password --region us-east-1 | \
  docker login --username AWS --password-stdin \
  <AWS_ACCOUNT_ID>.dkr.ecr.ap-southeast-2.amazonaws.com

Cross-Account Access Required

The ECR repository containing the forwarder images is typically hosted in the Central Monitoring team's AWS account, which may be separate from your application's AWS account.

Before proceeding, you must:

1. Contact the Central Monitoring team to request access to the ECR repository
2. Provide your AWS account ID and the IAM role/user that will pull the images
3. Wait for confirmation that cross-account permissions have been granted

The Central Monitoring team will configure the necessary ECR repository policies to allow your AWS account to pull the forwarder images.

---

Step 3: Network Setup

Docker Networks

The forwarders need to be on the same Docker network as your application containers for several reasons:

• Container Discovery: Forwarders need to discover your application containers
• Data Collection: Forwarders need access to container logs and metadata
• Network Communication: The forwarders may need to communicate with your application for health checks and metrics collection
• Security: Isolating monitoring traffic within a dedicated network improves security

Create Docker Network

# Create the network that forwarders will use (or use your existing application network)
docker network create <YOUR_DOCKER_NETWORK>

Network Configuration

Replace <YOUR_DOCKER_NETWORK> with your application's Docker network name in all subsequent commands.

If your application containers are already running:

• Use the same network name as your application containers instead of creating a new one
• This ensures the forwarders can discover and monitor your application containers

To find your existing networks:

docker network ls

To see which network your application containers are using:

docker inspect <your-app-container-name> | grep NetworkMode

---

Step 4: Deploy Forwarders

Deployment Order Required

Deploy Fluent Bit first, then Filebeat and Metricbeat.

Filebeat and Metricbeat have a dependency on the Fluent Bit container (see FLUENTBIT_CONTAINER="fluent-bit" environment variable) and will fail to start properly if Fluent Bit is not already running.

Pipeline Endpoint and Image Usage

The values shown in the deployment commands below are examples only. Each application has its own unique values provided during onboarding:

Pipeline Endpoints:

• Replace <YOUR_LOGS_PIPELINE> with your actual logs endpoint URL
• Replace <YOUR_METRICS_PIPELINE> with your actual metrics endpoint URL
• Replace <YOUR_LOGS_PIPELINE_EU> with your actual EU logs endpoint URL
• Replace <YOUR_METRICS_PIPELINE_EU> with your actual EU metrics endpoint URL

Docker Images:

• Replace <AWS_ACCOUNT_ID> with the AWS Account ID provided by Central Monitoring
• Replace <FORWARDER_IMAGE_NAME> with the specific forwarder image name provided by Central Monitoring during onboarding
• The version is set to latest as recommended

Deploy Fluent Bit

Fluent Bit handles log forwarding and processing:

docker run -d \
  --name fluent-bit \
  --hostname fluent-bit \
  --network <YOUR_DOCKER_NETWORK> \
  --restart always \
  -e LOGS_PIPELINE="<YOUR_LOGS_PIPELINE>" \
  -e METRICS_PIPELINE="<YOUR_METRICS_PIPELINE>" \
  -e LOGS_PIPELINE_EU="<YOUR_LOGS_PIPELINE_EU>" \
  -e METRICS_PIPELINE_EU="<YOUR_METRICS_PIPELINE_EU>" \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  -v /var/lib/docker/containers/:/var/lib/docker/containers/:ro \
  --log-driver json-file \
  <AWS_ACCOUNT_ID>.dkr.ecr.ap-southeast-2.amazonaws.com/<FORWARDER_IMAGE_NAME>:latest \
  --config /fluent-bit/etc/fluent-bit.yaml

Deploy Filebeat

Filebeat collects and ships log files:

docker run -d \
  --name filebeat \
  --hostname filebeat \
  --network <YOUR_DOCKER_NETWORK> \
  --restart unless-stopped \
  --user root \
  --security-opt label:disable \
  -e PRIVATE_DNS="$(hostname)" \
  -e DOMAIN_NAME="<YOUR_DOMAIN_NAME>" \
  -e CLIENT="<YOUR_CLIENT>" \
  -e AWS_ENV="<YOUR_AWS_ENV>" \
  -e REGION="<YOUR_REGION>" \
  -v /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro \
  -v /var/lib/docker/containers:/var/lib/docker/containers:ro \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  --log-driver json-file \
  --log-opt max-file=5 \
  --log-opt max-size=10m \
  --label restart-group=client \
  <AWS_ACCOUNT_ID>.dkr.ecr.ap-southeast-2.amazonaws.com/<FORWARDER_IMAGE_NAME>:latest

Deploy Metricbeat

Metricbeat collects system and application metrics:

docker run -d \
  --name metricbeat \
  --hostname metricbeat \
  --network <YOUR_DOCKER_NETWORK> \
  --restart unless-stopped \
  --user root \
  -e FLUENTBIT_CONTAINER="fluent-bit" \
  -e PRIVATE_DNS="$(hostname)" \
  -e DOMAIN_NAME="<YOUR_DOMAIN_NAME>" \
  -e CLIENT="<YOUR_CLIENT>" \
  -e AWS_ENV="<YOUR_AWS_ENV>" \
  -e REGION="<YOUR_REGION>" \
  -v /var/run/docker.sock:/var/run/docker.sock:ro \
  -v /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro \
  -v /proc:/hostfs/proc:ro \
  -v /:/hostfs:ro \
  --log-driver json-file \
  <AWS_ACCOUNT_ID>.dkr.ecr.ap-southeast-2.amazonaws.com/<FORWARDER_IMAGE_NAME>:latest

---

Step 5: Verification

Wait for Stabilization

# Allow containers to start and stabilize
sleep 30

Check Container Status

# Verify all containers are running
docker ps --filter "name=fluent-bit" --filter "name=filebeat" --filter "name=metricbeat"

Expected output should show all three containers with "Up" status:

CONTAINER ID   IMAGE                                    COMMAND                  CREATED         STATUS         PORTS     NAMES
abc123def456   .../<FLUENT_BIT_IMAGE_NAME>:latest      "--config /fluent-bi…"   2 minutes ago   Up 2 minutes             fluent-bit
def456ghi789   .../<FILEBEAT_IMAGE_NAME>:latest        "/usr/local/bin/dock…"   2 minutes ago   Up 2 minutes             filebeat
ghi789jkl012   .../<METRICBEAT_IMAGE_NAME>:latest      "/usr/local/bin/dock…"   2 minutes ago   Up 2 minutes             metricbeat

Health Check Script

Create and run this health check script:

#!/bin/bash
echo "=== Container Health Check ==="

# Check container statuses
FLUENT_BIT_STATUS=$(docker ps --filter "name=fluent-bit" --format "{{.Status}}" | head -1)
FILEBEAT_STATUS=$(docker ps --filter "name=filebeat" --format "{{.Status}}" | head -1)
METRICBEAT_STATUS=$(docker ps --filter "name=metricbeat" --format "{{.Status}}" | head -1)

echo "Fluent Bit Status: $FLUENT_BIT_STATUS"
echo "Filebeat Status: $FILEBEAT_STATUS"
echo "Metricbeat Status: $METRICBEAT_STATUS"

# Check for failed containers
FAILED_CONTAINERS=$(docker ps --filter "status=restarting" --format "{{.Names}}" | grep -E "(fluent-bit|filebeat|metricbeat)" || true)

if [ ! -z "$FAILED_CONTAINERS" ]; then
    echo "Failed containers detected: $FAILED_CONTAINERS"
    exit 1
fi

# Verify all containers are running
RUNNING_COUNT=$(docker ps --filter "name=fluent-bit" --filter "name=filebeat" --filter "name=metricbeat" --format "{{.Names}}" | wc -l)

if [ "$RUNNING_COUNT" -ne 3 ]; then
    echo "Expected 3 containers running, found: $RUNNING_COUNT"
    docker ps --filter "name=fluent-bit" --filter "name=filebeat" --filter "name=metricbeat" --format "table {{.Names}}\t{{.Status}}\t{{.Image}}"
    exit 1
fi

echo "All forwarder containers are running successfully"

---

Troubleshooting

Common Issues

Container Won't Start

Check the container logs to identify why the container failed to start:

# Check container logs for startup errors
docker logs fluent-bit
docker logs filebeat
docker logs metricbeat

# Check for resource constraints
docker stats

Network Issues

# Verify network exists
docker network ls | grep <YOUR_DOCKER_NETWORK>

# Check network connectivity
docker network inspect <YOUR_DOCKER_NETWORK>

Permission Issues

# Ensure Docker socket permissions
ls -la /var/run/docker.sock

---

Maintenance

Updating Forwarders

To update to a new version:

1. Pull new images:

   docker pull <AWS_ACCOUNT_ID>.dkr.ecr.ap-southeast-2.amazonaws.com/<FLUENT_BIT_IMAGE_NAME>:latest
   docker pull <AWS_ACCOUNT_ID>.dkr.ecr.ap-southeast-2.amazonaws.com/<FILEBEAT_IMAGE_NAME>:latest
   docker pull <AWS_ACCOUNT_ID>.dkr.ecr.ap-southeast-2.amazonaws.com/<METRICBEAT_IMAGE_NAME>:latest

2. Stop and remove old containers:

   docker stop fluent-bit filebeat metricbeat
   docker rm fluent-bit filebeat metricbeat

3. Deploy new versions using the commands from Step 4 with updated version tags.

---

Support

If you encounter issues during deployment:

1. Review container logs for error messages
2. Verify network connectivity to Central Monitoring endpoints
3. Contact Central Monitoring team with:
   • Container logs
   • System information (uname -a, docker version)
   • Network configuration