Skip to main content

Structured Logging

Structured logging is a practice of formatting log messages as structured data (JSON) rather than plain text. This makes logs machine-readable and enables powerful querying, filtering, and analysis capabilities in Central Monitoring.

Contents​

Why Structured Logging?​

Benefits:

  • Searchable and filterable - Query specific fields across all logs
  • Consistent format - Standardized structure across all applications
  • Better analytics - Enables dashboards, alerts, and metrics
  • Debugging efficiency - Quickly locate issues using structured queries

Required Log Format​

Logs must be in JSON (structured) format to be searchable, for example:

{
  "timestamp": "2024-10-24T10:30:00.000Z",
  "level": "INFO",
  "message": "User authentication successful",
  "service": "auth-service",
  "userId": "12345",
  "requestId": "req-abc-123",
  "duration": 150
}

What is NOT Structured Logging​

Many developers mistakenly think that putting JSON in the log message makes it structured. This is incorrect.

❌ Incorrect - JSON in Message Only​

Log Output:

2024-10-24T10:30:00.000Z INFO {"userId": "12345", "action": "login", "status": "success"}

Problems with this approach:

  • The log is plain text, not structured data
  • Log parsing tools cannot automatically extract fields
  • Cannot query or filter on individual fields (userId, action, status) without complex and costly text parsing

✅ Correct - Proper Structured Logging​

Log Output:

{
  "timestamp": "2024-10-24T10:30:00.000Z",
  "level": "INFO",
  "message": "User login successful",
  "service": "my-app",
  "userId": "12345",
  "action": "login",
  "status": "success"
}

Why this works:

  • Each field is a separate, queryable property
  • Can filter logs by userId = "12345"
  • Can aggregate all action = "login" events
  • Enables powerful analytics and monitoring
warning

DEBUG logs are not ingested to reduce volume and costs. Use INFO level and above for important operational information.

Examples by Language​

Node.js (Winston)​

const winston = require('winston');

const logger = winston.createLogger({
  format: winston.format.json(),
  defaultMeta: { service: 'my-app' },
  transports: [
    new winston.transports.Console()
  ]
});

logger.info('User login attempt', {
  userId: '12345',
  email: 'user@example.com',
  requestId: 'req-abc-123'
});

Python (structlog)​

import structlog

logger = structlog.get_logger()

logger.info(
    "User login attempt",
    user_id="12345",
    email="user@example.com",
    request_id="req-abc-123"
)

Java (Logback with JSON)​

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import net.logstash.logback.marker.Markers;

Logger logger = LoggerFactory.getLogger(MyClass.class);

logger.info(Markers.append("userId", "12345")
    .and(Markers.append("requestId", "req-abc-123")), 
    "User login attempt");

Best Practices​

  1. Use consistent field names across all services
  2. Include correlation IDs (requestId, traceId) for request tracking
  3. Add contextual information (userId, sessionId, etc.)
  4. Avoid sensitive data in logs (passwords, tokens, PII)
  5. Use appropriate log levels based on the severity
  6. Include error details in ERROR level logs (stack traces, error codes)

Data Type Consistency​

Important

All fields with the same name across different services must have the same data type. For example:

  • If userId is a string in one service, it must be a string in all services
  • If duration is a number in milliseconds, use the same format everywhere