Setting Up Alerts
To set up alerting for your application, you'll need to define the specific conditions you want to monitor and configure how you want to be notified when those conditions are met.
Contents
- Prerequisites
- Step 1: Identify What to Monitor
- Step 2: Define Alert Rules
- Step 3: Submit Alert Request
- Step 4: EventOps Configuration
- Step 5: Testing and Validation
- Getting Help
Prerequisites
Before setting up alerts, ensure:
- Your application is properly onboarded to Central Monitoring
- Your application logs and metrics are being ingested and are visible in Central Monitoring
- You have access to EventOps for notification configuration
Step 1: Identify What to Monitor
Review your application logs and metrics in Central Monitoring to identify:
Fields to Monitor
Examine the structured fields in your logs and metrics. Common examples are:
level(ERROR, WARN, INFO, etc.)status_code(HTTP response codes)response_time(API response times)error_message(specific error text)component(application component or service)
Patterns to Detect
Consider what conditions indicate problems. Common examples are:
- Specific error messages or exception types
- HTTP status codes (4xx, 5xx errors)
- Performance thresholds (response times > 5 seconds)
- Missing expected log entries
- Unusual frequency of events
Step 2: Define Alert Rules
For each alert you want to create, prepare the following information using this template structure:
Alert Request Template
Copy and paste this template into your email to CM-Support@accenture.com:
Subject: Alert Setup Request - [Alert Name] - [Your Application Name]
Application: [Your application name]
ALERT DETAILS:
Field Name: [Metric/Field Name]
Log/Metric/Other: [Is the field from logs, metrics, or other]
Description: [What this metric measures and why it's important]
Trigger Condition: [Specific condition and threshold that triggers the alert, including time window and action to take]
RESPONSE TEAM AND ACTIONS:
Team: [Team Name]
Actions:
1. [First troubleshooting step]
2. [Second troubleshooting step]
3. [Additional steps as needed]
Example Alert Request
Subject: Alert Setup Request - API 5xx Error Alert - MyApplication
Application: MyApplication
ALERT DETAILS:
Field Name: api.status_code
Log/Metric/Other: Log
Description: HTTP status codes from API responses to detect server errors
Trigger Condition: When api.status_code is 5xx (500-599) and occurs more than once in 5 minutes
RESPONSE TEAM AND ACTIONS:
Team: Support Team
Actions:
1. Check application logs for specific error messages and stack traces
2. Verify database connectivity and external service dependencies
3. Check server resource utilization (CPU, memory, disk space)
4. Review recent deployments or configuration changes
Another Example: API Integration Failure
Subject: Alert Setup Request - API Integration Failure Alert - MyApplication
Application: MyApplication
ALERT DETAILS:
Field Name: error.message
Log/Metric/Other: Log
Description: When external API calls keep failing due to configuration issues
Trigger Condition: When error.message equals "The API deployment for this resource does not exist"
RESPONSE TEAM AND ACTIONS:
Team: Support Team
Actions:
1. Send email to relevant support team with the instance info for the project team to check their API configuration
Step 3: Submit Alert Request
Send your completed alert definition to CM-Support@accenture.com using the template format shown above in the "Alert Request Template" section.
Step 4: EventOps Configuration
Once the CM team sets up the alert rule, you'll need to configure the notification handling in EventOps:
Tasks You Configure in EventOps:
- Email Recipients: Add team members who should receive alert notifications
- Notification Preferences: Set email frequency and formatting
- Ticket Creation Rules: Configure when to automatically create ServiceNow tickets
Tasks Configured by Central Monitoring:
- Alert rule logic and field mapping
- Data source connections
- Alert payload structure
- EventOps Integration
Step 5: Testing and Validation
After setup:
- Test Alert Generation: Verify alerts trigger under expected conditions
- Validate Notifications: Confirm emails are delivered to correct recipients
- Check Alert Content: Ensure alert messages contain useful troubleshooting information
- Adjust Thresholds: Fine-tune sensitivity based on initial results
Getting Help
For questions about alert setup or to request modifications:
- Email: CM-Support@accenture.com
- Include your application name and specific alert rule in question