Setting Up Alerts

Alert configuration is crucial for effective monitoring. This guide walks you through setting up notification channels and creating alert rules using the 9n9s web interface so you’re notified immediately when issues occur.

Overview

9n9s alerting works in two parts:

1. Notification Channels: Where alerts are sent (email, Slack, PagerDuty, etc.)
2. Alert Rules: When alerts are triggered and which channels to use

Step 1: Create Notification Channels

Email Notifications

1. Navigate to Notification Channels

   • Log into app.9n9s.com
   • Go to Organization Settings in the sidebar
   • Click “Notification Channels”

2. Add Email Channel

   • Click “Add Channel” button
   • Select “Email” from the channel types
   • Fill in the configuration:
     • Channel Name: “Operations Team”
     • Email Address: [email protected]
     • Template: Choose “Detailed” for comprehensive alerts
   • Click “Create Channel”

The email channel will now appear in your list and can be used in alert rules.

Slack Integration

Method 1: Slack App (Recommended)

1. Install Slack App

   • In 9n9s, go to Organization Settings > Notification Channels
   • Click “Add Channel” → “Slack”
   • Click “Install Slack App”
   • You’ll be redirected to Slack to authorize the app
   • Select the workspace and channels you want 9n9s to access
   • Return to 9n9s and your Slack channels will be available

2. Configure Channel Settings

   • Select the Slack channel for notifications
   • Choose message formatting options
   • Set up any channel-specific preferences
   • Click “Create Channel”

Method 2: Webhook

1. Create Slack Webhook

   • In your Slack workspace, go to your app settings
   • Create an “Incoming Webhook” integration
   • Copy the webhook URL

2. Add Webhook in 9n9s

   • In 9n9s, click “Add Channel” → “Slack Webhook”
   • Enter:
     • Channel Name: “Alerts Channel”
     • Webhook URL: Paste your Slack webhook URL
     • Channel: #alerts (optional, for display)
     • Username: 9n9s Monitor (optional)
   • Click “Create Channel”

PagerDuty Integration

1. Get PagerDuty Integration Key

   • In PagerDuty, go to Services → Select your service
   • Go to Integrations tab
   • Click “Add Integration” → “9n9s” (or “Generic API”)
   • Copy the Integration Key

2. Add PagerDuty Channel in 9n9s

   • Click “Add Channel” → “PagerDuty”
   • Enter:
     • Channel Name: “On-Call Team”
     • Integration Key: Paste your PagerDuty key
     • Routing Key: (if using Events API v2)
   • Click “Create Channel”

SMS Notifications

1. Add SMS Channel

   • Click “Add Channel” → “SMS”
   • Configure:
     • Channel Name: “Critical Alerts SMS”
     • Provider: Select Twilio
     • Phone Numbers: Add recipient numbers (one per line)
     • From Number: Your Twilio number
   • Click “Create Channel”

2. Twilio Configuration

   • You’ll need to provide your Twilio credentials in Organization Settings
   • Go to Organization Settings > Integrations > Twilio
   • Enter your Account SID and Auth Token

Step 2: Create Alert Rules

Basic Alert Rule for a Specific Monitor

1. Navigate to Alert Rules

   • Go to your project dashboard
   • Click “Alert Rules” in the navigation
   • Click “Create Alert Rule”

2. Configure Basic Rule

   • Rule Name: “Database Backup Failure”
   • Description: “Alert when daily backup fails”

3. Set Conditions

   • Monitor Selection: Choose “Specific Monitors”
   • Select your “Daily Database Backup” monitor
   • Status Changes: Check “Down” and “Failed”
   • Duration: Leave blank for immediate alerts

4. Configure Actions

   • Click “Add Action”
   • Notification Channel: Select “Operations Team” (email)
   • Delay: 0 minutes (immediate)
   • Priority: High

5. Save Rule

   • Review your configuration
   • Click “Create Alert Rule”

Tag-Based Alert Rules

Create rules that apply to multiple monitors based on tags:

1. Create Tag-Based Rule

   • Rule Name: “Production Service Alerts”
   • Description: “Alert for all production service failures”

2. Set Tag Conditions

   • Monitor Selection: Choose “Monitors with tags”
   • Tag Filters:
     • environment equals production
     • criticality equals high
   • Status Changes: Select “Down” and “Degraded”

3. Configure Multiple Actions

   • Action 1: Slack alert (immediate)
     • Channel: “Alerts Channel”
     • Delay: 0 minutes
   • Action 2: PagerDuty escalation
     • Channel: “On-Call Team”
     • Delay: 0 minutes

Escalation Rules

Set up progressive alerting with increasing urgency:

1. Create Escalation Rule

   • Rule Name: “Critical Service Escalation”
   • Monitor Selection: Tags where criticality equals critical

2. Configure Escalation Chain

   • Action 1: Slack notification
     • Delay: 0 minutes
     • Channel: “#alerts”
   • Action 2: Email notification
     • Delay: 5 minutes
     • Channel: “Operations Team”
   • Action 3: PagerDuty incident
     • Delay: 15 minutes
     • Channel: “On-Call Team”
   • Action 4: SMS alerts
     • Delay: 30 minutes
     • Channel: “Critical Alerts SMS”

3. Set Conditions

   • Only escalate if monitor is still down
   • Skip escalation if incident is acknowledged

Step 3: Advanced Alert Configuration

Alert Grouping

Prevent alert spam by grouping related alerts:

1. Enable Grouping in Rule Settings

   • In your alert rule, go to Advanced Settings
   • Enable “Group similar alerts”
   • Grouping Window: 5 minutes
   • Maximum Alerts: 5 alerts per group
   • Group By: Tags (environment, service)

2. Deduplication Settings

   • Enable “Deduplicate alerts”
   • Deduplication Window: 30 minutes
   • This prevents the same alert from being sent multiple times

Maintenance Windows

Suppress alerts during planned maintenance:

1. Schedule Maintenance

   • Go to Organization Settings > Maintenance Windows
   • Click “Schedule Maintenance”
   • Title: “Database Upgrade”
   • Start Time: Select date and time
   • Duration: 4 hours
   • Affected Monitors: Select specific monitors or use tags

2. Maintenance Settings

   • Suppress All Alerts: Yes
   • Notify Subscribers: Yes (for status page subscribers)
   • Automatic Status: Set monitors to “Maintenance” status

Conditional Alerting

Set up time-based and condition-based alerting:

1. Business Hours Alerting

   • In alert rule settings, enable “Time Windows”
   • Active Days: Monday-Friday
   • Active Hours: 9:00 AM - 5:00 PM
   • Timezone: America/New_York

2. After-Hours Critical Only

   • Create separate rule for after-hours
   • Time Windows: Evenings and weekends
   • Additional Conditions: Only criticality equals critical
   • Different Channels: Direct to on-call team

Step 4: Testing Your Alerts

Test Notification Channels

1. Test Individual Channels

   • Go to Organization Settings > Notification Channels
   • Find your channel in the list
   • Click the “Test” button next to it
   • Enter a test message
   • Choose severity level
   • Click “Send Test”

2. Verify Delivery

   • Check that the test message arrives
   • Verify formatting and content
   • Ensure links work correctly

Simulate Monitor Failures

For Heartbeat Monitors:

1. Temporary Failure Simulation

   • Go to your monitor’s detail page
   • Click “Simulate” → “Failure”
   • This sends a test failure pulse
   • Watch for alerts in your channels

2. Late Pulse Simulation

   • Click “Simulate” → “Late”
   • This simulates missing a scheduled pulse
   • Alerts should trigger after the grace period

For Uptime Monitors:

1. Edit Monitor Temporarily

   • Go to monitor settings
   • Change URL to a non-existent endpoint
   • Save changes
   • Wait for next check to fail
   • Restore correct URL afterward

2. Modify Assertions

   • Temporarily change expected status code
   • Or modify content assertions to fail
   • Monitor next check results

Verify Alert Delivery and Recovery

1. Check Alert History

   • Go to Project Dashboard > Alerts
   • View recent alert activity
   • Verify alerts were sent to correct channels

2. Test Recovery Notifications

   • After simulating failure, restore monitor to working state
   • Verify that recovery notifications are sent
   • Check that alert is marked as resolved

Common Alert Patterns

Production Environment Setup

1. Critical Services

   • Monitors: All monitors tagged with criticality=critical
   • Immediate Actions: Slack + PagerDuty
   • Escalation: SMS after 15 minutes if unacknowledged

2. Important Services

   • Monitors: All monitors tagged with criticality=high
   • Actions: Slack immediate, email after 5 minutes
   • Business Hours: Different channels for business vs after hours

Development and Staging

1. Development Environment

   • Monitors: All monitors tagged with environment=development
   • Actions: Email only, 10-minute delay
   • Quiet Hours: No alerts during off-hours

2. Staging Environment

   • Monitors: All monitors tagged with environment=staging
   • Actions: Slack alerts during business hours
   • Pre-production: Higher priority during testing periods

Team-Based Routing

1. Backend Team Alerts

   • Tag Filter: team=backend or service=api
   • Channel: Backend team Slack channel
   • Escalation: Team lead email after 10 minutes

2. Infrastructure Team Alerts

   • Tag Filter: team=infrastructure or component=database
   • Channel: Infrastructure Slack + email
   • Escalation: On-call rotation via PagerDuty

Best Practices

Alert Rule Organization

1. Use Descriptive Names: “Production API Failures” not “Rule 1”
2. Add Descriptions: Explain what the rule does and why
3. Regular Review: Monthly review of alert rules and their effectiveness
4. Test Regularly: Quarterly testing of alert delivery

Preventing Alert Fatigue

1. Start Conservative: Begin with critical monitors only
2. Gradual Expansion: Add more monitors as you tune thresholds
3. Use Escalation: Not everything needs immediate PagerDuty alerts
4. Monitor Alert Volume: Review alert frequency in dashboard

Effective Alert Content

1. Clear Subject Lines: Include service name and status
2. Actionable Information: Include links to dashboards and runbooks
3. Context: Environment, duration, affected users
4. Next Steps: What should the recipient do first?

Troubleshooting Alerts

Common Issues

Alerts Not Being Sent

1. Check Rule Status: Ensure alert rule is enabled
2. Verify Conditions: Check if conditions match actual monitor state
3. Test Channels: Use test feature to verify channel connectivity
4. Review Maintenance: Check for active maintenance windows

Too Many Alerts

1. Adjust Thresholds: Increase grace periods for flappy monitors
2. Enable Grouping: Group similar alerts together
3. Review Sensitivity: Check if monitors are too sensitive
4. Use Filtering: Add more specific tag filters

Missing Critical Alerts

1. Check Coverage: Ensure all critical monitors have alert rules
2. Verify Tagging: Confirm monitors have correct tags
3. Test Rules: Use simulation to test rule execution
4. Review Channels: Ensure notification channels are working

Getting Help

If you’re still having issues with alerts:

1. Check Status Page: Visit status.9n9s.com for service status
2. Contact Support: Use the chat widget or email [email protected]
3. Community: Join our Discord community for peer support
4. Documentation: Check the Advanced Alerting Reference

CLI Reference

For automation and advanced configurations, you can also manage alerts via CLI:

• CLI Alert Commands
• Configuration as Code
• API Reference

Next Steps

Once your alerts are configured:

1. Set up your team: Add team members and configure permissions
2. Create status pages: Provide transparency to your users
3. Explore integrations: Connect with your existing tools
4. Advanced alerting: Learn about complex alert configurations

Effective alerting is the foundation of reliable monitoring. Start with simple rules for your most critical services and gradually expand your alerting strategy as your team’s needs evolve.