Email Notifications

Email notifications provide a reliable, universal way to receive 9n9s alerts. This guide covers email setup, template customization, and delivery optimization to ensure your team receives critical alerts promptly.

Quick Setup

1. Add Email Channel

   • Go to Organization Settings > Notification Channels
   • Click “Add Channel” → “Email”
   • Enter email address and configure options
   • Click “Create Channel”

2. Test Delivery

   • Click “Send Test Email”
   • Check your inbox (and spam folder)
   • Verify formatting and content

3. Use in Alert Rules

   • Create or edit alert rules
   • Add your email channel as an action
   • Configure timing and conditions

Prerequisites

Email Account Requirements

• Valid Email Address: Any email provider (Gmail, Outlook, corporate email)
• Inbox Access: Ability to receive emails and manage filters
• Spam Management: Access to whitelist 9n9s sending addresses
• Mobile Compatibility: Email client should support HTML formatting

9n9s Requirements

• Organization Access: Admin or member role with notification permissions
• Active Subscription: Email notifications included in all plans
• Proper Configuration: Valid email addresses and delivery settings

Setting Up Email Notifications

Step 1: Add Email Channel

1. Navigate to Notification Channels

   • Log into app.9n9s.com
   • Go to Organization Settings in the sidebar
   • Click “Notification Channels”

2. Create Email Channel

   • Click “Add Channel” button
   • Select “Email” from the channel types
   • This opens the email configuration form

3. Basic Configuration

   • Channel Name: “Operations Team Email” (descriptive name for internal use)
   • Email Address: [email protected] (recipient email address)
   • Display Name: “Operations Team” (appears in alert rules)
   • Description: Optional description of channel purpose

Step 2: Configure Email Settings

1. Message Template

   • Template Type: Choose from predefined templates:
     • Minimal: Basic alert information only
     • Standard: Balanced detail and brevity (recommended)
     • Detailed: Comprehensive alert information
     • Custom: Create your own template

2. Formatting Options

   • HTML Formatting: Enable rich HTML emails (recommended)
   • Plain Text Fallback: Ensure compatibility with basic email clients
   • Subject Line Format: Customize email subject structure
   • Priority Indicators: Use email priority flags for critical alerts

3. Advanced Settings

   • Rate Limiting: Control email frequency (default: 10 per hour)
   • Quiet Hours: Suppress non-critical emails during specified times
   • Grouping: Combine multiple alerts into digest emails
   • Retry Settings: Configure retry attempts for failed deliveries

Step 3: Test and Verify

1. Send Test Email

   • Click “Send Test Email” in the channel configuration
   • Choose test alert type (Info, Warning, Critical)
   • Check your email within 1-2 minutes

2. Verify Delivery

   • Check inbox and spam/junk folders
   • Verify HTML formatting displays correctly
   • Test links to monitor dashboards
   • Confirm mobile email client compatibility

3. Whitelist Sender

   • Add [email protected] to your email whitelist
   • Add [email protected] to safe senders
   • Configure email filters if needed

Email Templates

Template Options

Minimal Template:

Subject: [9n9s] API Health Check - DOWN

Monitor: API Health Check
Status: DOWN (was UP)
Duration: 5 minutes

View Monitor: https://app.9n9s.com/monitors/mon_123

Standard Template:

Subject: [9n9s Alert] API Health Check is DOWN

Monitor Details:
• Name: API Health Check
• Status: DOWN (was UP)
• Project: Production Services
• Duration: 5 minutes
• Environment: production

Incident Information:
• Started: 2024-01-15 at 14:30 UTC
• Last Check: 14:29 UTC (successful)
• Response Code: 500 Internal Server Error
• Response Time: 5000ms

Actions:
• View Monitor: https://app.9n9s.com/monitors/mon_123
• View Logs: https://app.9n9s.com/monitors/mon_123/logs
• Create Incident: https://app.9n9s.com/incidents/new?monitor=mon_123

This is an automated message from 9n9s Monitoring.

Detailed Template:

Subject: [CRITICAL] API Health Check - DOWN for 5 minutes

Alert Summary:
Monitor "API Health Check" in project "Production Services"
has been DOWN for 5 minutes.

Monitor Information:
• Name: API Health Check
• Type: Uptime Monitor
• URL: https://api.company.com/health
• Status: DOWN (previously UP)
• Project: Production Services
• Environment: production
• Criticality: high

Incident Timeline:
• 14:30 UTC - Monitor failed (HTTP 500)
• 14:29 UTC - Last successful check (HTTP 200, 245ms)
• 14:28 UTC - Successful check (HTTP 200, 198ms)
• 14:27 UTC - Successful check (HTTP 200, 223ms)

Check Details:
• Response Code: 500 Internal Server Error
• Response Time: 5000ms (threshold: 2000ms)
• Error Message: "Database connection timeout"
• Check Region: us-east-1

Monitor Configuration:
• Check Frequency: Every 1 minute
• Timeout: 10 seconds
• Assertions: Status Code = 200, Response Time < 2000ms

Tags:
• environment: production
• service: api
• team: backend
• criticality: high

Recent Logs:
[14:30:15] ERROR: HTTP 500 - Database connection timeout
[14:29:45] SUCCESS: HTTP 200 - 245ms response time
[14:29:30] SUCCESS: HTTP 200 - 198ms response time

Suggested Actions:
1. Check database connectivity and performance
2. Review application logs for errors
3. Verify load balancer health
4. Contact on-call engineer if issue persists

Quick Actions:
• View Monitor Dashboard: https://app.9n9s.com/monitors/mon_123
• View Detailed Logs: https://app.9n9s.com/monitors/mon_123/logs
• Acknowledge Alert: https://app.9n9s.com/alerts/alert_456/ack
• Create Incident: https://app.9n9s.com/incidents/new?monitor=mon_123
• Snooze Alerts: https://app.9n9s.com/monitors/mon_123/snooze

Organization: Company Inc
Project: Production Services
Generated: 2024-01-15 14:30 UTC

This is an automated alert from 9n9s Monitoring System.
To unsubscribe or modify notification settings, visit:
https://app.9n9s.com/settings/notifications

Custom Template Variables

Use these variables in custom email templates:

Monitor Information:

• {{monitor.name}} - Monitor name
• {{monitor.type}} - Heartbeat or Uptime
• {{monitor.status}} - Current status
• {{monitor.previous_status}} - Previous status
• {{monitor.url}} - Monitor URL (for uptime monitors)

Project and Organization:

• {{project.name}} - Project name
• {{organization.name}} - Organization name
• {{environment}} - Environment from tags

Incident Details:

• {{incident.started_at}} - When incident started
• {{incident.duration}} - How long monitor has been down
• {{incident.description}} - Incident description

Alert Context:

• {{alert.severity}} - Alert severity level
• {{alert.timestamp}} - When alert was generated
• {{alert.rule_name}} - Alert rule that triggered

Tags and Metadata:

• {{tags.environment}} - Environment tag
• {{tags.team}} - Team tag
• {{tags.service}} - Service tag
• {{metadata.response_code}} - HTTP response code
• {{metadata.response_time}} - Response time

Links:

• {{links.monitor}} - Direct link to monitor
• {{links.logs}} - Link to monitor logs
• {{links.project}} - Link to project dashboard

Advanced Configuration

Multiple Email Addresses

Distribution Lists:

Email Address: [email protected]
Description: Distribution list reaching:
• [email protected]
• [email protected]
• [email protected]

Individual Recipients:

Channel 1: [email protected] (Backend Lead)
Channel 2: [email protected] (DevOps Engineer)
Channel 3: [email protected] (Operations Team)

Email Routing Rules

Configure different email channels for different alert types:

Critical Alerts:

Channel: "Critical Alerts Email"
Email: [email protected]
Conditions: criticality = critical
Template: Detailed
Subject: [CRITICAL] {{monitor.name}} - {{monitor.status}}

Team-Specific Alerts:

Channel: "Backend Team Email"
Email: [email protected]
Conditions: team = backend
Template: Standard
Subject: [Backend] {{monitor.name}} - {{monitor.status}}

Environment-Based Routing:

Production Alerts:
• Email: [email protected]
• Template: Detailed
• Subject: [PROD] {{monitor.name}} - {{monitor.status}}

Development Alerts:
• Email: [email protected]
• Template: Minimal
• Subject: [DEV] {{monitor.name}} - {{monitor.status}}

Digest and Grouping

Alert Grouping Settings:

• Group Similar Alerts: Combine alerts from same service
• Grouping Window: 5-minute window for combining alerts
• Maximum Group Size: 5 alerts per email
• Digest Schedule: Send grouped alerts every 15 minutes

Digest Email Format:

Subject: [9n9s Digest] 3 alerts in Production Services

Alert Summary (Last 15 minutes):
1. API Health Check - DOWN (5 minutes)
2. Database Connection - DEGRADED (3 minutes)
3. Cache Service - DOWN (2 minutes)

Detailed Alerts:
[Content of individual alerts...]

Email Client Optimization

HTML Email Support

Responsive Design:

• Emails automatically adapt to screen size
• Mobile-optimized layout and buttons
• Readable fonts and appropriate spacing
• Touch-friendly action buttons

Client Compatibility:

• Gmail (web, mobile, desktop)
• Outlook (all versions)
• Apple Mail (macOS, iOS)
• Thunderbird and other clients

Plain Text Fallback

For email clients that don’t support HTML:

[9n9s Alert] API Health Check - DOWN

Monitor: API Health Check
Status: DOWN (was UP)
Duration: 5 minutes
Project: Production Services

Incident started: 2024-01-15 14:30 UTC
Response code: 500 Internal Server Error
Response time: 5000ms

View monitor: https://app.9n9s.com/monitors/mon_123
View logs: https://app.9n9s.com/monitors/mon_123/logs

Tags: environment=production, service=api, team=backend

This is an automated alert from 9n9s.

Email Filters

Set up email filters for better organization:

Gmail Filter Example:

Filter Criteria:
• From: [email protected]
• Subject contains: [CRITICAL]

Actions:
• Apply label: "9n9s-Critical"
• Star the message
• Mark as important
• Forward to: [email protected]

Outlook Rule Example:

Rule Conditions:
• Sender: [email protected]
• Subject contains: "DOWN"

Actions:
• Move to folder: "Monitoring/Critical"
• Set importance: High
• Play sound: Alert.wav

Delivery Optimization

Avoiding Spam Filters

Best Practices:

1. Whitelist Senders: Add 9n9s email addresses to safe senders
2. Corporate Email: Use corporate email addresses when possible
3. Consistent Formatting: Use standard email templates
4. Valid SPF/DKIM: 9n9s emails are properly authenticated

SPF Record for Allowlisting:

9n9s sending IPs are included in SPF records for:
• [email protected]
• [email protected]
• [email protected]

Rate Limiting and Throttling

Rate Limit Configuration:

• Default: 10 emails per hour per channel
• Burst Handling: Allow up to 5 emails in 5 minutes
• Critical Override: Critical alerts bypass rate limits
• Escalation: Rate limits increase for sustained incidents

Throttling Options:

Conservative: 5 emails/hour (low-volume alerts)
Standard: 10 emails/hour (recommended)
Aggressive: 20 emails/hour (high-volume monitoring)
No Limit: Unlimited (critical services only)

Delivery Monitoring

Delivery Tracking:

• Success Rate: Track successful email deliveries
• Bounce Handling: Automatic retry for soft bounces
• Hard Bounce Management: Disable channels with permanent failures
• Delivery Time: Monitor email delivery latency

Email Delivery Logs:

[14:30:15] Email sent to [email protected] - Success (250ms)
[14:30:16] Email sent to [email protected] - Bounced (Invalid address)
[14:30:17] Email sent to [email protected] - Deferred (Mailbox full)
[14:30:45] Email retry to [email protected] - Success (180ms)

Security and Privacy

Email Security

Encryption:

• All emails sent via TLS/SSL encryption
• Email content encrypted at rest in 9n9s systems
• Secure email headers and authentication

Data Protection:

• Minimal sensitive data in email content
• Links to detailed information instead of inline data
• Configurable data retention for email logs

Access Control

Channel Permissions:

• Organization admins can create/modify email channels
• Project members can use assigned email channels
• Audit logging for all email channel changes

Email Address Management:

• Validate email addresses before adding
• Regular cleanup of invalid/bounced addresses
• Team member approval for sensitive distribution lists

Troubleshooting

Common Issues

Emails Not Received:

1. Check Spam/Junk Folders

   • Look for emails from [email protected]
   • Check quarantine folders in enterprise email systems
   • Verify corporate email security hasn’t blocked messages

2. Verify Email Address

   • Confirm email address is correctly entered
   • Test with a simple personal email address
   • Check for typos in domain names

3. Delivery Logs

   • Check email delivery status in 9n9s notification logs
   • Look for bounce messages or delivery errors
   • Review rate limiting and throttling settings

Formatting Issues:

1. HTML Not Displaying

   • Check if email client supports HTML
   • Verify HTML email option is enabled in 9n9s
   • Test with different email clients

2. Broken Links

   • Ensure organization URLs are correctly configured
   • Test links in test emails before deploying
   • Check for URL encoding issues

Performance Issues:

1. Delayed Emails

   • Check 9n9s system status page
   • Review email server response times
   • Verify no rate limiting is causing delays

2. Missing Emails During High Volume

   • Check rate limiting settings
   • Review bounce and delivery failure logs
   • Consider using multiple email addresses for load distribution

Getting Help

Support Resources:

1. 9n9s Support: Use in-app chat or email [email protected]
2. Email Provider Documentation: Check your email provider’s help docs
3. System Status: Monitor status.9n9s.com for email delivery issues
4. Community: Join our Discord for peer support

Best Practices

Email Management

1. Use Distribution Lists

   • Create team-based distribution lists
   • Keep individual email addresses as backups
   • Regularly update distribution list membership

2. Template Selection

   • Use minimal templates for high-frequency alerts
   • Use detailed templates for critical alerts only
   • Customize templates for specific team needs

3. Organization

   • Set up email filters for automatic organization
   • Use consistent subject line prefixes
   • Create folders for different alert types

Alert Hygiene

1. Avoid Email Fatigue

   • Use appropriate rate limiting
   • Escalate to email only after chat notifications
   • Regularly review and tune alert sensitivity

2. Clear Action Items

   • Include specific next steps in email content
   • Provide direct links to relevant dashboards
   • Use clear, non-technical language when possible

3. Escalation Paths

   • Document clear escalation procedures
   • Include backup contact information
   • Set up out-of-office handling procedures

---

Email notifications provide a reliable foundation for your alerting strategy. Configure multiple email channels for different teams and alert types, use appropriate templates for different situations, and ensure proper delivery optimization for critical alerts.