MeetNotify - Cloud-Grade Notification Platform System Design

Version: 2.0
Target Scale: 100K-10M events/day (Medium-Large, Startup Production)
Architecture Type: Multi-tenant SaaS Platform
Deployment Model: Cloud-native, horizontally scalable

Executive Summary

This document outlines the architecture for MeetNotify, a cloud-grade, multi-tenant notification platform designed to handle event-driven notification delivery at startup production scale. Unlike application-level notification systems, MeetNotify operates as infrastructure - a distributed event bus with delivery guarantees across multiple channels.

Core Capabilities:

• Multi-channel delivery (Email, SMS, Push, Webhook, In-app)
• Topic-based pub/sub for microservices
• Multi-tenant isolation with per-tenant rate limits
• At-least-once delivery guarantees
• Retry logic with exponential backoff
• Template engine with localization
• User preference management
• Observability and delivery tracking

1. Architecture Principles

1.1 Design Philosophy

Events, Not Notifications
At infrastructure level, we deal with events that have delivery contracts, not just "notifications."

Separation of Concerns

• Event ingestion ≠ Event processing ≠ Delivery
• Each layer scales independently
• Failures in one channel don't affect others

Multi-tenant First

• Tenant isolation by design
• Per-tenant metrics, limits, and billing
• Noisy neighbor protection

Delivery Guarantees

• At-least-once delivery (may duplicate)
• Per-channel idempotency
• Ordered within partition, not globally

1.2 Non-Goals (Scope Boundaries)

• ❌ Exactly-once delivery (too expensive for this scale)
• ❌ Global ordering (infeasible at scale)
• ❌ Synchronous delivery (all async)
• ❌ Message content filtering/transformation at scale

2. High-Level Architecture

┌─────────────────────────────────────────────────────────────────────────┐
│                          Client Applications                             │
│                 (Tenant Services, Microservices, APIs)                   │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                         API Gateway / Load Balancer                      │
│                   (Auth, Rate Limiting, Tenant Routing)                  │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                          Ingestion Service                               │
│              (Event Validation, Schema Registry, Publishing)             │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                                    ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                         Event Bus (Kafka/Pulsar)                         │
│                    Partitioned by tenant_id + user_id                    │
│                         Retention: 7 days                                │
└─────────────────────────────────────────────────────────────────────────┘
                                    │
                    ┌───────────────┴───────────────┐
                    ▼                               ▼
┌─────────────────────────────────┐   ┌─────────────────────────────────┐
│   Orchestration Service         │   │   Subscription Service          │
│   (Consumer Group)              │   │   (Topic → Subscriber Mapping)  │
│                                 │   │                                 │
│   - Fanout expansion            │◄──┤   - Manages subscriptions      │
│   - Recipient resolution        │   │   - Topic filtering             │
│   - Preference application      │   │   - Permission checks           │
│   - Channel routing             │   │                                 │
└─────────────────────────────────┘   └─────────────────────────────────┘
                    │
        ┌───────────┼───────────┬───────────┬───────────┐
        ▼           ▼           ▼           ▼           ▼
┌─────────────┐ ┌────────┐ ┌────────┐ ┌─────────┐ ┌────────────┐
│Email Worker │ │SMS     │ │Push    │ │Webhook  │ │In-App      │
│Pool         │ │Worker  │ │Worker  │ │Worker   │ │Worker      │
│             │ │Pool    │ │Pool    │ │Pool     │ │Pool        │
│SendGrid/SES │ │Twilio  │ │FCM/APNS│ │HTTP POST│ │WebSocket   │
└─────────────┘ └────────┘ └────────┘ └─────────┘ └────────────┘
        │           │           │           │           │
        └───────────┴───────────┴───────────┴───────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                      Delivery State Store                                │
│                    (Cassandra / DynamoDB / ScyllaDB)                     │
│                                                                          │
│  - Delivery status tracking                                              │
│  - Retry state management                                                │
│  - Idempotency keys                                                      │
│  - Deduplication window                                                  │
└─────────────────────────────────────────────────────────────────────────┘
                                │
                                ▼
┌─────────────────────────────────────────────────────────────────────────┐
│                        Dead Letter Queue (DLQ)                           │
│                    (Failed deliveries after max retries)                 │
└─────────────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────────────┐
│                        Supporting Services                               │
│                                                                          │
│  ┌──────────────┐  ┌──────────────┐  ┌────────────────┐                │
│  │ Preference   │  │ Template     │  │ Analytics      │                │
│  │ Service      │  │ Service      │  │ Service        │                │
│  └──────────────┘  └──────────────┘  └────────────────┘                │
│                                                                          │
│  ┌──────────────┐  ┌──────────────┐  ┌────────────────┐                │
│  │ Tenant       │  │ Rate Limiter │  │ Observability  │                │
│  │ Management   │  │ Service      │  │ (Metrics/Logs) │                │
│  └──────────────┘  └──────────────┘  └────────────────┘                │
└─────────────────────────────────────────────────────────────────────────┘

3. Core Components Deep Dive

3.1 Ingestion Service

Responsibility: Accept events from tenant applications, validate, enrich, and publish to event bus.

API Endpoints:

# Publish single event
POST /v1/events
Content-Type: application/json
Authorization: Bearer <tenant_api_key>

{
  "event_type": "order.created",
  "user_id": "user_123",
  "data": {
    "order_id": "order_456",
    "amount": 99.99
  },
  "metadata": {
    "idempotency_key": "unique_key_123",
    "priority": "high"
  }
}

# Publish batch events (up to 100)
POST /v1/events/batch

# Subscribe to topic (for pub/sub)
POST /v1/subscriptions
{
  "topic": "order.created",
  "endpoint": "https://service.example.com/webhook",
  "filter": {
    "amount": { "gt": 50 }
  }
}

# Publish to topic (fanout to all subscribers)
POST /v1/topics/{topic_name}/publish

Key Features:

• Schema validation against tenant-defined schemas
• Idempotency key checking (24-hour window)
• Rate limiting per tenant (configurable)
• Event enrichment (timestamps, tenant_id, trace_id)
• Async acknowledgment (returns event_id immediately)

Technology:

• Language: Go (high throughput, low latency)
• Framework: Fiber or Gin
• Validation: JSON Schema
• Rate Limiting: Redis Token Bucket

Scaling:

• Stateless service, horizontally scalable
• 1 instance can handle ~5,000 req/s
• For 10M events/day = ~115 events/s average, ~1,000 peak
• Deploy 3-5 instances for redundancy

3.2 Event Bus (Kafka)

Why Kafka:

• Durable, replicated log
• High throughput (millions of messages/sec)
• Consumer groups for parallel processing
• Partition-level ordering guarantees
• Industry-proven for event streaming

Topic Structure:

events.raw               # All incoming events (main topic)
deliveries.email        # Email delivery tasks
deliveries.sms          # SMS delivery tasks
deliveries.push         # Push notification tasks
deliveries.webhook      # Webhook delivery tasks
deliveries.inapp        # In-app notification tasks
dlq.email               # Failed email deliveries
dlq.sms                 # Failed SMS deliveries
dlq.push                # Failed push deliveries
dlq.webhook             # Failed webhook deliveries

Partitioning Strategy:

Primary partition key: hash(tenant_id + user_id)

Why this approach:

• Maintains ordering per user
• Distributes load across partitions
• Prevents single tenant from monopolizing partition
• Enables partition-level consumer scaling

Configuration:

topics:
  events.raw:
    partitions: 30
    replication_factor: 3
    retention_ms: 604800000  # 7 days
    compression_type: lz4
  
  deliveries.*:
    partitions: 20
    replication_factor: 3
    retention_ms: 259200000  # 3 days

Throughput Calculation:

• Target: 10M events/day
• Peak: 5x average = 578 events/sec
• With 30 partitions = ~20 events/sec per partition
• Well within Kafka limits (thousands/sec per partition)

3.3 Orchestration Service

Responsibility: Consume raw events, expand fanout, apply preferences, route to delivery channels.

Core Logic Flow:

1. Consume event from events.raw
2. Lookup event_type in subscription registry
3. Resolve all recipients:
   - Direct user_id from event
   - Topic subscribers (pub/sub model)
   - Follower fanout (if applicable)
4. For each recipient:
   a. Check tenant rate limits
   b. Lookup user preferences
   c. Apply quiet hours / digest mode
   d. Determine active channels
   e. Render templates (if needed)
5. Publish delivery tasks to channel-specific topics
6. Commit Kafka offset

Fanout Explosion Protection:

const (
    MAX_FANOUT_PER_EVENT = 10_000
    MAX_FANOUT_PER_TENANT_PER_MINUTE = 100_000
)

If fanout exceeds limits:

• Break into batches
• Rate-limit publishing
• Track in metrics
• Alert on excessive fanout

Data Enrichment:

{
  "delivery_id": "uuid",
  "event_id": "original_event_uuid",
  "tenant_id": "tenant_123",
  "user_id": "user_456",
  "channel": "email",
  "template_id": "order_confirmation",
  "template_data": { ... },
  "priority": "high",
  "max_retries": 3,
  "created_at": "2024-01-15T10:30:00Z"
}

Technology:

• Language: Go
• Kafka Consumer: confluent-kafka-go
• Database: PostgreSQL (subscriptions, preferences)
• Cache: Redis (hot preferences, rate limits)

Scaling:

• Consumer group with multiple instances
• Each instance processes subset of partitions
• Add more instances to scale throughput
• For 578 events/sec peak, 3-5 instances sufficient

3.4 Channel Workers

Each channel has dedicated worker pool consuming from channel-specific topic.

3.4.1 Email Worker

Provider Integration:

• Primary: SendGrid / AWS SES
• Fallback: Mailgun / Postmark

Retry Policy:

Attempt 1: Immediate
Attempt 2: 1 minute delay
Attempt 3: 10 minutes delay
Attempt 4: 1 hour delay
Attempt 5: DLQ

Idempotency:

• Store delivered message_id in state store
• Check before sending
• Prevents duplicate emails during retries

Rate Limiting:

• Per-tenant sending limits
• Provider rate limits (SES: 14 emails/sec)
• Backpressure when limits reached

Template Rendering:

• Liquid / Handlebars syntax
• Variables from event data
• Localization support (i18n)

Worker Pool Size: 10-20 workers (based on provider rate limits)

3.4.2 SMS Worker

Provider: Twilio / AWS SNS

Key Differences:

• Higher cost per message
• Strict character limits
• Regional restrictions
• Carrier filtering

Rate Limiting:

• More aggressive than email
• Track costs per tenant
• Block on budget exceeded

Worker Pool Size: 5-10 workers

3.4.3 Push Notification Worker

Providers:

• iOS: APNs (Apple Push Notification service)
• Android: FCM (Firebase Cloud Messaging)

Device Token Management:

• Store user → device_token mapping
• Handle token expiration
• Remove invalid tokens
• Support multiple devices per user

Payload:

{
  "notification": {
    "title": "New Order",
    "body": "Your order #123 has shipped!",
    "icon": "order_icon"
  },
  "data": {
    "order_id": "123",
    "deep_link": "app://orders/123"
  },
  "priority": "high"
}

Worker Pool Size: 10-15 workers

3.4.4 Webhook Worker

Challenge: External endpoints can be slow or unavailable

Timeout Strategy:

• Connection timeout: 5 seconds
• Read timeout: 30 seconds
• Total timeout: 35 seconds

Circuit Breaker:

If endpoint fails 5 consecutive times:
  - Mark as unhealthy
  - Stop sending for 5 minutes
  - Gradually retry (exponential backoff)

Retry on:

• 5xx errors
• Timeouts
• Connection refused

No retry on:

• 4xx errors (client errors)
• Invalid response (malformed JSON)

Worker Pool Size: 20-30 workers (higher due to network latency)

3.4.5 In-App Worker

Delivery Mechanism:

• WebSocket for connected users
• Store in database for offline users
• Poll-based fallback

WebSocket Hub:

type Hub struct {
    connections map[string]map[*WebSocketConn]bool
    register    chan *WebSocketConn
    unregister  chan *WebSocketConn
    broadcast   chan *Notification
}

Scaling WebSocket:

• Use Redis Pub/Sub for multi-instance coordination
• Each server maintains own connections
• Publish to Redis, all servers receive

Worker Pool Size: 5 workers + WebSocket servers

3.5 Delivery State Store

Purpose: Track delivery attempts, status, retry state

Database Choice: Cassandra / ScyllaDB / DynamoDB

Why NoSQL:

• High write throughput
• Time-series data pattern
• Don't need complex joins
• Horizontal scaling

Schema:

-- Using Cassandra as example
CREATE TABLE delivery_status (
    delivery_id UUID,
    tenant_id UUID,
    user_id UUID,
    event_id UUID,
    channel TEXT,
    status TEXT,  -- pending, sent, delivered, failed
    attempt_count INT,
    last_attempt_at TIMESTAMP,
    next_retry_at TIMESTAMP,
    error_message TEXT,
    provider_response TEXT,
    created_at TIMESTAMP,
    updated_at TIMESTAMP,
    PRIMARY KEY (delivery_id)
);

CREATE INDEX idx_tenant_user_status 
ON delivery_status (tenant_id, user_id, status);

CREATE INDEX idx_next_retry 
ON delivery_status (next_retry_at) 
WHERE status = 'pending';

Retention:

• Keep successful deliveries: 30 days
• Keep failed deliveries: 90 days
• Archive to S3 after retention period

Write Pattern:

• Async writes (don't block worker)
• Batch writes when possible
• Eventual consistency acceptable

3.6 Preference Engine

User Preferences Schema:

{
  "user_id": "user_123",
  "tenant_id": "tenant_456",
  "channels": {
    "email": {
      "enabled": true,
      "categories": {
        "marketing": false,
        "transactional": true,
        "social": true
      }
    },
    "sms": {
      "enabled": false
    },
    "push": {
      "enabled": true,
      "quiet_hours": {
        "enabled": true,
        "start": "22:00",
        "end": "08:00",
        "timezone": "America/New_York"
      }
    },
    "inapp": {
      "enabled": true
    }
  },
  "digest_mode": {
    "enabled": false,
    "frequency": "daily",
    "time": "09:00"
  },
  "language": "en",
  "updated_at": "2024-01-15T10:30:00Z"
}

API Endpoints:

# Get preferences
GET /v1/users/{user_id}/preferences

# Update preferences
PUT /v1/users/{user_id}/preferences

# Unsubscribe from category
POST /v1/users/{user_id}/unsubscribe
{
  "channel": "email",
  "category": "marketing"
}

Caching Strategy:

• Cache in Redis with 1-hour TTL
• Invalidate on update
• Cache hit rate should be >90%

3.7 Template Service

Template Storage: PostgreSQL

Schema:

CREATE TABLE templates (
    id UUID PRIMARY KEY,
    tenant_id UUID NOT NULL,
    name VARCHAR(255) NOT NULL,
    channel VARCHAR(50) NOT NULL,
    language VARCHAR(10) DEFAULT 'en',
    subject TEXT,  -- for email
    body TEXT NOT NULL,
    variables JSONB,  -- list of required variables
    version INT DEFAULT 1,
    active BOOLEAN DEFAULT true,
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW(),
    UNIQUE(tenant_id, name, channel, language)
);

Template Example (Email):

Subject: Your order #{{order_id}} has been {{status}}

Hi {{user_name}},

Great news! Your order #{{order_id}} has been {{status}}.

{{#if status == "shipped"}}
Track your package: {{tracking_url}}
Estimated delivery: {{delivery_date}}
{{/if}}

{{#if status == "delivered"}}
We hope you enjoy your purchase!
{{/if}}

Questions? Reply to this email or visit our help center.

Best,
The {{company_name}} Team

Template Rendering:

• Use Go template engine or Liquid
• Cache compiled templates
• Support conditionals and loops
• Validate variables before rendering

Localization:

• Store templates per language
• Fallback to default language if missing
• User language from preference or event

4. Multi-Tenancy Architecture

4.1 Tenant Isolation

Data Isolation:

• All tables include tenant_id column
• All queries include WHERE tenant_id = ?
• Database-level row-level security (RLS)

Compute Isolation:

• Per-tenant rate limits
• Per-tenant queue priorities
• Noisy neighbor detection

Kafka Partitioning:

• Include tenant_id in partition key
• Prevents single tenant from monopolizing

4.2 Tenant Management

Tenant Schema:

CREATE TABLE tenants (
    id UUID PRIMARY KEY,
    name VARCHAR(255) NOT NULL,
    api_key VARCHAR(255) UNIQUE NOT NULL,
    api_secret VARCHAR(255) NOT NULL,
    status VARCHAR(50) DEFAULT 'active',
    
    -- Rate limits
    rate_limit_events_per_second INT DEFAULT 100,
    rate_limit_events_per_day INT DEFAULT 100000,
    rate_limit_fanout_per_event INT DEFAULT 1000,
    
    -- Quotas
    quota_emails_per_month INT DEFAULT 10000,
    quota_sms_per_month INT DEFAULT 1000,
    quota_push_per_month INT DEFAULT 50000,
    
    -- Billing
    billing_plan VARCHAR(50) DEFAULT 'starter',
    billing_cycle_start DATE,
    
    -- Metadata
    webhook_signing_secret VARCHAR(255),
    allowed_domains JSONB,
    custom_metadata JSONB,
    
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);

API Key Management:

• Use HMAC-SHA256 for signing
• Support key rotation
• Track key usage in metrics

Rate Limiting:

Global → Per-Tenant → Per-User

Enforcement Points:

1. API Gateway (coarse-grained)
2. Ingestion Service (fine-grained)
3. Orchestrator (fanout limits)

4.3 Billing & Metering

Metrics to Track:

• Events ingested
• Deliveries attempted per channel
• Deliveries successful per channel
• Storage used (MB)
• API calls made

Metering Service:

• Consume from Kafka (shadow consumer)
• Aggregate by tenant per hour
• Write to time-series database (InfluxDB / TimescaleDB)
• Export to billing system

Overage Handling:

• Soft limit: Warn tenant
• Hard limit: Throttle or block
• Grace period: 24 hours

5. Pub/Sub for Microservices

5.1 Topic-Based Routing

Concept: Internal services subscribe to event types

Example Use Case:

Event: "order.created"

Subscribers:
- inventory-service (webhook)
- analytics-service (webhook)
- email-service (internal)
- slack-notifications (webhook)

Topic Naming Convention:

{domain}.{entity}.{action}

Examples:
- order.created
- order.updated
- order.cancelled
- user.registered
- payment.succeeded
- payment.failed

5.2 Subscription Management

Subscription Schema:

CREATE TABLE subscriptions (
    id UUID PRIMARY KEY,
    tenant_id UUID NOT NULL,
    topic_pattern VARCHAR(255) NOT NULL,  -- supports wildcards
    subscriber_type VARCHAR(50) NOT NULL,  -- webhook, internal, queue
    endpoint TEXT,  -- webhook URL or queue name
    
    -- Filtering
    filter_expression JSONB,  -- e.g., {"amount": {"gt": 100}}
    
    -- Configuration
    retry_policy JSONB,
    timeout_seconds INT DEFAULT 30,
    
    -- Status
    status VARCHAR(50) DEFAULT 'active',
    failure_count INT DEFAULT 0,
    last_success_at TIMESTAMP,
    last_failure_at TIMESTAMP,
    
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);

CREATE INDEX idx_subscriptions_tenant_topic 
ON subscriptions(tenant_id, topic_pattern);

Topic Pattern Matching:

order.*          → matches order.created, order.updated, etc.
order.created    → exact match
*.created        → matches any entity's created event

Filter Expression Example:

{
  "amount": {"gt": 100},
  "status": {"in": ["pending", "confirmed"]},
  "country": {"eq": "US"}
}

5.3 Webhook Delivery

Webhook Payload:

{
  "event_id": "evt_123",
  "event_type": "order.created",
  "created_at": "2024-01-15T10:30:00Z",
  "data": {
    "order_id": "order_456",
    "amount": 99.99,
    "customer_id": "cust_789"
  },
  "metadata": {
    "tenant_id": "tenant_123"
  }
}

Webhook Security:

1. Signature Verification:

X-MeetNotify-Signature: t=1234567890,v1=<signature>

Signature = HMAC-SHA256(tenant_secret, timestamp + "." + body)

2. IP Whitelisting (optional)

3. HTTPS Only

Webhook Retry Policy:

max_retries: 5
backoff_multiplier: 2
initial_delay: 1s
max_delay: 1h

Schedule:
- Attempt 1: Immediate
- Attempt 2: 2s
- Attempt 3: 4s
- Attempt 4: 8s
- Attempt 5: 16s
- DLQ if all fail

6. Data Models

6.1 Core Domain Models

Event

type Event struct {
    ID             string                 `json:"id"`
    TenantID       string                 `json:"tenant_id"`
    EventType      string                 `json:"event_type"`
    UserID         string                 `json:"user_id,omitempty"`
    Data           map[string]interface{} `json:"data"`
    Metadata       EventMetadata          `json:"metadata"`
    IdempotencyKey string                 `json:"idempotency_key,omitempty"`
    CreatedAt      time.Time              `json:"created_at"`
}

type EventMetadata struct {
    Priority  string `json:"priority"`
    Source    string `json:"source"`
    TraceID   string `json:"trace_id"`
    UserAgent string `json:"user_agent,omitempty"`
}

Delivery Task

type DeliveryTask struct {
    DeliveryID   string                 `json:"delivery_id"`
    EventID      string                 `json:"event_id"`
    TenantID     string                 `json:"tenant_id"`
    UserID       string                 `json:"user_id"`
    Channel      string                 `json:"channel"`
    TemplateID   string                 `json:"template_id,omitempty"`
    TemplateData map[string]interface{} `json:"template_data"`
    Priority     string                 `json:"priority"`
    MaxRetries   int                    `json:"max_retries"`
    CreatedAt    time.Time              `json:"created_at"`
}

Delivery Status

type DeliveryStatus struct {
    DeliveryID       string    `json:"delivery_id"`
    Status           string    `json:"status"` // pending, sent, delivered, failed
    AttemptCount     int       `json:"attempt_count"`
    LastAttemptAt    time.Time `json:"last_attempt_at"`
    NextRetryAt      time.Time `json:"next_retry_at,omitempty"`
    ErrorMessage     string    `json:"error_message,omitempty"`
    ProviderResponse string    `json:"provider_response,omitempty"`
    UpdatedAt        time.Time `json:"updated_at"`
}

6.2 PostgreSQL Schema

-- Tenants
CREATE TABLE tenants (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    name VARCHAR(255) NOT NULL,
    api_key VARCHAR(255) UNIQUE NOT NULL,
    api_secret_hash VARCHAR(255) NOT NULL,
    status VARCHAR(50) DEFAULT 'active',
    rate_limit_config JSONB,
    quota_config JSONB,
    billing_plan VARCHAR(50),
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);

-- Templates
CREATE TABLE templates (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID REFERENCES tenants(id),
    name VARCHAR(255) NOT NULL,
    channel VARCHAR(50) NOT NULL,
    language VARCHAR(10) DEFAULT 'en',
    subject TEXT,
    body TEXT NOT NULL,
    variables JSONB,
    version INT DEFAULT 1,
    active BOOLEAN DEFAULT true,
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW(),
    UNIQUE(tenant_id, name, channel, language, version)
);

-- Subscriptions (pub/sub)
CREATE TABLE subscriptions (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID REFERENCES tenants(id),
    topic_pattern VARCHAR(255) NOT NULL,
    subscriber_type VARCHAR(50) NOT NULL,
    endpoint TEXT,
    filter_expression JSONB,
    retry_policy JSONB,
    status VARCHAR(50) DEFAULT 'active',
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW()
);

-- User Preferences
CREATE TABLE user_preferences (
    user_id UUID,
    tenant_id UUID REFERENCES tenants(id),
    preferences JSONB NOT NULL,
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW(),
    PRIMARY KEY (tenant_id, user_id)
);

-- Idempotency tracking
CREATE TABLE idempotency_keys (
    key VARCHAR(255) PRIMARY KEY,
    tenant_id UUID REFERENCES tenants(id),
    event_id UUID NOT NULL,
    created_at TIMESTAMP DEFAULT NOW()
);
CREATE INDEX idx_idempotency_created ON idempotency_keys(created_at);
-- Auto-delete after 24 hours (use TTL or cron job)

-- Device tokens (for push notifications)
CREATE TABLE device_tokens (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id UUID REFERENCES tenants(id),
    user_id UUID NOT NULL,
    platform VARCHAR(20) NOT NULL,  -- ios, android
    token TEXT NOT NULL,
    active BOOLEAN DEFAULT true,
    last_used_at TIMESTAMP,
    created_at TIMESTAMP DEFAULT NOW(),
    UNIQUE(tenant_id, user_id, token)
);

-- Metrics aggregation
CREATE TABLE delivery_metrics (
    tenant_id UUID REFERENCES tenants(id),
    hour TIMESTAMP NOT NULL,
    channel VARCHAR(50) NOT NULL,
    status VARCHAR(50) NOT NULL,
    count BIGINT DEFAULT 0,
    PRIMARY KEY (tenant_id, hour, channel, status)
);
CREATE INDEX idx_delivery_metrics_hour ON delivery_metrics(hour);

7. Scalability & Performance

7.1 Throughput Calculations

Target: 10M events/day

Average Rate:

• 10,000,000 / 86,400 seconds = 115 events/sec

Peak Rate (5x average):

• 578 events/sec

With Fanout (avg 5 recipients per event):

• 578 * 5 = 2,890 deliveries/sec peak

Per Channel (equal distribution):

• Email: 578 deliveries/sec
• SMS: 115 deliveries/sec
• Push: 578 deliveries/sec
• Webhook: 289 deliveries/sec
• In-app: 578 deliveries/sec

7.2 Component Sizing

Kafka Cluster:

• 3 brokers (minimum for replication)
• 6 cores, 32 GB RAM each
• SSD storage (1 TB per broker)

PostgreSQL:

• 1 primary + 2 read replicas
• 8 cores, 64 GB RAM
• SSD storage (500 GB)

Redis:

• 3-node cluster (primary + 2 replicas)
• 4 cores, 16 GB RAM each

Cassandra/ScyllaDB:

• 3-node cluster (replication factor 3)
• 8 cores, 32 GB RAM each
• SSD storage (1 TB per node)

7.3 Database Optimization

PostgreSQL (Hot Data):

• Templates: ~10K rows, fully cached
• Subscriptions: ~100K rows, indexed
• Preferences: ~1M rows, cached in Redis
• Idempotency keys: 24-hour rolling window

Indexes:

CREATE INDEX idx_templates_tenant_active 
ON templates(tenant_id, active) WHERE active = true;

CREATE INDEX idx_subscriptions_tenant_topic 
ON subscriptions(tenant_id, topic_pattern) WHERE status = 'active';

CREATE INDEX idx_preferences_tenant_user 
ON user_preferences(tenant_id, user_id);

Connection Pooling:

MaxOpenConns: 100
MaxIdleConns: 25
ConnMaxLifetime: 5 minutes

Cassandra (Delivery State):

• Writes: ~3,000/sec peak
• Reads: ~500/sec (status checks)
• TTL on rows (auto-delete after 30 days)

Partition Strategy:

PRIMARY KEY ((tenant_id, delivery_date), delivery_id)

• Groups by tenant and date
• Efficient for time-range queries
• Even distribution

7.4 Caching Strategy

Redis Cache Layers:

1. Preferences Cache

   • Key: pref:{tenant_id}:{user_id}
   • TTL: 1 hour
   • Invalidate on update
   • Hit rate target: >95%

2. Rate Limit Counters

   • Key: ratelimit:{tenant_id}:{window}
   • TTL: 1 hour
   • Sliding window counter

3. Template Cache

   • Key: template:{tenant_id}:{name}:{channel}:{lang}
   • TTL: 24 hours
   • Compiled templates stored

4. Idempotency Cache

   • Key: idempotency:{key}
   • TTL: 24 hours
   • Faster than DB lookup

Cache Eviction:

• LRU policy
• Memory limit: 80% of total RAM
• Monitor eviction rate (should be <5%)

8. Reliability & Fault Tolerance

8.1 Retry Policies

Email Retry:

max_retries: 5
backoff_type: exponential
initial_delay: 30s
max_delay: 1h

Schedule:
- Attempt 1: Immediate
- Attempt 2: 30s
- Attempt 3: 2m (30s * 2^1)
- Attempt 4: 8m (30s * 2^2)
- Attempt 5: 32m (30s * 2^3)
- Attempt 6: 1h (capped)

SMS Retry:

max_retries: 3
backoff_type: exponential
initial_delay: 1m
max_delay: 30m

Push Retry:

max_retries: 3
backoff_type: exponential
initial_delay: 10s
max_delay: 10m

Webhook Retry:

max_retries: 5
backoff_type: exponential
initial_delay: 1s
max_delay: 1h

8.2 Circuit Breakers

Per-Channel Circuit Breaker:

type CircuitBreaker struct {
    FailureThreshold int           // 5 failures
    SuccessThreshold int           // 2 successes to close
    Timeout          time.Duration // 5 minutes
    State            string         // closed, open, half-open
}

State Transitions:

Closed → (5 failures) → Open
Open → (5 min timeout) → Half-Open
Half-Open → (2 successes) → Closed
Half-Open → (1 failure) → Open

Per-Endpoint Circuit Breaker (Webhooks):

• Track per subscription endpoint
• Protect against slow/failing endpoints
• Alert tenant on circuit open

8.3 Dead Letter Queue (DLQ)

When to DLQ:

• Max retries exceeded
• Permanent errors (4xx from webhook)
• Invalid data (malformed templates)
• Circuit breaker open for extended period

DLQ Structure:

{
  "delivery_id": "uuid",
  "event_id": "uuid",
  "tenant_id": "uuid",
  "channel": "email",
  "failure_reason": "max_retries_exceeded",
  "attempts": [
    {
      "attempt_num": 1,
      "timestamp": "2024-01-15T10:30:00Z",
      "error": "SMTP timeout"
    }
  ],
  "original_payload": { ... },
  "dlq_timestamp": "2024-01-15T12:30:00Z"
}

DLQ Processing:

• Manual retry via API
• Batch retry for systemic issues
• Export for analysis
• Tenant notifications

8.4 Failure Modes

8.5 Data Consistency

Event Processing:

• Kafka offset commit AFTER successful processing
• Idempotency keys prevent duplicates
• At-least-once delivery guarantee

Delivery State:

• Eventual consistency acceptable
• Status updates may lag reality
• Compensate with retry logic

Preference Updates:

• Strong consistency (PostgreSQL)
• Cache invalidation on write
• Short TTL to limit staleness

9. Security

9.1 Authentication & Authorization

Tenant API Keys:

Format: mtn_<environment>_<random>
Example: mtn_prod_kj34h5k2j3h4k5j2h3k4j5h2

Secret: mtn_sec_<random>

Key Storage:

• Hash secrets with bcrypt (cost 12)
• Store only hash in database
• Rotate keys with zero downtime

JWT for User Sessions:

{
  "sub": "user_123",
  "tenant_id": "tenant_456",
  "scope": "read:notifications write:preferences",
  "exp": 1642262400
}

API Request Signing:

X-MeetNotify-Timestamp: 1642262400
X-MeetNotify-Signature: <HMAC-SHA256>

Signature = HMAC-SHA256(
  api_secret,
  timestamp + method + path + body
)

Timestamp Validation:

• Reject requests older than 5 minutes
• Prevents replay attacks

9.2 Data Encryption

In Transit:

• TLS 1.3 for all API endpoints
• mTLS for inter-service communication (optional)
• WebSocket over WSS

At Rest:

• Database encryption (PostgreSQL: pgcrypto)
• Kafka encryption (optional, performance trade-off)
• S3 encryption for archived data

PII Handling:

• Encrypt sensitive fields (email, phone)
• Hashed user identifiers when possible
• GDPR compliance (right to deletion)

9.3 Rate Limiting

Multi-Level Rate Limiting:

1. Global Rate Limit:

   • 10,000 req/sec per API endpoint
   • Protects infrastructure

2. Per-Tenant Rate Limit:

   • Configured per tenant (e.g., 100 req/sec)
   • Based on billing plan

3. Per-User Rate Limit (for end users):

   • 10 req/sec per user
   • Prevents abuse

Implementation:

Algorithm: Token Bucket
Storage: Redis
Key: ratelimit:{tenant_id}:{window}
Window: 1 second sliding window

Response:

HTTP 429 Too Many Requests
Retry-After: 5
X-RateLimit-Limit: 100
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1642262405

9.4 Input Validation

Event Payload Validation:

{
  "event_type": {
    "type": "string",
    "pattern": "^[a-z_]+\\.[a-z_]+$",
    "maxLength": 100
  },
  "user_id": {
    "type": "string",
    "format": "uuid"
  },
  "data": {
    "type": "object",
    "maxProperties": 50,
    "maxSizeBytes": 10000
  }
}

Template Validation:

• Whitelist allowed template tags
• Prevent code injection
• Limit template size (100 KB)

Webhook URL Validation:

• Must be HTTPS
• No internal IPs (prevent SSRF)
• Whitelist allowed domains (optional)

10. Observability

10.1 Metrics (Prometheus)

Key Metrics:

Ingestion:

events_ingested_total{tenant_id, event_type}
events_rejected_total{tenant_id, reason}
ingestion_latency_seconds{quantile}

Processing:

events_processed_total{tenant_id, event_type}
fanout_size{tenant_id, event_type, quantile}
orchestration_latency_seconds{quantile}

Delivery:

deliveries_attempted_total{tenant_id, channel, status}
deliveries_succeeded_total{tenant_id, channel}
deliveries_failed_total{tenant_id, channel, error_type}
delivery_latency_seconds{channel, quantile}
retry_count_total{channel}

System:

kafka_lag{topic, partition, consumer_group}
redis_hit_rate{cache_type}
worker_queue_depth{channel}
circuit_breaker_state{endpoint}

SLA Metrics:

p50_delivery_latency_seconds{channel}
p95_delivery_latency_seconds{channel}
p99_delivery_latency_seconds{channel}
error_rate{channel}

10.2 Logging (Structured JSON)

Log Levels:

• DEBUG: Development only
• INFO: Normal operations
• WARN: Degraded performance, retries
• ERROR: Failures requiring attention

Log Format:

{
  "timestamp": "2024-01-15T10:30:00.123Z",
  "level": "INFO",
  "service": "orchestrator",
  "trace_id": "abc123",
  "tenant_id": "tenant_456",
  "event_id": "evt_789",
  "message": "Event processed successfully",
  "fields": {
    "event_type": "order.created",
    "fanout_size": 5,
    "processing_time_ms": 45
  }
}

Log Aggregation:

• ELK Stack (Elasticsearch, Logstash, Kibana)
• Or Loki + Grafana
• Retention: 30 days

10.3 Tracing (OpenTelemetry)

Trace Spans:

Ingestion → Event Bus → Orchestration → Channel Queue → Worker → Provider

Trace Context Propagation:

• W3C Trace Context standard
• Inject trace_id in all messages
• Correlate across services

Example Trace:

Trace ID: abc123
├─ Span: API Request (10ms)
├─ Span: Publish to Kafka (5ms)
├─ Span: Orchestration (50ms)
│  ├─ Span: Preference Lookup (10ms)
│  ├─ Span: Template Render (15ms)
│  └─ Span: Fanout (25ms)
├─ Span: Email Delivery (500ms)
│  ├─ Span: SendGrid API (480ms)
│  └─ Span: Status Update (20ms)

10.4 Alerting (PagerDuty / OpsGenie)

Critical Alerts:

• Kafka consumer lag > 1000 messages
• Error rate > 5% for any channel
• Circuit breaker open for > 10 minutes
• Database connection pool exhausted
• Worker queue depth > 10,000

Warning Alerts:

• Error rate > 1%
• p99 latency > 5 seconds
• Redis hit rate < 80%
• Disk usage > 80%

Alert Routing:

• Critical → PagerDuty → On-call engineer
• Warning → Slack #alerts channel

10.5 Health Checks

Liveness Probe:

GET /health/live

Response:
200 OK
{
  "status": "ok",
  "timestamp": "2024-01-15T10:30:00Z"
}

Readiness Probe:

GET /health/ready

Checks:
- Database connection
- Kafka connection
- Redis connection
- Worker queue health

Response:
200 OK (all healthy)
503 Service Unavailable (any unhealthy)
{
  "status": "ready",
  "checks": {
    "database": "ok",
    "kafka": "ok",
    "redis": "ok",
    "workers": "ok"
  }
}

11. Deployment Architecture

11.1 Infrastructure (AWS Example)

Compute:

• EKS (Kubernetes) for services
• EC2 for Kafka cluster
• RDS for PostgreSQL
• ElastiCache for Redis
• Keyspaces (managed Cassandra) or self-hosted

Networking:

• VPC with public/private subnets
• ALB for API Gateway
• NLB for WebSocket servers
• NAT Gateway for outbound traffic

Storage:

• EBS for Kafka storage
• S3 for archived logs/data
• EFS for shared config (optional)

Regions:

• Primary: us-east-1
• DR: us-west-2 (optional for this scale)

11.2 Kubernetes Deployment

Namespace Structure:

meetnotify-prod/
├── ingestion
├── orchestrator
├── workers-email
├── workers-sms
├── workers-push
├── workers-webhook
├── workers-inapp
└── websocket

Sample Deployment (Email Worker):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: email-worker
  namespace: meetnotify-prod
spec:
  replicas: 10
  selector:
    matchLabels:
      app: email-worker
  template:
    metadata:
      labels:
        app: email-worker
    spec:
      containers:
      - name: worker
        image: meetnotify/email-worker:v1.2.3
        resources:
          requests:
            cpu: 500m
            memory: 1Gi
          limits:
            cpu: 2000m
            memory: 4Gi
        env:
        - name: KAFKA_BROKERS
          value: "kafka-1:9092,kafka-2:9092,kafka-3:9092"
        - name: KAFKA_TOPIC
          value: "deliveries.email"
        - name: KAFKA_GROUP_ID
          value: "email-workers"
        - name: SENDGRID_API_KEY
          valueFrom:
            secretKeyRef:
              name: sendgrid-credentials
              key: api-key
        livenessProbe:
          httpGet:
            path: /health/live
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /health/ready
            port: 8080
          initialDelaySeconds: 10
          periodSeconds: 5

HorizontalPodAutoscaler:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: email-worker-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: email-worker
  minReplicas: 5
  maxReplicas: 20
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Pods
    pods:
      metric:
        name: kafka_consumer_lag
      target:
        type: AverageValue
        averageValue: "500"

11.3 CI/CD Pipeline

GitOps with ArgoCD:

Developer → Git Push → GitHub Actions
                          ↓
                    Build & Test
                          ↓
                    Build Docker Image
                          ↓
                    Push to ECR
                          ↓
                    Update Helm Chart
                          ↓
                    ArgoCD Detects Change
                          ↓
                    Deploy to K8s

GitHub Actions Workflow:

name: Deploy

on:
  push:
    branches: [main]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Run tests
        run: go test ./...
      
      - name: Build image
        run: docker build -t meetnotify/email-worker:${{ github.sha }} .
      
      - name: Push to ECR
        run: docker push meetnotify/email-worker:${{ github.sha }}
      
      - name: Update Helm chart
        run: |
          helm upgrade email-worker ./charts/email-worker \
            --set image.tag=${{ github.sha }}

11.4 Disaster Recovery

Backup Strategy:

PostgreSQL:

• Automated daily snapshots
• Point-in-time recovery (7 days)
• Cross-region replication (optional)

Kafka:

• Replicated across 3 brokers
• Data loss tolerance: acceptable (events expire after 7 days)
• Reconstruct from delivery logs if needed

Redis:

• Not critical (cache only)
• AOF persistence for preference cache
• Reload from PostgreSQL on failure

RPO (Recovery Point Objective): 1 hour
RTO (Recovery Time Objective): 4 hours

DR Runbook:

1. Detect failure (automated alerts)
2. Assess impact
3. Restore from backups
4. Replay missed events (if possible)
5. Validate system health
6. Resume traffic

12. Cost Estimation

12.1 Infrastructure Costs (AWS)

Compute (EKS):

• 10 t3.xlarge (4 vCPU, 16 GB) @ $0.1664/hr
• Total: $1,195/month

Kafka (EC2):

• 3 m5.2xlarge (8 vCPU, 32 GB) @ $0.384/hr
• Total: $829/month

PostgreSQL (RDS):

• 1 db.r6g.2xlarge (8 vCPU, 64 GB) @ $0.504/hr
• Total: $363/month

Redis (ElastiCache):

• 3 cache.r6g.xlarge (4 vCPU, 26 GB) @ $0.252/hr
• Total: $544/month

Cassandra (Keyspaces):

• On-demand pricing: ~$200/month (estimated)

Load Balancers:

• 2 ALB @ $16/month + traffic
• Total: ~$50/month

Data Transfer:

• Outbound: ~500 GB/month @ $0.09/GB
• Total: $45/month

Storage:

• S3: 100 GB @ $0.023/GB
• EBS: 3 TB @ $0.10/GB
• Total: ~$303/month

Total Infrastructure: ~$3,500/month

12.2 Third-Party Services

Email (SendGrid):

• 10M emails/month: $90/month

SMS (Twilio):

• 100K SMS/month @ $0.0075 each: $750/month

Push (FCM/APNs):

• Free

Observability:

• Datadog: ~$500/month
• PagerDuty: $50/month

Total Third-Party: ~$1,390/month

12.3 Total Cost of Ownership

Monthly Costs:

• Infrastructure: $3,500
• Third-party: $1,390
• Total: $4,890/month

Per Event:

• $4,890 / 10M events = $0.000489 per event

Per Delivery (with 5x fanout):

• $4,890 / 50M deliveries = $0.0000978 per delivery

Scaling to 100M events/day:

• Estimate: ~$35,000/month
• Requires multi-region, more workers, larger DB

13. Implementation Roadmap

Phase 1: Core Infrastructure (Weeks 1-4)

Week 1-2: Foundation

• Project structure & repository setup
• Kafka cluster deployment
• PostgreSQL setup with schemas
• Redis cluster setup
• Basic CI/CD pipeline

Week 3-4: Ingestion & Event Bus

• Ingestion API (REST endpoints)
• Authentication & rate limiting
• Event validation & publishing
• Kafka producer integration
• Unit tests

Phase 2: Orchestration (Weeks 5-6)

• Orchestration service skeleton
• Kafka consumer integration
• Preference lookup logic
• Template rendering
• Channel routing logic
• Publishing to delivery topics

Phase 3: Channel Workers (Weeks 7-10)

Week 7-8:

• Email worker (SendGrid integration)
• Push worker (FCM/APNs integration)
• Retry logic implementation
• Circuit breaker implementation

Week 9-10:

• SMS worker (Twilio integration)
• Webhook worker
• In-app worker + WebSocket server
• Delivery state tracking

Phase 4: Supporting Services (Weeks 11-12)

• Preference service API
• Template management API
• Subscription management (pub/sub)
• Dead letter queue processing
• Admin dashboard (basic)

Phase 5: Multi-Tenancy (Weeks 13-14)

• Tenant management API
• API key generation & rotation
• Per-tenant rate limiting
• Per-tenant metrics
• Billing hooks

Phase 6: Observability (Weeks 15-16)

• Prometheus metrics integration
• Structured logging setup
• OpenTelemetry tracing
• Grafana dashboards
• Alert rules & PagerDuty

Phase 7: Production Hardening (Weeks 17-20)

• Load testing (10x target load)
• Chaos engineering tests
• Security audit
• Performance optimization
• Documentation
• Runbooks

Phase 8: Launch (Week 21+)

• Beta release to pilot tenants
• Monitoring & incident response
• Feature iteration based on feedback
• Public launch

14. Testing Strategy

14.1 Unit Tests

Coverage Target: >80%

Key Areas:

• Event validation logic
• Template rendering
• Preference application
• Retry logic
• Circuit breaker state transitions

Example Test:

func TestRetryWithExponentialBackoff(t *testing.T) {
    policy := RetryPolicy{
        MaxRetries: 5,
        InitialDelay: 1 * time.Second,
        MaxDelay: 1 * time.Hour,
    }
    
    delays := policy.CalculateDelays()
    
    assert.Equal(t, 1*time.Second, delays[0])
    assert.Equal(t, 2*time.Second, delays[1])
    assert.Equal(t, 4*time.Second, delays[2])
    assert.Equal(t, 8*time.Second, delays[3])
    assert.Equal(t, 16*time.Second, delays[4])
}

14.2 Integration Tests

Test Scenarios:

• Publish event → Verify in Kafka
• Orchestrator consumes event → Verify delivery tasks created
• Worker delivers email → Verify status updated
• Webhook fails → Verify retry scheduled
• Circuit breaker opens → Verify deliveries paused

Test Environment:

• Docker Compose with all services
• Localstack for AWS services
• Mock email/SMS providers

14.3 Load Tests (Locust / k6)

Scenarios:

Sustained Load:

• 500 events/sec for 1 hour
• Verify no errors, stable latency

Spike Test:

• Ramp from 100 to 2000 events/sec in 1 minute
• Verify system handles spike

Endurance Test:

• 300 events/sec for 24 hours
• Check for memory leaks, degradation

Example k6 Script:

import http from 'k6/http';
import { check } from 'k6';

export let options = {
  stages: [
    { duration: '5m', target: 100 },
    { duration: '10m', target: 500 },
    { duration: '5m', target: 0 },
  ],
};

export default function () {
  const payload = JSON.stringify({
    event_type: 'order.created',
    user_id: `user_${Math.floor(Math.random() * 10000)}`,
    data: { order_id: `order_${Date.now()}` },
  });

  const res = http.post('https://api.meetnotify.com/v1/events', payload, {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': 'Bearer test_key',
    },
  });

  check(res, {
    'status is 201': (r) => r.status === 201,
    'response time < 500ms': (r) => r.timings.duration < 500,
  });
}

14.4 Chaos Engineering

Experiments:

1. Kill Random Worker:

   • Verify messages are redelivered by other workers
   • No data loss

2. Network Partition:

   • Simulate network split between services
   • Verify eventual consistency

3. Kafka Broker Failure:

   • Take down 1 of 3 brokers
   • Verify automatic failover

4. Database Slowdown:

   • Inject latency into PostgreSQL
   • Verify graceful degradation

5. Third-Party Outage:

   • Simulate SendGrid downtime
   • Verify circuit breaker opens
   • Verify retry to DLQ

Tools:

• Chaos Mesh (Kubernetes)
• Gremlin
• Custom scripts

15. API Reference

15.1 Event Publishing

Publish Single Event

POST /v1/events
Content-Type: application/json
Authorization: Bearer <tenant_api_key>

{
  "event_type": "order.created",
  "user_id": "user_123",
  "data": {
    "order_id": "order_456",
    "amount": 99.99,
    "customer_name": "John Doe"
  },
  "metadata": {
    "idempotency_key": "order_456",
    "priority": "high",
    "source": "checkout-service"
  },
  "channels": ["email", "push"],  // Optional: override default
  "template_id": "order_confirmation"  // Optional
}

Response (201 Created):
{
  "event_id": "evt_abc123",
  "status": "accepted",
  "created_at": "2024-01-15T10:30:00Z"
}

Publish Batch Events

POST /v1/events/batch
Content-Type: application/json
Authorization: Bearer <tenant_api_key>

{
  "events": [
    { "event_type": "order.created", "user_id": "user_1", ... },
    { "event_type": "order.created", "user_id": "user_2", ... }
  ]
}

Response (207 Multi-Status):
{
  "results": [
    { "event_id": "evt_1", "status": "accepted" },
    { "event_id": null, "status": "rejected", "error": "Invalid user_id" }
  ]
}

15.2 Subscription Management

Create Subscription

POST /v1/subscriptions
Content-Type: application/json
Authorization: Bearer <tenant_api_key>

{
  "topic_pattern": "order.*",
  "subscriber_type": "webhook",
  "endpoint": "https://api.example.com/webhooks/orders",
  "filter_expression": {
    "amount": { "gt": 100 }
  },
  "retry_policy": {
    "max_retries": 5,
    "initial_delay": "1s"
  }
}

Response (201 Created):
{
  "subscription_id": "sub_xyz789",
  "status": "active",
  "created_at": "2024-01-15T10:30:00Z"
}

List Subscriptions

GET /v1/subscriptions?topic=order.created
Authorization: Bearer <tenant_api_key>

Response (200 OK):
{
  "subscriptions": [
    {
      "subscription_id": "sub_xyz789",
      "topic_pattern": "order.*",
      "endpoint": "https://api.example.com/webhooks/orders",
      "status": "active"
    }
  ]
}

15.3 User Preferences

Get User Preferences

GET /v1/users/{user_id}/preferences
Authorization: Bearer <tenant_api_key>

Response (200 OK):
{
  "user_id": "user_123",
  "preferences": {
    "channels": {
      "email": { "enabled": true },
      "sms": { "enabled": false },
      "push": { "enabled": true }
    },
    "quiet_hours": {
      "enabled": true,
      "start": "22:00",
      "end": "08:00",
      "timezone": "America/New_York"
    }
  }
}

Update User Preferences

PUT /v1/users/{user_id}/preferences
Content-Type: application/json
Authorization: Bearer <tenant_api_key>

{
  "channels": {
    "email": { "enabled": false }
  }
}

Response (200 OK):
{
  "user_id": "user_123",
  "updated_at": "2024-01-15T10:30:00Z"
}

15.4 Delivery Status

Get Delivery Status

GET /v1/deliveries/{delivery_id}
Authorization: Bearer <tenant_api_key>

Response (200 OK):
{
  "delivery_id": "dlv_123",
  "event_id": "evt_456",
  "channel": "email",
  "status": "delivered",
  "attempts": 1,
  "delivered_at": "2024-01-15T10:31:00Z",
  "provider_response": {
    "message_id": "sendgrid_id_123"
  }
}

List Deliveries for Event

GET /v1/events/{event_id}/deliveries
Authorization: Bearer <tenant_api_key>

Response (200 OK):
{
  "deliveries": [
    {
      "delivery_id": "dlv_123",
      "channel": "email",
      "status": "delivered"
    },
    {
      "delivery_id": "dlv_124",
      "channel": "push",
      "status": "pending"
    }
  ]
}

15.5 Template Management

Create Template

POST /v1/templates
Content-Type: application/json
Authorization: Bearer <tenant_api_key>

{
  "name": "order_confirmation",
  "channel": "email",
  "language": "en",
  "subject": "Order #{{order_id}} Confirmed",
  "body": "Hi {{customer_name}}, your order has been confirmed...",
  "variables": ["order_id", "customer_name", "amount"]
}

Response (201 Created):
{
  "template_id": "tpl_abc123",
  "version": 1,
  "created_at": "2024-01-15T10:30:00Z"
}

16. Monitoring Dashboards

16.1 Operational Dashboard

Panels:

1. Events Ingested (rate/sec)
2. Deliveries Attempted (by channel)
3. Deliveries Succeeded (by channel)
4. Error Rate (%)
5. p95 Delivery Latency (by channel)
6. Kafka Consumer Lag
7. Worker Queue Depth
8. Circuit Breaker Status

16.2 Tenant Dashboard

Per-Tenant Metrics:

1. Events Published (count)
2. Deliveries by Channel
3. Error Rate
4. Quota Usage (%)
5. Cost Estimate ($)

16.3 SLA Dashboard

SLIs:

• Availability: 99.9% uptime
• Latency: p95 < 5 seconds (ingestion to delivery)
• Error Rate: < 1% per channel
• Throughput: Handle 2x peak load

17. Comparison: MeetNotify vs AWS SNS/GCP Pub/Sub

When to Use MeetNotify:

• Need multi-channel orchestration (not just pub/sub)
• Need user preferences & templates
• Multi-tenant SaaS product
• Want full control & customization

When to Use AWS SNS/GCP:

• Pure pub/sub (no preferences/templates)
• Massive scale (billions of events)
• Don't want to manage infrastructure
• Cost-sensitive

18. Security Checklist

• API key authentication
• HTTPS/TLS for all endpoints
• Rate limiting per tenant
• Input validation & sanitization
• SQL injection prevention (parameterized queries)
• XSS prevention (template escaping)
• SSRF prevention (webhook URL validation)
• Secrets stored in vault (not code)
• Database encryption at rest
• PII encryption (email, phone)
• GDPR compliance (right to deletion)
• Audit logging (who did what when)
• DDoS protection (CloudFlare / AWS Shield)
• Webhook signature verification
• Request timestamp validation
• Circuit breakers for external services

19. Operational Runbooks

19.1 High Kafka Consumer Lag

Symptoms:

• Alert: Kafka lag > 1000 messages
• Dashboard shows lag increasing

Diagnosis:

1. Check if workers are running: kubectl get pods
2. Check worker logs for errors
3. Check database connection pool
4. Check external provider rate limits

Resolution:

1. Scale up workers: kubectl scale deployment orchestrator --replicas=10
2. If database slow, check slow query log
3. If provider rate limited, wait or switch provider
4. If critical, pause non-essential event types

19.2 Email Delivery Failures

Symptoms:

• Alert: Email error rate > 5%
• Customers report missing emails

Diagnosis:

1. Check SendGrid status page
2. Check circuit breaker state
3. Check email worker logs
4. Check for blacklist/spam issues

Resolution:

1. If provider down, switch to fallback (Mailgun)
2. If blacklist, contact provider support
3. If quota exceeded, increase limits
4. If temporary issue, retries will handle

19.3 Database Connection Exhaustion

Symptoms:

• Alert: DB connections > 90%
• Slow queries, timeouts

Diagnosis:

1. Check connection pool metrics
2. Check slow query log
3. Check for connection leaks

Resolution:

1. Scale read replicas for read traffic
2. Increase connection pool size (temporarily)
3. Fix connection leaks in code
4. Add connection timeout

20. Future Enhancements

Phase 2 Features (6-12 months)

1. Advanced Analytics

   • Delivery heatmaps
   • A/B testing for templates
   • Predictive delivery times

2. Smart Delivery

   • ML-based send time optimization
   • Channel preference learning
   • Spam prediction

3. Multi-Region

   • Deploy in EU, APAC regions
   • Data residency compliance
   • Cross-region replication

4. Advanced Templates

   • WYSIWYG editor
   • Template versioning
   • Dynamic content blocks

5. Digest Mode

   • Batch notifications
   • Scheduled delivery
   • Smart grouping

6. 2-Way Messaging

   • Reply handling for email/SMS
   • Conversation threading
   • Sentiment analysis

7. Self-Service Portal

   • Tenant dashboard
   • Usage analytics
   • Billing management

21. Conclusion

MeetNotify is architected as a production-ready, cloud-grade notification platform capable of handling startup-scale workloads (100K-10M events/day) across multiple channels and tenants.

Key Strengths: ✅ Multi-tenant isolation by design
✅ Horizontal scalability at every layer
✅ At-least-once delivery guarantees
✅ Multi-channel orchestration (Email, SMS, Push, Webhook, In-app)
✅ Pub/sub for microservices communication
✅ User preference management
✅ Template engine with localization
✅ Comprehensive observability
✅ Cost-effective for target scale

Production Readiness:

• Retry logic with exponential backoff
• Circuit breakers for external services
• Dead letter queues for failed deliveries
• Multi-layer rate limiting
• Health checks & liveness probes
• Structured logging & distributed tracing
• Automated alerting & incident response

Next Steps:

1. Complete Phase 1 implementation (Core Infrastructure)
2. Deploy to staging environment
3. Run load tests at 10x target scale
4. Onboard pilot tenants for beta testing
5. Iterate based on feedback
6. Public launch

This is not a "normie notification service."
This is infrastructure.

why does thios did not directly like show up on the localhost:3000 why plz make it show up