Monitoring System

Assume 100,000 monitored resources across hosts, containers, serverless functions, databases, and managed cloud services. Each resource emits 200 active metric series on average, sampled once every 10 seconds. Assume logs average 1 KB per second per resource after filtering, and traces are sampled to 500,000 spans per second across the fleet.

Assume 80 bytes per encoded metric sample before compression, 250 bytes per sampled trace span after basic attribute pruning, 30 days of log retention, 45 days of metric retention, 7 days of raw trace retention, 200,000 active alert rules, and a 10x burst multiplier during large incidents or regional outages.

• Metric series: 100,000 monitored resources times 200 active series is 20M active series.
• Metric ingest: 20M series sampled once every 10 seconds is 2M samples per second on average. A 10x burst gives 20M samples per second.
• Metric storage: 2M samples per second times 80 bytes is 160 MB per second raw. That is about 13.8 TB per day raw. With 4:1 compression, store about 3.45 TB per day. Over 45 days, reserve about 155 TB before replicas and index overhead.
• Log ingest: 100,000 resources times 1 KB per second is 100 MB per second. That is about 8.6 TB per day raw. With 3:1 compression over 30 days, reserve about 86 TB before replicas and search index overhead.
• Trace ingest: 500,000 sampled spans per second times 250 bytes is 125 MB per second. That is about 10.8 TB per day. Over 7 days, reserve about 76 TB before replication.
• Alert evaluations: 200,000 active rules evaluated every 60 seconds means about 3,333 rule evaluations per second. The engine should evaluate from precomputed rollups rather than scanning raw samples.
• Notifications: if 1 percent of alert rules fire during a major outage, that is 2,000 initial alert events. Grouping by service, region, and severity should collapse this into far fewer incidents before paging humans.

• metric_series.tenant_id, metric_name, tag_hash supports series lookup during ingestion and query planning.
• metric_chunks.tenant_id, series_id, bucket_start is the primary access path for dashboard and alert window scans.
• telemetry_event_index.tenant_id, event_time, service supports recent log and trace exploration by service and time.
• telemetry_event_index.trace_id supports direct trace waterfall retrieval and log-to-trace correlation.
• alert_rules.tenant_id, enabled lets scheduler shards find active rules for a tenant.
• alert_incidents.tenant_id, fingerprint, state enforces deduplication of active incidents.

Each tenant owns metric series, event indexes, alert rules, dashboards, notification routes, SLOs, and incidents. Alert rules reference metric queries or synthetic check streams and create alert incidents through a dedupe fingerprint. Logs and spans share correlation identifiers so a dashboard point can jump to related logs or a trace waterfall without joining on the ingestion hot path.

Metrics are usually stored in a specialized time-series database or columnar store such as M3, Cortex, Thanos, VictoriaMetrics, ClickHouse, Druid, Bigtable, or a custom LSM-backed TSDB. Logs fit object storage plus a search index or columnar lake. Traces fit a span store indexed by trace ID, service, operation, and time.

Rule definitions, silences, routes, SLO definitions, audit logs, and incident state can live in a strongly consistent relational database or a replicated key-value store because they are smaller but more correctness-sensitive than raw telemetry.

A monitoring system should explain the three pillars clearly. Metrics are compact numerical time series that answer what changed and when. Logs are detailed event records that explain why a specific error happened. Traces connect work across services and show where latency or errors occurred in a distributed request.

The platform should not treat them as interchangeable. Metrics power alerting, dashboards, and SLO burn-rate calculations because they are cheap to aggregate. Logs support debugging and audit trails, but they are more expensive to index and query. Traces provide request-level causality, but sampling is usually required. Correlation fields such as trace ID, service, region, deploy version, customer tier, and timestamp let users move from an alert to the relevant logs and traces quickly.

The time-series store is optimized around tenant, metric name, tag set, and time bucket. Compression works well when samples for the same series arrive in order and have similar timestamps or values. Rollups at 1 minute, 5 minutes, 1 hour, and 1 day resolutions keep long-range dashboards affordable.

The biggest operational risk is cardinality explosion. A label such as request_id, user_id, session_id, or raw URL can create millions of unique series and make both storage and queries fail. The ingestion gateway and processors should enforce label allow lists, drop unbounded labels, limit unique series per tenant, and surface cardinality reports so customers can fix instrumentation.

Alert rules should evaluate on stable, bounded windows instead of querying arbitrary raw data every second. A threshold rule might evaluate the last 5 minutes every 1 minute. A burn-rate rule might evaluate both 5-minute and 1-hour windows to catch fast outages without paging on tiny blips.

Rule workers should shard by tenant and rule ID, read rollup chunks, apply no-data semantics, and write an idempotent evaluation result. For counters, the engine must handle resets. For percentiles, it should use histogram summaries or sketches rather than averaging p95 values incorrectly. Late data should be either ignored after a grace period or handled through explicit re-evaluation policies.

Paging one person for every matching time series is a common failure mode. The system should compute an incident fingerprint from tenant, rule, service, environment, region, and chosen group labels. Repeated firings update the same incident with dedupe_count and last_fired_at rather than creating a new page every minute.

Grouping collapses related alerts into one incident view. Silences and maintenance windows suppress known work. Escalation policies decide when to page primary on-call, secondary on-call, managers, or a customer webhook. Recovery windows prevent flapping by requiring the condition to remain healthy for several evaluations before resolving.

An SLI is a measured reliability signal such as successful requests divided by total requests, p99 latency under a threshold, or synthetic checkout success rate. An SLO is the target for that SLI over a window, such as 99.9 percent success over 30 days. The error budget is the allowed badness left before the team violates the SLO.

The monitoring system should compute SLO windows from metrics and synthetic checks, show remaining budget, and alert on burn rate instead of only absolute error count. Multi-window burn-rate alerts reduce noise by paging when the service is burning budget both quickly and consistently.

Logs and traces can dwarf metric volume. The system should encourage structured logs, field redaction, per-tenant quotas, dynamic sampling, and retention tiers. High-value error traces can be kept at a higher rate than routine successful traces.

Tail sampling can keep traces with errors, high latency, or rare routes after seeing the full request. Head sampling is cheaper and simpler but may drop the one trace needed during an incident. A premium design lets teams configure sampling by service, route, status, tenant tier, and incident mode.

Alternative one is a metrics-only monitoring system. It is simpler and cheaper, and it may be enough for infrastructure health, but it cannot provide full incident investigation because logs, traces, SLO context, and notification workflows are missing.

Alternative two is a log-first observability system that stores all telemetry as events in a search engine. It gives flexible debugging, but metric alerting and long-range dashboards become expensive unless rollups and specialized time-series indexes are added.

Alternative three is to rely on separate best-of-breed tools for metrics, logs, tracing, paging, and dashboards. This reduces platform scope but weakens correlation, ownership, consistent RBAC, and alert fatigue controls.

Do not build a full Datadog-style platform for a small application with a few services and modest reliability needs. Managed monitoring, cloud provider metrics, and a simple paging integration are usually enough until telemetry volume, cross-service debugging, compliance, or alert fatigue justify the complexity.