Metrics Collection

Assume 3M active time series on average, 5M active series at peak, a 15-second default scrape interval, and 30 days per month. A raw sample has an 8-byte timestamp and an 8-byte float value before compression. Labels are stored once per series in the index and metadata store rather than repeated on every sample.

Assume compressed TSDB chunks average 2 to 4 bytes per sample with delta-of-delta and XOR encoding. Use 3 replicas for recent data, 15 days of raw retention, 180 days of 5-minute rollups, and 2 years of 1-hour rollups. Query traffic is assumed to be 1,000 range queries per second at peak across dashboards, alerts, and ad hoc users.

• Average ingest: 3M active series divided by a 15-second scrape interval is 200,000 samples per second.
• Daily samples: 200,000 samples per second times 86,400 seconds is 17.28B samples per day.
• Raw bytes: 17.28B samples times 16 bytes for timestamp plus value is about 276 GB per day, before labels, WAL, replication, and indexes.
• Compression: at 2 to 4 bytes per sample, daily compressed chunks are about 35 to 70 GB. This assumes stable scrape intervals, similar neighboring values, and chunk encoding with delta-of-delta timestamps and XOR values.
• Raw retention: 15 days times 35 to 70 GB is about 525 GB to 1.05 TB of compressed chunks. Add indexes, metadata, WAL, compaction workspace, and safety headroom to plan around 1.0 to 1.5 TB before replication.
• Replication: 3 replicas plus indexes and write amplification make recent storage roughly 5 to 8 TB.
• Rollups: 5-minute buckets create 288 points per series per day. For 3M series, that is 864M rollup points per day before label reduction. Dropping high-cardinality dimensions in rollups is essential.
• Sharding: at 500,000 peak samples per second, 128 shards receive about 3,900 samples per second each on average. 256 shards reduce per-shard load and provide room for hot series and tenant skew.

• metric_series.tenant_id, label_set_hash supports idempotent series creation and lookup during ingestion.
• metric_series.tenant_id, metric_name supports metric discovery and query planning.
• label_index.tenant_id, label_name, label_value, block_id is the inverted index used to resolve label matchers into series ids.
• tsdb_chunks.tenant_id, shard_id, series_id, chunk_start_ts supports range scans for one series on the owning shard.
• rollup_blocks.tenant_id, rollup_id, resolution, bucket_start_ts supports long-range queries without reading raw chunks.

Each metric series has many compressed chunks. Label index entries point to sets of series ids, not to individual samples. Rollup blocks can be derived from raw chunks and may intentionally drop high-cardinality labels. Scrape targets and ingestion clients produce samples, but the durable storage model is organized around series and time.

At production scale, use a purpose-built distributed TSDB rather than a generic row store for samples. The hot write path resembles Kafka or a partitioned log feeding shard-local TSDB writers. Recent chunks can live on fast SSDs, compacted immutable blocks can move to object storage, and metadata can live in a strongly consistent key-value store.

Cassandra, Bigtable, or DynamoDB can store time-bucketed samples for a simpler design, but they often waste space and struggle with PromQL-style label matching unless paired with an inverted index. ClickHouse, Druid, Pinot, or other columnar OLAP systems can work well for aggregated metrics, but raw high-frequency per-series writes still need careful batching and cardinality controls.

Counters, gauges, histograms, and summaries have different semantics. A counter is monotonically increasing except for resets, so queries usually use rate over a time window. A gauge is an instantaneous value such as memory usage, queue depth, or temperature. A histogram stores counts in configured buckets plus sum and count, which allows aggregation across instances and later percentile estimation. A summary often calculates client-side quantiles, which can be useful locally but is hard to aggregate correctly across many instances.

The ingestion layer should enforce type consistency for a metric name within a tenant. If one service emits request_latency_seconds as a gauge and another emits it as a histogram, queries become misleading. Metadata should record the metric type and reject or quarantine incompatible samples.

Histograms are usually the best fit for distributed percentiles because bucket counts can be summed by service, region, or cluster. Summaries can be retained for application-specific local quantiles, but interview answers should explain why they are not generally aggregatable.

Pull collection, popularized by Prometheus, lets the monitoring system control scrape intervals, discover targets, detect missing targets, and apply backpressure centrally. It is operationally clean for long-running services and Kubernetes pods because the scraper knows whether a target failed to respond.

Push collection is useful for short-lived jobs, serverless functions, network-isolated environments, and StatsD-style UDP clients. It can be simpler for developers but makes liveness harder: if a client stops pushing, the backend must distinguish silence from success unless heartbeat metrics exist.

A production platform can support both by converging on the same internal sample format. The key is to keep push gateways stateless or lightly stateful, avoid unbounded buffering, and make pull scrape health visible as first-class metadata.

A time series is defined by metric name plus the complete label set. Labels such as service, region, endpoint, method, and status are valuable because they allow slicing and aggregation. Labels such as user_id, request_id, session_id, trace_id, or raw URL can create one series per request and overwhelm the system.

Cardinality grows multiplicatively. If a metric has 100 services, 20 endpoints, 5 status classes, 10 regions, and 4 methods, it can create 400,000 series before considering instances. Adding a label with 1M possible user ids makes the metric impossible to store and query economically.

Protect the system with per-tenant active series quotas, per-metric limits, label name allowlists or denylists, value length limits, high-cardinality detection, and ingestion-time rejection for new series once budgets are exceeded. Good user feedback matters: teams need to know which metric and label caused the explosion.

Metrics are append-heavy and mostly ordered by time. A TSDB writer keeps an in-memory head block for recent samples and a write-ahead log for crash recovery. Periodically it cuts immutable chunks or blocks, writes them to disk, and later compacts them into larger files.

Delta-of-delta timestamp compression stores the change in interval rather than every full timestamp. With a stable 15-second scrape interval, the delta-of-delta is often zero or very small. XOR value compression stores the XOR between neighboring floating-point values, which is compact when values change slowly or repeat. These techniques can reduce samples from 16 raw bytes to a few bytes.

Labels should not be repeated on each sample. Store labels once in the series catalog and index, then store sample chunks by series id. This avoids huge write amplification and keeps the hot path append-oriented.

A query engine first resolves selectors. For example, a selector for metric http_requests_total with labels service equals checkout and region equals us-east-1 becomes an intersection of postings lists from the inverted index. The result is a set of series ids to read.

The engine then fetches chunks for the requested time range, decodes samples, aligns them to evaluation steps, and applies functions. Rate calculations need enough points before and after counter resets. Aggregations such as sum by service merge series while preserving chosen grouping labels.

The main risk is fanout. A small-looking query can match millions of series if a label matcher is broad. The frontend should estimate cost before execution, enforce maximum series and sample counts, split work by time and shard, cache common dashboard results, and cancel queries that exceed budget.

Raw 15-second samples are valuable during active incidents but expensive for multi-year retention. Older data is usually queried at coarser resolution, such as 5 minutes or 1 hour. Rollup workers can aggregate count, sum, min, max, last value, and histogram buckets into lower-resolution blocks.

Rollups are not just storage compression; they are also query acceleration. A 30-day dashboard with a 5-minute step should not decode every 15-second raw sample if a precomputed 5-minute rollup exists. The query planner should choose the lowest-cost resolution that satisfies the requested step and accuracy.

Downsampling can lose detail. Spikes shorter than the rollup interval may be hidden unless min and max are retained. Percentiles require histograms or sketches, not just average values. A good design clearly states what accuracy is preserved in each tier.

Alternative one is a pure Prometheus-per-team model. It is simple, easy to operate for small teams, and excellent for local reliability, but global querying, long retention, and cross-team governance become harder.

Alternative two is a StatsD-only push pipeline into a stream processor and columnar OLAP store. It is easy for applications to emit counters and timers, but target liveness, scrape health, and rich label discovery are weaker than pull-based monitoring.

Alternative three is to store metrics as logs and aggregate them later. This can reuse logging infrastructure, but it is far more expensive for high-frequency numeric time series and usually cannot match TSDB query latency or compression.

Do not use a metrics platform as the source of truth for billing, financial transactions, audit logs, or request-level debugging. Metrics are aggregated, sampled, rolled up, and eventually expired. Use logs, traces, event streams, or transactional databases when every individual event must be preserved exactly.