Logging System

Assume 200,000 hosts across production, staging, and shared infrastructure. Each host emits 10 log events per second on average, with a 5x incident burst. Average event size is 1 KB before compression, including timestamp, service, severity, message, labels, trace_id, and request context.

Assume one Kafka replica factor of 3, four-to-one compression for long-term object storage, one hot full-text searchable tier for 7 days, one warm searchable tier for 23 additional days, and one cold object-storage tier for 365 days. Hot full-text indexing is estimated at 1.5x raw event size for primary index structures and one replica for availability.

• Event rate: 200,000 hosts times 10 events per second is 2,000,000 events per second average.
• Daily events: 2,000,000 events per second times 86,400 seconds is 172,800,000,000 events per day.
• Raw volume: 172.8B events times 1 KB is about 172.8 TB per day, rounded to 173 TB per day.
• Peak bandwidth: 10M events per second during a 5x burst times 1 KB is 10 GB per second before protocol overhead.
• Kafka buffer: keeping 72 hours of raw data at 173 TB per day is 519 TB. With three replicas, reserve roughly 1.6 PB before compression and operational headroom.
• Hot index: full-text indexing can cost about 1.5 times raw size for primary data. One replica doubles it, so 173 TB times 1.5 times 2 is about 519 TB per hot day. Seven days is about 3.6 PB.
• Cold storage: object storage with four-to-one compression stores 173 TB divided by 4, or about 43.25 TB per day. One year is about 15.8 PB.
• Partitions: if a Kafka partition safely handles about 20 to 30 MB per second sustained in this workload, 10 GB per second peak needs about 334 to 500 partitions. Choose 400 to 600 plus tenant-aware partitioning.
• Shards: if one hot primary shard should stay near 100 to 200 GB, a 173 TB raw day expanded to about 260 TB primary index data needs roughly 1,300 to 2,600 shards. With tenant and tier optimization, plan around 1,000 to 2,000 primary shards per hot day and adjust based on measured shard size.

• tenants.tenant_id is the primary key for control-plane lookups.
• log_streams.tenant_id, service, environment supports stream discovery and label validation.
• index_segments.tenant_id, start_time, end_time, tier supports query planning by tenant and time range.
• retention_policies.tenant_id, policy_version supports safe policy updates and rollbacks.
• query_audit_logs.tenant_id, created_at supports compliance review and quota analysis.

Each tenant owns many log streams, retention policies, index segments, and query audit rows. Index segments are the catalog entries that tell the query planner where data lives; they point to external search clusters and object storage rather than storing log events in the metadata database.

For the data plane, use Kafka or Pulsar for the durable ingestion buffer, Elasticsearch or OpenSearch for full-text hot search, Loki or a custom chunk store for label-indexed logs, ClickHouse or Druid for analytical aggregation, and S3, GCS, or Azure Blob Storage for compressed cold retention.

The metadata store can be PostgreSQL, MySQL, Spanner, FoundationDB, or DynamoDB depending on consistency and scale needs. It is low-volume compared with the log events, but it is critical for authorization, query planning, and retention enforcement.

The most important design choice is where the system acknowledges writes. If the gateway acknowledges before durable buffering, a gateway crash loses accepted logs. If it waits for full indexing, every index outage becomes an ingestion outage. A strong design acknowledges after Kafka quorum append.

Backpressure should be explicit and priority-aware. The gateway can return 429 with retry-after for tenants over quota, agents can buffer locally to disk for a bounded time, and the pipeline can shed low-value debug logs before audit or error logs. Kafka lag becomes the signal for autoscaling consumers, increasing sampling, or temporarily disabling expensive enrichments.

The system should also avoid infinite buffering. Local agent buffers need size limits, Kafka retention needs enough time for index recovery, and tenants need drop policies when they exceed contractual limits. Silent loss is worse than an explicit rejected count.

Structured JSON logs are much cheaper to process than arbitrary text because fields are already named and typed. The platform should encourage services to emit structured logs with standard fields such as timestamp, level, service, environment, trace_id, request_id, customer_id hash, and region.

Unstructured logs still matter for legacy systems. Pipeline workers can use configured parsers, but parser failures must not block ingestion. Store the original message, add a parse_error flag, and route failures to a dead-letter stream for rule fixes.

Enrichment increases query value but can also increase cost and cardinality. Add stable metadata from deployment systems and service catalogs, but be careful with high-cardinality labels such as user_id, request_id, and full URL. Those belong in searchable fields or raw payloads, not always in index labels.

Elasticsearch and OpenSearch build inverted indexes for text and selected fields. This is excellent for ad hoc searches such as finding an error phrase across services, but it multiplies storage and CPU cost. Mapping explosions, high-cardinality fields, and too many small shards can take down the cluster.

Loki-style systems intentionally index only labels such as tenant, service, environment, region, and level. The actual log lines are compressed into chunks in object storage. Queries first use labels and time ranges to find chunks, then scan those chunks. This is cheaper and operationally simpler at very high volume, but arbitrary free-text search over broad time ranges is slower.

A premium answer proposes a hybrid. Keep full-text indexes for recent high-value logs, errors, audits, and selected services. Store debug and trace-heavy logs as label-indexed chunks. Let tenants choose policies by value and budget rather than forcing one indexing model on all data.

Hot storage is optimized for low query latency and write throughput. It uses fast disks, enough replicas, and recent time-based indexes. This is the most expensive tier, so retention should be short, often 3 to 14 days depending on customer needs.

Warm storage keeps data searchable but cheaper. It may use slower nodes, fewer replicas, force-merged segments, searchable snapshots, or reduced field indexes. Query latency is higher, but incident investigations can still search the last few weeks.

Cold storage is compressed object storage. It is durable and inexpensive per byte, but search is slower and should run as an asynchronous job. Lifecycle jobs must update the segment catalog atomically so queries know which tier owns each time range.

Log search is dangerous because a single query can fan out to thousands of shards and scan terabytes. Require tenant scope and time bounds. Estimate query cost before execution using segment metadata, field statistics, and tier information.

For hot queries, fan out to relevant shards in parallel and stream the first results as soon as possible. For cold queries, create an asynchronous job with progress, partial results, and cancellation. Use time-sorted pagination so users can continue investigations without rescanning from the beginning.

Protect the cluster with query budgets, concurrency limits, shard fan-out caps, result limits, and cancellation on client disconnect. Cache common field lists, recent dashboard queries, and small repeated searches, but avoid caching sensitive cross-tenant data.

The platform will see log storms during bad deploys, dependency outages, retry loops, and security incidents. Those are exactly the moments when logs are most useful, so the design should degrade by priority instead of failing uniformly.

Use per-tenant and per-service quotas, priority classes, and sampling rules. For example, keep all ERROR and audit logs, sample DEBUG logs after a threshold, and aggregate repeated identical messages into counters. Make dropped and sampled counts visible so engineers do not mistake absence of logs for absence of failures.

Isolation can be logical or physical. Small tenants can share Kafka topics and index clusters with strict quotas. Regulated or high-volume tenants may need dedicated topics, index namespaces, encryption keys, or even clusters. Query isolation is as important as ingest isolation because one expensive search can starve everyone.

Alternative one is a simple ELK stack where Logstash writes all events into Elasticsearch. It is easy to understand and powerful for search, but it becomes expensive and fragile at very high volume unless mappings, shards, and retention are tightly controlled.

Alternative two is a Loki-style architecture with label indexes and compressed chunks in object storage. It is cost-effective for high-volume logs when users usually query by labels and time, but broad free-text search over cold data is slower.

Alternative three is an analytics database such as ClickHouse for structured logs. It can be excellent for aggregations, dashboards, and columnar scans, but it is not a drop-in replacement for arbitrary full-text debugging unless paired with text indexes or careful schema design.

Do not use the logging system as the primary source for precise metrics, distributed tracing, financial ledgers, or user-facing audit state. Logs are high-volume diagnostic evidence and may be sampled, delayed, redacted, or retained by policy. Use metrics for alerting math, traces for request causality, and transactional databases for authoritative business state.