Discord

Assume 200M monthly active users, 50M daily active users, 10M peak concurrent WebSocket connections, 3M peak concurrent voice participants, 20M guilds, and 4B text messages per day. Average text message storage record is 1 KB after metadata, indexes, and small attachment references, but not including large file blobs.

Assume a peak multiplier of 10x over average message writes. Average online fanout per message is 20 recipients for normal channels, but very large guilds may have millions of members and only a much smaller active channel subscriber set. Voice uses an SFU where each participant sends one upstream audio stream and receives selected downstream streams.

• Message writes: 4B messages per day divided by 86,400 seconds is about 46,300 writes per second on average.
• Peak writes: using a 10x multiplier gives about 463,000 message writes per second.
• Gateway fanout: if each message reaches 20 online recipients on average, average delivery traffic is about 926,000 gateway events per second and peak is about 9.3M events per second.
• Storage: 4B messages times 1 KB is about 4 TB raw per day. With 3 replicas plus compaction and index overhead, plan for 12 to 20 TB per day.
• Hot storage: 30 days of replicated message history is about 360 to 600 TB. Older messages can move to colder storage tiers while remaining queryable.
• Voice ingress: 3M participants times roughly 50 Kbps upstream is about 150 Gbps entering media servers. SFU egress is higher because each listener receives selected streams from active speakers.
• Presence: 10M concurrent users times 6 presence transitions per hour is 60M transitions per hour, or about 16,700 per second before fanout. Naive fanout to every guild member would be unbounded, so presence must be subscription-based, batched, and suppressed for very large guilds.

• guilds.guild_id and channels.channel_id are primary keys for metadata lookups.
• guild_memberships.guild_id, user_id supports permission checks and member listing. A reverse index by user_id, guild_id supports listing a user's guilds at login.
• messages_by_channel_bucket.channel_id, bucket_start, message_id supports channel history range reads without scanning across all messages.
• read_state_by_user_channel.user_id, channel_id supports cross-device unread synchronization.
• Avoid secondary indexes on message content in the serving store. Search should use a separate indexing pipeline.

A guild owns many channels and memberships. A channel owns many message buckets. A user's read state references the latest message id they have read in each channel. Attachment blobs, search indexes, analytics, and audit logs are derived or side stores and should not be required for the message send hot path.

Cassandra or ScyllaDB fits message history because writes are append-heavy and reads are usually bounded ranges within a channel. Use time buckets to prevent a single popular channel partition from growing without bound. Store recent messages on fast nodes and use retention or tiering for cold history.

Redis or a similar in-memory store fits presence because presence is ephemeral and naturally TTL-based. Kafka, Pulsar, or a managed pub-sub service fits gateway fanout because message delivery should be asynchronous and retryable after persistence.

A Discord-like product has millions of long-lived WebSocket connections. One gateway node cannot hold arbitrary users, so the system shards sessions by user id, guild id affinity, or assigned shard id. Clients receive a gateway URL and shard plan, then identify and heartbeat at a fixed interval.

Session resume is critical because mobile networks and deploys cause frequent disconnects. The gateway attaches monotonically increasing sequence numbers to events and stores a short replay buffer per session or shard. On reconnect, the client presents its session id and last sequence. If the buffer still contains missed events, the gateway resumes; otherwise the client performs a full state sync.

Backpressure must be explicit. If a client cannot consume events, the gateway can drop typing indicators and presence deltas first, then require a resync for less critical state. It should not let one slow client block a shard's event loop.

The naive model sends every guild message to every guild member. That fails when a public guild has millions of members. The correct model is channel-scoped and activity-aware. A message event should go to online sessions that have permission to read the channel and are currently subscribed to that guild or channel view.

For normal guilds, fanout-on-write through gateway shards is efficient. For very large guilds or announcement channels, combine fanout-on-write for active viewers with fanout-on-read for inactive members. Users who open the channel later fetch history from the message store rather than receiving every event while away.

Membership and permission checks also need caching. Include a permission version in fanout events, and make gateways refresh or reject delivery if their cached view is stale. Large guilds should use lazy member lists, page-based member search, and aggregate counts rather than full rosters.

Presence looks simple but becomes one of the largest fanout sources. If a user belongs to 100 guilds, and each guild has thousands of members, broadcasting every online or activity change to every shared member is impossible.

The system should store presence as ephemeral state with a TTL and version. Gateway nodes subscribe users only to presence needed for open guilds, visible channel member lists, direct message friends, and active voice channels. For very large guilds, show approximate online counts or presence for a visible member window rather than every member.

Coalescing matters. Batch frequent activity updates, suppress no-op transitions, debounce flapping mobile clients, and prioritize status changes over rich activity changes. Presence should degrade before text or voice when the system is under pressure.

Message history is append-heavy and read by channel ranges. Partitioning only by channel can create huge partitions for active channels. Partitioning by channel and time bucket, such as day or hour, bounds partition size and keeps recent history efficient.

Snowflake-style ids encode time, worker, and sequence. They give globally unique ids and make pagination simple: fetch messages before or after a known id. Within a channel, order by message id. Cross-channel total ordering is unnecessary and would add coordination cost.

Edits and deletes should not rewrite large histories. Store edit version and delete state, append audit events if needed, and allow compaction to clean old tombstones. Attachments should live in blob storage with metadata references in the message row.

Unread count is not a separate counter that must be incremented on every message for every member. That would recreate the large fanout problem in the database. The scalable primitive is a per-user, per-channel last read message id.

When a user opens a channel, the client advances the read marker. Unread can be computed as messages after the marker, often with cached channel high-water marks and mention counters. Mention counts may be maintained asynchronously because exact counts for every inactive user are expensive.

Cross-device behavior should be monotonic. A stale mobile client should not move the read marker backward unless it explicitly requests a reset. Updates should be idempotent and compare message ids before writing.

Voice should not use the text gateway as the media transport. Clients join through a control API, receive media credentials, and send audio or video to a regional SFU. The SFU forwards selected streams to listeners, handles packet loss and jitter, and emits lightweight state such as speaking indicators.

Region selection is a product and reliability tradeoff. Pick a region near most participants, respect guild or channel preference, and watch load, packet loss, and round-trip time. If a region degrades, new joins should move elsewhere and existing rooms may be migrated with a brief reconnect.

Audio generally tolerates some packet loss better than high latency. Prefer UDP, use jitter buffers, forward error correction or packet loss concealment where appropriate, and avoid expensive server-side mixing unless the product needs recording or special effects.

Alternative one is a simpler WebSocket chat system with a relational database and Redis pub-sub. It works for small communities but does not handle large guild fanout, high write volume, or durable replay well.

Alternative two is fanout-on-read for all messages, where clients fetch new history instead of receiving gateway pushes. It reduces delivery fanout but makes chat feel less real-time and increases polling or sync costs.

Alternative three is server-side audio mixing instead of SFU forwarding. Mixing simplifies clients and can help recording, but it increases server CPU and can add latency. SFU forwarding is usually better for interactive voice and video rooms.

Do not use this full architecture for a small team chat or support widget. The cost of gateway sharding, wide-column storage, presence suppression, and SFU operations is only justified when the product needs large communities, persistent history, and high-concurrency voice.