How I'd design a ChatGPT-style chat service

I'd ask whether we're training a model or building an application around one. For this answer, it's the latter. A managed provider supplies inference; we own the accounts, conversation history, streaming and billing limits.

I'd also leave uploads, voice, web browsing and tool execution out of the first version. A document assistant needs retrieval and document-level permissions. An agent that can run code needs an execution boundary. Neither is required to answer this particular question well.

Here's what the product does need:

• A user can start a conversation, send a message and watch the answer arrive.
• The conversation is still there when they open another device.
• A short network interruption doesn't automatically start a second model call.
• They can stop a response, and the service has limits on how much work one account can start.

I'd allow one answer to be generated at a time within a conversation. Two tabs can read it, and two different conversations can run in parallel. Two simultaneous follow-ups in the same conversation need a defined order; rejecting the second one is a reasonable starting point.

For responsiveness, I'll use a p95 target of two seconds to the first token for traffic we admit. That's a target to test, not something the choice of SSE or a particular database can guarantee.

Suppose we have a million daily users, each sending ten messages a day. I'll call one user message and its assistant response a turn. Assume a five-times peak and a twenty-second average response stream.

Six hundred requests a second isn't an unusual API workload. Keeping twelve thousand response streams open takes some planning, but the bigger number here is the model traffic: about 1.8 million input tokens and 300,000 output tokens per second at peak.

That's where I'd spend time on capacity. A provider might accept the HTTP connection while still making us wait for inference. If we're self-hosting, prompt prefill, output length, batching and model size all affect throughput. There isn't enough information here to give a credible GPU count.

The 3,000 input tokens include conversation context sent with each request. They aren't all newly stored text. The 30 GB/day storage estimate is separate: roughly 3 KB of new user and assistant text per turn, before indexes, replication, backups and stream buffers. All storage figures use decimal units.

One implication is worth keeping in mind: adding API servers won't fix a model quota that's already exhausted. I'll need an admission limit as well as a queue.

Consider a timeout just after the user presses Send. The server may have received nothing. Or it may have saved the message and started generating an answer before the response got lost.

The browser can't tell those apart. I want it to retry the same operation, rather than guess whether to create another one. That means the submitted turn needs an identity of its own.

I'd keep conversations, messages and generations in a relational database. A generation is one attempt to produce an answer. Its ID stays the same when the browser reconnects or retries the original submission.

A small API could look like this. These are endpoints for the proposed chat service, not Skiva's API:

POST /v1/conversations
POST /v1/conversations/{id}/messages
  Idempotency-Key: turn_abc
  { "content": "...", "expected_version": 7 }
  → 202 { "generation_id": "gen_123", "state": "queued" }

GET  /v1/generations/{id}/events    # authenticated SSE
GET  /v1/generations/{id}           # authoritative state
POST /v1/generations/{id}/cancel
GET  /v1/conversations/{id}/messages?before=42

If the browser misses that 202 response, it sends turn_abc again. The server returns gen_123, not a newly created generation. A unique constraint on (tenant_id, conversation_id, request_key) enforces this. I'd store a hash of the request body too: the same key with different text should be a conflict, not a surprising replay of an old answer.

Acceptance happens in a short database transaction. Lock the conversation row, check its version and active generation, reserve the token budget, then save the user message, queued generation and an outbox event together. After commit, release the lock.

The model call happens later. Holding a row lock for a twenty-second response would make every other operation on that conversation wait on inference. The 202 only tells the client we've durably accepted the work.

The smallest version could call the model from the request handler and stream its output straight back. That works until the API process restarts, the client disconnects, or a busy model holds too many handlers open.

I'd move generation into workers and let the API return once it has accepted the turn. The browser then opens a separate connection for output. Now a browser reconnect doesn't have to restart inference, and the processes handling web connections can scale independently of the workers.

There's a failure to deal with before drawing that queue, though. What if the database commits the new turn, then the process crashes before publishing the job? The user has an accepted message that no worker knows about.

That's why the acceptance transaction also saves an outbox row. A relay reads committed rows and publishes them. It can publish twice if it loses an acknowledgement, so workers still need a conditional claim. The outbox closes the gap between saving work and remembering to dispatch it.

Follow gen_123 through the diagram: the conversation service accepts it, the relay publishes its job, and a worker claims it. The worker loads a bounded history, calls the model and writes output batches to the shared replay buffer. A streaming gateway forwards those batches to the browser. At the end, the worker saves the completed assistant message in the database.

I wouldn't deploy every box as a separate service on day one. The diagram separates responsibilities. The conversation API and streaming gateway can start in one deployment; the outbox relay and usage reconciliation can share a worker deployment. The useful split is between short API work and long-running inference.

I'd use server-sent events for text output. The browser submits messages and cancellation requests over normal HTTP; the long-lived connection mostly carries data in the other direction. WebSockets would also work, but I don't need their bidirectional channel for this version.

Each output event gets a sequence number within its generation. For example:

id: gen_123:17
event: token.delta
data: {"text":"a bounded output batch"}

id: gen_123:18
event: completed
data: {"message_id":"msg_456"}

Suppose the browser received event 17 before losing its connection. It reconnects with that last event ID, and the gateway sends later events from a shared buffer. Keeping this buffer on one gateway's local heap wouldn't help if the reconnect landed on another machine.

The gateway must check ownership before replaying anything. Knowing gen_123:17 doesn't grant access to the conversation. The browser should also ignore event IDs it has already applied; reconnects can deliver duplicates.

I'd start with a ten-minute replay window and a byte limit per generation. I'd batch small token fragments rather than send a separate database write for every fragment, and save occasional partial checkpoints to the conversation database.

This gives us a useful reconnect feature, with a limit. If the replay buffer is lost, we can return a completed message from the database. We can't promise to recover every token from an unfinished answer. In that case, show the last saved checkpoint as incomplete and report the interruption. Quietly skipping missing text would leave the user with an answer that looks complete but isn't.

There's one more case: the final answer has been saved, but the browser misses the completed event. On reconnect, check the database state as well as the replay buffer. Otherwise the UI can sit on "Generating..." while the finished answer is already there.

I wouldn't cancel inference the instant a socket closes. Mobile connections drop, and users refresh tabs. Give reconnects a short grace period; use an explicit Stop request for cancellation. Slow readers still need a bounded gateway buffer, so disconnect them when necessary and let them reconnect. Heartbeats and disabled proxy buffering are small configuration details that matter here.

The queue might deliver the same job to two workers. Before calling the model, each tries to change the generation from QUEUED to RUNNING. Only one conditional update succeeds. That worker records a run token, which it must supply with every checkpoint and final write.

Now suppose the user presses Stop just as the last tokens arrive. Cancellation and completion both try to make a terminal database update. Whichever commits first wins. A late completion can't overwrite CANCELLED; a cancellation arriving after completion returns the completed state. The stream gateway has to follow that result too.

The awkward case is a worker disappearing after the provider has started its call. Our database can prevent another worker from claiming the original queued job, but it doesn't control what the external provider has already done. Unless that provider supports the right idempotency or resume behavior, claiming "exactly once" generation would be misleading.

I'd choose a conservative first version: don't automatically take over a model call that has already started. Mark the lost attempt interrupted, reject late durable writes from its old run token and request upstream cancellation where possible. Keep any partial checkpoint. Let the user decide whether to start a new attempt.

That isn't seamless recovery, and it doesn't undo the cost of the abandoned call. It does avoid quietly launching a second paid answer while pretending to continue the first. Automatic retries are much easier to justify before inference has started.

If we keep accepting work faster than the model can finish it, the queue grows and the first-token target becomes fiction. I'd prefer an early, clear rejection to a spinner that waits indefinitely.

For the two-second target, a normal queue wait of around 250 ms leaves some room for the rest of the request. That's an initial budget to measure. Queued work also needs a hard start deadline. If it expires, mark the generation failed with a capacity error; don't reset its clock every time the job is redelivered.

Before acceptance, check account request rates, concurrent generations, token limits, the tenant's remaining budget and provider capacity. Reject new work with 429 or 503 and a retry hint when needed. After acceptance, a capacity timeout is a failed request and belongs in the service metrics.

I'd reserve an upper bound on the model cost before starting a call, then settle the actual usage once. Unused reservation is released, not charged again. If the final usage report is missing, keep the attempt marked for reconciliation rather than assuming it was free.

The earlier token estimates make it possible to put a rough price on this hypothetical workload:

30,000 × input price per million tokens
 + 5,000 × output price per million tokens
 = estimated model cost per day

Additional model attempts, hosting and storage are extra.

This isn't a Skiva bill or a provider quote. It shows why context length matters: sending an ever-growing conversation back to the model can be expensive even when relatively few new words are saved. Context needs a bound, with an explicit truncation or summarization policy.

Scheduling matters too. One account submitting many long prompts shouldn't consume all available model slots. I'd use per-tenant concurrency limits and fair scheduling, then scale streaming connections separately from inference. More workers don't buy more provider token quota.

A note on isolation

Every query for a conversation, message or generation needs the tenant and ownership check, including reconnects. Take tenant identity from authentication, never from a field the client submits. Keep provider credentials on the server and send only the authorized conversation context to the model.

I'd avoid logging prompt and response bodies by default. Shared caches need tenant-aware keys, and retention and deletion need a policy. Adding document retrieval or tool execution would require more work on permissions and prompt injection; neither becomes safe merely because the chat service is authenticated.

At this point, I'd stop adding boxes and try to break the path we've drawn. A few tests would tell me more than another round of component selection:

1. Lose the acceptance response. Commit the turn, drop the 202 and retry with the same key. There should be one generation. Change the text while keeping the key and check that the server returns a conflict.
2. Publish a job twice. Crash the relay after publication but before it records the acknowledgement. On redelivery, check that only one worker starts the model call.
3. Lose the last stream event. Save the answer, then close the connection before the browser sees completion. Reopening the conversation should show the saved reply, not an endless spinner.
4. Kill a worker during inference. The partial reply should stay marked incomplete. Check that the old worker can't later overwrite the terminal state and that uncertain provider usage isn't treated as zero.
5. Race Stop against completion. Repeat it under load. The database, stream and browser should agree on which result won.

I'd measure queue age and provider time separately, along with first-token latency, completion rate, interrupted attempts and cost per completed turn. If first-token latency rises while queue age stays low, adding queue consumers may not help. The provider or prompt size is a better place to look.

I'd keep SSE, the relational database and a managed model for this version. None is the only possible choice. They're enough to explain the behavior we agreed on without taking a detour into operating a GPU fleet.

The limitations are fairly specific now. We serialize turns within a conversation. Replay is bounded. An interrupted model call may require a new attempt. A model provider outage can still stop generation even while history and account APIs remain available.

Those are useful trade-offs to discuss with the interviewer. If branching conversations are essential, the history model needs to change. If voice is essential, I'd revisit the transport. If provider cost or control justifies self-hosting, I'd benchmark real prompts and output lengths before sizing that fleet.

For the original text-chat problem, I'd spend the remaining time on those questions rather than add another service to the diagram. We can explain where a turn is saved, who owns its generation, what survives a disconnect and when another model call is allowed. That's a workable place to finish.

The figures can be downloaded as Skiva JSON and imported into your workspace. They contain editable nodes, connections and notes, so you don't have to redraw the example to argue with it.

A useful next exercise is to remove the managed provider. Which parts of the design actually change when inference is self-hosted? If you're using an MCP agent, ask it to read the current diagram and propose that change:

Read the AI chat diagram in my Skiva workspace.
Replace the managed provider with self-hosted inference.
Explain how scheduling, capacity and failures would change.
Keep the conversation API and tenant boundaries intact.
Submit a proposal for review; don't approve it.

Or keep the diagram as it is and ask the agent to trace a worker crash after event 17. Check its answer against the saved generation state and replay window. If it says "just retry," ask what happens to the first model call and who pays for it.