Long Compute Inference

3 min read

The user submits a request that is going to take 30 seconds or more. Image generation. A complex analytical query. A long-context reasoning task. The user has crossed the unit-task boundary; static skeletons no longer carry their weight; engagement is the last move before they leave.

This scenario sits in the 10 S+ band. Block & Zakay 1997's Block & Zakay 1997 meta-analysis frames the trade-off — engagement compresses prospective duration while expanding retrospective duration; the design decision is whether you have made that trade deliberately. Fitch's Fitch Slack and FIFA examples are the canonical references; Myers 1985 Myers 1985 is the determinate-progress fallback where the inference reports phases.

What is happening

The ai-streaming demo stands in. For a real long inference, the tuned flow stacks more layers:

1. Thinking state during the time-to-first-token gap — the same dots-and-cursor pattern from ai-chat-streaming-response.
2. Streaming render as soon as the first token arrives, paced to a natural reading rhythm.
3. Tool-call transparency if the inference involves visible tool calls — narrate them ("Searching…", "Reading…", "Reasoning…").
4. Determinate progress if the inference can report phases ("Step 3 of 7"); fall back to engagement otherwise.
5. Cancellation always available — the stop button must respond inside the perceptual frame even if the abort takes longer.
6. Background fallback — past 30–60 seconds, offer "do this in the background and notify me" so the user can leave the surface.

What to tune

1. Pre-action — submit button echo within ~50 ms; thinking dots cover the time-to-first-token gap.
2. First 1 s — thinking state in place. No spinner, no skeleton over content the model will produce.
3. 1 – 10 s — token streaming where text is the output. Tool-call transparency where the work is visible.
4. Past 10 s — engaging copy where applicable; determinate progress where the inference reports phases. Cancellation always visible.
5. Past 30–60 s — hand-off to background sync with notification. The foreground is no longer the right surface.

When perceived performance hurts you here

The engagement-vs-retrospective-duration trade is the central trap. A 30-second inference with rich engagement feels short while it runs and long in retrospect — the user remembers it as taking forever even when their session went smoothly. Slack and FIFA accept this; for inference where the user repeats the action many times in a session, the retrospective cost compounds.

The cleaner answer for repeat-use AI inference: ship determinate progress where measurable, tool-call transparency where applicable (the user is learning during the wait), and background sync past 60 s. Generic engaging content (motivational quotes, mini-games) belongs only on rare or one-off inferences.

Accessibility

1. aria-live="polite" on the streaming output and on tool-call narration.
2. aria-busy="true" during the inference; flip on completion.
3. prefers-reduced-motion: reduce — replace cross-fades and pulse animations with static states.
4. Always provide a way out — visible cancellation, visible "do in background", visible re-attempt on failure.

Other patterns that fit

Same band, same surface — switch in whichever maps best to your UI.