I almost shipped a bad AI integration once. The feature worked technically: the model answered questions correctly, the UI looked clean, and everyone nodded along in our internal demo. Then we put real users on it, and within 48 hours a product manager Slack'd me asking why 60% of sessions were ending with no response received.
The problem wasn't the API or the server. It was eight seconds of silence.
Users sent a message, saw nothing happen, assumed the request had failed, and closed the tab. The response would arrive shortly after, to nobody. We had built a functional AI feature that felt completely broken, and the fix wasn't an architectural overhaul or a performance optimization. It was streaming: start sending tokens the moment the model produces them, instead of batching the full response and delivering it at once.
I have seen this mistake in codebases across multiple companies now. The non-streaming version is simpler to write, tests pass, and nobody notices in development because developers wait for things, while real users usually do not.
This tutorial builds the complete streaming implementation: a Node.js backend using Server-Sent Events to push tokens to the browser as they arrive, with session management that gives the model memory across turns, and proper abort handling so you are not paying for tokens nobody reads.
The complete code is at github.com/ziaongit/nodejs-openai-streaming.
No OpenAI key? I tested this entire article using Groq instead. It is free, it speaks the same OpenAI SDK format, and getting a key takes about two minutes. The repo has a
USE_GROQ=trueflag already wired in. Copy.env.exampleto.env, paste your Groq key, and you are running. I usedllama-3.3-70b-versatile. The output is good enough that you would not notice the difference in a dev session.
Prerequisites
- Node.js v18.11 or later
- An OpenAI API key
- Solid familiarity with Express and async/await
The Architecture Before Any Code
Before we write any code, it helps to see what is actually happening across the three layers. Debugging streaming issues without this picture gets painful fast.
┌──────────────────────────────────────────────────────────────────┐
│ Browser │
│ │
│ 1. POST /session ─────────────────────► receives sessionId │
│ 2. POST /chat { message, sessionId } │
│ 3. response.body.getReader() ──► ReadableStream │
│ 4. decode chunks ──► parse SSE events ──► append tokens to UI │
└────────────────────────────┬─────────────────────────────────────┘
│ HTTP (text/event-stream)
▼
┌──────────────────────────────────────────────────────────────────┐
│ Express Server │
│ │
│ sessions Map ── { sessionId: { messages: [...] } } │
│ │
│ POST /chat │
│ ├─ load session history │
│ ├─ push user message │
│ ├─ set SSE headers + flushHeaders() │
│ ├─ openai.create({ stream: true, signal: controller.signal }) │
│ ├─ for await chunk → res.write(`data: ${token}\n\n`) │
│ ├─ accumulate fullResponse → push to session │
│ └─ send { done: true } → res.end() │
└────────────────────────────┬─────────────────────────────────────┘
│ HTTPS chunked transfer encoding
▼
┌──────────────────────────────────────────────────────────────────┐
│ OpenAI API │
│ gpt-4o-mini · stream: true │
└──────────────────────────────────────────────────────────────────┘
The session store lives on the server. Every time the user sends a message, that full conversation history goes to OpenAI inside the messages array, which is how the model knows what was said earlier. The API itself remembers nothing between calls, so you carry the state.
The mechanism is simpler than it sounds: open a regular HTTP response and never close it. The server keeps writing chunks into the socket as tokens arrive, and when the model finishes you call res.end() and the connection drops.
Project Setup
mkdir nodejs-openai-streaming
cd nodejs-openai-streaming
npm init \-y
npm install express openai cors dotenv
mkdir public
Add `"type": "module"` to `package.json`:
{
"name": "nodejs-openai-streaming",
"version": "1.0.0",
"type": "module",
"scripts": {
"start": "node server.js",
"dev": "node \--watch server.js"
},
"dependencies": {
"cors": "^2.8.6",
"dotenv": "^16.6.1",
"express": "^4.22.2",
"openai": "^4.104.0"
}
}
Create .env:
Check out our hands-on, practical guide to learning Git, with best-practices, industry-accepted standards, and included cheat sheet. Stop Googling Git commands and actually learn it!
OPENAI\_API\_KEY=your\_api\_key\_here
PORT=3000
Project structure:
nodejs-openai-streaming/
├── server.js
├── public/
│ └── index.html
├── .env
├── .env.example
├── .gitignore
└── package.json
Why SSE and Not WebSockets
People ask about WebSockets every time, usually because they sound more capable and therefore more correct.
For AI chat, they usually are not. Responses travel one way, server to client, and SSE is built for exactly that. It runs over plain HTTP with no protocol upgrade and no special reverse proxy configuration, and it works through HTTP/2 without touching a config file. I have shipped SSE to production multiple times and never once worried about browser support.
WebSockets make sense when the client needs to push data back at the same frequency the server pushes it: gaming, shared whiteboards, live bidding. In an AI chat, the user types one message and sits there reading, so SSE fits the traffic pattern better.
One genuine limitation of SSE worth knowing: the browser's built-in EventSource API only supports GET requests. Since you need to post a message body, you use fetch with a ReadableStream reader instead. The SSE wire protocol is identical; you are just reading the stream manually rather than through the browser's event wrapper.
Session Management
The first chatbot I built on top of this API had a bug I could not figure out for two days. Users would ask a follow-up question and the model would answer like it was hearing from them for the first time, with no context or memory of the earlier turns.
The API does not store anything between calls, so each request goes in cold. The messages array is how you fix that: pack the full conversation history into every request, user turns and assistant turns both, and the model responds as if it remembers the whole thread. Leave it out and you get that same broken experience I shipped by accident.
const sessions \= new Map();
const SESSION\_TTL\_MS \= 30 \* 60 \* 1000; // 30 minutes
function getOrCreateSession(sessionId) {
if (\!sessions.has(sessionId)) {
sessions.set(sessionId, {
messages: \[\],
lastActive: Date.now(),
});
}
const session \= sessions.get(sessionId);
session.lastActive \= Date.now();
return session;
}
setInterval(() \=\> {
const now \= Date.now();
for (const \[id, session\] of sessions) {
if (now \- session.lastActive \> SESSION\_TTL\_MS) {
sessions.delete(id);
}
}
}, 10 \* 60 \* 1000);
The Map is fine at development scale and in single-process deployments. The lastActive timestamp gets bumped on every request. The cleanup interval runs every ten minutes and removes sessions that have gone quiet for half an hour.
The hard limit of this approach is that a Node.js process restart wipes it entirely. If you run multiple instances behind a load balancer, each instance has its own Map, and a user whose requests hit different instances will lose their conversation mid-sentence. The fix is Redis: one client, same data structure, TTL handled natively. I am not going to build that here because it would double the length of this article without adding anything about streaming, but do not ship this to production without it.
The Streaming Endpoint
Four things happen in sequence before tokens start flowing: input validation, SSE setup, abort handling, and the OpenAI call with its token loop. Each step has at least one non-obvious decision.
SSE Setup
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.flushHeaders();
flushHeaders() is the line I see missing in more streaming PRs than I can count. Skip it and Express holds onto the response headers until the first res.write() call. The browser opens the request, gets nothing back, sits there, and after a few seconds the whole thing looks like a hung request. You will spend an hour checking your OpenAI call, your async logic, and your middleware, and the fix is one line at the top. Call flushHeaders() right after setting the headers; nothing else changes.
AbortController
const controller \= new AbortController();
res.on('close', () \=\> {
if (\!res.writableEnded) controller.abort();
});
One important detail: attach the close listener to res, not req. The req object fires close when the request body is fully consumed, which happens immediately after the POST body arrives and would abort the stream before it starts. res.on('close') fires when the actual response connection drops, which is what you want. The !res.writableEnded guard prevents calling abort() after a clean finish.
When the connection drops, controller.abort() propagates a cancellation signal to the OpenAI SDK and the for await loop exits on the next iteration. Without this, you keep pulling tokens from the API and writing them into a closed socket, which means you are spending money and achieving nothing.
The OpenAI Streaming Call
const stream \= await openai.chat.completions.create(
{
model: 'gpt-4o-mini',
messages: \[
{
role: 'system',
content: 'You are a helpful assistant. Be concise and clear.',
},
...session.messages,
\],
stream: true,
max\_tokens: 800,
},
{ signal: controller.signal }
);
Set max_tokens. This is not optional. I left it uncapped during a testing session and watched my usage dashboard spike in real time. Left alone, the model will write a dissertation in response to a three-word question. I use 800 for chat, which works fine for most things, and you can tune it once you see what your users actually send.
The signal parameter threads the AbortController through to the SDK. If the client disconnects, the SDK cancels the underlying HTTP request to OpenAI, not just the local loop.
Token Loop and History
let fullResponse = '';
for await (const chunk of stream) {
if (req.socket.destroyed) break;
const token = chunk.choices[0]?.delta?.content;
if (token) {
fullResponse += token;
res.write(`data: ${JSON.stringify({ token })}\n\n`);
}
}
if (fullResponse) {
session.messages.push({ role: 'assistant', content: fullResponse });
}
Each chunk contains one token in delta.content. You concatenate them into fullResponse as they arrive, relay each one to the client, and save the complete string to the session once the loop ends.
The double newline in res.write() is mandatory; it is how SSE signals the end of a single event. The browser buffers data until it sees \n\n before firing. Omit it and the client receives all tokens at once when the connection closes, which is the same broken behavior as not streaming at all.
Do not save fullResponse if the stream was aborted. A truncated assistant reply in the session history will corrupt the conversation context, and the model's next turn will be built on incomplete information.
The Complete server.js
import 'dotenv/config';
import express from 'express';
import cors from 'cors';
import OpenAI from 'openai';
import path from 'path';
import crypto from 'crypto';
import { fileURLToPath } from 'url';
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const app = express();
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
app.use(cors());
app.use(express.json());
app.use(express.static(path.join(__dirname, 'public')));
const sessions = new Map();
const SESSION_TTL_MS = 30 * 60 * 1000;
function getOrCreateSession(sessionId) {
if (!sessions.has(sessionId)) {
sessions.set(sessionId, { messages: [], lastActive: Date.now() });
}
const session = sessions.get(sessionId);
session.lastActive = Date.now();
return session;
}
setInterval(() => {
const now = Date.now();
for (const [id, session] of sessions) {
if (now - session.lastActive > SESSION_TTL_MS) sessions.delete(id);
}
}, 10 * 60 * 1000);
app.post('/session', (req, res) => {
const sessionId = crypto.randomUUID();
getOrCreateSession(sessionId);
console.log(`[/session] created: ${sessionId}`);
res.json({ sessionId });
});
app.delete('/session/:sessionId', (req, res) => {
sessions.delete(req.params.sessionId);
res.json({ ok: true });
});
app.post('/chat', async (req, res) => {
const { message, sessionId } = req.body;
if (!message || typeof message !== 'string') {
return res.status(400).json({ error: 'message is required' });
}
if (!sessionId || typeof sessionId !== 'string') {
return res.status(400).json({ error: 'sessionId is required' });
}
const session = getOrCreateSession(sessionId);
session.messages.push({ role: 'user', content: message });
console.log(`[/chat] received:`, { message, sessionId });
res.setHeader('Content-Type', 'text/event-stream');
res.setHeader('Cache-Control', 'no-cache');
res.setHeader('Connection', 'keep-alive');
res.flushHeaders();
const controller = new AbortController();
res.on('close', () => {
if (!res.writableEnded) controller.abort();
});
let fullResponse = '';
try {
const stream = await openai.chat.completions.create(
{
model: 'gpt-4o-mini',
messages: [
{ role: 'system', content: 'You are a helpful assistant. Be concise and clear.' },
...session.messages,
],
stream: true,
max_tokens: 800,
},
{ signal: controller.signal }
);
for await (const chunk of stream) {
if (req.socket.destroyed) break;
const token = chunk.choices[0]?.delta?.content;
if (token) {
fullResponse += token;
res.write(`data: ${JSON.stringify({ token })}\n\n`);
}
}
if (fullResponse) {
session.messages.push({ role: 'assistant', content: fullResponse });
console.log(`[stream complete] ${fullResponse.length} chars`);
}
res.write(`data: ${JSON.stringify({ done: true })}\n\n`);
res.end();
} catch (err) {
if (err.name === 'AbortError' || err.name === 'APIUserAbortError' || err.message === 'Request was aborted.') {
res.end();
return;
}
console.error('Stream error:', err.message);
res.write(`data: ${JSON.stringify({ error: err.message })}\n\n`);
res.end();
}
});
const PORT = process.env.PORT || 3000;
app.listen(PORT, () => console.log(`Server running at http://localhost:${PORT}`));
Building the Frontend
The client has two jobs: manage the session lifecycle and render the stream. The session lifecycle is simple (create on load, delete on "New Chat"), and the stream rendering is where most implementations get tripped up.
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>AI Streaming Chat</title>
<style>
* { box-sizing: border-box; margin: 0; padding: 0; }
body {
font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
background: #f0f2f5;
height: 100vh;
display: flex;
flex-direction: column;
align-items: center;
padding: 1.5rem 1rem;
}
.chat-wrapper { width: 100%; max-width: 700px; display: flex; flex-direction: column; height: 100%; }
h1 { font-size: 1.2rem; color: #333; margin-bottom: 1rem; }
#messages {
flex: 1; overflow-y: auto; background: white; border: 1px solid #e0e0e0;
border-radius: 10px; padding: 1rem; display: flex; flex-direction: column; gap: 0.75rem;
}
.message {
max-width: 85%; padding: 0.65rem 0.9rem; border-radius: 10px;
line-height: 1.6; font-size: 0.93rem; white-space: pre-wrap; word-break: break-word;
}
.user { background: #0070f3; color: white; align-self: flex-end; }
.assistant { background: #f1f1f1; color: #111; align-self: flex-start; }
.input-row { display: flex; gap: 0.5rem; margin-top: 0.75rem; align-items: flex-end; }
textarea {
flex: 1; padding: 0.7rem; border: 1px solid #ddd; border-radius: 8px;
font-size: 0.93rem; font-family: inherit; resize: none; height: 52px; overflow-y: auto;
}
button { padding: 0.65rem 1.2rem; border: none; border-radius: 8px; font-size: 0.93rem; cursor: pointer; }
#send { background: #0070f3; color: white; }
#send:disabled { background: #aaa; cursor: not-allowed; }
#clear { background: #f1f1f1; color: #555; }
.status { font-size: 0.78rem; color: #aaa; margin-top: 0.3rem; min-height: 1rem; }
</style>
</head>
<body>
<div class="chat-wrapper">
<h1>AI Chat</h1>
<div id="messages"></div>
<p class="status" id="status"></p>
<div class="input-row">
<textarea id="input" placeholder="Ask something... (Ctrl+Enter to send)"></textarea>
<button id="send">Send</button>
<button id="clear">New Chat</button>
</div>
</div>
<script>
let sessionId = null;
async function initSession() {
const res = await fetch('/session', { method: 'POST' });
const data = await res.json();
sessionId = data.sessionId;
}
function appendMessage(role, text) {
const div = document.createElement('div');
div.className = `message ${role}`;
div.textContent = text;
document.getElementById('messages').appendChild(div);
div.scrollIntoView({ behavior: 'smooth' });
return div;
}
document.getElementById('send').addEventListener('click', async () => {
const input = document.getElementById('input');
const message = input.value.trim();
if (!message || !sessionId) return;
const sendBtn = document.getElementById('send');
const status = document.getElementById('status');
input.value = '';
sendBtn.disabled = true;
status.textContent = 'Generating...';
appendMessage('user', message);
const assistantBubble = appendMessage('assistant', '');
try {
const response = await fetch('/chat', {
method: 'POST',
headers: { 'Content-Type': 'application/json' },
body: JSON.stringify({ message, sessionId }),
});
if (!response.ok) {
const err = await response.json();
assistantBubble.textContent = `Error: ${err.error}`;
return;
}
const reader = response.body.getReader();
const decoder = new TextDecoder();
while (true) {
const { value, done } = await reader.read();
if (done) break;
const lines = decoder.decode(value, { stream: true }).split('\\n');
for (const line of lines) {
if (!line.startsWith('data: ')) continue;
try {
const data = JSON.parse(line.slice(6));
if (data.done) break; // exits inner for loop; outer while loop ends on next read() → done: true
if (data.error) { assistantBubble.textContent += `\\n[Error: ${data.error}]`; break; }
if (data.token) {
assistantBubble.textContent += data.token;
assistantBubble.scrollIntoView({ behavior: 'smooth' });
}
} catch { /* split chunk; wait for next read */ }
}
}
status.textContent = '';
} catch (err) {
assistantBubble.textContent = `Request failed: ${err.message}`;
} finally {
sendBtn.disabled = false;
status.textContent = '';
input.focus();
}
});
document.getElementById('clear').addEventListener('click', async () => {
if (!sessionId) return;
await fetch(`/session/${sessionId}`, { method: 'DELETE' });
await initSession();
document.getElementById('messages').innerHTML = '';
document.getElementById('status').textContent = '';
});
document.getElementById('input').addEventListener('keydown', (e) => {
if (e.key === 'Enter' && e.ctrlKey) document.getElementById('send').click();
});
initSession();
</script>
</body>
</html>
The try/catch inside the inner loop handles a real edge case: SSE events occasionally span two network reads. When that happens, JSON.parse throws on the incomplete fragment, and catching it so the next read can deliver the rest keeps the stream alive. Without the catch, the whole stream crashes on what is essentially normal network behavior.
Running It
npm run dev
The terminal confirms the server started:
Server running at http://localhost:3000
Hit http://localhost:3000 in your browser and type something. The terminal should print:
\[/session\] created: 2dfb2586-4601-4781-932d-cb3dcca5d5a5
\[/chat\] received: {
message: 'Do you know about StackAbuse?',
sessionId: '2dfb2586-4601-4781-932d-cb3dcca5d5a5'
}
\[stream complete\] 304 chars
In the browser, you will see the response build out token by token, which means the streaming is working. The interface looks like this with a live response:
Send a follow-up that references the first answer and the model will remember it. The session history is included in every request, so context carries across turns. Click "New Chat" and the server deletes the session and creates a fresh one. The model starts over with no memory of the previous conversation.
To inspect the raw stream, open DevTools, go to Network, find the /chat request, and look at the EventStream tab. You will see each data: event arrive individually, which is the most useful thing I know of for debugging streaming behavior that looks correct in the UI but is not working the way you think.
Production Gaps to Address Before Launch
Before you ship this, know what the implementation does not handle. I have seen developers take tutorial code and deploy it unchanged.
Context overflow. Every request sends the full session history. gpt-4o-mini has a 128,000-token context window, which is large enough that most chat sessions will never hit it. But a session running for hours will eventually approach the limit. When it does, the API returns a context_length_exceeded error. The standard fix is a sliding window: keep the system prompt, trim old messages from the front of the history, optionally run a summarization call to compress old context before it falls off. This is a real engineering problem and there is no one-size answer.
Rate limiting. There is none here, so one client can exhaust your monthly OpenAI quota. Add express-rate-limit before this goes anywhere near the internet.
Authentication. The /chat endpoint accepts requests from anyone. In a real deployment, you need session-based auth or JWT validation before the route handler runs. Without it, anyone who knows your URL is spending your API credits.
Multi-process sessions. Covered above. Redis or equivalent before you scale horizontally.
Conclusion
I have shipped versions of this exact setup to production twice now, across different companies and products, with the same core pattern: session store on the server, SSE for transport, and AbortController to stop wasting money on disconnected clients. The gaps I listed are not hypothetical warnings; I hit most of them myself before figuring out the fixes.
The part that trips most developers is not the OpenAI API itself, but the plumbing around it: getting tokens to the browser the moment they arrive, keeping the connection alive without burning credits on disconnected clients, and maintaining conversation context without letting it grow unbounded. That is where the real work sits. Hopefully this article saved you the afternoon I spent learning most of it the hard way.
The full code is at github.com/ziaongit/nodejs-openai-streaming. Clone it, break it, and make it your own.
