Next.js Discord

Discord Forum

Different response behaviors when using OpenAIStream. How do you test your stream output?

Unanswered
Dwarf Crocodile posted this in #help-forum
Open in Discord
Dwarf CrocodileOP
Depending on what client/tool I use to test my API responses, I get troubling different behaviors when OpenAIStream is used to wrap the response. I can't tell if this is a bug, or the tools are implemented differently, or what the best practice for testing this is?

I've tested with nearly the same as the Vercel AI SDK example: https://github.com/StephenTangCook/vercel-ai-sdk-example/blob/b6d34406f7e7c636924c8504684f0dc8cfc24236/app/api/completion/route.ts#L32

Versions:
"ai": "^2.1.20",
"next": "13.4.9",
"openai-edge": "^1.2.0",
node v18.16.0

1. With OpenAiStream and StreamingTextResponse:
  const stream = OpenAIStream(response);
  return new StreamingTextResponse(stream, {
    headers: {"Content-Type": "text/event-stream"},
  });

- Browser (chrome): streams the text incrementally
- Postman: connection opens, then closes. No data chunks received.
- cURL: the entire text is shown at once (no stream)

2. Only StreamingTextResponse (no OpenAiStream):
  return new StreamingTextResponse(response.body, {
    headers: {"Content-Type": "text/event-stream"},
  });

- Browser (chrome): displays incremental data chunks of the stream
- Postman: connection opens, displays incremental data chunks of the stream, then closes
- cURL: receives incremental data chunks of the stream

As you can see, only # 2 consistently returns a stream. What does wrapping the response with OpenAIStream do in # 1 that makes these tools unhappy?

8 Replies

@Dwarf Crocodile Depending on what client/tool I use to test my API responses, I get troubling different behaviors when `OpenAIStream` is used to wrap the response. I can't tell if this is a bug, or the tools are implemented differently, or what the best practice for testing this is? I've tested with nearly the same as the Vercel AI SDK example: https://github.com/StephenTangCook/vercel-ai-sdk-example/blob/b6d34406f7e7c636924c8504684f0dc8cfc24236/app/api/completion/route.ts#L32 **Versions:** "ai": "^2.1.20", "next": "13.4.9", "openai-edge": "^1.2.0", node v18.16.0 **1. With `OpenAiStream` and `StreamingTextResponse`:** javascript const stream = OpenAIStream(response); return new StreamingTextResponse(stream, { headers: {"Content-Type": "text/event-stream"}, }); - Browser (chrome): streams the text incrementally - Postman: connection opens, then closes. No data chunks received. - cURL: the entire text is shown at once (no stream) **2. Only `StreamingTextResponse` (no `OpenAiStream`):** javascript return new StreamingTextResponse(response.body, { headers: {"Content-Type": "text/event-stream"}, }); - Browser (chrome): displays incremental data chunks of the stream - Postman: connection opens, displays incremental data chunks of the stream, then closes - cURL: receives incremental data chunks of the stream As you can see, only # 2 consistently returns a stream. What does wrapping the response with `OpenAIStream` do in # 1 that makes these tools unhappy?
Saltwater Crocodile
Running into the same problem! I ended up giving up last week because I couldn't figure it out
tafutada777
HTTP/2 or 1.1 chunk?
tafutada777
afaik, Vercel does not use SSE. it's streaming chunks.
Vercel AI SDK provides 2 utility helpers to make the above seamless: First, we pass the streaming response we receive from OpenAI to OpenAIStream. This method decodes/extracts the text tokens in the response and then re-encodes them properly for simple consumption. We can then pass that new stream directly to StreamingTextResponse. This is another utility class that extends the normal Node/Edge Runtime Response class with the default headers you probably want (hint: 'Content-Type': 'text/plain; charset=utf-8' is already set for you).
Dwarf CrocodileOP
Thanks for pointing me towards the headers. Another data point, if I don't set any custom headers:
  const stream = OpenAIStream(response);
  return new StreamingTextResponse(stream);


- Browser (chrome): displays entire text at once (no stream)
- Postman: displays entire text at once (no stream)
- cURL: displays entire text at once (no stream)
tafutada777
those tools are automatically restructure chunks, so you need to use low level tool like telnet, Wireshark.
i mean http/1.1 chunks