Next.js Discord

Discord Forum

How to send readable stream from Next JS route handler?

Answered
English Angora posted this in #help-forum
Open in Discord
English AngoraOP
As in title. I know there is a tutorial about the openAI usage for streaming chat msg. I only want to know how to send any kind of stream back to the client. Regardless if it is open ai and I want to chunk some other data.
Answered by Ray
something like this
export async function GET() {
  const stream = new ReadableStream({
    async start(controller) {
      const encoder = new TextEncoder();

      const chunks = [1, 2, 3, 4];
      for (const chunk of chunks) {
        controller.enqueue(encoder.encode(`data: ${chunk}\n\n`));
        await new Promise((resolve) => setTimeout(resolve, 2000));
      }

      controller.close();
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache",
    },
  });
}
View full answer

17 Replies

@English Angora As in title. I know there is a tutorial about the openAI usage for streaming chat msg. I only want to know how to send any kind of stream back to the client. Regardless if it is open ai and I want to chunk some other data.
Ray
something like this
export async function GET() {
  const stream = new ReadableStream({
    async start(controller) {
      const encoder = new TextEncoder();

      const chunks = [1, 2, 3, 4];
      for (const chunk of chunks) {
        controller.enqueue(encoder.encode(`data: ${chunk}\n\n`));
        await new Promise((resolve) => setTimeout(resolve, 2000));
      }

      controller.close();
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache",
    },
  });
}
Answer
English AngoraOP
@Ray Thank for this. Yes it does make a stream and I was able to read it on the client. However, for some reason it still waits for some time, and then gives all the chunks at once. It does not start streaming as soon as I fetch the data on the client, which is the whole purpose of a stream, not to wait for the whole data to be transfered 😕
English AngoraOP
    const document = new Document({ text: JSON.stringify(data) });

    // Split text and create embeddings. Store them in a VectorStoreIndex
    const index = await VectorStoreIndex.fromDocuments([document]);

    // Query the index
    const queryEngine = index.asQueryEngine();

    const stream = new ReadableStream({
      async start(controller) {
        const encoder = new TextEncoder();

        const chunks = await queryEngine.query({ query, stream: true });

        for await (const chunk of chunks) {
          controller.enqueue(encoder.encode(chunk.response));
        }

        controller.close();
      },
    });


I already have a "stream" from llama-index, but really it is a wrapper around openAI.
English AngoraOP
ok so something is off with the code related to llama index. Your code example does chunk as expected! Thanks this was already a massive help. Wonder why this hasn't been documented? Could be that it has nothing to do with Next js, but it is rather regular web api of node?
English AngoraOP
It actually work without it somehow.
In my case I suspect that this line tames a long time to execute and it actually does pause for some time:
const chunks = await queryEngine.query({ query, stream: true });
English AngoraOP
I am a noob man, sorry for uggly code format
hm not sure

look here in the docs >> https://ts.llamaindex.ai/modules/high_level/query_engine

chunk.response is a string type
Ray
do you get stream response with this?
@English Angora hm not sure look here in the docs >> https://ts.llamaindex.ai/modules/high_level/query_engine chunk.response is a string type
Ray
well I think you can just return the stream here
const stream = await queryEngine.query({ query: "query string", stream: true });
return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache",
    },
  });
English AngoraOP
Yeah, one of the very first thing I tried.

Argument of type 'AsyncIterable<Response>' is not assignable to parameter of type 'BodyInit | null | undefined'.

Getting this from "Response"

Maybe I should be using somehting else? I have also tried NextApiResponse and NextResponse, none of them takes 'AsyncIterable<Response>' as type 😕
Ray
oh
English AngoraOP
btw you helped me a lot, so thanks, I got it to work a bit faster with streaming and chunking but it does suck that it still pauses a bit.

Have a nice rest of the dat @Ray 🙏