Next.js Discord

Discord Forum

Serverless Function Streame Response

Unanswered
Ivory Gull posted this in #help-forum
Open in Discord
Ivory GullOP
I have a nextjs serverless function, which streams an openai response. It works well with my next frontend. But in my react native app I receive the complete response as one chunk. I tested calling the openai api directly from react native and there streaming works. So I figure that something with my function is wrong for this usecase? Has anyone a clue?? This is not a react native question I think :/

6 Replies

Ivory GullOP
This is the log in react native app and this is the code to fetch:
  fetch("http://192.168.2.182:3000/api/lotti", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          Connection: "keep-alive",
        },
        mode: "cors",
        body: JSON.stringify(message),
        reactNative: { textStreaming: true }
      }).then(response => response.body)
        .then(async stream => {
          console.log("Stream:", stream);
          const reader = stream.getReader();
          console.log("Reader:", reader);
          let done, value;
          while (!done) {
            ({done, value} = await reader.read());
            if (!done) {
              const chunk = new TextDecoder().decode(value);
              console.log("chunk", chunk);
            }
          }
          console.log('Streaming complete.');
        })
this is the code in next frontend and the logs:
const res = await fetch(url, {
    method: method,
    headers: {
      "Content-Type": "application/json",
    },
    body: payload && JSON.stringify(payload),
  });

  const data = res.body;

  if (data) {
    const reader = data.getReader();
    const decoder = new TextDecoder();
    let done = false;

    let buffer = "";

    while (!done) {
      const { value, done: doneReading } = await reader.read();
      done = doneReading;
      const chunkValue = decoder.decode(value);
      console.log("chunkValue", chunkValue);
      buffer += chunkValue;

      yield buffer;
    }
  }
this is the serverless function:
export const maxDuration = 60;

import {
  ChatGPTAgent,
  Prompt,
  streamCreateChatCompletion,
} from "@/server/openai";
import { getMessagesByChatroomId } from "@/server/messages/messages.repository";
import { lottiChatTemplate } from "@/server/ask-lotti.prompt";
import { Message } from "@/server/messages/messages.schema";
import { customLogger } from "@/utils/logger";
import { NextRequest, NextResponse } from "next/server";
import { OpenAIStream, StreamingTextResponse } from "ai";

export async function POST(req: NextRequest) {
  try {
    const body = await req.json();
    customLogger.debug("[lotti.ts handler] Request body:", body);
    const { content, userId, chatroomId } = body as Partial<Message>;

    if (!content || !userId || !chatroomId) {
      return;
    }

    let previousMessages = (await getMessagesByChatroomId(chatroomId)) ?? [];
    if (!previousMessages.length) {
      customLogger.debug("[lotti.ts handler] No previous messages found");
      return;
    }

    const prompts: Prompt<Partial<Message>>[] = previousMessages.map(
      (message) =>
        new Prompt(
          lottiChatTemplate(message.role as ChatGPTAgent),
          message as Partial<Message>,
        ),
    );
    prompts.push(new Prompt(lottiChatTemplate("user"), { content }));

    const responseStream = await streamCreateChatCompletion(prompts, {
      type: "json_object",
    });

    const stream = OpenAIStream(responseStream);

    const headers = {
      "Content-Type": "text/plain", // Adjust content type as needed
      "Cache-Control": "no-cache", // Add any other headers as required
      "Transfer-Encoding": "chunked",
    };

    return new StreamingTextResponse(stream, { headers });
  } catch (error) {
    customLogger.error("[lotti.ts] An error occurred:", error);
  }
}

export async function OPTIONS(req: NextRequest) {
  return new NextResponse(null, { status: 200 });
}
@Anay-208 can you show a preview of output in both frontend & In both react-native
Ivory GullOP
is this enough or do you need anything else?
Anay-208
I’ll see it when I get free