Next.js Discord

Discord Forum

Chunks Breaking in Production on Vercel when Streaming Tokens from LangChain app

Unanswered
Sloth bear posted this in #help-forum
Open in Discord
Sloth bearOP
Hi,

I'm facing a very weird issue that didn't appear before.

Background:

I've built an application that allows you to design production ready autonomous agents for clinical operations and chat with these agents using a chat interface. Its all low-code built on top of Next.js. User inputs a request on the chat, backend is written in FastAPI python, hosted elsewhere, which streams back tokens.

Tokens look like this in string form:

Token Type 1 -> {"type": "final_answer_token", "token": " Hello"}
Token Type 2 -> {"type": "action", "tool_name": "Current search", "input": "Some sample query"}
Token Type 3 -> {"type": "source", "title": "Some title goes here", "link": "https://some..."}


These are streamed over to the frontend using Vercel's AI SDK. Very simple.

import { StreamingTextResponse } from "ai";
import axios from "axios";
import { Transform } from "stream";

export async function POST(req: Request) {
  // Call made to FastAPI from where a generator yields stringified version of dictionary object
  const res = await axios.post(...);

  let readable_stream = res.data;

  const customTransform = new Transform({
    transform(chunk: any, encoding: any, callback: any) {
      const { type } = JSON.parse(chunk.toString());

      if (type === "POLL") {
        // console.log("Not sending POLL to client");
        callback(null);
        return;
      }      
      return callback(null, "<SEP>" + chunk.toString() + "<SEP>");
    },
  });

  readable_stream = readable_stream
    .on("data", (chunk: any) => {
      console.log("Chunk: ", chunk)
    })
    .pipe(customTransform);

  return new StreamingTextResponse(readable_stream);
}


Received in a standard way on Frontend.

import { useChat } from "ai/react";
const { messages, stop, input, handleInputChange, handleSubmit, isLoading } = useChat(...)

23 Replies

Sloth bearOP
The Problem:
All was working fine until something weird started to happen in production. Chunks start to cut off the moment they arrive on frontend.

"<SEP>" + chunk.toString() + "<SEP>"

I'm adding this "<SEP>" to make sure it doesn't cut off, but turns out that doesn't help. Maybe I don't understand fully what's going on. By cut-off I mean, arrived chunks are more like the following:

<SEP>{"type": "final_answer_token", "token": "!"}
<SEP>{"type": "final_answer_token", "token": "!"}<SE
<SEP>{"type": "final_answer_token", "token": "
<SEP>{"type": "final_answer_token", "token": "!"}<SEP
<SEP>{"type": "final_answer_token", "token":


And this throws parsing errors. They're being recieved in the API perfectly intact, but they break when arriving on frontend in "Production".

Can anyone kindly help?
tafutada777
did u deploy on AWS Lambda instead of Edge Runtime?
Sloth bearOP
I’m just calling an FastAPI that has my agent framework in there and the processing happens there.
I need to fetch it in. The only way remains to be Fetch API, but I can’t stream responses that way.
Any ideas how can I do that. I have searched all over internet, nothing very useful on streaming with Native Fetch API
The response has a body that is not interable and I can’t get a ReadableStream from it.
The axios body, I can define the response type as stream but not for fetch. Axios gives ReadableStream, very straightforward, not Fetch API.
tafutada777
you mean like this?

export async function POST(req: Request) {
    // Extract the `messages` from the body of the request
    const {messages} = await req.json()

    const response = await fetch('https://api.anthropic.com/v1/complete', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'x-api-key': process.env.ANTHROPIC_API_KEY!
        },
        body: JSON.stringify({
            prompt: buildPrompt(messages),
            model: 'claude-v1',
            max_tokens_to_sample: 300,
            temperature: 0.9,
            stream: true
        })
    })

    // Convert the response into a friendly text-stream
    const stream: ReadableStream<any> = AnthropicStream(response)

    // Respond with the stream
    return new StreamingTextResponse(stream)
}
actually i haven't tried it yet. so im not sure the details.
Sloth bearOP
Why didn’t I find this before. Very useful.
Do these edge runtime functions run inside the browser?
In that case the chunk shouldn’t break.
Let me play around with it. I’ve already handled caveat this article mentions, but not using edge runtime. I’ll swap it in to see how it benefits.
tafutada777
Vercel uses third party cloud servers, AWS Lambda and Cloudflare Worker.
Vercel's Edge Runtime runtime, which is subset of Node.js, runs on Cloudflare Worker.
in Vercel platform, middleware.ts file is deployed to Edge Runtime.
as for APIs, you can choose a runtime to deploy, Edge or Lambda.
Edge Runtime is suited for I/O bound apps like Chat apps, as it is measured by CPU Time instead of wall time(elapsed time), say OpenAI API is slow to response, it takes longer than AWS lambda cap, and charged based on wall time.
Plus, Edge is no cold start time as V8 isolate, while lambda has around 250ms cold start time.
tafutada777
it's Vercel magic. besides that Vercel is working on connection pooling mechanism to Postgress, im now looking into it.
tafutada777
plus, this is jaw dropping code from Vercel.
it sends chunks using React component RECURSIVELY.

export default async function Reader({
                                         reader
                                     }: {
    reader: ReadableStreamDefaultReader<any>
}) {
    const {done, value} = await reader.read()

    if (done) {
        return null
    }

    const text = new TextDecoder().decode(value)

    return (
        <span>
            {text}
            <Suspense>
                <Reader reader={reader}/>
            </Suspense>
        </span>
    )
}
it send HTTP/2 or HTTP 1.1 chunk
Sloth bearOP
Is this the underlying implementation of useChat, cuz I’m using that for now to handle all the stuff on frontend?
tafutada777
it's server side
Sloth bearOP
Btw, I’m seeing a recursive react component for the first time. Very interesting.
tafutada777
Yeah, amazing. Here is my poc code. it helps.
https://github.com/tfutada/zenn-vercel-ai-sdk/blob/main/app/server-component/page.tsx
Sloth bearOP
Let me take a look. I’m gonna do fixes today. Very useful stuff you’ve referred to.