Next.js Discord

Discord Forum

ai sdk langchain agent streaming from api to page

Unanswered
Golden northern bumble bee posted this in #help-forum
Open in Discord
Golden northern bumble beeOP
hi,
i am trying to use the new AI sdk from vercel to stream a response from a langchain agent in my api route, to a chat component in the frontend of my nextJS app.

It seems that if I put the handlers from LangChainStream() in the call to the agent, the stream ends prematurely (after the first llm call). If i put the handlers on the model, the stream never completes. I need it to complete to save the response to a database.

export const runtime = 'edge';

const handleRequest = async (messages) => {
  const { stream, handlers } = LangChainStream();

  const chat = new ChatOpenAI({
    streaming: true,
    temperature: 0.3,
    modelName: 'gpt-3.5-turbo',
    openAIApiKey: process.env.OPENAI_API_KEY,
    //can also put callbacks:[handlers] here
  });

  const tools = [calendlyTool, dbSearchTool];

  const prompt = ZeroShotAgent.createPrompt(tools, {
    prefix: '-'
    You have access to the following tools:
      TOOLS:`,
    suffix: '-'
  });

  const chatPrompt = ChatPromptTemplate.fromPromptMessages([
    new SystemMessagePromptTemplate(prompt),
    HumanMessagePromptTemplate.fromTemplate(`{input}
 
          This was your previous work (but I haven't seen any of it! I only see what you return as "Final Answer: [your final response here]".
          {agent_scratchpad}`),
  ]);

  const llmChain = new LLMChain({
    prompt: chatPrompt,
    llm: chat,
  });

  const agent = new ZeroShotAgent({
    llmChain,
    allowedTools: tools.map((tool) => tool.name),
  });

  const executor = AgentExecutor.fromAgentAndTools({ agent, tools });

  executor
    .run(
      JSON.stringify(
        messages.map((m) =>
          m.role == 'user'
            ? new HumanChatMessage(m.content)
            : new AIChatMessage(m.content)
        )
      ),
      [handlers]
    )
    .catch(console.error);

  return new StreamingTextResponse(stream);
};

export default async function POST(req) {
  const { messages } = await req.json();
  return handleRequest(messages);
}

1 Reply

@Golden northern bumble bee you might wanna use Vercel AI SDK to work with Edge Runtime, which supports subset of Node.js, so LangChain SDK might not work. indeed OpenAI SDK does not work in Edge Runtime, thus Vercel developed their own SDK.