Next.js Discord

Discord Forum

Service works first time, then times out on subsequent runs, fails for a minute, then works again.

Unanswered
Cape lion posted this in #help-forum
Open in Discord
Cape lionOP
I have a serverless streaming service I've written that is invoking an LLM service running on my computer. I'm using a subdomain redirect -> nginx -> ollama service to run this. It will work the first time, but then if I run the service again right after, it always times out. Then works again after a minute. Rinse/repeat.

It works 100% of the time on my local machine. The service is fast and returns responses quickly. How can I diagnose the error and fix? Really could use some help figuring this out, as the Vercel logs only say "[POST] /api/generate reason=EDGE_FUNCTION_INVOCATION_TIMEOUT, status=504, user_error=true"

22 Replies

Cape lionOP
I'm using Firebase
and can you send contents of /api/generate/page.tsx
and if there is any line number given, specially look around that line
Cape lionOP
I'll just put all the code here
ollama.ts (part 1)
import { readStream } from "@/app/utilities/readStream";

export async function stream(model: string, prompt: string, system: string): Promise<ReadableStream<any>> {
  return await buildStream(model, prompt, system);
}

function parseSequentialJSON(jsonString: string): { response: string }[] {
  // Add a comma between the }{ to transform it into a valid JSON array
  const validJsonString = `[${jsonString.replace(/}\s*{/g, '},{')}]`;

  try {
    // Parse the valid JSON string
    const jsonArray = JSON.parse(validJsonString);
    return jsonArray;
  } catch (error) {
    throw new Error("Failed to parse.")
  }
}
part 2:
async function buildStream(model: string, prompt: string, system: string): Promise<ReadableStream<any>> {
  const encoder = new TextEncoder();
  return new ReadableStream({
    async start(controller) {
      try {
        const { OLLAMA_HOST } = process.env;
        console.log(OLLAMA_HOST);
        const endpoint = '/api/generate'; // API endpoint

        const url = new URL(endpoint, OLLAMA_HOST).toString();

        console.log(url);

        const requestBody = {
          prompt,
          system,
          model,
          stream: true,
        };

        const response = await fetch(url, {
          method: 'POST',
          body: JSON.stringify(requestBody),
          headers: {
            'Content-Type': 'application/json'
          }
        });

        console.log(`Response ok: ${response.ok}`);

        if (!response.ok) {
          throw new Error(`HTTP error! Status: ${response.status}`);
        }

        console.log("Response is OK");

        // Handle the stream
        const reader = response.body;
        if (reader == null) {
          throw new Error("Reader is null.");
        }
        let buffer = '';
        await readStream(
          reader,
          value => {
            try {
              buffer += value;
              const next = parseSequentialJSON(buffer).map(element => element.response).join("");
              controller.enqueue(encoder.encode(next));
              buffer = '';
            } catch { }
          }
        );

        controller.close();
      } catch (e) {
        console.log('An error occurred while communicating with ollama.');
        throw e;
      }
    },
  });
}
That's where /api/generate resolves in my applicatin. Then it sends a requests to ollama's /api/generate, which does the LLM completion
Cape lionOP
I don't think this is particularly easy to diagnose either. It runs quite fast (ollama startup is less than a second).
It does work sometimes. But about 20% of the time.
try building and starting the app locally
Cape lionOP
Done. I tested it locally and it works no problem, although that may not mean much since the local execution does not have a timeout like production
you'll have to add console.log breakpoints, and see in logs where exactly the error occurs
Cape lionOP
Fair, I'll add more logs. Something weird is happening at some point and I need to diagnose it further.
Notably, it's not getting to the point where it hits my ollama service as evidenced by the lack of log outputs when running sudo tcpdump -i any port 11434. Thanks for the suggestion... hopefully the logs give more insight
Cape lionOP
Interesting. I think the previous stream is not closing properly, despite controller.close() being invoked. I'll try to confirm that. Could that affect the subsequent call to that service? Perhaps, but I'm not sure.
OK this is interesting.

The [POST] /api/generate status=200 log isn't generated after the controller is closed. That log is generated if I invoke the service again after waiting about 20+ seconds, and then it works again. But if I invoke the service before those 20 or so seconds, the next call is a timeout.

How do I ensure the controller is closed & the request is completed other than calling controller.close()?
Cape lionOP
I think I'm mistaken actually. That 200 is shown FIRST. My apologies for misunderstanding.
I'll work on this tomorrow. Really need to figure this out. I don't think I'm far from the solution...
Cape lionOP
Still having this issue unfortunately. It looks like the request goes through and closes successfully. Yet subsequent invocations that happen in under 20-30 seconds will always fail. They always succeed if the wait time is after 20-30 seconds. I know this is a weird and difficult to fix bug, but the logs look good on Vercel now, and the network tab doesn't show anything relevant to the error.

I need some ideas on how to diagnose this issue further.
Cape lionOP
It still fails sometimes but its working more or less better than before.
Cape lionOP
DISCOVERED THE PROBLEM!

It's Firebase hanging on the API key check. My AI service is working perfectly fine when the check is skipped.

I'll go to the Firebase folks and see if they can help me out here.