Api Function queue
Unanswered
Dickcissel posted this in #help-forum
DickcisselOP
Hey guys, so I have an issue with a webhook from an external service, the problem is that sometimes I get duplicated events that hit the endpoint at the exact same time, causing the updates that I need to be duplicated.
One possible fix I thought of is to create a queue, and do a first in first out (FIFO).
The problem is that when I deploy it to vercel it doesnt work, the approach I took is to create a variable that will hold the array with requests:
And for some reason this locally seemed to work, but not when deployed to Vercel.
The reason I suppose is that each call of that function is "stateless" in a way, and consecutive calls dont really have the same
The question then becomes, how can I solve that problem within the realm of Nextjs/Vercel, or do I have to look for other solutions?
One possible fix I thought of is to create a queue, and do a first in first out (FIFO).
The problem is that when I deploy it to vercel it doesnt work, the approach I took is to create a variable that will hold the array with requests:
const queue: { req: NextApiRequest; res: NextApiResponse }[] = []
let isProcessing = false
export async function handleApiFunction (req: NextApiRequest, res: NextApiResponse) {
queue.push({ req, res })
if (!isProcessing) {
console.log({ isProcessing })
await processQueue()
}
}And for some reason this locally seemed to work, but not when deployed to Vercel.
The reason I suppose is that each call of that function is "stateless" in a way, and consecutive calls dont really have the same
queue array.The question then becomes, how can I solve that problem within the realm of Nextjs/Vercel, or do I have to look for other solutions?
3 Replies
@Dickcissel yes. as you said, it is possible that duplicated events can be handled by different instances, i.e. AWS lambda Node.js runtimes, which Vercel platform employs behind the scenes, especially heavy traffic situation. Obviously only one runtime when getting it up and running locally on ur machine.
So to address ur issue, duplicate events, there are two options.
1. make the api idempotent with event_id
2. use Vercel KV, Redis, with rdb.SetNX(event_id), which is a kinda exclusive lock. I faced the similar issue with GCP Pubsub, emit duplicate events, so i implemented it in Golang. I can send it to you if you want.
So to address ur issue, duplicate events, there are two options.
1. make the api idempotent with event_id
2. use Vercel KV, Redis, with rdb.SetNX(event_id), which is a kinda exclusive lock. I faced the similar issue with GCP Pubsub, emit duplicate events, so i implemented it in Golang. I can send it to you if you want.
DickcisselOP
I was going to use kv but the problem is kv is not available to enterprise clients, and the project is under enterprise account, so then the idea becomes using a db to store the queue there or completely move away from nextjs for that feature and do a normal backe d
oh. or Upstash directly