Handling large multipart uploads
Unanswered
mjk134 posted this in #help-forum
mjk134OP
Hi, I'm using a nextjs rest api handler to forward a multipart (file) upload to another service, like so:
But i'm concerned about handling much larger file uploads. It might end up using the memory since it is being awaited. Is there any way to pass in a stream instead?
// Read form data from client
const formData = await req.formData();
const apiRequest = await fetch(
`http://localhost:3000/upload?userId=${user.user_id}&folderId=${folderId}&tempFileId=${tempFileId}`,
{
method: "POST",
body: formData,
},
);But i'm concerned about handling much larger file uploads. It might end up using the memory since it is being awaited. Is there any way to pass in a stream instead?
16 Replies
mjk134OP
I do some data/session checking in between hence the requirement of the handler.
@mjk134 Hi, I'm using a nextjs rest api handler to forward a multipart (file) upload to another service, like so:
ts
// Read form data from client
const formData = await req.formData();
const apiRequest = await fetch(
`http://localhost:3000/upload?userId=${user.user_id}&folderId=${folderId}&tempFileId=${tempFileId}`,
{
method: "POST",
body: formData,
},
);
But i'm concerned about handling much larger file uploads. It might end up using the memory since it is being awaited. Is there any way to pass in a stream instead?
I feel like a simple redirect would do the job here, assuming the service is in an external system.
If the service is just another route handler served at localhost:3000 then this new route handler makes little sense to me.
If the service is just another route handler served at localhost:3000 then this new route handler makes little sense to me.
@joulev I feel like a simple redirect would do the job here, assuming the service is in an external system.
If the service is just another route handler served at localhost:3000 then this new route handler makes little sense to me.
mjk134OP
The service is a fastify server, how would a redirect work
Instead of fetch() the service using the exact same request body, just NextResponse.redirect() to that service, the request body should be sent along as well
Actually nvm
That wouldn’t work, lemme see if I can come up with another way
@joulev That wouldn’t work, lemme see if I can come up with another way
mjk134OP
appreciate the help
@mjk134 The service is a fastify server, how would a redirect work
where do you host the file? in a s3-compatible bucket i assume?
mjk134OP
the file being uploaded or the actual location of file
as in what do you do with the file when it reaches fastify? you save it somewhere right?
mjk134OP
if its the location assume its not a bucket but the actual server disk
@joulev as in what do you do with the file when it reaches fastify? you save it somewhere right?
mjk134OP
yeah fastify saves it to the server disk then i proccess it there
the thing is, no matter what you do in this route handler, since the file is submitted to this route handler all of its content is always put to the server memory, so large uploads will always cost a lot of memory even if you run
therefore the popular approach if you use an s3-compatible bucket is: instead of uploading the file via the nextjs server, you ask the nextjs server to generate a presigned URL for you, then you upload to that presigned URL directly from the client's browser. the presigned URL will be a service that can take and process the big files without any issues.
here since you are not using a s3-compatible bucket, i assume your fastify server can already handle large uploads so you could do something similar to the approach above: nextjs server asks for a presigned URL (basically a direct upload URL with an expiration date) from fastify, then the browser uploads to fastify directly via that presigned URL without going thru the nextjs server.
req.formData() or not.therefore the popular approach if you use an s3-compatible bucket is: instead of uploading the file via the nextjs server, you ask the nextjs server to generate a presigned URL for you, then you upload to that presigned URL directly from the client's browser. the presigned URL will be a service that can take and process the big files without any issues.
here since you are not using a s3-compatible bucket, i assume your fastify server can already handle large uploads so you could do something similar to the approach above: nextjs server asks for a presigned URL (basically a direct upload URL with an expiration date) from fastify, then the browser uploads to fastify directly via that presigned URL without going thru the nextjs server.
@joulev the thing is, no matter what you do in this route handler, since the file is submitted to this route handler all of its content is always put to the server memory, so large uploads will always cost a lot of memory even if you run `req.formData()` or not.
therefore the popular approach if you use an s3-compatible bucket is: instead of uploading the file via the nextjs server, you ask the nextjs server to generate a presigned URL for you, then you upload to that presigned URL directly from the client's browser. the presigned URL will be a service that can take and process the big files without any issues.
here since you are not using a s3-compatible bucket, i assume your fastify server can already handle large uploads so you could do something similar to the approach above: nextjs server asks for a presigned URL (basically a direct upload URL with an expiration date) from fastify, then the browser uploads to fastify directly via that presigned URL without going thru the nextjs server.
mjk134OP
Thats kinda what i wanted to avoid
but if there is no alternative
its fine