what is the proper way of uploading files with the new NextJS 13 app router?
Unanswered
Pink salmon posted this in #help-forum
Pink salmonOP
(first time messing with uploading files btw, so im kinda lost)
should i use a route handler? an old pages/api endpoint? or maybe with a server action somehow?
i was trying to use formidable to save the file to the public directory, (following a tutorial), because it seems to accept
should i use a route handler? an old pages/api endpoint? or maybe with a server action somehow?
i was trying to use formidable to save the file to the public directory, (following a tutorial), because it seems to accept
NextApiRequest from NextApiHandler, the pages/api way, (for some reason). but i dont know how can i achieve the same thing within a route handler, it doesnt want to accept a NextRequest object54 Replies
@Pink salmon im abt to play around with file uploading in Vercel ecosystem, especially
https://vercel.com/docs/storage/vercel-blob/using-blob-sdk
Blob. it is still in beta, but it worth trying it. you see a sample code, which you get an idea.https://vercel.com/docs/storage/vercel-blob/using-blob-sdk
one thing you might wanna know is, Vercel provides two types of runtimes: Clodflare worker(Edge Runtime) and AWS lambda(Node.js runtime). The former is a brand new, cheaper, lighter weight compared to the latter. when it comes to have file uploading mechanism, the former would be better as the official doc uses as such
export const runtime = 'edge'; // choose Edge Runtimeimage that malicious users try to attack your services to upload a buncha big size files, CF would be nicely handle those requests.
@Pink salmon (first time messing with uploading files btw, so im kinda lost)
should i use a route handler? an old pages/api endpoint? or maybe with a server action somehow?
i was trying to use formidable to save the file to the public directory, (following a tutorial), because it seems to accept `NextApiRequest` from `NextApiHandler`, the `pages/api` way, (for some reason). but i dont know how can i achieve the same thing within a route handler, it doesnt want to accept a `NextRequest` object
you can use
FormDataexport async function POST(request: Request) {
const formData = await request.formData();
console.log(formData.get("photo"));
return new Response("ok");
}"use client";
export default function Page() {
const handleSubmit = (event: React.FormEvent<HTMLFormElement>) => {
event.preventDefault();
const formData = new FormData(event.currentTarget);
fetch("/upload", { method: "POST", body: formData });
};
return (
<form onSubmit={handleSubmit}>
<label>
File:
<input type="file" name="photo" />
</label>
<button type="submit">Submit</button>
</form>
);
}@tafutada777 one thing you might wanna know is, Vercel provides two types of runtimes: Clodflare worker(Edge Runtime) and AWS lambda(Node.js runtime). The former is a brand new, cheaper, lighter weight compared to the latter. when it comes to have file uploading mechanism, the former would be better as the official doc uses as such
typescript
export const runtime = 'edge'; // choose Edge Runtime
Pink salmonOP
seems cool, but i dont use and have never used vercel, since i only do personal projects for now (web dev i mean, i haven even applied for a job yet), and im from a third world country in economic crisis with a lot of restrictions, so pricing is a problem, even spending just a few dollars. i would like to know at least how it works tho, someday ill have to deploy a real application for someone, so any suggestions about that?
@joulev you can use `FormData`
ts
export async function POST(request: Request) {
const formData = await request.formData();
console.log(formData.get("photo"));
return new Response("ok");
}
ts
"use client";
export default function Page() {
const handleSubmit = (event: React.FormEvent<HTMLFormElement>) => {
event.preventDefault();
const formData = new FormData(event.currentTarget);
fetch("/upload", { method: "POST", body: formData });
};
return (
<form onSubmit={handleSubmit}>
<label>
File:
<input type="file" name="photo" />
</label>
<button type="submit">Submit</button>
</form>
);
}
Pink salmonOP
ok so i guess
formData.get('photo') returns a File object, but then how do i write it to the disk (the public folder)? formidable apparently saved it automaticallyor maybe there is a better approach and i shouldnt be writing those files to the disk? as i said this is the first time messing with files so i have no idea tbh
@Pink salmon ok so i guess `formData.get('photo')` returns a `File` object, but then how do i write it to the disk (the public folder)? formidable apparently saved it automatically
you can simply use fs to write to disk normally
btw when you write to the public folder, you need to rebuild the app to be able to access it
and if you deploy on vercel, you cannot write to the public folder or anywhere except
/tmp which is where formidable actually saves files toPink salmonOP
mmm so it worked bc it was in dev mode?
yes
Pink salmonOP
what is the proper way then?
maybe blob in some database? i really have no idea
hmm, im used to serverless so i just upload that file to e.g. aws s3 or cloudflare r2
if you deploy in your own server, i guess you can just save it to your file system then use fs to read it
Pink salmonOP
mmm i think those are vms, right?
what are vms?
Pink salmonOP
vm, virtual machine
as in, what do you mean by "those"
Pink salmonOP
ah, aws s3 and couldflare r2
no not really
Pink salmonOP
mmm, what are they then? like i said i never really deployed anything, and i think i should start knowing about that stuff
basically they are servers hosted by amazon/cloudflare that you can upload files to and download files from
optimised to storing file objects compared to e.g. databases that are optimised to store textual data
Pink salmonOP
right, so they are cdns? or like a cdn
no
the way i understand it, cdns are where you host files in several locations around the world
which is not the case for these storage services
Pink salmonOP
yea, like cdn but without the network part
i was saying it more in the sense that they are only for storing files
and how do you do that exactly? like the implementation
a fetch to some url, using some access token?
you can see the documentation of each service to know that
Great black wasp
Use this as a base
Some of these may be outdated
You use presigned url to upload your file
@joulev you can see the documentation of each service to know that
Pink salmonOP
yea, ill do that. but how can i know what... things exist? i mean, if you tell me i have to store files, i would think of a vm. what am i supposed to do to know what services people use? read everything that amazon aws, cloudflare, vercel, etc offers? any other provider thats also widely used?
@Great black wasp https://youtu.be/wbNyipJw9rI
Pink salmonOP
thanks, ill look into that
Great black wasp
Aws documentation is hard you can just use uploadthing ngl
@Pink salmon yea, ill do that. but how can i know what... things exist? i mean, if you tell me i have to store files, i would think of a vm. what am i supposed to do to know what services people use? read everything that amazon aws, cloudflare, vercel, etc offers? any other provider thats also widely used?
https://discord.com/channels/752553802359505017/1024406585012924486/1118899175728349245 Near had a talk the other day about exactly this
Pink salmonOP
looks just like what im looking for, thanks, ill look into it
@Pink salmon sorry to hear abt ur situation. actually Vercel hobby plan is free with no credit reg. so many high scool kids in Japan use it. Vercel is really generous, good company i think.
https://vercel.com/pricing
https://vercel.com/pricing
Great black wasp
Vercel blob still in private beta you have to sign up for waitlist
@tafutada777 <@354114876526034944> sorry to hear abt ur situation. actually Vercel hobby plan is free with no credit reg. so many high scool kids in Japan use it. Vercel is really generous, good company i think.
https://vercel.com/pricing
Pink salmonOP
yeah i just saw that, looks pretty good, im gonna use it from now on for my personal projects, thanks
if i were you no job yet, i would create a portfolio web site and get a freelance job, after that you can work on file uploading and so on, just my two cents
Great black wasp
a proper way to upload file in nextjs 13 is probably client form data -> client validation-> client side fetch your endpoint (api route) -> do upload logic on your api route(validation file naming, etc)
Gazami crab
Im kinda confused about how file upload works. It seems that depending on the use case there are different processes. Can someone clarify what do I need in term of services integration to perform a file upload to OpenAI Whisper API? Consider the following scenario
1) Client running in browser, performs file upload (Blob)
2) API route handler running on Vercel (Receives Blob and somehow passes it to WhisperAPI)
3) WhisperAPI running on OpenAI servers (Receives the file and returns transcription)
Im not really sure what's the process in (2) do I need to write the file to the filesystem in order to be able to pass it to the WhisperAPI? That's what Im doing, but it turns out it works during development but not on Vercel (throwing ENOENT error)
is there any mechanism to no store the file in (2) and just pass it to (3) ?
Here's my code (works locally, not on vercel)
1) Client running in browser, performs file upload (Blob)
2) API route handler running on Vercel (Receives Blob and somehow passes it to WhisperAPI)
3) WhisperAPI running on OpenAI servers (Receives the file and returns transcription)
Im not really sure what's the process in (2) do I need to write the file to the filesystem in order to be able to pass it to the WhisperAPI? That's what Im doing, but it turns out it works during development but not on Vercel (throwing ENOENT error)
is there any mechanism to no store the file in (2) and just pass it to (3) ?
Here's my code (works locally, not on vercel)
export async function POST(req: NextRequest, res: NextResponse) {
const form = await req.formData()
const blob = form.get('file')! as Blob
/**
* Simple form validation
*/
if (!blob) {
return new Response('Bad Request', {
status: 400,
})
}
const buffer = Buffer.from(await blob.arrayBuffer())
const timestamp = new Date().toISOString().replace(/[-:.]/g,"");
const random = ("" + Math.random()).substring(2, 8);
const randomName = timestamp+random;
const filename = `${randomName}.webm`
let filepath = `${path.join('public', 'uploads', filename)}`
fs.writeFileSync(filepath, buffer)
let header = {
'Content-Type': 'multipart/form-data',
'Accept': 'application/json',
'Authorization': `Bearer ${process.env.OPENAI_API_KEY}`
}
let formData = new FormData()
formData.append('file', fs.createReadStream(filepath))
formData.append('model', 'whisper-1')
try {
let res = await axios.post('https://api.openai.com/v1/audio/transcriptions', formData, {
headers: {
...header,
}
});
console.log('Transcription ', res.data?.text);
return NextResponse.json({ transcription: res.data?.text });
} catch (error) {
console.log(error);
return NextResponse.json({ error: error });
} finally {
fs.unlink(filepath, () => {})
}
}@Gazami crab in general, people use stream and pipe like piping linux shell.
even though AWS Lambda has an ephemeral storage, disk I/O operations are slow and it is sync call (os level), so node.js spawns an os thread to prevent the event loop from choking. and you don't wanna consume memory by opening entire payload.
i am not sure exactly how to do that, but this could help.
https://stackoverflow.com/questions/75793118/streaming-multipart-form-data-request-with-native-fetch-in-node-js
ls -l | moreeven though AWS Lambda has an ephemeral storage, disk I/O operations are slow and it is sync call (os level), so node.js spawns an os thread to prevent the event loop from choking. and you don't wanna consume memory by opening entire payload.
i am not sure exactly how to do that, but this could help.
https://stackoverflow.com/questions/75793118/streaming-multipart-form-data-request-with-native-fetch-in-node-js
node.js is single thread, event loop. writeFileSync chokes the event loop, which is similar to freezing in web browser.
@tafutada777 <@468141287464370188> in general, people use stream and pipe like piping linux shell.
ls -l | more
even though AWS Lambda has an ephemeral storage, disk I/O operations are slow and it is sync call (os level), so node.js spawns an os thread to prevent the event loop from choking. and you don't wanna consume memory by opening entire payload.
i am not sure exactly how to do that, but this could help.
https://stackoverflow.com/questions/75793118/streaming-multipart-form-data-request-with-native-fetch-in-node-js
Gazami crab
I see. Right now I just fixed the above code by using /tmp as the upload directory. The approach you are mentioning seems more scalable right? I think saving it temporarily as Im doing it, will incur in more processing / storage in Serverless?
and more money. AWS lambda, which Next.js Serverless uses under the hood, charges wall time(elapsed time) instead of CPU time. Cloudflare worker edge runtime is charged by CPU time btw.
Actually i use GCP, so i am not familiar with AWS, so you might want to double check it.
Actually i use GCP, so i am not familiar with AWS, so you might want to double check it.