Running out of vercel resources
Unanswered
Northern Pintail posted this in #help-forum
Northern PintailOP
So I have an open source website and its being used by a lot of students from my college and my vercel resources are depleting fast, any other hosting option I can host at for nextjs which is fast and serverless? I dont really wanna spend money on something that doesnt make money
77 Replies
is your nextjs website already cached properly?
what is your website's route setup?
build log?
@alfonsüs ardani is your nextjs website already cached properly?
Northern PintailOP
I get a lot of users actually like 1.5k daily visitors so I guess it’s expected?
It’s a PWA app with only 1 page and 7 endpoints which the user can click
I am running out of fluid CPU so fast because of that 7 endpoints
I scrape and process a lot of data so maybe
\
Route (app)
┌ ○ /
├ ○ /_not-found
├ ○ /~offline
├ ƒ /api/fetchAllGrades
├ ƒ /api/fetchAttendance
├ ƒ /api/fetchExamSchedule
├ ƒ /api/fetchGrades
├ ƒ /api/fetchHostelDetails
├ ƒ /api/fetchMarks
├ ƒ /api/getCaptcha
├ ƒ /api/login
└ ƒ /api/parseSemTT
┌ ○ /
├ ○ /_not-found
├ ○ /~offline
├ ƒ /api/fetchAllGrades
├ ƒ /api/fetchAttendance
├ ƒ /api/fetchExamSchedule
├ ƒ /api/fetchGrades
├ ƒ /api/fetchHostelDetails
├ ƒ /api/fetchMarks
├ ƒ /api/getCaptcha
├ ƒ /api/login
└ ƒ /api/parseSemTT
@Northern Pintail I get a lot of users actually like 1.5k daily visitors so I guess it’s expected?
yea but have you cached the functions?
@alfonsüs ardani yea but have you cached the functions?
Northern PintailOP
not sure
Traced Next.js server files in: 43.875ms
Created all serverless functions in: 168.902ms
Collected static files (public/, static/, .next/static): 10.873ms
Build Completed in /vercel/output [22s]
Deploying outputs...
Deployment completed
Creating build cache...
Created build cache: 26.254s
Uploading build cache [267.15 MB]
Build cache uploaded: 5.737sthese are the logs
do I have to manually cache the functions?
@Northern Pintail do I have to manually cache the functions?
i guess so? does your data changes very actively? it would be a waste if it scrapes the same data if it doesn't change for few seconds/mimutes/hour i think?
do you have an estimate on how often the data changes?
@alfonsüs ardani i guess so? does your data changes very actively? it would be a waste if it scrapes the same data if it doesn't change for few seconds/mimutes/hour i think?
Northern PintailOP
the button to sync the data is manual an the user presses it on their own, the data prolly only changes once a day but because its manual I expect a user to only click it once in a few days but again I have a big amount of users using it daily
sometimes lets say results/marks came so a lot of users will reload together
sometimes lets say results/marks came so a lot of users will reload together
its all manual and depending on the user
@Northern Pintail the button to sync the data is manual an the user presses it on their own, the data prolly only changes once a day but because its manual I expect a user to only click it once in a few days but again I have a big amount of users using it daily
sometimes lets say results/marks came so a lot of users will reload together
yeah so what you can do is just strictly cache the function to only rescrape once per day.
and cache results/marks once every 30 minute or an hour
or 10 minutes even
Northern PintailOP
but I dont know when the data updates, different teachers upload stuff differently so the data for everyone doesnt update together, I save everything in localhost aswell but still thats why I give the manual button
@alfonsüs ardani and cache results/marks once every 30 minute or an hour
Northern PintailOP
I save everything in localhost anyways as they are scraped
then what is the function doing in vercel :/
@alfonsüs ardani then what is the function doing in vercel :/
Northern PintailOP
the manual button calls those API's to scrape different endpoints of a website and get the data
there are 14-15 endpoints I scrape all with different data
so the scraping isn't being done in vercel?
Northern PintailOP
its being done inside the functions
you said you save everything in localhost yet you have users clicking a manual button that hits those endpoint? i am confused sorry
Northern PintailOP
mb localstorage*
i said localhost
had to say localstorage
i dont know how or why thats relevant
try caching it for 15 - 30 mins and see if the usage decreases
Northern PintailOP
I have cached every 30 mins, will see if the usage decreases
@Northern Pintail I have cached every 30 mins, will see if the usage decreases
make sure refreshing the endpoint in the browser shows instant result
even at and after the 30th minute
Northern PintailOP
Hmm it isn’t happening instantly
const nextConfig = {
reactStrictMode: true,
devIndicators: false,
images: {
remotePatterns: [
{
protocol: 'https',
hostname: 'avatars.githubusercontent.com',
},
{
protocol: 'https',
hostname: 'cdn.discordapp.com',
},
{
protocol: 'https',
hostname: 'lh3.googleusercontent.com',
},
],
},
async headers() {
return [
{
source: "/api/(.*)",
headers: [
{
key: "Cache-Control",
value: "public, s-maxage=1800, stale-while-revalidate=60",
},
],
},
{
source: "/:path*",
headers: [
{
key: "Cache-Control",
value: "public, s-maxage=1800, stale-while-revalidate=60",
},
],
},
];
},
webpack: (config, { isServer }) => {
if (isServer) {
config.externals.push("@napi-rs/canvas");
}
return config;
},
};This is the only thing we have to add right?
you have to wrap your api with unstable_cache or "use cache" (if next16 with cachedComponent: true)
i am eager to show you how fast it will be. hopefully it will decrease your usage
Northern PintailOP
Ok
I am using next 16 yes
that case you can just write "use cache" on file-top-level of the API endpoint and add
https://nextjs.org/docs/app/api-reference/functions/cacheLife
cacheLife("minutes") (or hours, or custom time with custom profiles)https://nextjs.org/docs/app/api-reference/functions/cacheLife
Northern PintailOP
Kk
Northern PintailOP
Cache Error: Cannot access json on the server. You cannot dot into a temporary client reference from a server component. You can only pass the value through to the client.
at POST (src\app\api\fetchMarks\route.js:13:62)
11 | try {
12 | cacheLife('halfHour');
> 13 | const { cookies, dashboardHtml, campus } = await req.json();
| ^
14 |
15 | const $ = cheerio.load(dashboardHtml);
16 | const cookieHeader = Array.isArray(cookies) ? cookies.join("; ") : cookies; {was getting this, what's the way around this? because well i cannot remove it
@Northern Pintail Cache Error: Cannot access json on the server. You cannot dot into a temporary client reference from a server component. You can only pass the value through to the client.
at POST (src\app\api\fetchMarks\route.js:13:62)
11 | try {
12 | cacheLife('halfHour');
> 13 | const { cookies, dashboardHtml, campus } = await req.json();
| ^
14 |
15 | const $ = cheerio.load(dashboardHtml);
16 | const cookieHeader = Array.isArray(cookies) ? cookies.join("; ") : cookies; {
was getting this, what's the way around this? because well i cannot remove it
interesting. there is an API that depends on inocming request?
i thought its just a GET without request data
that way you need to cache only the scraping function not the whole route.
something like
also make sure that cacheLife profile halfHour exists
something like
export async function scrapeSomething() {
"use cache"
cacheLife('halfHour')
const $ = cheerio.load(...)
...
// do not do check auth here. check auth before or after this function.
return data
}also make sure that cacheLife profile halfHour exists
Northern PintailOP
Ya I did that I did define halfHour
I will cache the function ok
@Northern Pintail I will cache the function ok
did it work? sorry for pinging
@alfonsüs ardani did it work? sorry for pinging
Northern PintailOP
I did try it and was having some errors, earlier I had all the functions bunched up into 1 button so the things that only changed maybe once per semester were being synced as much as attendance and marks, so now I have only attatched attednance and marks to the reload button and all the other stuff spaced out in their own pages so the user reloads the thing only when they think it might have changed so as to not so much ( everything does reload once on login )
I will do the caching now but this splitting of sync buttons so the useless functions dont sync so often should decrease the usage a lot
was working on the splitting yesterday all day, it just finished now
@Northern Pintail So I have an open source website and its being used by a lot of students from my college and my vercel resources are depleting fast, any other hosting option I can host at for nextjs which is fast and serverless? I dont really wanna spend money on something that doesnt make money
Cassin's Kingbird
cloudflare workers should do the job
unless ur hitting 100k requests per day
then the paid tier is just $5 / month
@Cassin's Kingbird cloudflare workers should do the job
Northern PintailOP
I have tried Cloudflare workers but was getting certificate errors which is most prolly because my college website gives invalid certificate sometimes ( idk why don’t ask )
Have opened a ticket in cloudflare’s server regarding if I could bypass that error but no response till now
Cassin's Kingbird
Wdym certificate errors
Like ssl certificates?
@Cassin's Kingbird Wdym certificate errors
Northern PintailOP
failed: TLS peer's certificate is not trusted; reason = unable to get local issuer certificate
I am assuming its because of my college website
Cassin's Kingbird
Well there are many other hosting providers too
Northern PintailOP
otherwise I am not sure
Cassin's Kingbird
Do you have GitHub student pack?
@Cassin's Kingbird Well there are many other hosting providers too
Northern PintailOP
could host in multiple websites and have a loadbalancer accross them ( my friend suggested this )
@Cassin's Kingbird Do you have GitHub student pack?
Northern PintailOP
yes
Cassin's Kingbird
Free for github students
@Northern Pintail could host in multiple websites and have a loadbalancer accross them ( my friend suggested this )
Cassin's Kingbird
no u don't need a loadbalancer if your hosting on an edge network
@Cassin's Kingbird Free for github students
Northern PintailOP
kk
will check that out