Unusual Data Cache Writes on Vercel Deployment
Unanswered
Cape lion posted this in #help-forum
Cape lionOP
I deploy my Next.js project on Vercel and in a few recen days, the Data Cache Writes just eat up all the 1M Write units in just a few hours. What happened?
I have checked out the Incremental Static Regeneration and increase the revalidate time but it does not help.
I have checked out the Incremental Static Regeneration and increase the revalidate time but it does not help.
57 Replies
Does your url have personalized parts in it? Even query params that you read
Cape lionOP
One thing I've noticed after working with the app directory for some time now is that setting the route segmentation configs directly on the page, otherwise lots of weird things can happen and you can't seem to figure out why
Especially the "dynamic" one
Cape lionOP
For this project I'm still working on the page router.
Ah okay hmm
Cape lionOP
It's still fine until a few deays ago and suddenly it eat up all the available units
Data-cache is app directory only afaik, that's why I got confused. What you mean is the incremental static generations shot up. (I know they are grouped in the same chart above)
But since we know it's page router now, things are a bit different
Cape lionOP
Yeah in the chart it stated that ISR and all the help link recommend to increase the revalidate time. But it does not work.
@MikeTheTechLead Data-cache is app directory only afaik, that's why I got confused. What you mean is the incremental static generations shot up. (I know they are grouped in the same chart above)
Cape lionOP
Is the Data Cache Writes only used for App Router? It's weird. As now it is counted as billable.
First we need to figure out from the logs if its the same pages that are generated every time or if it's a bot that is just pounding different pages that do not exist but will be counted towards ISR for some reason
Also you can also try and load a page up in your browser, go to the dev tools -> network and find the page load event and see if vercel is sending a "hit" or "miss" back
Just to make sure that pages actually get cached
Cape lionOP
I'm checking this page: https://www.nanohome.vn/en-US/products/sub-category/table-lamps
actually there are a lot of MISS for each product
I do see a lot of stales yep.
Cape lionOP
https://www.nanohome.vn/products
Let's say this page. This HIT. So it means, it cached correctly?
Let's say this page. This HIT. So it means, it cached correctly?
Yes. HIT means its served from Cloudflare (Vercels CDN cache)
STALE means that Vercel has deemed that its old, so next time someone visits it, it will be regenerated
What I see is that everything under ta normal page (like clearance.json) seems ok. but anything under /products/category/lighting.json?category=lighting seems to always be stale
Cape lionOP
I just deploy the projects a few hours ago, so the data should be fresh?
I don't change anything on the database, so it's weird that it's stale.
well the stale/hit depends on when vercel/next thinks that the generated page i stale, not when the data is old. at least when it comes to the page router
Cape lionOP
As I remember, the revalidate time is 600-6000 so at least after that amount of time, it start to consider fetching the new data?
that is how its supposed to be yes
Cape lionOP
hmmm I'm really suspecting the culprit is the recent change in vercel pricing.
I happened on 24 July
hmm it shouldnt be
Cape lionOP
Perhaps it is due to this 8Kb - how they calculate the Data Write
maybe that just exposed something that was wrong codewise
but from what I can tell, when loading this page: https://www.nanohome.vn/en-US/products/recTFbl9lJUhciher/usm-haller-cabinet-o2xoce2-metal-pure-orange-drop-down-door-fully-opened
it preloads this url:https://www.nanohome.vn/_next/data/rfbQfZ0ZeDJ6WjK52ToOx/en-US/products/category/lighting.json?category=lighting
which for some reason is always stale.
But if I load that url directly, its not stale
it preloads this url:https://www.nanohome.vn/_next/data/rfbQfZ0ZeDJ6WjK52ToOx/en-US/products/category/lighting.json?category=lighting
which for some reason is always stale.
But if I load that url directly, its not stale
Cape lionOP
The preload should be OK? I remember next.js will try to preload links available on the screen
Btw, is "stale" a problem? I mean if it's stale, then it should not be recorded in the cache data?
nope, the preload seems to trigger generation of a page because its stale.
Cape lionOP
Ah, I see what you mean
Stale => old version of the page gets deleted => new page version generated => ISR usage => costly
so every page load of that one product details page effectively regenerates 2 pages every time
now the question is why its happening
Cape lionOP
My best guest is it mess up somehow with react-query
hmm, now that one with lighting seems to be HIT, but some of the others arent
It seems like those pages are in the stale for a while before they are cached. Could it be that your backend is very slow at delivering data or that they timeout under generation somehow?
Cape lionOP
Yeah that might be a good guess
I'm using Airtable for this and recently I think they put some limit on the API
🤔
if any, you might want to consider moving to a setup where you have a revalidation of lets say 24 hours or more, and then revalidate via the revalidate tag instead on data change
if its possible
600s for revalidation is fairly low on pages. Can reek quite the cost on vercel if someone were to "lovebomb" your site, hitting all pages at one and every x seconds, if they do not get triggered by the spam/bot filter
Cape lionOP
Thank you a lot.
Let me check the above suggestions and see if it's the problem.
Also, lil mention, the cache is tied to the deploymentId in vercel, so when you do a deployment, you will invalidate the current cached pages
@Cape lion Is there a way to know if I'm under attack?
Easiest thing is to look at the firewall page in vercel and see if it has detected something
Cape lionOP
OK it seems my website is under attack. I remove all the ISR (revalidate) in the project, the problem is still there.
I turn on the Attack Challenge Mode in Firewall and it's OK now.
I turn on the Attack Challenge Mode in Firewall and it's OK now.