Kubernetes rollout static error.
Answered
Cuban Crocodile posted this in #help-forum
Cuban CrocodileOP
Hi guys. We run Next.JS app in kubernetes. We face an issue when we release new version. What happens is:
1. Client loads the page
2. We release new version and Kubernetes rolls out few pods on the new version
3. Clients from p1 have their page loaded and they continue requesting chunks. And if the request arrives to a new version it would return 404
4. If client refreshes the page and requests comes to the new version some subsequent requests could still land to the old version and get 404
One way of dealing with it is roll out update all at once to minimize the sliding window but I still feel there must be a better way.
My question is, what it the right way to roll out new version of next.js app, especially hosted in Kubernetes?
1. Client loads the page
2. We release new version and Kubernetes rolls out few pods on the new version
3. Clients from p1 have their page loaded and they continue requesting chunks. And if the request arrives to a new version it would return 404
4. If client refreshes the page and requests comes to the new version some subsequent requests could still land to the old version and get 404
One way of dealing with it is roll out update all at once to minimize the sliding window but I still feel there must be a better way.
My question is, what it the right way to roll out new version of next.js app, especially hosted in Kubernetes?
Answered by Toyger
found it, basically they use sticky session, just instead cookie they use different key from headers https://vercel.com/docs/deployments/skew-protection
9 Replies
Toyger
hmm, never think of this.
Can you show some screenshots of errors, for what specific files and path it show 404.
did you try to reproduce it on vercel, they basically have some kind of blue green deployment, so theoretically it should have same behaviour.
First thing that come to mind as one of possible workarounds you can attach in middleware some "version" to cookies, and check it on client with previously stored version in localstorage, if they mismatch force router.refresh which most likely should refetch new version.
Can you show some screenshots of errors, for what specific files and path it show 404.
did you try to reproduce it on vercel, they basically have some kind of blue green deployment, so theoretically it should have same behaviour.
First thing that come to mind as one of possible workarounds you can attach in middleware some "version" to cookies, and check it on client with previously stored version in localstorage, if they mismatch force router.refresh which most likely should refetch new version.
Cuban CrocodileOP
Don’t have a screenshot right now. But the root cause is simple. One index.html has one set of chunks, another index.html has different chunks. And if you serve two versions at the same time, you are going to have mix and match.
Sticky sessions and cookies is a workaround to a certain extent. But you’d also need to endure that any connection with no cookies always goes to the new version.
What is the exact blue green mechanism for vercel?
@Cuban Crocodile What is the exact blue green mechanism for vercel?
Toyger
I don't know what they use, but site remain available when new is deploying so theoretically if they don't have some additional workaround on vercel should be same behaviour with deployment.
why I asked about paths, because my assumption that you already have resources in cache from previous deployment, so when you make new request it should go for RSC which will should generate new Etag which theoretically should force resource or page update. But if bug persist same on vercel then probably you can submit new issue to github with this behaviour and maybe they will fix it.
why I asked about paths, because my assumption that you already have resources in cache from previous deployment, so when you make new request it should go for RSC which will should generate new Etag which theoretically should force resource or page update. But if bug persist same on vercel then probably you can submit new issue to github with this behaviour and maybe they will fix it.
Cuban CrocodileOP
The way it works is:
Client loads index.html -
inside we have:
include chunk-123.js
include chunk-456.js
Then we have version 2 of the same app, it has index.html with different chunks
include chunk-789.js
include chunk-101112.js
If we have both versions running at the same time, then request for index.html may land to v1, but request for include chunk-123.js might land to v2 so it would result into 404.
What is the strategy for a rolling update?
Client loads index.html -
inside we have:
include chunk-123.js
include chunk-456.js
Then we have version 2 of the same app, it has index.html with different chunks
include chunk-789.js
include chunk-101112.js
If we have both versions running at the same time, then request for index.html may land to v1, but request for include chunk-123.js might land to v2 so it would result into 404.
What is the strategy for a rolling update?
Does vercel use sticky sessions?
@Cuban Crocodile Does vercel use sticky sessions?
Toyger
found it, basically they use sticky session, just instead cookie they use different key from headers https://vercel.com/docs/deployments/skew-protection
Answer
Cuban CrocodileOP
I knew there is no magic! )