Internal Server Error (500) on Production Standalone Mode in Alpine Image - Next.js 13.4.12, 13.4.19
Answered
JTwhizzkid posted this in #help-forum
Hello! I'm currently having issues when trying to deploy a NextJS app using standalone mode in a Docker image. I've spent all day trying to figure out the issue and apply 'fixes' that people have found online, but none of them seem to work for me.
On Next.js 13.4.12, the whole website returns the Internal Server Error 500 error, with ECONNREFUSED 0.0.0.0:36993 (or a similar port) errors in the logs.
On Next.js 13.4.19 (latest), the website loads however just the image endpoint receives the error instead.
Looking online, this seems to have been a common issue introduced in 13.4.13 and fixed in 13.4.19, but I'm thinking this may be a different issue?
Will post additional logs and info in the thread.
On Next.js 13.4.12, the whole website returns the Internal Server Error 500 error, with ECONNREFUSED 0.0.0.0:36993 (or a similar port) errors in the logs.
On Next.js 13.4.19 (latest), the website loads however just the image endpoint receives the error instead.
Looking online, this seems to have been a common issue introduced in 13.4.13 and fixed in 13.4.19, but I'm thinking this may be a different issue?
Will post additional logs and info in the thread.
Answered by JTwhizzkid
Hello - managed to resolve this, but just an update if anyone else has this issue/is interested:
The issue was (I believe) with the server having an older CPU - some sort of Intel Xeon 3GHz 4 core one, not sure on the model. I managed to fix this by pinning the service on Docker Swarm to a different host.
I think it was completely unrelated to all the other issues introduced in Next 13.4.13.
In hindsight, I remember having a similar problem with the
Thanks for anyone who helped!
The issue was (I believe) with the server having an older CPU - some sort of Intel Xeon 3GHz 4 core one, not sure on the model. I managed to fix this by pinning the service on Docker Swarm to a different host.
I think it was completely unrelated to all the other issues introduced in Next 13.4.13.
In hindsight, I remember having a similar problem with the
next build command in the GitLab CI/CD runner - I was receiving a random Illegal Instruction Core Dumped fault on this node, which I accidentally discovered the fix when I restarted the runner and it restarted on a different host. But as I couldn't see any errors or logs anywhere and as that was done a while ago, I'd completely forgotten about it.Thanks for anyone who helped!
21 Replies
Dockerfile:
(Have tried adding
ENV NODE_TLS_REJECT_UNAUTHORIZED 0
and ENV HOSTNAME "127.0.0.1" but did not work)
FROM node:18-alpine
ENV NODE_ENV production
ENV NEXT_TELEMETRY_DISABLED 1
RUN addgroup --system --gid 1001 nodejs
RUN adduser --system --uid 1001 nextjs
COPY ./public /app/public
COPY --chown=nextjs:nodejs ./.next/standalone /app
COPY --chown=nextjs:nodejs ./.next/static /app/.next/static
USER nextjs
EXPOSE 3000
ENV PORT 3000
ENV HOSTNAME "0.0.0.0"
WORKDIR /app
CMD ["node", "server.js"](Have tried adding
ENV NODE_TLS_REJECT_UNAUTHORIZED 0
and ENV HOSTNAME "127.0.0.1" but did not work)
Logs on 13.4.12
(Summary for web forum index)
(Summary for web forum index)
- info Loaded env from /app/.env
Listening on port 3000 url: http://0.0.0.0:3000
- info Loaded env from /app/.env
Error: socket hang up
at connResetException (node:internal/errors:720:14)
at Socket.socketOnEnd (node:_http_client:525:23)
at Socket.emit (node:events:526:35)
at endReadableNT (node:internal/streams/readable:1359:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
code: 'ECONNRESET'
}
...
Error: connect ECONNREFUSED 0.0.0.0:36993
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1495:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '0.0.0.0',
port: 36993
}Logs on 13.4.19
(Summary for web forum indexing)
(Summary for web forum indexing)
website-v6-dev.1.9adq5q5216lm@Panavision | - ready started server on 127.0.0.1:3000, url: http://127.0.0.1:3000
...
ready started server on [::]:3000, url: http://localhost:3000
website-v6-dev.1.c0xg42joylic@Panavision | - info Loaded env from /app/.env
website-v6-dev.1.c0xg42joylic@Panavision | - info Loaded env from /app/.env
website-v6-dev.1.c0xg42joylic@Panavision | TypeError: fetch failed
website-v6-dev.1.c0xg42joylic@Panavision | at Object.fetch (node:internal/deps/undici/undici:11576:11)
website-v6-dev.1.c0xg42joylic@Panavision | at async invokeRequest (/app/node_modules/next/dist/server/lib/server-ipc/invoke-request.js:17:12)
website-v6-dev.1.c0xg42joylic@Panavision | at async invokeRender (/app/node_modules/next/dist/server/lib/router-server.js:254:29)
website-v6-dev.1.c0xg42joylic@Panavision | at async handleRequest (/app/node_modules/next/dist/server/lib/router-server.js:447:24)
website-v6-dev.1.c0xg42joylic@Panavision | at async requestHandler (/app/node_modules/next/dist/server/lib/router-server.js:464:13)
website-v6-dev.1.c0xg42joylic@Panavision | at async Server.<anonymous> (/app/node_modules/next/dist/server/lib/start-server.js:117:13) {
website-v6-dev.1.c0xg42joylic@Panavision | cause: Error: read ECONNRESET
website-v6-dev.1.c0xg42joylic@Panavision | at TCP.onStreamRead (node:internal/stream_base_commons:217:20) {
website-v6-dev.1.c0xg42joylic@Panavision | errno: -104,
website-v6-dev.1.c0xg42joylic@Panavision | code: 'ECONNRESET',
website-v6-dev.1.c0xg42joylic@Panavision | syscall: 'read'
website-v6-dev.1.c0xg42joylic@Panavision | }
website-v6-dev.1.c0xg42joylic@Panavision | }If I ssh into the container and use netstat, there doesn't seem to be anything on the port that's having the issue on 13.4.12. On 13.4.19, there's a next-render-work process on the port
Running the standalone server locally on my Windows development machine works fine - the issue seems to just be on the Docker image (tried both node:16-alpine and node:18-alpine). I have a GitLab CI/CD pipeline that successfully installs deps, lints and builds.
Sharp is also installed in the package.json. I'm using Prisma linked to a read-only MySQL database and a couple external apis. Also using google fonts using the new next/font/google system.
Thanks in advance!
Sharp is also installed in the package.json. I'm using Prisma linked to a read-only MySQL database and a couple external apis. Also using google fonts using the new next/font/google system.
Thanks in advance!
European sprat
Have you tried a different node version? My alpine docker image is set to 18.17.1 and it's working
There's been a whole slew of issues with standalone builds and things that affected docker deployments but it all seems to be resolved as of 13.4.19 although I had to make a bunch of changes and ensure I'm passing in ENV HOSTNAME 0.0.0.0 (AWS ECS deployments set this to the internal AWS host/address if you don't specifically set it yourself and it needs to be 0.0.0.0)
Where are you deploying it?
It’s deployed on a Docker Swarm that goes through a nginx reverse proxy on another swarm service. The env var should already be set in the dockerfile.
Will try updating the node version tomorrow
European sprat
Yea just ensure the swarm isn't overriding it by default. I assume it wouldn't but worth checking
It's really been a nightmare over the past two weeks with all the bugs that appeared related to standalone
@European sprat Have you tried a different node version? My alpine docker image is set to 18.17.1 and it's working
It worked for the images on the first page then stopped working again with the same error
On alpine 18.7.1 with next 13.4.9, have confirmed the hostname is 0.0.0.0 in the env
TypeError: fetch failed
at Object.fetch (node:internal/deps/undici/undici:11576:11)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async invokeRequest (/app/node_modules/next/dist/server/lib/server-ipc/invoke-request.js:17:12)
at async invokeRender (/app/node_modules/next/dist/server/lib/router-server.js:254:29)
at async requestHandler (/app/node_modules/next/dist/server/lib/router-server.js:475:24)
at async Server.<anonymous> (/app/node_modules/next/dist/server/lib/start-server.js:117:13) {
cause: Error: connect ECONNREFUSED 0.0.0.0:41955
at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1495:16) {
errno: -111,
code: 'ECONNREFUSED',
syscall: 'connect',
address: '0.0.0.0',
port: 41955
}
}For some reason it seems to be trying to fetch a port that doesn't have anything listening on it
The line on router-server.js seems to think there is a worker running on that port, which there isn't.
So it appears that when the container boots then the port is correct, but there is this error.
Then after a bit the next render worker program stops and starts on a new port, but the new port is not updated in the router worker so it then shows the other error
Then after a bit the next render worker program stops and starts on a new port, but the new port is not updated in the router worker so it then shows the other error
Reproduced on a full node 18.17.1 docker image, same issue. Also tried directly installing sharp within the docker image.
If I take the .next directory and manually run it on my windows machine it works fine
If I take the .next directory and manually run it on my windows machine it works fine
So done a bit more testing, it seems if I run the Docker image on the debian linux server in the swarm (through nginx) or individually (directly) it doesn't work, but if I run the exact same image on my windows machine with wsl2 it's fine
Running it on my ubuntu linux server is also fine. Have also tried running the container in privileged more and it doesn't make any difference. The ubuntu and debian docker inspect responses are pretty much identical. Only diff apart from the OS is my debian install is on docker 24.0.4 and ubuntu is on 24.0.5
Hello - managed to resolve this, but just an update if anyone else has this issue/is interested:
The issue was (I believe) with the server having an older CPU - some sort of Intel Xeon 3GHz 4 core one, not sure on the model. I managed to fix this by pinning the service on Docker Swarm to a different host.
I think it was completely unrelated to all the other issues introduced in Next 13.4.13.
In hindsight, I remember having a similar problem with the
Thanks for anyone who helped!
The issue was (I believe) with the server having an older CPU - some sort of Intel Xeon 3GHz 4 core one, not sure on the model. I managed to fix this by pinning the service on Docker Swarm to a different host.
I think it was completely unrelated to all the other issues introduced in Next 13.4.13.
In hindsight, I remember having a similar problem with the
next build command in the GitLab CI/CD runner - I was receiving a random Illegal Instruction Core Dumped fault on this node, which I accidentally discovered the fix when I restarted the runner and it restarted on a different host. But as I couldn't see any errors or logs anywhere and as that was done a while ago, I'd completely forgotten about it.Thanks for anyone who helped!
Answer