That's to be expected. Emulation is not performant at all. If your project is public, then Linux ARM runners should be available for free; otherwise, if the project is private you have no other option than to host your own runner until ARM runners becomes available for private projects as well.

One thing you could play around with is to change the build platform for unrelated stages if you use docker buildx.

For example I do that in my Dockerfile:

FROM --platform=$BUILDPLATFORM node:22-alpine@sha256:5539840ce9d013fa13e3b9814c9353024be7ac75aca5db6d039504a56c04ea59 AS base
# ...
FROM --platform=$BUILDPLATFORM base AS builder
# ...
FROM --platform=$BUILDPLATFORM base AS installer
# ...
FROM node:22-alpine@sha256:5539840ce9d013fa13e3b9814c9353024be7ac75aca5db6d039504a56c04ea59 AS runner

I build a linux/arm64 image in the end, but by overriding the platform for the stages, they are run natively, since the raw build doesn't need to be running on emulated ARM in my case. Only the final image.