Using Node.js worker threads on an API route
Unanswered
Sun bear posted this in #help-forum
Sun bearOP
I'm trying to use worker threads in my API route using the app router. Specifically, I have a piece of code that uses threads (https://www.npmjs.com/package/threads) to run some CPU intensive operations in a worker thread. Originally, my code looked like this:
This resulted in the following error:
I understood this to mean that Webpack never bundled/copied the worker code anywhere, which would make sense since it's not directly imported or referenced other than by a string.
Looking up how workers should be used in Webpack 5, I refactored the code to this:
The error I got now was slightly different:
This is promising, it's definitely being picked up by Webpack now. I can also see the file in
import { Pool, spawn, Worker } from "threads";
const pool = Pool(() => spawn(new Worker("./cryptWorker.js"), { timeout: 10 * MINUTES }), 8);This resulted in the following error:
Error: Cannot find module '<path>/.next/server/app/schema/cryptWorker.js'I understood this to mean that Webpack never bundled/copied the worker code anywhere, which would make sense since it's not directly imported or referenced other than by a string.
Looking up how workers should be used in Webpack 5, I refactored the code to this:
import { Pool, spawn } from "threads";
import { Worker } from "node:worker_threads";
const pool = Pool(() => spawn(new Worker(new URL("./cryptWorker.js", import.meta.url)), { timeout: 10 * MINUTES }), 8);The error I got now was slightly different:
Error: Cannot find module '/_next/_rsc_backend_dist_out_utils_cryptWorker_js.js'This is promising, it's definitely being picked up by Webpack now. I can also see the file in
.next/server. However, I'm not sure where to go from here. What is the /_next directory it's looking it up in? Is this an alias for .next/server or something different? Any help would be appreciated!6 Replies
Sun bearOP
The full stack trace of the latter error is:
⨯ uncaughtException: Error: Cannot find module '/_next/_rsc_backend_dist_out_utils_cryptWorker_js.js'
at Module._resolveFilename (node:internal/modules/cjs/loader:1144:15)
at Module._load (node:internal/modules/cjs/loader:985:27)
at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:135:12)
at MessagePort.<anonymous> (node:internal/main/worker_thread:186:26)
at [nodejs.internal.kHybridDispatch] (node:internal/event_target:826:20)
at exports.emitMessage (node:internal/per_context/messageport:23:28)Sun bearOP
I've found a workaround. Fortunately in my case the code using workers is in a separate package within the monorepo, so I was able to work around the issue by tweaking my Webpack config:
...which just makes it so the imported package in question (
This is a solution for my specific use case, but I'd still like to keep this open to explore options where the code using workers is bundled.
config.externals.push(({ context, request }, callback) => {
if (/^@myproject/.test(request)) {
return callback(null, `module ${request}`);
}
callback();
});...which just makes it so the imported package in question (
@myproject/xxx) is not bundled at all. Note that I couldn't use serverComponentsExternalPackages as that doesn't work for workspace packages (https://github.com/vercel/next.js/issues/43433).This is a solution for my specific use case, but I'd still like to keep this open to explore options where the code using workers is bundled.
Schneider’s Smooth-fronted Caiman
im in a similar situation but my case is worse because my solution is in typescript so I need to make sure that I load the ts file and make sure is compiled as js. my worker file depends on various workspaces within my monorepo too. I have no solution yet but I hope someone can shed some light on this
@Schneider’s Smooth-fronted Caiman im in a similar situation but my case is worse because my solution is in typescript so I need to make sure that I load the ts file and make sure is compiled as js. my worker file depends on various workspaces within my monorepo too. I have no solution yet but I hope someone can shed some light on this
African Slender-snouted Crocodile
same problem did you get any solution?
@African Slender-snouted Crocodile same problem did you get any solution?
Schneider’s Smooth-fronted Caiman
Nope. Stopped working on this
Chow Chow
hi, check out this discussion, I've been experimenting with it now and posted some new findings there https://github.com/vercel/next.js/discussions/56635