Next.js Discord

Discord Forum

Webpack bundling issues (tiktokken)

Unanswered
Chinese Egret posted this in #help-forum
Open in Discord
Chinese EgretOP
I added a library (@effect/ai-anthropic) which pulls in tiktokken, and I am getting wasm webpack bundle errors

//next config
import "./src/env.js";

/** @type {import("next").NextConfig} */
const config = {};

export default config;



  ~/../4nswer.me ❯ pnpm build

> 4nswer.me@0.1.0 build /home/riley/programming/4nswer.me
> next build

   ▲ Next.js 15.3.2
   - Environments: .env

   Creating an optimized production build ...


Retrying 1/3...


Retrying 2/3...
Failed to compile.

./node_modules/.pnpm/tiktoken@1.0.21/node_modules/tiktoken/lite/tiktoken_bg.wasm
Module parse failed: Unexpected character '' (1:0)
The module seem to be a WebAssembly module, but module is not flagged as WebAssembly module for webpack.
BREAKING CHANGE: Since webpack 5 WebAssembly is not enabled by default and flagged as experimental feature.
You need to enable one of the WebAssembly experiments via 'experiments.asyncWebAssembly: true' (based on async modules) or 'experiments.syncWebAssembly: true' (like webpack 4, deprecated).
For files that transpile to WebAssembly, make sure to set the module type in the 'module.rules' section of the config (e. g. 'type: "webassembly/async"').
(Source code omitted for this binary file)

Import trace for requested module:
./node_modules/.pnpm/tiktoken@1.0.21/node_modules/tiktoken/lite/tiktoken_bg.wasm
./node_modules/.pnpm/tiktoken@1.0.21/node_modules/tiktoken/lite/tiktoken.js
./node_modules/.pnpm/@anthropic-ai+tokenizer@0.0.4/node_modules/@anthropic-ai/tokenizer/dist/cjs/index.js
./node_modules/.pnpm/@effect+ai-anthropic@0.6.5_@effect+ai@0.16.5_@effect+experimental@0.46.4_@effect+platfo_37ce30aee9d8a070baae66b1689224a0/node_modules/@effect/ai-anthropic/dist/esm/AnthropicTokenizer.js
./node_modules/.pnpm/@effect+ai-anthropic@0.6.5_@effect+ai@0.16.5_@effect+experimental@0.46.4_@effect+platfo_37ce30aee9d8a070baae66b1689224a0/node_modules/@effect/ai-anthropic/dist/esm/index.js
./src/types/types.ts
./src/components/MainArea.tsx
./src/app/page.tsx


> Build failed because of webpack errors
 ELIFECYCLE  Command failed with exit code 1.

1 Reply

Chinese EgretOP
after updating my next config as it suggests:
//next config:
      8  const config = {
      9    webpack: (config) => {
     10      config.experiments = {
     11        ...config.experiments,
     12        asyncWebAssembly: true,
     13      };
     14      return config;
     15    },
     16  };


  ~/../4nswer.me ❯ pnpm build

> 4nswer.me@0.1.0 build /home/riley/programming/4nswer.me
> next build

   ▲ Next.js 15.3.2
   - Environments: .env

   Creating an optimized production build ...


Retrying 1/3...


Retrying 1/3...
 ⚠ Compiled with warnings in 12.0s

./node_modules/.pnpm/tiktoken@1.0.21/node_modules/tiktoken/lite/tiktoken_bg.wasm
The generated code contains 'async/await' because this module is using "asyncWebAssembly".
However, your target environment does not appear to support 'async/await'.
As a result, the code may not run as expected or may cause runtime errors.

Import trace for requested module:
./node_modules/.pnpm/tiktoken@1.0.21/node_modules/tiktoken/lite/tiktoken_bg.wasm
./node_modules/.pnpm/tiktoken@1.0.21/node_modules/tiktoken/lite/tiktoken.js
./node_modules/.pnpm/@anthropic-ai+tokenizer@0.0.4/node_modules/@anthropic-ai/tokenizer/dist/cjs/index.js
./node_modules/.pnpm/@effect+ai-anthropic@0.6.5_@effect+ai@0.16.5_@effect+experimental@0.46.4_@effect+platfo_37ce30aee9d8a070baae66b1689224a0/node_modules/@effect/ai-anthropic/dist/esm/AnthropicTokenizer.js
./node_modules/.pnpm/@effect+ai-anthropic@0.6.5_@effect+ai@0.16.5_@effect+experimental@0.46.4_@effect+platfo_37ce30aee9d8a070baae66b1689224a0/node_modules/@effect/ai-anthropic/dist/esm/index.js
./src/types/types.ts
./src/components/MainArea.tsx
./src/app/page.tsx

 ✓ Linting and checking validity of types
   Collecting page data  ...Error: Missing tiktoken_bg.wasm
    at 5843 (.next/server/app/api/rpc/route.js:14:1100815)
    at t (.next/server/webpack-runtime.js:1:143)
    at 9062 (.next/server/app/api/rpc/route.js:14:1107254)
    at t (.next/server/webpack-runtime.js:1:143)
    at 4239 (.next/server/app/api/rpc/route.js:14:1089870)
    at t (.next/server/webpack-runtime.js:1:143)
    at a (.next/server/app/api/rpc/route.js:14:1107704)
    at <unknown> (.next/server/app/api/rpc/route.js:14:1107731)
    at t.X (.next/server/webpack-runtime.js:1:1275)
    at <unknown> (.next/server/app/api/rpc/route.js:14:1107717)

> Build error occurred
[Error: Failed to collect page data for /api/rpc] { type: 'Error' }
 ELIFECYCLE  Command failed with exit code 1.