Webpack bundling issues (tiktokken)
Unanswered
Chinese Egret posted this in #help-forum
Chinese EgretOP
I added a library (
@effect/ai-anthropic
) which pulls in tiktokken, and I am getting wasm webpack bundle errors//next config
import "./src/env.js";
/** @type {import("next").NextConfig} */
const config = {};
export default config;
~/../4nswer.me ❯ pnpm build
> 4nswer.me@0.1.0 build /home/riley/programming/4nswer.me
> next build
▲ Next.js 15.3.2
- Environments: .env
Creating an optimized production build ...
Retrying 1/3...
Retrying 2/3...
Failed to compile.
./node_modules/.pnpm/tiktoken@1.0.21/node_modules/tiktoken/lite/tiktoken_bg.wasm
Module parse failed: Unexpected character '' (1:0)
The module seem to be a WebAssembly module, but module is not flagged as WebAssembly module for webpack.
BREAKING CHANGE: Since webpack 5 WebAssembly is not enabled by default and flagged as experimental feature.
You need to enable one of the WebAssembly experiments via 'experiments.asyncWebAssembly: true' (based on async modules) or 'experiments.syncWebAssembly: true' (like webpack 4, deprecated).
For files that transpile to WebAssembly, make sure to set the module type in the 'module.rules' section of the config (e. g. 'type: "webassembly/async"').
(Source code omitted for this binary file)
Import trace for requested module:
./node_modules/.pnpm/tiktoken@1.0.21/node_modules/tiktoken/lite/tiktoken_bg.wasm
./node_modules/.pnpm/tiktoken@1.0.21/node_modules/tiktoken/lite/tiktoken.js
./node_modules/.pnpm/@anthropic-ai+tokenizer@0.0.4/node_modules/@anthropic-ai/tokenizer/dist/cjs/index.js
./node_modules/.pnpm/@effect+ai-anthropic@0.6.5_@effect+ai@0.16.5_@effect+experimental@0.46.4_@effect+platfo_37ce30aee9d8a070baae66b1689224a0/node_modules/@effect/ai-anthropic/dist/esm/AnthropicTokenizer.js
./node_modules/.pnpm/@effect+ai-anthropic@0.6.5_@effect+ai@0.16.5_@effect+experimental@0.46.4_@effect+platfo_37ce30aee9d8a070baae66b1689224a0/node_modules/@effect/ai-anthropic/dist/esm/index.js
./src/types/types.ts
./src/components/MainArea.tsx
./src/app/page.tsx
> Build failed because of webpack errors
ELIFECYCLE Command failed with exit code 1.
1 Reply
Chinese EgretOP
after updating my next config as it suggests:
//next config:
8 const config = {
9 webpack: (config) => {
10 config.experiments = {
11 ...config.experiments,
12 asyncWebAssembly: true,
13 };
14 return config;
15 },
16 };
~/../4nswer.me ❯ pnpm build
> 4nswer.me@0.1.0 build /home/riley/programming/4nswer.me
> next build
▲ Next.js 15.3.2
- Environments: .env
Creating an optimized production build ...
Retrying 1/3...
Retrying 1/3...
⚠ Compiled with warnings in 12.0s
./node_modules/.pnpm/tiktoken@1.0.21/node_modules/tiktoken/lite/tiktoken_bg.wasm
The generated code contains 'async/await' because this module is using "asyncWebAssembly".
However, your target environment does not appear to support 'async/await'.
As a result, the code may not run as expected or may cause runtime errors.
Import trace for requested module:
./node_modules/.pnpm/tiktoken@1.0.21/node_modules/tiktoken/lite/tiktoken_bg.wasm
./node_modules/.pnpm/tiktoken@1.0.21/node_modules/tiktoken/lite/tiktoken.js
./node_modules/.pnpm/@anthropic-ai+tokenizer@0.0.4/node_modules/@anthropic-ai/tokenizer/dist/cjs/index.js
./node_modules/.pnpm/@effect+ai-anthropic@0.6.5_@effect+ai@0.16.5_@effect+experimental@0.46.4_@effect+platfo_37ce30aee9d8a070baae66b1689224a0/node_modules/@effect/ai-anthropic/dist/esm/AnthropicTokenizer.js
./node_modules/.pnpm/@effect+ai-anthropic@0.6.5_@effect+ai@0.16.5_@effect+experimental@0.46.4_@effect+platfo_37ce30aee9d8a070baae66b1689224a0/node_modules/@effect/ai-anthropic/dist/esm/index.js
./src/types/types.ts
./src/components/MainArea.tsx
./src/app/page.tsx
✓ Linting and checking validity of types
Collecting page data ...Error: Missing tiktoken_bg.wasm
at 5843 (.next/server/app/api/rpc/route.js:14:1100815)
at t (.next/server/webpack-runtime.js:1:143)
at 9062 (.next/server/app/api/rpc/route.js:14:1107254)
at t (.next/server/webpack-runtime.js:1:143)
at 4239 (.next/server/app/api/rpc/route.js:14:1089870)
at t (.next/server/webpack-runtime.js:1:143)
at a (.next/server/app/api/rpc/route.js:14:1107704)
at <unknown> (.next/server/app/api/rpc/route.js:14:1107731)
at t.X (.next/server/webpack-runtime.js:1:1275)
at <unknown> (.next/server/app/api/rpc/route.js:14:1107717)
> Build error occurred
[Error: Failed to collect page data for /api/rpc] { type: 'Error' }
ELIFECYCLE Command failed with exit code 1.