Sitemap generation for large scale use case
Unanswered
West African Lion posted this in #help-forum
West African LionOP
I’m using Next 14 App Router to build my app, which has millions of pages. Im generating the pages using SSR while generating the path using id fetched from my headless CMS. Im using generating the page with generateStaticParams and dynamic = ‘force-dynamic’.
Meanwhile I also generate the sitemap statically using Next.js built in function sitemap.ts. It works well with a small number of records. However, with millions of paths put into sitemap. The sitemaps file grows too large for my serverless deployment (>1Gb). I have split the sitemaps with 50,000 records but the overall size is too large.
Do anyone have experience on how to work on this with minimal effort? Thanks!
PS: Im using AWS Amplify
Meanwhile I also generate the sitemap statically using Next.js built in function sitemap.ts. It works well with a small number of records. However, with millions of paths put into sitemap. The sitemaps file grows too large for my serverless deployment (>1Gb). I have split the sitemaps with 50,000 records but the overall size is too large.
Do anyone have experience on how to work on this with minimal effort? Thanks!
PS: Im using AWS Amplify
3 Replies
@West African Lion I’m using Next 14 App Router to build my app, which has millions of pages. Im generating the pages using SSR while generating the path using id fetched from my headless CMS. Im using generating the page with generateStaticParams and dynamic = ‘force-dynamic’.
Meanwhile I also generate the sitemap statically using Next.js built in function sitemap.ts. It works well with a small number of records. However, with millions of paths put into sitemap. The sitemaps file grows too large for my serverless deployment (>1Gb). I have split the sitemaps with 50,000 records but the overall size is too large.
Do anyone have experience on how to work on this with minimal effort? Thanks!
PS: Im using AWS Amplify
you can try to do what most website builder do: build for every page it's own sitemap. Like that, you can easily handle the creation of the sitemaps and if you need to update one, you can directly update and only update the correct one. Also you get rid of the storage limits 👍
West African LionOP
Hi @B33fb0n3 , thank you for the suggestion! I have created a quick outline on what my project structure looks like. In my situation, generating a sitemap for each page statically might still create large build size. Im exploring something like ssr dynamic sitemap but cannot find too much resources on that. Next-sitemap seems not an available solution
West African LionOP
Sorry for typo, the 1st Sitemap.ts should be under app directory. The 1st sitemap.ts is pointing to sitemap under color. The sitemap will point to the url for each color/[id] and split into multiple files when >5000. Eg color/00000, color/00001, etc.