Next.js Discord

Discord Forum

Configure robots.txt based on environment

Unanswered
Black-bellied Whistling-Duck posted this in #help-forum
Open in Discord
Black-bellied Whistling-DuckOP
I want to have different robots.txt for production and staging. So I had two files robots.txt and robots.txt.dev.

I tried the below solution but it didn't work. Is there any way to do it?

async rewrites() {
    return [

      {
        source: '/robots.txt',
        destination: `/public/robots.${process.env.NODE_ENV === 'production' ? 'txt' : 'txt.dev'}`,
      }
    ];
  }


async rewrites() {
    return [

      {
        source: '/robots.txt',
        destination: `/robots.${process.env.NODE_ENV === 'production' ? 'txt' : 'txt.dev'}`,
      }
    ];
  }

5 Replies

Ray
how about using robot.ts

import { MetadataRoute } from 'next'
 
export default function robots(): MetadataRoute.Robots {
  if (process.env.NODE_ENV === 'production') 
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: '/private/',
    },
    sitemap: 'https://acme.com/sitemap.xml',
  }

  // development
  return {
    rules: {
      userAgent: '*',
      allow: '/',
      disallow: '/private/',
    },
    sitemap: 'https://acme.com/sitemap.xml',
  }
}
Black-bellied Whistling-DuckOP
I will try that.
@Ray Thanks that worked for me. But in disallow I want to pass two different paths so how I can do that?
Ray
type Robots = {
  rules:
    | {
        userAgent?: string | string[]
        allow?: string | string[]
        disallow?: string | string[]
        crawlDelay?: number
      }
    | Array<{
        userAgent: string | string[]
        allow?: string | string[]
        disallow?: string | string[]
        crawlDelay?: number
      }>
  sitemap?: string | string[]
  host?: string
}

disallow accept string or string[]
Black-bellied Whistling-DuckOP
thanks that worked