Hey everyone, I'm facing a security challenge in my Next.js app:

1. Current Setup:
- Using @edge-csrf/nextjs for CSRF protection
- CSRF token generated in middleware
- Token passed to client via headers and cookies

2. The Problem:
- Python scraper still gets 200 responses or copies requests from the network tab and tests it
- Need to prevent successful scraping attempts

3. What I've Tried:
- Implementing dynamic CSRF token generation
- Adding custom headers to detect scraping

4. What I Need:
- Robust solution to block scraping attempts
- Ensure CSRF protection works against automated tools
- Next.js-specific best practices for this scenario

Any ideas on how to strengthen our defenses? Thanks in advance! 🙏

middleware.ts

const csrfProtect = createCsrfProtect({
  cookie: {
    secure: process.env.NODE_ENV === "production",
    sameSite: "strict",
  },
});

// Next.js middleware function
export const middleware = async (request: NextRequest) => {
  const response = NextResponse.next();

  try {
    await csrfProtect(request, response);
  } catch (err) {
    if (err instanceof CsrfError)
      return new NextResponse("invalid csrf token", { status: 403 });
    throw err;
  }

  return response;
};

search/page.tsx

export default function SearcPage() {
  const csrfToken = headers().get("X-CSRF-Token") || "missing";
  return <SearchClientPage csrfToken={csrfToken} />;
}

SearchClientPage .tsx

const handleSubmit = async (event: React.FormEvent<HTMLFormElement>) => {
    event.preventDefault();
    let formData = new FormData(event.currentTarget);
    formData.set("csrf_token", csrfToken);
    formData.set("from", "search");
    formAction(formData);
  };

Response from copy request from network tab and test it with PowerShell or py

1:{"success":true,"message":"­","data":{.....data}}}