why dont use the timeline in a event handler instead of a useEffect ? and also you should use the useGSAP hooks https://gsap.com/resources/React#usegsap-hook-

and the thing in your code actually is that your animation is already reversed. I guess it should be something like this

const tl = useRef(gsap.timeline({ paused: true }));

useEffect(() => {
  // Create the timeline once
  tl.current.to(overlayRef.current, {
    clipPath: "polygon(0 0, 100% 0, 100% 100%, 0 100%)",
    duration: 4,
    ease: "power4.inOut",
  });

  // Clean up function
  return () => tl.current.kill();
}, []);

useEffect(() => {
  if (isMenuOpen) {
    tl.current.play();
  } else {
    tl.current.reverse();
  }
}, [isMenuOpen]);

useGSAP helps you with clean up function and other thing with react