I started using AI to generate Playwright test scaffolding a few months ago. Feed it a user flow, whether it's a login form or a multi-step checkout, and you get back a working spec in under a minute that runs and mostly passes on the first try.
I asked a model to write a test for a password reset flow, and it built a spec that filled in the email field, clicked submit, and checked for a success message on screen. That passed every time I ran it, but it never checked whether an email actually went out, whether the link inside it worked, or whether that same link still worked after being used once. The page reported success no matter what the backend did, so the test just checked the page's opinion of itself.
Before I touched it, the spec looked like this:
await page.getByLabel('Email').fill(testUser.email);
await page.getByRole('button', { name: 'Send reset link' }).click();
