Can a developer use ChatGPT to write our tests?
Yes, and the tests are often good. Give ChatGPT or Claude a flow and it will write a clean Playwright or Cypress script in seconds.
But writing a test is one small step of QA. Someone still has to decide what to test, run the tests, keep them alive when the product changes, and work out why one failed. With a chat assistant, every one of those steps lands on a developer.
The five differences below are what that looks like in practice. The first one is about money, so we start there.
What does it cost to have developers run QA with an AI assistant?
Developer time is one of the most expensive things a startup has. The assistant is cheap. The hours around it are not.
Someone writes the prompts. Someone reviews the generated code, runs it, fixes the flaky ones, and debugs the failures. That work does not stop after the first suite, because the suite needs care every time the product moves. It turns into a quiet, permanent line in the engineering budget.
Testifly is an out-of-the-box QA solution that delivers continuous QA at a fraction of developer-hour costs. Plans start at $29 a month, with a 30-day free trial and no card. See pricing for what each plan includes.
Who decides what to test?
With an assistant, the developer does. ChatGPT can generate excellent tests from a prompt, but it can only test what it was told about. If the developer forgot the invite flow, the account settings page, or the search filter, those never get a test.
Testifly connects to the running application and discovers its real pages, flows and features on its own. The tests come from what actually exists, not from what someone remembered to describe. Read more about AI flow discovery.
This matters most for the failures nobody reports. A form that looks like it submitted but never saved the data. A discount code that silently stops applying at the last step. Nobody files a ticket for those. They only get tested if something goes looking.
Who runs the tests, and when?
A test suite written with an assistant is code, and code needs someone to run it. Someone has to remember to execute it, wire it into CI, keep the environments in order and watch the results.
Testifly runs on every deploy, across environments, in CI/CD and on a schedule. QA becomes a continuous process and not another developer task. It also keeps testing between deploys, so a problem that appears on a Tuesday afternoon, with no release, still gets found. More on runs and monitoring.
What happens when the product changes?
AI can create a test today. The real problem starts when the product changes tomorrow.
A button moves, a label changes, a step is added. The scripts break, and the developer who has the context is usually busy with something else. Then the suite is either fixed by hand or quietly ignored. Both are how test automation turns into technical debt.
Testifly detects UI changes, heals broken selectors, and adapts tests when the product behavior changes, without a developer rewriting the suite. When something is a real regression and not just UI drift, it is reported as a regression and not quietly repaired. Details on self-healing and adaptation.
What does a developer get when a test fails?
With most generated suites, “test failed” is where the report ends. The investigation is still a manual job: open the logs, reproduce it, guess.
Testifly gives you a root cause and a session video of the failure, so the team can see what actually broke and start from an answer. See failed test root cause analysis.
Why does it matter that QA is independent from the developer?
When the same workflow builds a feature and generates its tests, the tests can end up checking the implementation against its own assumptions. If the model misunderstood the requirement, it can write a test that agrees with the mistake.
Independent research points the same way. A study of ChatGPT-generated unit tests found that they still suffer from correctness issues, including compilation errors and execution failures (Yuan et al., 2023). It looked at unit tests and an earlier ChatGPT model, not end-to-end tests, but the lesson carries over: someone has to check the tests. GitHub says the same about its own assistant, and tells developers to be skeptical of code that “looks right” but does not match their intent.
Testifly looks at the running application from the outside, the way a user does, and keeps exploring and checking real flows. Your QA process no longer depends on developers remembering, describing or maintaining every test.
We wrote more about this in why letting an AI agent write your E2E tests falls short.
How do the two compare?
| Question | Testifly discovers & runs itself | ChatGPT or another assistant prompt-driven, developer-run |
|---|---|---|
| Who decides what to test? | Testifly discovers the app's real pages and flows | The developer, in a prompt |
| When do tests run? | On every deploy, across environments, in CI/CD and on a schedule | When someone runs them |
| When the product changes | Broken selectors are healed, and tests adapt | A developer rewrites the tests |
| When a test fails | Root cause and session video are attached | A developer investigates |
| Independent of the feature's author? | Yes, it tests the running app from outside | No, it works from what the developer provides |
| Cost of ownership | A fraction of developer-hour costs, from $29 a month | Developer hours for writing, running, fixing and debugging |
When is a general-purpose AI the better choice?
There are real cases where it is. If you need one script for one flow, and you have a developer with the time to own it, an assistant is fast and costs almost nothing. It is also good for exploring a new idea, drafting test cases to think with, or learning a testing framework.
And if you already use an assistant, you can connect it to Testifly through the MCP server. The two are not opposites. The assistant is good at writing. Testifly is what makes sure QA keeps happening after the chat is closed.
What should a small team do first?
Point Testifly at your app and see what it finds without writing anything. You can check your site first with the readiness checker on the homepage, no signup needed, or start the 30-day free trial.
