What is a feature in Testifly?

A feature is one part of your web app that can be tested, usually one page or one area of a page. Each feature has a URL, an overview, a detailed definition, a list of UI components and a set of tests. Features can have sub-features.

What is the difference between automating a test and running it?

Automating means an AI agent turns the written steps of a test into working test code. This happens once, and again only when the test changes. Running means Testifly executes that code against your app to check that it still works. Runs happen many times - on demand, on a schedule or through the API.

What does Awaiting Your Input mean?

A test run failed, and Testifly could not decide on its own whether your app changed or has a bug. Open the run, read the explanation and choose one of the answers. Testifly then updates the test, retries it or marks a bug, based on your answer.

What does Product Bug Detected mean?

While writing or running the test, the AI agent found behaviour in your app that looks like a real bug, for example a button that does nothing. The test page explains the problem. If you disagree, you can re-automate the test.

What does Over plan limit mean?

Your project reached the test limit of its current plan, so Testifly could not automate this test. The test has no code and does not run. Choose a bigger plan in Settings → Billing, or delete tests you no longer need, then automate the test again.

How Testifly Works

Testifly tests your web app in four stages. First, AI agents discover your app and describe its features. Then they write tests in plain language. Next, they automate each test, which means they turn it into test code. Finally, Testifly runs the code and reports the results. This page explains the words Testifly uses and what each status means.

What are the building blocks of a project?

Term	What it is
Project	One web app you want to test. It has its own settings, users, features and tests.
Environment	One copy of your app, for example staging or QA. Each environment has its own base URL, test users and settings. Every project starts with Main Environment.
Feature	One testable part of your app, usually a page or an area of a page. A feature has a URL, an Overview, a detailed Definition and a list of UI components. Features can contain sub-features.
UI component	One element on a page, such as a button, a field, a menu or a table. Testifly stores a name, a description and a screenshot of each component. Test steps refer to components by name.
Test	A check of one behaviour. It has a title, a description and numbered steps. Each step has an action and an expected result.
Prerequisite	A test that must pass before another test can start, for example “Page Loaded Verification”. Testifly always runs prerequisites first.
Test code	The automated version of a test. Testifly writes it for you. You can read it on the test page.
Test suite	A named group of tests that you run together, for example Sanity, Smoke or Regression. Suites can run on a schedule.
Run	One execution of a set of tests. Each run records the result, a video and a timeline for every test.
Job	A piece of background work, such as feature discovery, automation or a run. You can follow jobs on the Jobs page.
Runner	The machine that opens a browser and runs the tests. Testifly provides cloud runners. For apps behind a VPN, you install a Remote Runner in your own network.

How does work flow through Testifly?

Discover. An agent visits your app, finds its pages and creates a feature for each one. It records the UI components and clicks on them to learn what they do.
Write tests. For each feature, the agent writes tests in plain language. Tests fall into four categories: Visibility, Functionality, Business Logic and Negative (checks that the app handles wrong input correctly). The agent also places tests into your test suites, based on each suite’s name and description.
Automate. For each test, an agent opens the page in a real browser, follows the steps one by one and writes the code for each step. It checks that the code works before it moves on.
Run. Testifly executes the code, records a video and saves the result. If a step fails because the page changed, Testifly tries to repair the test by itself.
Ask you. When Testifly cannot tell whether a failure is a change in your app or a bug, it marks the test Awaiting Your Input and asks you.

You can also skip discovery and write features and tests yourself. The later stages are the same.

What do the test statuses mean?

Every test has one status. The dashboard groups the statuses into two families.

Automation statuses

These statuses describe the test before it has working code, or why it could not get code.

Status	What it means	What you can do
Draft	The test is saved but not sent for automation. A test also returns to Draft when you change its description or steps.	Review the steps and click Automate test.
Automating	An agent is writing the code for this test now, or the test is waiting in line.	Wait. You get an email when automation finishes.
Automation Failed	The agent could not write working code, for example because a step was unclear or the process took too long.	Read the error on the test page, improve the steps and click Automate test.
Product Bug Detected	The agent found behaviour that looks like a real bug in your app.	Read the explanation. Fix the bug, or re-automate the test if you think the agent was wrong.
Not Applicable	The test cannot work as written, for reasons outside the test itself. A common cause is missing data, for example an empty list.	Add the missing data or delete the test.
Authentication Failed	The agent could not log in to your app.	Check the test user in Settings → Environments.
Faulty Prerequisite	A prerequisite of this test does not work, so this test cannot start.	Fix or re-automate the prerequisite test first.
Excluded Path	The test reached a URL that you excluded in Settings → Exclusions.	Change the test or the exclusion list.
Over plan limit	Your project reached the test limit of its current plan, so Testifly could not automate this test.	Choose a bigger plan in Settings → Billing, or delete tests you no longer need. Then click Automate test.

Run statuses

These statuses describe a test that has working code.

Status	What it means
Ready To Run	The test has code and is waiting for its next run.
Run Scheduled	The test is part of a run that is about to start.
Running	The test is running now.
Test Run Passed	The last run passed.
Test Run Failed	The last run failed. Open the run to see the video and the error.
Test Run Skipped	The test did not start, for example because the login or a prerequisite failed, or a bot check blocked the browser.
Awaiting Your Input	The last run failed and Testifly needs your answer to continue. See Runs and Results.

Run results

A whole run gets one of three results: Pass (all tests passed), Fail (all tests failed) or Partial Pass (some passed and some did not).

Which tests does Testifly run?

A test must have code to run. Tests in Draft, Automating, Automation Failed or Over plan limit have no working code, so they never run.

When you run a whole feature or a test suite, Testifly also leaves out:

Tests marked Product Bug Detected or Not Applicable. Their status stays in place so the problem stays visible until you act on it.
Tests that are Awaiting Your Input. Answer the question on the run page first.

When you pick specific tests – with Run test on a test page or Run filtered tests on the features list – every chosen test that has code runs, whatever its status.

Where do I go next?

Features – add, discover and manage features.
Tests – write, edit and automate tests.
Runs and Results – run tests and read the results.