How do I run tests in Testifly?

Click Run test on a test page, Run all tests on a feature, Run filtered tests on the features list, or the run icon of a test suite. Suites can also run on a schedule, and you can start runs through the Test Execution API.

Does Testifly fix tests by itself when my app changes?

Often, yes. When a step fails because an element changed, Testifly looks for the element on the page again, updates the test and retries, up to three attempts. If it cannot tell what changed, it marks the test Awaiting Your Input and asks you.

What is the difference between Failed and Skipped?

Failed means the test ran and a step did not give the expected result. Skipped means the test could not start, for example because the login or a prerequisite failed, a bot check blocked the browser, or the test reached an excluded URL.

What do Regressed and Resolved mean on a run?

Regressed tests failed in this run but had a different result in the previous run. Resolved tests passed in this run but had a different result in the previous run.

Runs and Results

A run executes a set of tests against your app and records what happened. Each test in a run gets a result, a video recording and a timeline of its steps. When a run ends, Testifly emails the results and can post a summary to Slack.

How do I start a run?

Where	Button	What runs
Test page	RUN TEST	That one test and its prerequisites
Feature page or quick view	RUN ALL TESTS	All runnable tests in the feature
Features list	RUN FILTERED TESTS	Every test that matches your filter and has code
Test Suites list	Run icon (▶)	All runnable tests in the suite
Test suite schedule	Automatic	The suite, at the times you set. See Test Suites and Scheduling.
Your CI/CD pipeline	API call	See Test Execution API.

Which tests count as runnable is explained in How Testifly Works.

Choosing where to run

If your project has more than one environment, a dialog asks which Environment to run against. If the environment is behind a VPN, the dialog also asks which of your Remote Runners should run the tests. Otherwise the run starts at once on Testifly’s cloud runners.

What happens during a run?

A run is an Execute Tests job. You can follow it live on the Jobs page.

Collect the tests. Testifly picks the tests you asked for and adds the prerequisites they need. A test whose prerequisite is missing or has no code is left out.
Start browsers. Tests run in parallel, each in its own browser. Prerequisites run first.
For each test:
1. Load any hooks you set for the environment, such as a page-load script or extra HTTP headers.
2. Sign in with the project’s login test, if the app needs a login.
3. Run the prerequisites.
4. Open the feature URL.
5. Run the steps one by one, while recording a video and a timeline.
Repair when possible. If a step fails, Testifly investigates. When a UI element has moved or changed, it finds the element again, updates the test and runs it again. A test gets up to three attempts.
Save the results. Testifly uploads the video, saves the timeline and sets the result of each test.
Clean up. If Run cleanup after each run is on in Settings → Misc., the cleanup test runs at the end. It is skipped while other jobs are still running in the project.
Report. Testifly sends a results email and, if Slack is connected, a Slack summary. See Notifications and Emails.

How is each test result decided?

Result	When
Passed	All steps gave the expected result.
Failed	A step did not give the expected result, or the feature URL could not be opened.
Skipped	The test could not start: the browser could not start, the login failed, a prerequisite failed, a bot check blocked the browser, or the test reached an excluded URL.
Awaiting Your Input	A step failed and Testifly could not work out whether your app changed or has a bug.

After a run, the test’s status on the test page changes to match its newest result. Tests marked Product Bug Detected or Not Applicable keep that status.

Where do I see past runs?

Click Runs in the side menu. The list shows every run with its name, start time and counts of Tests, Passed, Failed, Skipped, Scheduled, Running and Awaiting Input. Click FILTER to show only runs of a given test suite or status.

What is on a run page?

Click a run to open it.

Tabs filter the tests in the run: All, Passed, Failed, Skipped, Running, Regressed, Resolved, Scheduled and Awaiting Your Input. A tab appears only when it has tests.
- Regressed – tests that failed now but had a different result in the previous run.
- Resolved – tests that passed now but had a different result in the previous run.
Details – the Test Run ID, the Status, the Start time and the Environment.

Click a test to open its run review page.

How do I review one test in a run?

The run review page shows everything you need to understand what happened:

Video – the recording of the test in the browser. It starts where the test starts.
Steps – the steps of the test with their results.
Details – the test, its run status and the start time.
Timeline – the key moments of the run, such as login, prerequisites and each step. Click an item to jump to that moment in the video.
Error box – when the test did not pass, the explanation of what went wrong.

What does Awaiting Your Input mean?

Testifly could not decide on its own why a step failed. For example, a button it needs may no longer be on the page. It explains the problem in a red box on the run review page and asks you to choose one answer:

Your answer	What Testifly does next
The application changed. Please align the test	Sends the test back to the agent, which updates the steps and code to match your app. You get an email when it is done.
The application changed but differently	Same, but you describe what changed. The agent uses your description.
The application had a temporary glitch, please try again	Sets the test back to Ready To Run, so it runs again next time.
The application did not change, there is a bug in my application	Marks the test Product Bug Detected and the run result Failed.
The application did not change, it’s a bug in the test creation	Sends the test back to the agent together with your description of the problem.

Choose an answer, add a description if asked, and click Submit. Your answer and your name are added to the run’s error note, so your team can see who decided what.

The same box appears for tests marked Product Bug Detected, so you can tell Testifly if it was wrong.

A test that is awaiting your input is not included in suite or feature runs until you answer.