Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 21 additions & 0 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -264,6 +264,27 @@ Exception classes are also accepted and match any subclass:
raise AssertionError()


To request a re-run from *inside* a test -- for example when generated data
happens to hit a known-bad case, similar to ``hypothesis.assume()`` -- use

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

— Comment created by Claude

The hypothesis.assume() comparison is misleading. Inside a @given test, Hypothesis treats RerunRequest as a failure: it shrinks the input, saves the failing example to its database and replays it first on the next attempt. Each rerun then hits the same input again, so the reruns cannot succeed and only add shrinking time. Please mention this, or drop the analogy.

``rerun_if``:

.. code-block:: python

from pytest_rerunfailures import RerunRequest, rerun_if

@pytest.mark.flaky(reruns=2, only_rerun=[RerunRequest])
def test_something_with_different_people():
person1, person2 = PersonFactory.build_batch(2)
rerun_if(person1.name == person2.name)
...

``rerun_if`` aborts the current attempt by raising ``RerunRequest`` if the
condition is true, and the test is re-run while it still has re-runs left
(``@pytest.mark.flaky(reruns=...)`` or ``--reruns``). Without a re-run
budget the request behaves like a regular test failure. Filtering with
``only_rerun=[RerunRequest]`` ensures only the requested condition triggers
a re-run, not other failures.

You can use ``@pytest.mark.flaky(condition)`` similarly as ``@pytest.mark.skipif(condition)``, see `pytest-mark-skipif <https://docs.pytest.org/en/6.2.x/reference.html#pytest-mark-skipif>`_

.. code-block:: python
Expand Down
4 changes: 4 additions & 0 deletions changes/58.feature.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
Add ``rerun_if()`` helper and ``RerunRequest`` exception to request a re-run
from inside a test (similar to ``hypothesis.assume()``), e.g. combined with
``@pytest.mark.flaky(reruns=N, only_rerun=[RerunRequest])`` to retry only on
the requested condition.
19 changes: 19 additions & 0 deletions src/pytest_rerunfailures.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,25 @@ def works_with_current_xdist():
"exponential backoff (delay * factor ** (attempt - 1)). defaults to 1.0, "
"i.e. a constant delay."
)


class RerunRequest(Exception):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

— Comment created by Claude

RerunRequest subclasses Exception, so any except Exception: (or pytest.raises(Exception)) between rerun_if() and the test function swallows it. If a test helper or fixture wraps the call like that, no rerun is requested and the test keeps running with the bad data it was trying to reject. pytest's own skip/fail outcomes subclass BaseException for this reason. Consider deriving from BaseException; only_rerun=[RerunRequest] still matches because of the isinstance check.

"""Raised by :func:`rerun_if` to discard an attempt and request a rerun."""


def rerun_if(condition, reason=None):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

— Comment created by Claude

Cleanup: the new class and function sit between RERUNS_DELAY_BACKOFF_FACTOR_DESC and ONLY_RERUN_DESC, splitting the block of option description constants. Moving them below the constants or near the other public helpers keeps that block together.

"""Request a rerun from inside a test when ``condition`` is true.

The current attempt is aborted and reported as a failure that can be
retried if the test still has reruns left (via ``@pytest.mark.flaky`` or
``--reruns``). Without a rerun budget it behaves like a regular test
failure. Combine with ``only_rerun=[RerunRequest]`` to rerun *only* on
this condition and not on other failures.
"""
if condition:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

— Comment created by Claude

Minor: reason or "rerun requested from test" replaces any falsy reason, so an explicit reason="" cannot be passed through. "rerun requested from test" if reason is None else reason matches the None default exactly.

raise RerunRequest(reason or "rerun requested from test")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

— Comment created by Claude

rerun_if() only raises a normal exception, so a request can still be turned into a hard failure:

  • a global or ini filter like --only-rerun ConnectionError (or only_rerun = ... in ini) does not match RerunRequest: ..., so the test fails without being rerun;
  • a broad --rerun-except regex can block it too;
  • so can a marker condition= that evaluates to false.

This contradicts "request a rerun". A more general fix: have _should_hard_fail_on_error treat RerunRequest as always rerunnable when the test has reruns left. Users would then not need to add only_rerun=[RerunRequest] themselves.



ONLY_RERUN_DESC = (
"If passed, only rerun errors matching the regex provided. "
"Pass this flag multiple times (or list one regex per line in the ini "
Expand Down
49 changes: 49 additions & 0 deletions tests/test_pytest_rerunfailures.py
Original file line number Diff line number Diff line change
Expand Up @@ -617,6 +617,55 @@ def test_pass():
assert_outcomes(result, passed=1, rerun=2)


def test_rerun_if_requests_rerun_from_inside_test(testdir):
testdir.makepyfile(
"""
import pytest
from pytest_rerunfailures import rerun_if

import py
path = py.path.local(__file__).dirpath().ensure('test.res')

@pytest.mark.flaky(reruns=1)
def test_pass():
count = int(path.read() or 0)
path.write(count + 1)
rerun_if(count == 0, 'first attempt has bad data')
"""
)
result = testdir.runpytest()
assert_outcomes(result, passed=1, rerun=1)


def test_rerun_if_only_rerun_restricts_to_requested_reruns(testdir):
testdir.makepyfile(
"""
import pytest
from pytest_rerunfailures import RerunRequest, rerun_if

@pytest.mark.flaky(reruns=2, only_rerun=[RerunRequest])
def test_fail():
rerun_if(False)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

— Comment created by Claude

This test never raises RerunRequest (rerun_if(False)), so it would also pass if RerunRequest never matched only_rerun. The main README example (only_rerun=[RerunRequest] plus rerun_if(True) on the first attempt, which should then rerun) has no test. Neither does the global --reruns path that the docs describe.

assert False
"""
)
result = testdir.runpytest()
assert_outcomes(result, passed=0, failed=1, rerun=0)


def test_rerun_if_without_rerun_budget_fails_normally(testdir):
testdir.makepyfile(
"""
from pytest_rerunfailures import rerun_if

def test_fail():
rerun_if(True)
"""
)
result = testdir.runpytest()
assert_outcomes(result, passed=0, failed=1, rerun=0)


def test_flaky_marker_with_zero_reruns_disables_rerun(testdir):
testdir.makepyfile(
f"""
Expand Down
Loading