Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
66 changes: 37 additions & 29 deletions .github/CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,13 +190,13 @@ use for each.
It provides easy to use auto-documenting via docstrings.
- [Tutorials](../examples/tutorials) - Notebooks with tutorials.

- [hypex](../hypex) - The code of the HypEx library.
- [aa.py](../hypex/aa.py), [ab.py](../hypex/ab.py), [matching.py](../hypex/matching.py) and
[homogeneity.py](../hypex/homogeneity.py) - Public experiment entry points.
- [dataset](../hypex/dataset) - Dataset abstractions, roles, pandas and Spark backends.
- [comparators](../hypex/comparators) - Group comparisons and statistical tests.
- [experiments](../hypex/experiments) - Composable experiment pipelines.
- [utils](../hypex/utils) - Shared enums, errors, backend registry, logging and profiling.
- [hypex](../hypex) - The code of HypEx library.
- [ab_test](../hypex/ab_test) - The main module for A/B testing
- [algorithms](../hypex/algorithms) - Modules with different algorithms for calculation (Like Knn)
blenders and ready-made presets.
- [dataset](../hypex/dataset) - The internal interface for working with data. Also contains data generator
- [selectors](../hypex/selectors) - Modules with different pre-processing data (Filters, feature selectors)
- [utils](../hypex/utils) - Common util tools (Timer, Profiler, Logging).

## Developing HypEx

Expand Down Expand Up @@ -226,14 +226,14 @@ If you are installing from the source, you will need Python 3.8 or later.
poetry install
```

To install all dependencies and development tools:
To install all dependencies:
```bash
poetry install --all-extras --with dev
poetry install --all-extras
```

To install the base package with development tools:
```bash
poetry install --with dev
To install only specific dependency groups:
```
poetry install -E cv -E report
```

After that, there is virtual environment, where you can test and implement your own code.
Expand Down Expand Up @@ -291,18 +291,7 @@ hooks accept is a commit CI accepts. When you bump one, bump the other --
`.github/workflows/ci.yml` fails the lint job if the two pins drift apart.

### Testing

Run the focused test suite or a complete interpreter environment with tox:

```console
tox -e py311 -- tests/unit/path/to/test_file.py -q
tox -e py311
tox -e py311 -- -m "not spark"
tox -e coverage
```

The coverage environment enforces the minimum project coverage configured in
`pyproject.toml`. Tests that require Spark also require a local Java runtime.
(in progress)

## Documentation

Expand All @@ -317,14 +306,33 @@ By default, functions, that have no description will be mock from overall docume

### Building Documentation

Build the documentation from the repository root:
To build the documentation:

```console
tox -e docs
1. Clone repository to your device.

```bash
git clone git@github.com:sb-ai-lab/HypEx.git
cd HypEx
```

This runs Sphinx with warnings treated as errors and checks the reStructuredText
sources with doc8. Generated HTML files are written to `docs/_build/html`.
2. Make environment and install requirements.

```bash
poetry install -E cv -E nlp
```

3. Remove existing html files:

```bash
cd docs
poetry run make clean html
```

4. Generate HTML documentation files. The generated files will be in `docs/_build/html`.

```bash
poetry build
```

### Writing Documentation

Expand Down
117 changes: 63 additions & 54 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,6 @@ on:
push:
branches: [ master, dev/master ]
pull_request:
# Retargeting a PR emits edited rather than synchronize.
types: [ opened, synchronize, reopened, edited ]
branches: [ master, dev/master ]
workflow_dispatch:

Expand Down Expand Up @@ -36,16 +34,16 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v4

- uses: actions/setup-python@v7
- uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: pip
cache-dependency-path: poetry.lock

- name: Cache tox environments
uses: actions/cache@v6
uses: actions/cache@v4
with:
path: .tox
key: tox-lint-${{ runner.os }}-${{ hashFiles('tox.ini', 'pyproject.toml') }}
Expand All @@ -64,7 +62,7 @@ jobs:
tox_ver=$(grep -oP 'ruff == \K[0-9.]+' tox.ini | sort -u)
pc_ver=$(grep -A1 'astral-sh/ruff-pre-commit' .pre-commit-config.yaml \
| grep -oP 'rev: v\K[0-9.]+')
pp_ver=$(python -c 'import pathlib, tomllib; print(tomllib.loads(pathlib.Path("pyproject.toml").read_text())["tool"]["poetry"]["group"]["dev"]["dependencies"]["ruff"])')
pp_ver=$(grep -oP '^ruff = "\K[0-9.]+' pyproject.toml)
echo "tox=$tox_ver pre-commit=$pc_ver pyproject=$pp_ver"
if [ "$(printf '%s\n' "$tox_ver" "$pc_ver" "$pp_ver" | sort -u | wc -l)" != "1" ]; then
echo "::error::Ruff versions disagree between tox.ini, .pre-commit-config.yaml and pyproject.toml"
Expand All @@ -78,9 +76,9 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v4

- uses: actions/setup-python@v7
- uses: actions/setup-python@v5
with:
python-version: "3.11"

Expand All @@ -89,25 +87,6 @@ jobs:
# Catches a pyproject.toml edited without re-running `poetry lock`.
- run: poetry check --lock

no-spark:
name: Tests without PySpark
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v7

- uses: actions/setup-python@v7
with:
python-version: "3.11"
cache: pip
cache-dependency-path: poetry.lock

- run: python -m pip install tox

# Installs the wheel without extras and checks imports, pandas workflows,
# and the actionable errors exposed by Spark-only entry points.
- run: tox -e no-spark

# ---------------------------------------------------------------------------
# The test matrix. Narrowed from 17 jobs to 10: every supported interpreter
# on Linux, plus the oldest and newest on macOS and Windows, which is where
Expand All @@ -124,14 +103,14 @@ jobs:
name: Tests ${{ matrix.python-version }} / ${{ matrix.os }}
runs-on: ${{ matrix.os }}
timeout-minutes: 45
continue-on-error: ${{ matrix.experimental || false }}
strategy:
# One red interpreter must not hide the state of the others.
fail-fast: false
matrix:
include:
# Ubuntu 26.04 has no setup-python builds for these interpreters.
- { os: ubuntu-22.04, python-version: "3.8", spark: full }
- { os: ubuntu-22.04, python-version: "3.9", spark: full }
- { os: ubuntu-latest, python-version: "3.8", spark: full }
- { os: ubuntu-latest, python-version: "3.9", spark: full }
- { os: ubuntu-latest, python-version: "3.10", spark: full }
- { os: ubuntu-latest, python-version: "3.11", spark: full }
- { os: ubuntu-latest, python-version: "3.12", spark: full }
Expand All @@ -140,30 +119,33 @@ jobs:
- { os: macos-15-intel, python-version: "3.8", spark: smoke }
- { os: macos-latest, python-version: "3.12", spark: smoke }
- { os: windows-latest, python-version: "3.8", spark: smoke }
# PySpark 3.5 crashes its Python worker on this leg. Keep the
# non-Spark tests blocking; restore Spark coverage with PySpark 4.
- { os: windows-latest, python-version: "3.12", spark: none }
# Every Spark-backed test that ships a Python function to an executor
# crashes the Python worker on this leg ("Python worker exited
# unexpectedly"); the same tests pass on Linux and macOS 3.12 and on
# Windows 3.8. pyspark 3.5 does not officially support 3.12. Not
# reproducible locally. Make it blocking again once pyspark 4 is in.
- { os: windows-latest, python-version: "3.12", spark: smoke, experimental: true }

steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v4

- name: Set up Python ${{ matrix.python-version }}
uses: actions/setup-python@v7
uses: actions/setup-python@v5
with:
python-version: ${{ matrix.python-version }}
cache: pip
cache-dependency-path: poetry.lock

- name: Cache tox environments
uses: actions/cache@v6
uses: actions/cache@v4
with:
path: .tox
key: >-
tox-${{ runner.os }}-py${{ matrix.python-version }}-${{
hashFiles('tox.ini', 'pyproject.toml') }}

# Spark-backed tests start a local JVM.
- uses: actions/setup-java@v6
- uses: actions/setup-java@v5
with:
distribution: temurin
java-version: "17"
Expand All @@ -180,7 +162,7 @@ jobs:
run: tox -- --durations=25

- name: Run the test suite without Spark
if: matrix.spark != 'full'
if: matrix.spark == 'smoke'
run: tox -- -m "not spark" --durations=25

# Runs even when the previous step failed, so one run shows both. These
Expand All @@ -202,21 +184,17 @@ jobs:
name: Coverage (locked)
runs-on: ubuntu-latest
timeout-minutes: 30
env:
# Poetry >=2.3 defaults to marker-only installation. This lock contains
# overlapping numeric-stack variants; select one using the active extras.
POETRY_INSTALLER_RE_RESOLVE: "true"
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v4

- uses: actions/setup-python@v7
- uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: pip
cache-dependency-path: poetry.lock

# Spark-backed tests start a local JVM.
- uses: actions/setup-java@v6
- uses: actions/setup-java@v5
with:
distribution: temurin
java-version: "17"
Expand All @@ -232,21 +210,31 @@ jobs:
--cov=hypex --cov-report=term-missing --cov-report=xml

- name: Upload coverage artifact
uses: actions/upload-artifact@v7
uses: actions/upload-artifact@v4
if: always()
with:
name: coverage-xml
path: coverage.xml
if-no-files-found: warn

# Codecov is optional: without a token on a private fork it is skipped
# rather than failing the run.
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
continue-on-error: true
with:
files: coverage.xml
token: ${{ secrets.CODECOV_TOKEN }}
fail_ci_if_error: false

docs:
name: Documentation
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v4

- uses: actions/setup-python@v7
- uses: actions/setup-python@v5
with:
python-version: "3.10"
cache: pip
Expand All @@ -264,7 +252,7 @@ jobs:
# means catching it before merge rather than after.
- run: tox -e docs

- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@v4
if: always()
with:
name: docs-html
Expand All @@ -276,9 +264,9 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v4

- uses: actions/setup-python@v7
- uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: pip
Expand All @@ -290,7 +278,7 @@ jobs:
# cannot be the first time we discover the metadata is invalid.
- run: tox -e build

- uses: actions/upload-artifact@v7
- uses: actions/upload-artifact@v4
with:
name: dist
path: dist/
Expand All @@ -307,9 +295,9 @@ jobs:
# zero; tox -e typing already fails honestly.
continue-on-error: true
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v4

- uses: actions/setup-python@v7
- uses: actions/setup-python@v5
with:
python-version: "3.10"
cache: pip
Expand All @@ -318,3 +306,24 @@ jobs:
- run: python -m pip install tox

- run: tox -e typing

audit:
name: Dependency audit (advisory)
runs-on: ubuntu-latest
timeout-minutes: 20
# Advisory because a new CVE in a transitive dependency should not turn an
# unrelated pull request red. The scheduled run in security.yml is what
# gets watched.
continue-on-error: true
steps:
- uses: actions/checkout@v4

- uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: pip
cache-dependency-path: poetry.lock

- run: python -m pip install tox

- run: tox -e audit
8 changes: 3 additions & 5 deletions .github/workflows/notebooks.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,6 @@ on:
- "hypex/**"
- ".github/workflows/notebooks.yml"
pull_request:
# Retargeting a PR emits edited rather than synchronize.
types: [ opened, synchronize, reopened, edited ]
branches: [ master, dev/master ]
paths:
- "examples/tutorials/**"
Expand All @@ -34,16 +32,16 @@ jobs:
runs-on: ubuntu-latest
timeout-minutes: 90
steps:
- uses: actions/checkout@v7
- uses: actions/checkout@v4

- uses: actions/setup-python@v7
- uses: actions/setup-python@v5
with:
python-version: "3.11"
cache: pip
cache-dependency-path: poetry.lock

# Spark-backed tests start a local JVM.
- uses: actions/setup-java@v6
- uses: actions/setup-java@v5
with:
distribution: temurin
java-version: "17"
Expand Down
Loading
Loading