diff --git a/README.md b/README.md index 8b2d9d63..17efeb24 100644 --- a/README.md +++ b/README.md @@ -48,6 +48,14 @@ or any other Python package manager that consumes PyPI. + The client compresses request bodies with `gzip` by default (no extra dependencies required). To opt in to + `brotli` (better compression ratio), install the optional extra and pass `compression='brotli'`: + + ```bash + pip install "apify-client[brotli]" + # or + uv add "apify-client[brotli]" + ``` - From [conda-forge](https://anaconda.org/conda-forge/apify-client), it can be installed with [conda](https://docs.conda.io/en/latest/): diff --git a/docs/01_introduction/index.mdx b/docs/01_introduction/index.mdx index 2661b7a2..1502ae8a 100644 --- a/docs/01_introduction/index.mdx +++ b/docs/01_introduction/index.mdx @@ -46,6 +46,23 @@ The Apify client is available as the `apify-client` package [on PyPI](https://py +For better request-body compression, opt in to `brotli`, which compresses better than `gzip`, especially for large payloads. Install the optional extra: + + + + ```bash + pip install "apify-client[brotli]" + ``` + + + ```bash + conda install conda-forge::apify-client conda-forge::brotli + ``` + + + +For details, see [HTTP compression](../02_concepts/13_http_compression.mdx). + ## Quick example The following example shows how to run an Actor and retrieve its results: diff --git a/docs/02_concepts/13_http_compression.mdx b/docs/02_concepts/13_http_compression.mdx new file mode 100644 index 00000000..3073ed3c --- /dev/null +++ b/docs/02_concepts/13_http_compression.mdx @@ -0,0 +1,95 @@ +--- +id: http-compression +title: HTTP compression +description: The client compresses every request body automatically using gzip by default, with optional brotli via an explicit opt-in. +--- + +The Apify client compresses every request body before sending it to the API. It reduces the amount of data transferred over the network, resulting in faster requests and lower bandwidth usage, especially for large payloads such as Actor inputs, dataset uploads, or key-value store records. + +## How it works + +The client compresses request bodies using the compressor configured via the `compression` parameter (default `'gzip'`). The server supports both gzip and brotli and decompresses the request body transparently. + +## Configuration + +To choose the compression algorithm, pass `compression` to the client constructor: + +```python +from apify_client import ApifyClient + +# Default: gzip, no extra dependency required +client = ApifyClient(token='MY-APIFY-TOKEN') + +# Opt in to brotli (requires apify-client[brotli]) +client = ApifyClient(token='MY-APIFY-TOKEN', compression='brotli') +``` + +## Enabling brotli + +Brotli is available as an optional extra. Install it alongside the client: + +```bash +pip install "apify-client[brotli]" +# or +uv add "apify-client[brotli]" +``` + +Then pass `compression='brotli'` to the client constructor. If you request brotli without installing the extra, the client raises a clear `ImportError`. There is no silent fallback. + +## Custom quality and advanced control + +For fine-grained control over compression quality, inject an `HttpCompressor` instance directly instead of a string literal: + +```python +from apify_client import ApifyClient +from apify_client.http_compressors import BrotliHttpCompressor, GzipHttpCompressor + +# Brotli at maximum quality +client = ApifyClient(token='MY-APIFY-TOKEN', compression=BrotliHttpCompressor(quality=11)) + +# Gzip at maximum quality +client = ApifyClient(token='MY-APIFY-TOKEN', compression=GzipHttpCompressor(quality=9)) +``` + +You can also implement a fully custom compressor by subclassing `HttpCompressor`: + +```python +from apify_client import ApifyClient +from apify_client.http_compressors import HttpCompressor + + +class IdentityCompressor(HttpCompressor): + content_encoding = 'identity' + """Value sent in the `Content-Encoding` header.""" + + def compress(self, data: bytes) -> bytes: + """Compress a request body. + + Args: + data: The raw bytes to compress. + + Returns: + The compressed bytes. + """ + return data + + +client = ApifyClient(token='MY-APIFY-TOKEN', compression=IdentityCompressor()) +``` + +## Comparison + +| | Brotli | Gzip | +|-------------------------------|-----------------------------------------|---------------------------------| +| **Compression ratio** | Typically better than gzip | Good | +| **CPU cost** | Moderate, depends on quality | Low | +| **Availability** | Requires the `brotli` extra | Built-in, no extra needed | +| **`Content-Encoding` header** | `br` | `gzip` | +| **Quality range** | `0–11` | `1–9` | +| **Default quality** | `6` | `9` | +| **Enable via config** | `compression='brotli'` | `compression='gzip'` (default) | +| **Best for** | Large payloads where bandwidth matters | Minimal-dependency environments | + +:::tip +For most workloads, the bandwidth savings from brotli outweigh the CPU costs. Install the `brotli` extra and pass `compression='brotli'` unless you can't install additional packages in your environment. +::: diff --git a/pyproject.toml b/pyproject.toml index 6753ec8c..a8fb2b51 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -30,6 +30,9 @@ dependencies = [ "pydantic[email]>=2.11.0", ] +[project.optional-dependencies] +brotli = ["brotli>=1.0.9"] + [project.urls] "Apify Homepage" = "https://apify.com" "Homepage" = "https://docs.apify.com/api/client/python/" diff --git a/src/apify_client/_apify_client.py b/src/apify_client/_apify_client.py index 30137aa8..2c5595a7 100644 --- a/src/apify_client/_apify_client.py +++ b/src/apify_client/_apify_client.py @@ -72,12 +72,16 @@ WebhookDispatchCollectionClientAsync, ) from apify_client._statistics import ClientStatistics -from apify_client._utils import check_custom_headers +from apify_client._utils.http import check_custom_headers from apify_client.http_clients import HttpClient, HttpClientAsync, ImpitHttpClient, ImpitHttpClientAsync +from apify_client.http_compressors._resolve import resolve_compressor if TYPE_CHECKING: from datetime import timedelta + from apify_client.http_compressors._base import HttpCompressor + from apify_client.types import HttpCompressionAlgorithm + @docs_group('Apify API clients') class ApifyClient: @@ -122,6 +126,7 @@ def __init__( timeout_long: timedelta = DEFAULT_TIMEOUT_LONG, timeout_max: timedelta = DEFAULT_TIMEOUT_MAX, headers: dict[str, str] | None = None, + compression: HttpCompressionAlgorithm | HttpCompressor = 'gzip', ) -> None: """Initialize the Apify API client. @@ -143,6 +148,8 @@ def __init__( timeout_long: Default timeout for long-duration API operations (long-polling, streaming, ...). timeout_max: Maximum timeout cap for exponential timeout growth across retries. headers: Additional HTTP headers to include in all API requests. + compression: Compression algorithm for request bodies. Pass a string literal to select an algorithm, + or an `HttpCompressor` instance for finer-grained control. """ # We need to do this because of mocking in tests and default mutable arguments. api_url = DEFAULT_API_URL if api_url is None else api_url @@ -205,6 +212,7 @@ def __init__( self._timeout_long = timeout_long self._timeout_max = timeout_max self._headers = headers + self._http_compressor = resolve_compressor(compression) @classmethod def with_custom_http_client( @@ -270,6 +278,7 @@ def http_client(self) -> HttpClient: min_delay_between_retries=self._min_delay_between_retries, statistics=self._statistics, headers=self._headers, + http_compressor=self._http_compressor, ) return self._http_client @@ -476,6 +485,7 @@ def __init__( timeout_long: timedelta = DEFAULT_TIMEOUT_LONG, timeout_max: timedelta = DEFAULT_TIMEOUT_MAX, headers: dict[str, str] | None = None, + compression: HttpCompressionAlgorithm | HttpCompressor = 'gzip', ) -> None: """Initialize the Apify API client. @@ -497,6 +507,8 @@ def __init__( timeout_long: Default timeout for long-duration API operations (long-polling, streaming, ...). timeout_max: Maximum timeout cap for exponential timeout growth across retries. headers: Additional HTTP headers to include in all API requests. + compression: Compression algorithm for request bodies. Pass a string literal to select an algorithm, + or an `HttpCompressor` instance for finer-grained control. """ # We need to do this because of mocking in tests and default mutable arguments. api_url = DEFAULT_API_URL if api_url is None else api_url @@ -559,6 +571,7 @@ def __init__( self._timeout_long = timeout_long self._timeout_max = timeout_max self._headers = headers + self._http_compressor = resolve_compressor(compression) @classmethod def with_custom_http_client( @@ -624,6 +637,7 @@ def http_client(self) -> HttpClientAsync: min_delay_between_retries=self._min_delay_between_retries, statistics=self._statistics, headers=self._headers, + http_compressor=self._http_compressor, ) return self._http_client diff --git a/src/apify_client/_resource_clients/_resource_client.py b/src/apify_client/_resource_clients/_resource_client.py index 59060572..a2cba1e3 100644 --- a/src/apify_client/_resource_clients/_resource_client.py +++ b/src/apify_client/_resource_clients/_resource_client.py @@ -9,13 +9,9 @@ from apify_client._consts import DEFAULT_WAIT_FOR_FINISH, DEFAULT_WAIT_WHEN_JOB_NOT_EXIST from apify_client._docs import docs_group from apify_client._logging import WithLogDetailsClient -from apify_client._utils import ( - catch_not_found_for_resource_or_throw, - catch_not_found_or_throw, - response_to_dict, - to_safe_id, - to_seconds, -) +from apify_client._utils.errors import catch_not_found_for_resource_or_throw, catch_not_found_or_throw +from apify_client._utils.http import response_to_dict, to_safe_id +from apify_client._utils.time import to_seconds from apify_client.errors import ApifyApiError if TYPE_CHECKING: diff --git a/src/apify_client/_resource_clients/actor.py b/src/apify_client/_resource_clients/actor.py index 4afeb921..01655352 100644 --- a/src/apify_client/_resource_clients/actor.py +++ b/src/apify_client/_resource_clients/actor.py @@ -23,12 +23,9 @@ UpdateActorRequest, ) from apify_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync -from apify_client._utils import ( - encode_key_value_store_record_value, - encode_webhooks_to_base64, - response_to_dict, - to_seconds, -) +from apify_client._utils.encoding import encode_key_value_store_record_value, encode_webhooks_to_base64 +from apify_client._utils.http import response_to_dict +from apify_client._utils.time import to_seconds if TYPE_CHECKING: from datetime import timedelta diff --git a/src/apify_client/_resource_clients/actor_collection.py b/src/apify_client/_resource_clients/actor_collection.py index 892f4f58..f30146a0 100644 --- a/src/apify_client/_resource_clients/actor_collection.py +++ b/src/apify_client/_resource_clients/actor_collection.py @@ -15,7 +15,7 @@ ) from apify_client._pagination import get_items_iterator, get_items_iterator_async from apify_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync -from apify_client._utils import to_seconds +from apify_client._utils.time import to_seconds if TYPE_CHECKING: from collections.abc import AsyncIterator, Iterator diff --git a/src/apify_client/_resource_clients/build.py b/src/apify_client/_resource_clients/build.py index ed68fbe6..c51981a0 100644 --- a/src/apify_client/_resource_clients/build.py +++ b/src/apify_client/_resource_clients/build.py @@ -5,7 +5,7 @@ from apify_client._docs import docs_group from apify_client._models import Build, BuildResponse from apify_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync -from apify_client._utils import response_to_dict +from apify_client._utils.http import response_to_dict if TYPE_CHECKING: from datetime import timedelta diff --git a/src/apify_client/_resource_clients/dataset.py b/src/apify_client/_resource_clients/dataset.py index d5f5b430..dabc9b03 100644 --- a/src/apify_client/_resource_clients/dataset.py +++ b/src/apify_client/_resource_clients/dataset.py @@ -9,11 +9,8 @@ from apify_client._models import Dataset, DatasetResponse, DatasetStatistics, DatasetStatisticsResponse from apify_client._pagination import DEFAULT_CHUNK_SIZE, get_items_iterator, get_items_iterator_async from apify_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync -from apify_client._utils import ( - create_storage_content_signature, - response_to_dict, - response_to_list, -) +from apify_client._utils.crypto import create_storage_content_signature +from apify_client._utils.http import response_to_dict, response_to_list if TYPE_CHECKING: from collections.abc import AsyncIterator, Iterator diff --git a/src/apify_client/_resource_clients/key_value_store.py b/src/apify_client/_resource_clients/key_value_store.py index 5b83bbb8..391c0969 100644 --- a/src/apify_client/_resource_clients/key_value_store.py +++ b/src/apify_client/_resource_clients/key_value_store.py @@ -16,13 +16,10 @@ ) from apify_client._pagination import DEFAULT_CHUNK_SIZE, get_cursor_iterator, get_cursor_iterator_async from apify_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync -from apify_client._utils import ( - catch_not_found_or_throw, - create_hmac_signature, - create_storage_content_signature, - encode_key_value_store_record_value, - response_to_dict, -) +from apify_client._utils.crypto import create_hmac_signature, create_storage_content_signature +from apify_client._utils.encoding import encode_key_value_store_record_value +from apify_client._utils.errors import catch_not_found_or_throw +from apify_client._utils.http import response_to_dict from apify_client.errors import ApifyApiError, InvalidResponseBodyError if TYPE_CHECKING: diff --git a/src/apify_client/_resource_clients/log.py b/src/apify_client/_resource_clients/log.py index 723b0312..f8f8cdd5 100644 --- a/src/apify_client/_resource_clients/log.py +++ b/src/apify_client/_resource_clients/log.py @@ -5,7 +5,7 @@ from apify_client._docs import docs_group from apify_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync -from apify_client._utils import catch_not_found_for_resource_or_throw +from apify_client._utils.errors import catch_not_found_for_resource_or_throw from apify_client.errors import ApifyApiError if TYPE_CHECKING: diff --git a/src/apify_client/_resource_clients/request_queue.py b/src/apify_client/_resource_clients/request_queue.py index 0e2c8ab2..639c1835 100644 --- a/src/apify_client/_resource_clients/request_queue.py +++ b/src/apify_client/_resource_clients/request_queue.py @@ -36,7 +36,9 @@ ) from apify_client._pagination import DEFAULT_CHUNK_SIZE, get_cursor_iterator, get_cursor_iterator_async from apify_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync -from apify_client._utils import catch_not_found_or_throw, response_to_dict, to_seconds +from apify_client._utils.errors import catch_not_found_or_throw +from apify_client._utils.http import response_to_dict +from apify_client._utils.time import to_seconds from apify_client.errors import ApifyApiError if TYPE_CHECKING: diff --git a/src/apify_client/_resource_clients/run.py b/src/apify_client/_resource_clients/run.py index fa1e8977..cd006793 100644 --- a/src/apify_client/_resource_clients/run.py +++ b/src/apify_client/_resource_clients/run.py @@ -13,7 +13,9 @@ from apify_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync from apify_client._status_message_watcher import StatusMessageWatcher, StatusMessageWatcherAsync from apify_client._streamed_log import StreamedLog, StreamedLogAsync -from apify_client._utils import encode_key_value_store_record_value, response_to_dict, to_safe_id, to_seconds +from apify_client._utils.encoding import encode_key_value_store_record_value +from apify_client._utils.http import response_to_dict, to_safe_id +from apify_client._utils.time import to_seconds if TYPE_CHECKING: import logging diff --git a/src/apify_client/_resource_clients/schedule.py b/src/apify_client/_resource_clients/schedule.py index b6d2b3e2..8820b942 100644 --- a/src/apify_client/_resource_clients/schedule.py +++ b/src/apify_client/_resource_clients/schedule.py @@ -11,7 +11,7 @@ ScheduleResponse, ) from apify_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync -from apify_client._utils import response_to_dict +from apify_client._utils.http import response_to_dict if TYPE_CHECKING: from apify_client.types import Timeout diff --git a/src/apify_client/_resource_clients/task.py b/src/apify_client/_resource_clients/task.py index 0492e981..ff79e72d 100644 --- a/src/apify_client/_resource_clients/task.py +++ b/src/apify_client/_resource_clients/task.py @@ -14,7 +14,9 @@ UpdateTaskRequest, ) from apify_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync -from apify_client._utils import encode_webhooks_to_base64, response_to_dict, to_seconds +from apify_client._utils.encoding import encode_webhooks_to_base64 +from apify_client._utils.http import response_to_dict +from apify_client._utils.time import to_seconds if TYPE_CHECKING: from datetime import timedelta diff --git a/src/apify_client/_resource_clients/task_collection.py b/src/apify_client/_resource_clients/task_collection.py index 22075129..cabef51f 100644 --- a/src/apify_client/_resource_clients/task_collection.py +++ b/src/apify_client/_resource_clients/task_collection.py @@ -15,7 +15,7 @@ ) from apify_client._pagination import get_items_iterator, get_items_iterator_async from apify_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync -from apify_client._utils import to_seconds +from apify_client._utils.time import to_seconds if TYPE_CHECKING: from collections.abc import AsyncIterator, Iterator diff --git a/src/apify_client/_resource_clients/user.py b/src/apify_client/_resource_clients/user.py index 26a18acd..9d7c3754 100644 --- a/src/apify_client/_resource_clients/user.py +++ b/src/apify_client/_resource_clients/user.py @@ -16,7 +16,7 @@ UserPublicInfo, ) from apify_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync -from apify_client._utils import response_to_dict +from apify_client._utils.http import response_to_dict if TYPE_CHECKING: from apify_client.types import Timeout diff --git a/src/apify_client/_resource_clients/webhook.py b/src/apify_client/_resource_clients/webhook.py index 439ba34c..18cb44da 100644 --- a/src/apify_client/_resource_clients/webhook.py +++ b/src/apify_client/_resource_clients/webhook.py @@ -14,7 +14,7 @@ WebhookUpdate, ) from apify_client._resource_clients._resource_client import ResourceClient, ResourceClientAsync -from apify_client._utils import response_to_dict +from apify_client._utils.http import response_to_dict if TYPE_CHECKING: from apify_client._literals import WebhookEventType diff --git a/src/apify_client/_status_message_watcher.py b/src/apify_client/_status_message_watcher.py index 61febdca..a1bf1e38 100644 --- a/src/apify_client/_status_message_watcher.py +++ b/src/apify_client/_status_message_watcher.py @@ -9,7 +9,7 @@ from typing import TYPE_CHECKING, Self from apify_client._docs import docs_group -from apify_client._utils import to_seconds +from apify_client._utils.time import to_seconds if TYPE_CHECKING: import logging diff --git a/src/apify_client/_utils.py b/src/apify_client/_utils.py deleted file mode 100644 index 92ef5c63..00000000 --- a/src/apify_client/_utils.py +++ /dev/null @@ -1,313 +0,0 @@ -from __future__ import annotations - -import hashlib -import hmac -import io -import json -import string -import time -import warnings -from base64 import b64encode, urlsafe_b64encode -from functools import cache -from typing import TYPE_CHECKING, Any, Literal, TypeVar, overload - -import impit - -from apify_client._consts import OVERRIDABLE_DEFAULT_HEADERS -from apify_client._models import WebhookCreate, WebhookRepresentation -from apify_client.errors import InvalidResponseBodyError, NotFoundError - -if TYPE_CHECKING: - from datetime import timedelta - - from apify_client.errors import ApifyApiError - from apify_client.http_clients import HttpResponse - from apify_client.types import WebhooksList - -T = TypeVar('T') - -_BASE62_CHARSET = string.digits + string.ascii_letters -"""Module-level constant for base62 encoding.""" - - -@overload -def to_seconds(td: None, *, as_int: bool = ...) -> None: ... -@overload -def to_seconds(td: timedelta) -> float: ... -@overload -def to_seconds(td: timedelta, *, as_int: Literal[True]) -> int: ... -@overload -def to_seconds(td: timedelta, *, as_int: Literal[False]) -> float: ... - - -def to_seconds(td: timedelta | None, *, as_int: bool = False) -> float | int | None: - """Convert timedelta to seconds. - - Args: - td: The timedelta to convert, or None. - as_int: If True, round and return as int. Defaults to False. - - Returns: - The total seconds as a float (or int if as_int=True), or None if input is None. - """ - if td is None: - return None - seconds = td.total_seconds() - return int(seconds) if as_int else seconds - - -def catch_not_found_or_throw(exc: ApifyApiError) -> None: - """Suppress 404 Not Found errors and re-raise all other API errors. - - Args: - exc: The API error to check. - - Raises: - ApifyApiError: If the error is not a 404 Not Found error. - """ - if not isinstance(exc, NotFoundError): - raise exc - - -def catch_not_found_for_resource_or_throw(exc: ApifyApiError, resource_id: str | None) -> None: - """Like `catch_not_found_or_throw`, but only suppress 404s when the client targets a specific resource by ID. - - For chained clients without a `resource_id` (e.g. `run.dataset()`, `run.log()`), a 404 could mean either the - parent or the default sub-resource is missing — the API body cannot disambiguate — so the error propagates - rather than being swallowed. - """ - if resource_id is None: - raise exc - catch_not_found_or_throw(exc) - - -def encode_key_value_store_record_value(value: Any, *, content_type: str | None = None) -> tuple[Any, str]: - """Encode a value for storage in a key-value store record. - - Args: - value: The value to encode (can be dict, str, bytes, or file-like object). - content_type: The content type; if None, it's inferred from the value type. - - Returns: - A tuple of (encoded_value, content_type). - """ - if not content_type: - if isinstance(value, (bytes, bytearray, io.IOBase)): - content_type = 'application/octet-stream' - elif isinstance(value, str): - content_type = 'text/plain; charset=utf-8' - else: - content_type = 'application/json; charset=utf-8' - - if ( - 'application/json' in content_type - and not isinstance(value, (bytes, bytearray, io.IOBase)) - and not isinstance(value, str) - ): - # Don't use indentation to reduce size. - value = json.dumps( - value, - ensure_ascii=False, - allow_nan=False, - default=str, - ).encode('utf-8') - - return (value, content_type) - - -def is_retryable_error(exc: Exception) -> bool: - """Check if the given error is retryable. - - All `impit.HTTPError` subclasses are considered retryable because they represent transport-level failures - (network issues, timeouts, protocol errors, body decoding errors) that are typically transient. HTTP status - code errors are handled separately in `_make_request` based on the response status code, not here. - """ - return isinstance( - exc, - ( - InvalidResponseBodyError, - impit.HTTPError, - ), - ) - - -def to_safe_id(id: str) -> str: - """Convert a resource ID to URL-safe format by replacing forward slashes with tildes. - - Args: - id: The resource identifier in format `resource_id` or `username/resource_id`. - - Returns: - The resource identifier with `/` characters replaced by `~`. - """ - return id.replace('/', '~') - - -def response_to_dict(response: HttpResponse) -> dict: - """Parse the API response as a dictionary and validate its type. - - Args: - response: The HTTP response object from the API. - - Returns: - The parsed response as a dictionary. - - Raises: - ValueError: If the response is not a dictionary. - """ - data = response.json() - - if isinstance(data, dict): - return data - - raise ValueError(f'The response is not a dictionary. Got: {type(data).__name__}') - - -def response_to_list(response: HttpResponse) -> list: - """Parse the API response as a list and validate its type. - - Args: - response: The HTTP response object from the API. - - Returns: - The parsed response as a list. - - Raises: - ValueError: If the response is not a list. - """ - data = response.json() - - if isinstance(data, list): - return data - - if isinstance(data, dict): - return [data] - - raise ValueError(f'The response is not a list. Got: {type(data).__name__}') - - -def encode_base62(num: int) -> str: - """Encode an integer to a base62 string. - - Args: - num: The number to encode. - - Returns: - The base62-encoded string. - """ - if num == 0: - return _BASE62_CHARSET[0] - - # Use list to build result for O(n) complexity instead of O(n^2) string concatenation. - parts = [] - while num > 0: - num, remainder = divmod(num, 62) - parts.append(_BASE62_CHARSET[remainder]) - - # Reverse and join once at the end. - return ''.join(reversed(parts)) - - -def create_hmac_signature(secret_key: str, message: str) -> str: - """Generate an HMAC-SHA256 signature and encode it using base62. - - The HMAC signature is truncated to 30 characters and then encoded in base62 to reduce the signature length. - - Args: - secret_key: The secret key used for signing. - message: The message to be signed. - - Returns: - The base62-encoded signature. - """ - signature = hmac.new(secret_key.encode('utf-8'), message.encode('utf-8'), hashlib.sha256).hexdigest()[:30] - - decimal_signature = int(signature, 16) - - return encode_base62(decimal_signature) - - -def create_storage_content_signature( - resource_id: str, - url_signing_secret_key: str, - *, - expires_in: timedelta | None = None, - version: int = 0, -) -> str: - """Create a secure signature for a storage resource like a dataset or key-value store. - - This signature is used to generate a signed URL for authenticated access, which can be expiring or permanent. - The signature is created using HMAC with the provided secret key and includes the resource ID, expiration time, - and version. - - Args: - resource_id: The unique identifier of the storage resource. - url_signing_secret_key: The secret key for signing the URL. - expires_in: Optional expiration duration; if None, the signature never expires. - version: The signature version number (default: 0). - - Returns: - The base64url-encoded signature string. - """ - expires_at = int(time.time() * 1000) + int(to_seconds(expires_in) * 1000) if expires_in is not None else 0 - - message_to_sign = f'{version}.{expires_at}.{resource_id}' - hmac_sig = create_hmac_signature(url_signing_secret_key, message_to_sign) - - base64url_encoded_payload = urlsafe_b64encode(f'{version}.{expires_at}.{hmac_sig}'.encode()) - return base64url_encoded_payload.decode('utf-8') - - -def check_custom_headers(class_name: str, headers: dict[str, str]) -> None: - """Warn if custom headers override important default headers.""" - overwrite_headers = [key for key in headers if key.title() in OVERRIDABLE_DEFAULT_HEADERS] - - if overwrite_headers: - warnings.warn( - f'{", ".join(overwrite_headers)} headers of {class_name} was overridden with an ' - 'explicit value. A wrong header value can lead to API errors, it is recommended to use the default ' - f'value for following headers: {", ".join(OVERRIDABLE_DEFAULT_HEADERS)}.', - category=UserWarning, - stacklevel=3, - ) - - -@cache -def _webhook_representation_keys() -> frozenset[str]: - """Return all field names and aliases declared on `WebhookRepresentation`.""" - keys = set[str]() - for name, info in WebhookRepresentation.model_fields.items(): - keys.add(name) - if info.alias is not None: - keys.add(info.alias) - return frozenset(keys) - - -def encode_webhooks_to_base64(webhooks: WebhooksList | None) -> str | None: - """Encode a list of ad-hoc webhooks to a base64 string for the `webhooks` query parameter. - - Returns `None` for `None` or an empty list, so the query parameter is omitted. - - See `WebhooksList` for the accepted shapes. `WebhookRepresentation` instances are used as-is. `WebhookCreate` - instances and dict shapes are projected onto the fields `WebhookRepresentation` declares, dropping anything else - (e.g. persistent-only fields like `condition`). Filtering by the declared field names and aliases means new - ad-hoc fields added to `WebhookRepresentation` flow through automatically, without touching this function. - """ - if not webhooks: - return None - - representations = list[WebhookRepresentation]() - allowed = _webhook_representation_keys() - - for webhook in webhooks: - if isinstance(webhook, WebhookRepresentation): - representations.append(webhook) - continue - - data = webhook.model_dump(by_alias=True) if isinstance(webhook, WebhookCreate) else dict(webhook) - filtered = {key: value for key, value in data.items() if key in allowed} - representations.append(WebhookRepresentation.model_validate(filtered)) - - data = [r.model_dump(by_alias=True, exclude_none=True) for r in representations] - json_string = json.dumps(data).encode(encoding='utf-8') - return b64encode(json_string).decode(encoding='ascii') diff --git a/src/apify_client/_utils/__init__.py b/src/apify_client/_utils/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/src/apify_client/_utils/crypto.py b/src/apify_client/_utils/crypto.py new file mode 100644 index 00000000..5f09fa48 --- /dev/null +++ b/src/apify_client/_utils/crypto.py @@ -0,0 +1,87 @@ +from __future__ import annotations + +import hashlib +import hmac +import string +import time +from base64 import urlsafe_b64encode +from typing import TYPE_CHECKING + +from apify_client._utils.time import to_seconds + +if TYPE_CHECKING: + from datetime import timedelta + +_BASE62_CHARSET = string.digits + string.ascii_letters + + +def encode_base62(num: int) -> str: + """Encode an integer to a base62 string. + + Args: + num: The number to encode. + + Returns: + The base62-encoded string. + """ + if num == 0: + return _BASE62_CHARSET[0] + + # Use list to build result for O(n) complexity instead of O(n^2) string concatenation. + parts = [] + while num > 0: + num, remainder = divmod(num, 62) + parts.append(_BASE62_CHARSET[remainder]) + + # Reverse and join once at the end. + return ''.join(reversed(parts)) + + +def create_hmac_signature(secret_key: str, message: str) -> str: + """Generate an HMAC-SHA256 signature and encode it using base62. + + The HMAC signature is truncated to 30 characters and then encoded in base62 to reduce the signature length. + + Args: + secret_key: The secret key used for signing. + message: The message to be signed. + + Returns: + The base62-encoded signature. + """ + signature = hmac.new(secret_key.encode('utf-8'), message.encode('utf-8'), hashlib.sha256).hexdigest()[:30] + + decimal_signature = int(signature, 16) + + return encode_base62(decimal_signature) + + +def create_storage_content_signature( + resource_id: str, + url_signing_secret_key: str, + *, + expires_in: timedelta | None = None, + version: int = 0, +) -> str: + """Create a secure signature for a storage resource like a dataset or key-value store. + + This signature is used to generate a signed URL for authenticated access, which can be expiring or permanent. + The signature is created using HMAC with the provided secret key and includes the resource ID, expiration time, + and version. + + Args: + resource_id: The unique identifier of the storage resource. + url_signing_secret_key: The secret key for signing the URL. + expires_in: Optional expiration duration; if None, the signature never expires. + version: The signature version number (default: 0). + + Returns: + The base64url-encoded signature string. + """ + expires_at = int(time.time() * 1000) + int(to_seconds(expires_in) * 1000) if expires_in is not None else 0 + + message_to_sign = f'{version}.{expires_at}.{resource_id}' + hmac_sig = create_hmac_signature(url_signing_secret_key, message_to_sign) + + base64url_encoded_payload = urlsafe_b64encode(f'{version}.{expires_at}.{hmac_sig}'.encode()) + return base64url_encoded_payload.decode('utf-8') diff --git a/src/apify_client/_utils/encoding.py b/src/apify_client/_utils/encoding.py new file mode 100644 index 00000000..b4adf385 --- /dev/null +++ b/src/apify_client/_utils/encoding.py @@ -0,0 +1,87 @@ +from __future__ import annotations + +import io +import json +from base64 import b64encode +from functools import cache +from typing import TYPE_CHECKING, Any + +from apify_client._models import WebhookCreate, WebhookRepresentation + +if TYPE_CHECKING: + from apify_client.types import WebhooksList + + +def encode_key_value_store_record_value(value: Any, *, content_type: str | None = None) -> tuple[Any, str]: + """Encode a value for storage in a key-value store record. + + Args: + value: The value to encode (can be dict, str, bytes, or file-like object). + content_type: The content type; if None, it's inferred from the value type. + + Returns: + A tuple of (encoded_value, content_type). + """ + if not content_type: + if isinstance(value, (bytes, bytearray, io.IOBase)): + content_type = 'application/octet-stream' + elif isinstance(value, str): + content_type = 'text/plain; charset=utf-8' + else: + content_type = 'application/json; charset=utf-8' + + if ( + 'application/json' in content_type + and not isinstance(value, (bytes, bytearray, io.IOBase)) + and not isinstance(value, str) + ): + # Don't use indentation to reduce size. + value = json.dumps( + value, + ensure_ascii=False, + allow_nan=False, + default=str, + ).encode('utf-8') + + return (value, content_type) + + +@cache +def _webhook_representation_keys() -> frozenset[str]: + """Return all field names and aliases declared on `WebhookRepresentation`.""" + keys = set[str]() + for name, info in WebhookRepresentation.model_fields.items(): + keys.add(name) + if info.alias is not None: + keys.add(info.alias) + return frozenset(keys) + + +def encode_webhooks_to_base64(webhooks: WebhooksList | None) -> str | None: + """Encode a list of ad-hoc webhooks to a base64 string for the `webhooks` query parameter. + + Returns `None` for `None` or an empty list, so the query parameter is omitted. + + See `WebhooksList` for the accepted shapes. `WebhookRepresentation` instances are used as-is. `WebhookCreate` + instances and dict shapes are projected onto the fields `WebhookRepresentation` declares, dropping anything else + (e.g. persistent-only fields like `condition`). Filtering by the declared field names and aliases means new + ad-hoc fields added to `WebhookRepresentation` flow through automatically, without touching this function. + """ + if not webhooks: + return None + + representations = list[WebhookRepresentation]() + allowed = _webhook_representation_keys() + + for webhook in webhooks: + if isinstance(webhook, WebhookRepresentation): + representations.append(webhook) + continue + + data = webhook.model_dump(by_alias=True) if isinstance(webhook, WebhookCreate) else dict(webhook) + filtered = {key: value for key, value in data.items() if key in allowed} + representations.append(WebhookRepresentation.model_validate(filtered)) + + data = [r.model_dump(by_alias=True, exclude_none=True) for r in representations] + json_string = json.dumps(data).encode(encoding='utf-8') + return b64encode(json_string).decode(encoding='ascii') diff --git a/src/apify_client/_utils/errors.py b/src/apify_client/_utils/errors.py new file mode 100644 index 00000000..989d3966 --- /dev/null +++ b/src/apify_client/_utils/errors.py @@ -0,0 +1,51 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +import impit + +from apify_client.errors import InvalidResponseBodyError, NotFoundError + +if TYPE_CHECKING: + from apify_client.errors import ApifyApiError + + +def catch_not_found_or_throw(exc: ApifyApiError) -> None: + """Suppress 404 Not Found errors and re-raise all other API errors. + + Args: + exc: The API error to check. + + Raises: + ApifyApiError: If the error is not a 404 Not Found error. + """ + if not isinstance(exc, NotFoundError): + raise exc + + +def catch_not_found_for_resource_or_throw(exc: ApifyApiError, resource_id: str | None) -> None: + """Like `catch_not_found_or_throw`, but only suppress 404s when the client targets a specific resource by ID. + + For chained clients without a `resource_id` (e.g. `run.dataset()`, `run.log()`), a 404 could mean either the + parent or the default sub-resource is missing — the API body cannot disambiguate — so the error propagates + rather than being swallowed. + """ + if resource_id is None: + raise exc + catch_not_found_or_throw(exc) + + +def is_retryable_error(exc: Exception) -> bool: + """Check if the given error is retryable. + + All `impit.HTTPError` subclasses are considered retryable because they represent transport-level failures + (network issues, timeouts, protocol errors, body decoding errors) that are typically transient. HTTP status + code errors are handled separately in `_make_request` based on the response status code, not here. + """ + return isinstance( + exc, + ( + InvalidResponseBodyError, + impit.HTTPError, + ), + ) diff --git a/src/apify_client/_utils/http.py b/src/apify_client/_utils/http.py new file mode 100644 index 00000000..448746bd --- /dev/null +++ b/src/apify_client/_utils/http.py @@ -0,0 +1,78 @@ +from __future__ import annotations + +import warnings +from typing import TYPE_CHECKING + +from apify_client._consts import OVERRIDABLE_DEFAULT_HEADERS + +if TYPE_CHECKING: + from apify_client.http_clients import HttpResponse + + +def to_safe_id(id: str) -> str: + """Convert a resource ID to URL-safe format by replacing forward slashes with tildes. + + Args: + id: The resource identifier in format `resource_id` or `username/resource_id`. + + Returns: + The resource identifier with `/` characters replaced by `~`. + """ + return id.replace('/', '~') + + +def response_to_dict(response: HttpResponse) -> dict: + """Parse the API response as a dictionary and validate its type. + + Args: + response: The HTTP response object from the API. + + Returns: + The parsed response as a dictionary. + + Raises: + ValueError: If the response is not a dictionary. + """ + data = response.json() + + if isinstance(data, dict): + return data + + raise ValueError(f'The response is not a dictionary. Got: {type(data).__name__}') + + +def response_to_list(response: HttpResponse) -> list: + """Parse the API response as a list and validate its type. + + Args: + response: The HTTP response object from the API. + + Returns: + The parsed response as a list. + + Raises: + ValueError: If the response is not a list. + """ + data = response.json() + + if isinstance(data, list): + return data + + if isinstance(data, dict): + return [data] + + raise ValueError(f'The response is not a list. Got: {type(data).__name__}') + + +def check_custom_headers(class_name: str, headers: dict[str, str]) -> None: + """Warn if custom headers override important default headers.""" + overwrite_headers = [key for key in headers if key.title() in OVERRIDABLE_DEFAULT_HEADERS] + + if overwrite_headers: + warnings.warn( + f'{", ".join(overwrite_headers)} headers of {class_name} was overridden with an ' + 'explicit value. A wrong header value can lead to API errors, it is recommended to use the default ' + f'value for following headers: {", ".join(OVERRIDABLE_DEFAULT_HEADERS)}.', + category=UserWarning, + stacklevel=3, + ) diff --git a/src/apify_client/_utils/time.py b/src/apify_client/_utils/time.py new file mode 100644 index 00000000..4e8c761b --- /dev/null +++ b/src/apify_client/_utils/time.py @@ -0,0 +1,32 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING, Literal, overload + +if TYPE_CHECKING: + from datetime import timedelta + + +@overload +def to_seconds(td: None, *, as_int: bool = ...) -> None: ... +@overload +def to_seconds(td: timedelta) -> float: ... +@overload +def to_seconds(td: timedelta, *, as_int: Literal[True]) -> int: ... +@overload +def to_seconds(td: timedelta, *, as_int: Literal[False]) -> float: ... + + +def to_seconds(td: timedelta | None, *, as_int: bool = False) -> float | int | None: + """Convert timedelta to seconds. + + Args: + td: The timedelta to convert, or None. + as_int: If True, round and return as int. Defaults to False. + + Returns: + The total seconds as a float (or int if as_int=True), or None if input is None. + """ + if td is None: + return None + seconds = td.total_seconds() + return int(seconds) if as_int else seconds diff --git a/src/apify_client/_utils/try_import.py b/src/apify_client/_utils/try_import.py new file mode 100644 index 00000000..450e63cf --- /dev/null +++ b/src/apify_client/_utils/try_import.py @@ -0,0 +1,49 @@ +from __future__ import annotations + +import sys +from contextlib import contextmanager +from dataclasses import dataclass +from types import ModuleType +from typing import TYPE_CHECKING + +if TYPE_CHECKING: + from collections.abc import Iterator + from typing import Any + + +@contextmanager +def try_import(module_name: str, *symbol_names: str) -> Iterator[None]: + """Context manager to attempt importing symbols into a module. + + If an `ImportError` is raised during the import, the symbol will be replaced with a `FailedImport` object. + """ + try: + yield + except ImportError as e: + for symbol_name in symbol_names: + setattr(sys.modules[module_name], symbol_name, FailedImport(e.args[0])) + + +def install_import_hook(module_name: str) -> None: + """Install an import hook for a specified module.""" + sys.modules[module_name].__class__ = ImportWrapper + + +@dataclass +class FailedImport: + """Represent a placeholder for a failed import.""" + + message: str + """The error message associated with the failed import.""" + + +class ImportWrapper(ModuleType): + """A wrapper class for modules to handle attribute access for failed imports.""" + + def __getattribute__(self, name: str) -> Any: + result = super().__getattribute__(name) + + if isinstance(result, FailedImport): + raise ImportError(result.message) # noqa: TRY004 + + return result diff --git a/src/apify_client/http_clients/_base.py b/src/apify_client/http_clients/_base.py index 6f97f16e..2fd2aeea 100644 --- a/src/apify_client/http_clients/_base.py +++ b/src/apify_client/http_clients/_base.py @@ -1,6 +1,5 @@ from __future__ import annotations -import gzip import json as jsonlib import os import sys @@ -20,11 +19,13 @@ ) from apify_client._docs import docs_group from apify_client._statistics import ClientStatistics -from apify_client._utils import to_seconds +from apify_client._utils.time import to_seconds +from apify_client.http_compressors._gzip import GzipHttpCompressor if TYPE_CHECKING: from collections.abc import AsyncIterator, Iterator, Mapping + from apify_client.http_compressors._base import HttpCompressor from apify_client.types import JsonSerializable, Timeout @@ -100,6 +101,7 @@ def __init__( min_delay_between_retries: timedelta = DEFAULT_MIN_DELAY_BETWEEN_RETRIES, statistics: ClientStatistics | None = None, headers: dict[str, str] | None = None, + http_compressor: HttpCompressor | None = None, ) -> None: """Initialize the HTTP client base. @@ -113,7 +115,9 @@ def __init__( min_delay_between_retries: Minimum delay between retries. statistics: Statistics tracker for API calls. Created automatically if not provided. headers: Additional HTTP headers to include in all requests. + http_compressor: Compressor used to compress request bodies. Defaults to `GzipHttpCompressor`. """ + self._http_compressor = http_compressor if http_compressor is not None else GzipHttpCompressor() self._timeout_short = timeout_short self._timeout_medium = timeout_medium self._timeout_long = timeout_long @@ -203,13 +207,13 @@ def _prepare_request_call( data: str | bytes | bytearray | None = None, json: JsonSerializable | None = None, ) -> tuple[dict[str, str], dict[str, Any] | None, bytes | None]: - """Prepare headers, params, and body for an HTTP request. Serializes JSON and applies gzip compression.""" + """Prepare headers, params, and body for an HTTP request. Serializes JSON and compresses the body.""" if json is not None and data is not None: raise ValueError('Cannot pass both "json" and "data" parameters at the same time!') headers = dict(headers) if headers else {} - # Dump JSON data to string so it can be gzipped. + # Dump JSON data to string so it can be compressed. if json is not None: data = jsonlib.dumps(json, ensure_ascii=False, allow_nan=False, default=str).encode('utf-8') headers['Content-Type'] = 'application/json' @@ -217,8 +221,10 @@ def _prepare_request_call( if isinstance(data, (str, bytes, bytearray)): if isinstance(data, str): data = data.encode('utf-8') - data = gzip.compress(data) - headers['Content-Encoding'] = 'gzip' + elif isinstance(data, bytearray): + data = bytes(data) + data = self._http_compressor.compress(data) + headers['Content-Encoding'] = self._http_compressor.content_encoding return (headers, self._parse_params(params), data) diff --git a/src/apify_client/http_clients/_impit.py b/src/apify_client/http_clients/_impit.py index c137dde7..02e9a3d4 100644 --- a/src/apify_client/http_clients/_impit.py +++ b/src/apify_client/http_clients/_impit.py @@ -20,7 +20,7 @@ ) from apify_client._docs import docs_group from apify_client._logging import log_context, logger_name -from apify_client._utils import to_seconds +from apify_client._utils.time import to_seconds from apify_client.errors import ApifyApiError, InvalidResponseBodyError from apify_client.http_clients._base import HttpClient, HttpClientAsync @@ -29,6 +29,7 @@ from apify_client._statistics import ClientStatistics from apify_client.http_clients._base import HttpResponse + from apify_client.http_compressors._base import HttpCompressor from apify_client.types import JsonSerializable, Timeout T = TypeVar('T') @@ -73,6 +74,7 @@ def __init__( min_delay_between_retries: timedelta = DEFAULT_MIN_DELAY_BETWEEN_RETRIES, statistics: ClientStatistics | None = None, headers: dict[str, str] | None = None, + http_compressor: HttpCompressor | None = None, ) -> None: """Initialize the Impit-based synchronous HTTP client. @@ -86,6 +88,7 @@ def __init__( min_delay_between_retries: Minimum delay between retries (increases exponentially with each attempt). statistics: Statistics tracker for API calls. Created automatically if not provided. headers: Additional HTTP headers to include in all requests. + http_compressor: Compressor used to compress request bodies. Defaults to `GzipHttpCompressor`. """ super().__init__( token=token, @@ -97,6 +100,7 @@ def __init__( min_delay_between_retries=min_delay_between_retries, statistics=statistics, headers=headers, + http_compressor=http_compressor, ) self._impit_client = impit.Client( @@ -320,6 +324,7 @@ def __init__( min_delay_between_retries: timedelta = DEFAULT_MIN_DELAY_BETWEEN_RETRIES, statistics: ClientStatistics | None = None, headers: dict[str, str] | None = None, + http_compressor: HttpCompressor | None = None, ) -> None: """Initialize the Impit-based asynchronous HTTP client. @@ -333,6 +338,7 @@ def __init__( min_delay_between_retries: Minimum delay between retries (increases exponentially with each attempt). statistics: Statistics tracker for API calls. Created automatically if not provided. headers: Additional HTTP headers to include in all requests. + http_compressor: Compressor used to compress request bodies. Defaults to `GzipHttpCompressor`. """ super().__init__( token=token, @@ -344,6 +350,7 @@ def __init__( min_delay_between_retries=min_delay_between_retries, statistics=statistics, headers=headers, + http_compressor=http_compressor, ) self._impit_async_client = impit.AsyncClient( diff --git a/src/apify_client/http_compressors/__init__.py b/src/apify_client/http_compressors/__init__.py new file mode 100644 index 00000000..bd0edf5d --- /dev/null +++ b/src/apify_client/http_compressors/__init__.py @@ -0,0 +1,15 @@ +from apify_client._utils.try_import import install_import_hook as _install_import_hook +from apify_client._utils.try_import import try_import as _try_import + +# These imports have only mandatory dependencies, so they are imported directly. +from apify_client.http_compressors._base import HttpCompressor +from apify_client.http_compressors._gzip import GzipHttpCompressor + +_install_import_hook(__name__) + +# `brotli` is an optional extra, so it's wrapped in try_import. Accessing `BrotliHttpCompressor` +# without the extra installed raises a clear ImportError instead of failing at package import time. +with _try_import(__name__, 'BrotliHttpCompressor'): + from apify_client.http_compressors._brotli import BrotliHttpCompressor + +__all__ = ['BrotliHttpCompressor', 'GzipHttpCompressor', 'HttpCompressor'] diff --git a/src/apify_client/http_compressors/_base.py b/src/apify_client/http_compressors/_base.py new file mode 100644 index 00000000..60d3f774 --- /dev/null +++ b/src/apify_client/http_compressors/_base.py @@ -0,0 +1,25 @@ +from __future__ import annotations + +from abc import ABC, abstractmethod + + +class HttpCompressor(ABC): + """Strategy for compressing HTTP request bodies. + + Extend this class to create a custom compressor. Set `content_encoding` to the value + that should be sent in the `Content-Encoding` header and implement `compress`. + """ + + content_encoding: str + """Value sent in the `Content-Encoding` header, for example `gzip` or `br`.""" + + @abstractmethod + def compress(self, data: bytes) -> bytes: + """Compress a request body. + + Args: + data: The raw bytes to compress. + + Returns: + The compressed bytes. + """ diff --git a/src/apify_client/http_compressors/_brotli.py b/src/apify_client/http_compressors/_brotli.py new file mode 100644 index 00000000..fcd1e78c --- /dev/null +++ b/src/apify_client/http_compressors/_brotli.py @@ -0,0 +1,38 @@ +from __future__ import annotations + +import brotli + +from apify_client.http_compressors._base import HttpCompressor + + +class BrotliHttpCompressor(HttpCompressor): + """Compresses request bodies using brotli. + + Requires the `brotli` extra: `pip install "apify-client[brotli]"`. + """ + + content_encoding = 'br' + + _min_quality = 0 + """Lowest valid quality (fastest, least compression).""" + + _max_quality = 11 + """Highest valid quality (slowest, best compression).""" + + def __init__(self, *, quality: int = 6) -> None: + """Initialize the brotli compressor. + + Args: + quality: Compression level, from the fastest to the best compression. + + Raises: + ValueError: If `quality` is out of the valid range. + """ + if not self._min_quality <= quality <= self._max_quality: + raise ValueError( + f'brotli quality must be between {self._min_quality} and {self._max_quality}, got {quality}.' + ) + self._quality = quality + + def compress(self, data: bytes) -> bytes: + return brotli.compress(data, quality=self._quality) diff --git a/src/apify_client/http_compressors/_gzip.py b/src/apify_client/http_compressors/_gzip.py new file mode 100644 index 00000000..d4a3c6f4 --- /dev/null +++ b/src/apify_client/http_compressors/_gzip.py @@ -0,0 +1,38 @@ +from __future__ import annotations + +import gzip + +from apify_client.http_compressors._base import HttpCompressor + + +class GzipHttpCompressor(HttpCompressor): + """Compresses request bodies using gzip. + + Uses the standard library `gzip` module. No extra dependencies required. + """ + + content_encoding = 'gzip' + + _min_quality = 1 + """Lowest valid quality (fastest, least compression).""" + + _max_quality = 9 + """Highest valid quality (slowest, best compression).""" + + def __init__(self, *, quality: int = _max_quality) -> None: + """Initialize the gzip compressor. + + Args: + quality: Compression level, from the fastest to the best compression. + + Raises: + ValueError: If `quality` is out of the valid range. + """ + if not self._min_quality <= quality <= self._max_quality: + raise ValueError( + f'gzip quality must be between {self._min_quality} and {self._max_quality}, got {quality}.' + ) + self._quality = quality + + def compress(self, data: bytes) -> bytes: + return gzip.compress(data, compresslevel=self._quality) diff --git a/src/apify_client/http_compressors/_resolve.py b/src/apify_client/http_compressors/_resolve.py new file mode 100644 index 00000000..ccd5d77c --- /dev/null +++ b/src/apify_client/http_compressors/_resolve.py @@ -0,0 +1,38 @@ +from __future__ import annotations + +from typing import TYPE_CHECKING + +from apify_client.http_compressors._base import HttpCompressor +from apify_client.http_compressors._gzip import GzipHttpCompressor + +if TYPE_CHECKING: + from apify_client.types import HttpCompressionAlgorithm + + +def resolve_compressor(compression: HttpCompressionAlgorithm | HttpCompressor) -> HttpCompressor: + """Convert a compression string or `HttpCompressor` instance into a concrete `HttpCompressor`. + + Args: + compression: A string literal naming an HTTP compression algorithm, or an `HttpCompressor` instance. + + Returns: + A ready-to-use `HttpCompressor`. + + Raises: + ImportError: If the requested algorithm needs an optional extra that is not installed. + ValueError: If `compression` is not a recognized compression algorithm. + """ + if isinstance(compression, HttpCompressor): + return compression + if compression == 'gzip': + return GzipHttpCompressor() + if compression == 'brotli': + # The import is here so the ImportError is raised at call time, + # not at module import time, giving users a clear message. + from apify_client.http_compressors import BrotliHttpCompressor # noqa: PLC0415 + + return BrotliHttpCompressor() + + # The backend supports also `deflate` and `identity` (no compression). One can build + # a custom compressor if needed. + raise ValueError(f'Unsupported compression algorithm: {compression!r}') diff --git a/src/apify_client/types.py b/src/apify_client/types.py index 748fb846..47c5d7b6 100644 --- a/src/apify_client/types.py +++ b/src/apify_client/types.py @@ -11,6 +11,9 @@ WebhookRepresentationDict, ) +HttpCompressionAlgorithm = Literal['brotli', 'gzip'] +"""Accepted string literals for the `compression` parameter on `ApifyClient` and `ApifyClientAsync`.""" + Timeout = timedelta | Literal['no_timeout', 'short', 'medium', 'long'] """Type for the `timeout` parameter on resource client methods. @@ -42,6 +45,7 @@ """ __all__ = [ + 'HttpCompressionAlgorithm', 'JsonSerializable', 'Timeout', 'WebhooksList', diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py index 1cdb3689..1db53b71 100644 --- a/tests/integration/conftest.py +++ b/tests/integration/conftest.py @@ -16,7 +16,7 @@ ) from apify_client import ApifyClient, ApifyClientAsync from apify_client._consts import DEFAULT_API_URL -from apify_client._utils import create_hmac_signature, create_storage_content_signature +from apify_client._utils.crypto import create_hmac_signature, create_storage_content_signature if TYPE_CHECKING: from collections.abc import Generator diff --git a/tests/unit/conftest.py b/tests/unit/conftest.py index 64d73cd9..8ee3d0a8 100644 --- a/tests/unit/conftest.py +++ b/tests/unit/conftest.py @@ -1,9 +1,14 @@ -from collections.abc import Iterable +from __future__ import annotations + from logging import getLogger +from typing import TYPE_CHECKING import pytest from pytest_httpserver import HTTPServer +if TYPE_CHECKING: + from collections.abc import Iterable + @pytest.fixture(scope='session') def make_httpserver() -> Iterable[HTTPServer]: diff --git a/tests/unit/test_http_clients.py b/tests/unit/test_http_clients.py index b54c9fee..e748d647 100644 --- a/tests/unit/test_http_clients.py +++ b/tests/unit/test_http_clients.py @@ -1,10 +1,12 @@ +from __future__ import annotations + import gzip import time -from collections.abc import Callable from datetime import UTC, datetime, timedelta -from typing import Any +from typing import TYPE_CHECKING from unittest.mock import Mock +import brotli import impit import pytest @@ -12,6 +14,12 @@ from apify_client.errors import InvalidResponseBodyError from apify_client.http_clients import HttpClient, HttpClientAsync, HttpResponse, ImpitHttpClient, ImpitHttpClientAsync from apify_client.http_clients._impit import _is_retryable_error +from apify_client.http_compressors._brotli import BrotliHttpCompressor +from apify_client.http_compressors._gzip import GzipHttpCompressor + +if TYPE_CHECKING: + from collections.abc import Callable + from typing import Any class _ConcreteHttpClient(HttpClient): @@ -260,6 +268,16 @@ def test_is_retryable_error() -> None: assert not _is_retryable_error(Exception('test')) +@pytest.fixture( + params=[ + pytest.param((GzipHttpCompressor(), 'gzip', gzip.decompress), id='gzip'), + pytest.param((BrotliHttpCompressor(), 'br', brotli.decompress), id='brotli'), + ] +) +def compressor_case(request: pytest.FixtureRequest) -> tuple: + return request.param + + def test_prepare_request_call_basic() -> None: """Test _prepare_request_call with basic parameters.""" client = _ConcreteHttpClient() @@ -270,112 +288,125 @@ def test_prepare_request_call_basic() -> None: assert data is None -def test_prepare_request_call_with_json() -> None: +def test_prepare_request_call_with_json(compressor_case: tuple) -> None: """Test _prepare_request_call with JSON data.""" - client = _ConcreteHttpClient() + compressor, content_encoding, decompress = compressor_case + client = _ConcreteHttpClient(http_compressor=compressor) json_data = {'key': 'value', 'number': 42} headers, _params, data = client._prepare_request_call(json=json_data) assert headers['Content-Type'] == 'application/json' - assert headers['Content-Encoding'] == 'gzip' + assert headers['Content-Encoding'] == content_encoding assert data is not None assert isinstance(data, bytes) + assert decompress(data) == b'{"key": "value", "number": 42}' -def test_prepare_request_call_with_empty_dict_json() -> None: +def test_prepare_request_call_with_empty_dict_json(compressor_case: tuple) -> None: """Test _prepare_request_call with empty dict JSON (falsy but valid).""" - client = _ConcreteHttpClient() + compressor, content_encoding, decompress = compressor_case + client = _ConcreteHttpClient(http_compressor=compressor) headers, _params, data = client._prepare_request_call(json={}) assert headers['Content-Type'] == 'application/json' - assert headers['Content-Encoding'] == 'gzip' + assert headers['Content-Encoding'] == content_encoding assert data is not None assert isinstance(data, bytes) - # Verify the gzipped data contains the JSON - decompressed = gzip.decompress(data) - assert decompressed == b'{}' + assert decompress(data) == b'{}' -def test_prepare_request_call_with_empty_list_json() -> None: +def test_prepare_request_call_with_empty_list_json(compressor_case: tuple) -> None: """Test _prepare_request_call with empty list JSON (falsy but valid).""" - client = _ConcreteHttpClient() + compressor, content_encoding, decompress = compressor_case + client = _ConcreteHttpClient(http_compressor=compressor) headers, _params, data = client._prepare_request_call(json=[]) assert headers['Content-Type'] == 'application/json' - assert headers['Content-Encoding'] == 'gzip' + assert headers['Content-Encoding'] == content_encoding assert data is not None assert isinstance(data, bytes) - # Verify the gzipped data contains the JSON - decompressed = gzip.decompress(data) - assert decompressed == b'[]' + assert decompress(data) == b'[]' -def test_prepare_request_call_with_zero_json() -> None: +def test_prepare_request_call_with_zero_json(compressor_case: tuple) -> None: """Test _prepare_request_call with zero JSON (falsy but valid).""" - client = _ConcreteHttpClient() + compressor, content_encoding, decompress = compressor_case + client = _ConcreteHttpClient(http_compressor=compressor) headers, _params, data = client._prepare_request_call(json=0) assert headers['Content-Type'] == 'application/json' - assert headers['Content-Encoding'] == 'gzip' + assert headers['Content-Encoding'] == content_encoding assert data is not None assert isinstance(data, bytes) - # Verify the gzipped data contains the JSON - decompressed = gzip.decompress(data) - assert decompressed == b'0' + assert decompress(data) == b'0' -def test_prepare_request_call_with_false_json() -> None: +def test_prepare_request_call_with_false_json(compressor_case: tuple) -> None: """Test _prepare_request_call with False JSON (falsy but valid).""" - client = _ConcreteHttpClient() + compressor, content_encoding, decompress = compressor_case + client = _ConcreteHttpClient(http_compressor=compressor) headers, _params, data = client._prepare_request_call(json=False) assert headers['Content-Type'] == 'application/json' - assert headers['Content-Encoding'] == 'gzip' + assert headers['Content-Encoding'] == content_encoding assert data is not None assert isinstance(data, bytes) - # Verify the gzipped data contains the JSON - decompressed = gzip.decompress(data) - assert decompressed == b'false' + assert decompress(data) == b'false' -def test_prepare_request_call_with_empty_string_json() -> None: +def test_prepare_request_call_with_empty_string_json(compressor_case: tuple) -> None: """Test _prepare_request_call with empty string JSON (falsy but valid).""" - client = _ConcreteHttpClient() + compressor, content_encoding, decompress = compressor_case + client = _ConcreteHttpClient(http_compressor=compressor) headers, _params, data = client._prepare_request_call(json='') assert headers['Content-Type'] == 'application/json' - assert headers['Content-Encoding'] == 'gzip' + assert headers['Content-Encoding'] == content_encoding assert data is not None assert isinstance(data, bytes) - # Verify the gzipped data contains the JSON - decompressed = gzip.decompress(data) - assert decompressed == b'""' + assert decompress(data) == b'""' -def test_prepare_request_call_with_string_data() -> None: +def test_prepare_request_call_with_string_data(compressor_case: tuple) -> None: """Test _prepare_request_call with string data.""" - client = _ConcreteHttpClient() + compressor, content_encoding, decompress = compressor_case + client = _ConcreteHttpClient(http_compressor=compressor) headers, _params, data = client._prepare_request_call(data='test string') - assert headers['Content-Encoding'] == 'gzip' + assert headers['Content-Encoding'] == content_encoding assert isinstance(data, bytes) + assert decompress(data) == b'test string' -def test_prepare_request_call_with_bytes_data() -> None: +def test_prepare_request_call_with_bytes_data(compressor_case: tuple) -> None: """Test _prepare_request_call with bytes data.""" - client = _ConcreteHttpClient() + compressor, content_encoding, decompress = compressor_case + client = _ConcreteHttpClient(http_compressor=compressor) headers, _params, data = client._prepare_request_call(data=b'test bytes') - assert headers['Content-Encoding'] == 'gzip' + assert headers['Content-Encoding'] == content_encoding + assert isinstance(data, bytes) + assert decompress(data) == b'test bytes' + + +def test_prepare_request_call_with_bytearray_data(compressor_case: tuple) -> None: + """Test _prepare_request_call with bytearray data (regression: must compress without error).""" + compressor, content_encoding, decompress = compressor_case + client = _ConcreteHttpClient(http_compressor=compressor) + + headers, _params, data = client._prepare_request_call(data=bytearray(b'test bytearray')) + + assert headers['Content-Encoding'] == content_encoding assert isinstance(data, bytes) + assert decompress(data) == b'test bytearray' def test_prepare_request_call_json_and_data_error() -> None: diff --git a/tests/unit/test_http_compressors.py b/tests/unit/test_http_compressors.py new file mode 100644 index 00000000..de858f45 --- /dev/null +++ b/tests/unit/test_http_compressors.py @@ -0,0 +1,149 @@ +from __future__ import annotations + +import gzip +import importlib +import sys +from contextlib import contextmanager +from typing import TYPE_CHECKING + +import brotli +import pytest + +from apify_client.http_compressors import BrotliHttpCompressor, GzipHttpCompressor +from apify_client.http_compressors._resolve import resolve_compressor + +if TYPE_CHECKING: + from collections.abc import Iterator + + +@contextmanager +def _brotli_unavailable() -> Iterator[None]: + """Reimport the `http_compressors` package with the `brotli` import forced to fail. + + Simulates an environment where the optional `brotli` extra is not installed, and restores the + original module state afterwards so other tests in the same worker are unaffected. + """ + + class _Blocker: + def find_spec(self, name: str, *_args: object) -> None: + if name == 'brotli' or name.startswith('brotli.'): + raise ModuleNotFoundError(f"No module named '{name}'") + + def _affected(name: str) -> bool: + return name == 'brotli' or name.startswith(('brotli.', 'apify_client.http_compressors')) + + saved = {name: mod for name, mod in list(sys.modules.items()) if _affected(name)} + blocker = _Blocker() + sys.meta_path.insert(0, blocker) + for name in saved: + del sys.modules[name] + + try: + importlib.import_module('apify_client.http_compressors') + yield + finally: + sys.meta_path.remove(blocker) + for name in [name for name in sys.modules if _affected(name)]: + del sys.modules[name] + sys.modules.update(saved) + + +@pytest.mark.parametrize( + 'quality', + [ + pytest.param(1, id='fastest'), + pytest.param(9, id='best'), + ], +) +def test_gzip_compressor_round_trips_at_quality(quality: int) -> None: + """Gzip compressor round-trips data at the boundary quality levels.""" + assert gzip.decompress(GzipHttpCompressor(quality=quality).compress(b'payload')) == b'payload' + + +@pytest.mark.parametrize( + 'quality', + [ + pytest.param(0, id='fastest'), + pytest.param(11, id='best'), + ], +) +def test_brotli_compressor_round_trips_at_quality(quality: int) -> None: + """Brotli compressor round-trips data at the boundary quality levels.""" + assert brotli.decompress(BrotliHttpCompressor(quality=quality).compress(b'payload')) == b'payload' + + +@pytest.mark.parametrize( + 'quality', + [ + pytest.param(0, id='below minimum'), + pytest.param(10, id='above maximum'), + pytest.param(-1, id='negative'), + ], +) +def test_gzip_compressor_rejects_out_of_range_quality(quality: int) -> None: + """Gzip compressor raises `ValueError` at construction for a quality not between `1` and `9`.""" + with pytest.raises(ValueError, match='gzip quality must be between 1 and 9'): + GzipHttpCompressor(quality=quality) + + +@pytest.mark.parametrize( + 'quality', + [ + pytest.param(-1, id='negative'), + pytest.param(12, id='above maximum'), + ], +) +def test_brotli_compressor_rejects_out_of_range_quality(quality: int) -> None: + """Brotli compressor raises `ValueError` at construction for a quality not between `0` and `11`.""" + with pytest.raises(ValueError, match='brotli quality must be between 0 and 11'): + BrotliHttpCompressor(quality=quality) + + +def test_gzip_compressor_round_trips_and_sets_content_encoding() -> None: + """Gzip compressor round-trips data and reports `gzip` as its content encoding.""" + compressor = GzipHttpCompressor() + assert compressor.content_encoding == 'gzip' + assert gzip.decompress(compressor.compress(b'hello world')) == b'hello world' + + +def test_brotli_compressor_round_trips_and_sets_content_encoding() -> None: + """Brotli compressor round-trips data and reports `br` as its content encoding.""" + compressor = BrotliHttpCompressor() + assert compressor.content_encoding == 'br' + assert brotli.decompress(compressor.compress(b'hello world')) == b'hello world' + + +def test_resolve_compressor_gzip() -> None: + """The `'gzip'` literal resolves to a `GzipHttpCompressor`.""" + assert isinstance(resolve_compressor('gzip'), GzipHttpCompressor) + + +def test_resolve_compressor_brotli() -> None: + """The `'brotli'` literal resolves to a `BrotliHttpCompressor`.""" + assert isinstance(resolve_compressor('brotli'), BrotliHttpCompressor) + + +def test_resolve_compressor_passes_through_instance() -> None: + """An `HttpCompressor` instance is returned unchanged.""" + compressor = BrotliHttpCompressor(quality=11) + assert resolve_compressor(compressor) is compressor + + +def test_resolve_compressor_rejects_unknown_algorithm() -> None: + """An unrecognized encoding raises `ValueError`.""" + with pytest.raises(ValueError, match='Unsupported compression algorithm'): + resolve_compressor('deflate') # ty: ignore[invalid-argument-type] + + +def test_brotli_import_hook_raises_clear_error_when_extra_missing() -> None: + """Accessing `BrotliHttpCompressor` without the `brotli` extra raises `ImportError`, not `TypeError`.""" + with _brotli_unavailable(): + module = importlib.import_module('apify_client.http_compressors') + with pytest.raises(ImportError): + _ = module.BrotliHttpCompressor + + +def test_resolve_compressor_brotli_raises_clear_error_when_extra_missing() -> None: + """Resolving `compression='brotli'` without the extra raises `ImportError` at resolution time.""" + with _brotli_unavailable(), pytest.raises(ImportError): + resolve_compressor('brotli') diff --git a/tests/unit/test_run_charge.py b/tests/unit/test_run_charge.py index 6b030651..aa176734 100644 --- a/tests/unit/test_run_charge.py +++ b/tests/unit/test_run_charge.py @@ -4,6 +4,7 @@ import json from typing import TYPE_CHECKING +import brotli import pytest from werkzeug import Request, Response @@ -12,26 +13,43 @@ if TYPE_CHECKING: from pytest_httpserver import HTTPServer + from apify_client.types import HttpCompressionAlgorithm + _MOCKED_RUN_ID = 'test_run_id' _CHARGE_PATH = f'/v2/actor-runs/{_MOCKED_RUN_ID}/charge' def _decode_body(request: Request) -> dict: raw = request.get_data() - if request.headers.get('Content-Encoding') == 'gzip': + encoding = request.headers.get('Content-Encoding') + if encoding == 'br': + raw = brotli.decompress(raw) + elif encoding == 'gzip': raw = gzip.decompress(raw) return json.loads(raw) +@pytest.mark.parametrize( + 'compression', + [ + pytest.param('gzip', id='gzip'), + pytest.param('brotli', id='brotli'), + ], +) @pytest.mark.parametrize( 'count', - [0, 1, 5], + [ + pytest.param(0, id='zero'), + pytest.param(1, id='one'), + pytest.param(5, id='five'), + ], ) def test_run_charge_preserves_count_sync( httpserver: HTTPServer, count: int, + compression: HttpCompressionAlgorithm, ) -> None: - """Ensure `count` is sent as-is (in particular, `0` is preserved).""" + """Ensure `count` is sent as-is (in particular, `0` is preserved), regardless of the request-body compression.""" captured_requests: list[Request] = [] def capture_request(request: Request) -> Response: @@ -41,7 +59,7 @@ def capture_request(request: Request) -> Response: httpserver.expect_request(_CHARGE_PATH, method='POST').respond_with_handler(capture_request) api_url = httpserver.url_for('/').removesuffix('/') - client = ApifyClient(token='test_token', api_url=api_url) + client = ApifyClient(token='test_token', api_url=api_url, compression=compression) client.run(_MOCKED_RUN_ID).charge('test-event', count=count) @@ -50,13 +68,25 @@ def capture_request(request: Request) -> Response: assert body['count'] == count +@pytest.mark.parametrize( + 'compression', + [ + pytest.param('gzip', id='gzip'), + pytest.param('brotli', id='brotli'), + ], +) @pytest.mark.parametrize( 'count', - [0, 1, 5], + [ + pytest.param(0, id='zero'), + pytest.param(1, id='one'), + pytest.param(5, id='five'), + ], ) async def test_run_charge_preserves_count_async( httpserver: HTTPServer, count: int, + compression: HttpCompressionAlgorithm, ) -> None: """Async variant of `test_run_charge_preserves_count_sync`.""" captured_requests: list[Request] = [] @@ -68,7 +98,7 @@ def capture_request(request: Request) -> Response: httpserver.expect_request(_CHARGE_PATH, method='POST').respond_with_handler(capture_request) api_url = httpserver.url_for('/').removesuffix('/') - client = ApifyClientAsync(token='test_token', api_url=api_url) + client = ApifyClientAsync(token='test_token', api_url=api_url, compression=compression) await client.run(_MOCKED_RUN_ID).charge('test-event', count=count) diff --git a/tests/unit/test_statistics.py b/tests/unit/test_statistics.py index 69fbc3c4..37634a7b 100644 --- a/tests/unit/test_statistics.py +++ b/tests/unit/test_statistics.py @@ -1,3 +1,5 @@ +from __future__ import annotations + import pytest from apify_client._statistics import ClientStatistics diff --git a/tests/unit/test_url_generation.py b/tests/unit/test_url_generation.py index a1a2aec6..a48b704d 100644 --- a/tests/unit/test_url_generation.py +++ b/tests/unit/test_url_generation.py @@ -8,7 +8,7 @@ from apify_client import ApifyClient, ApifyClientAsync from apify_client._consts import DEFAULT_API_URL -from apify_client._utils import create_hmac_signature, create_storage_content_signature +from apify_client._utils.crypto import create_hmac_signature, create_storage_content_signature # ============================================================================ # Test data and helpers diff --git a/tests/unit/test_utils.py b/tests/unit/test_utils.py index 90e918af..a52d3fb4 100644 --- a/tests/unit/test_utils.py +++ b/tests/unit/test_utils.py @@ -13,18 +13,10 @@ from apify_client._models import WebhookCondition, WebhookCreate from apify_client._resource_clients._resource_client import ResourceClientBase -from apify_client._utils import ( - catch_not_found_or_throw, - create_hmac_signature, - create_storage_content_signature, - encode_base62, - encode_key_value_store_record_value, - encode_webhooks_to_base64, - is_retryable_error, - response_to_dict, - response_to_list, - to_safe_id, -) +from apify_client._utils.crypto import create_hmac_signature, create_storage_content_signature, encode_base62 +from apify_client._utils.encoding import encode_key_value_store_record_value, encode_webhooks_to_base64 +from apify_client._utils.errors import catch_not_found_or_throw, is_retryable_error +from apify_client._utils.http import response_to_dict, response_to_list, to_safe_id from apify_client.errors import ApifyApiError, InvalidResponseBodyError if TYPE_CHECKING: diff --git a/uv.lock b/uv.lock index ea6ec2b7..fb279a6b 100644 --- a/uv.lock +++ b/uv.lock @@ -3,7 +3,8 @@ revision = 3 requires-python = ">=3.11" resolution-markers = [ "python_full_version >= '3.14'", - "python_full_version < '3.14'", + "python_full_version == '3.13.*'", + "python_full_version < '3.13'", ] [options] @@ -49,6 +50,11 @@ dependencies = [ { name = "pydantic", extra = ["email"] }, ] +[package.optional-dependencies] +brotli = [ + { name = "brotli" }, +] + [package.dev-dependencies] dev = [ { name = "black" }, @@ -73,11 +79,13 @@ dev = [ [package.metadata] requires-dist = [ + { name = "brotli", marker = "extra == 'brotli'", specifier = ">=1.0.9" }, { name = "colorama", specifier = ">=0.4.0" }, { name = "impit", specifier = "~=0.13.0" }, { name = "more-itertools", specifier = ">=10.0.0" }, { name = "pydantic", extras = ["email"], specifier = ">=2.11.0" }, ] +provides-extras = ["brotli"] [package.metadata.requires-dev] dev = [ @@ -147,6 +155,54 @@ wheels = [ { url = "https://files.pythonhosted.org/packages/94/51/f975cae76d44274cc2868dc9040ac5d58d464784610234455b4e7b19c6ef/black-26.5.1-py3-none-any.whl", hash = "sha256:4ed7f7da04046d2e488437170797d3b4a4ad83906683bcb7dfc68b673bbce5e2", size = 213693, upload-time = "2026-05-18T16:53:33.964Z" }, ] +[[package]] +name = "brotli" +version = "1.2.0" +source = { registry = "https://pypi.org/simple" } +sdist = { url = "https://files.pythonhosted.org/packages/f7/16/c92ca344d646e71a43b8bb353f0a6490d7f6e06210f8554c8f874e454285/brotli-1.2.0.tar.gz", hash = "sha256:e310f77e41941c13340a95976fe66a8a95b01e783d430eeaf7a2f87e0a57dd0a", size = 7388632, upload-time = "2025-11-05T18:39:42.86Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/7a/ef/f285668811a9e1ddb47a18cb0b437d5fc2760d537a2fe8a57875ad6f8448/brotli-1.2.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:15b33fe93cedc4caaff8a0bd1eb7e3dab1c61bb22a0bf5bdfdfd97cd7da79744", size = 863110, upload-time = "2025-11-05T18:38:12.978Z" }, + { url = "https://files.pythonhosted.org/packages/50/62/a3b77593587010c789a9d6eaa527c79e0848b7b860402cc64bc0bc28a86c/brotli-1.2.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:898be2be399c221d2671d29eed26b6b2713a02c2119168ed914e7d00ceadb56f", size = 445438, upload-time = "2025-11-05T18:38:14.208Z" }, + { url = "https://files.pythonhosted.org/packages/cd/e1/7fadd47f40ce5549dc44493877db40292277db373da5053aff181656e16e/brotli-1.2.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:350c8348f0e76fff0a0fd6c26755d2653863279d086d3aa2c290a6a7251135dd", size = 1534420, upload-time = "2025-11-05T18:38:15.111Z" }, + { url = "https://files.pythonhosted.org/packages/12/8b/1ed2f64054a5a008a4ccd2f271dbba7a5fb1a3067a99f5ceadedd4c1d5a7/brotli-1.2.0-cp311-cp311-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:2e1ad3fda65ae0d93fec742a128d72e145c9c7a99ee2fcd667785d99eb25a7fe", size = 1632619, upload-time = "2025-11-05T18:38:16.094Z" }, + { url = "https://files.pythonhosted.org/packages/89/5a/7071a621eb2d052d64efd5da2ef55ecdac7c3b0c6e4f9d519e9c66d987ef/brotli-1.2.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:40d918bce2b427a0c4ba189df7a006ac0c7277c180aee4617d99e9ccaaf59e6a", size = 1426014, upload-time = "2025-11-05T18:38:17.177Z" }, + { url = "https://files.pythonhosted.org/packages/26/6d/0971a8ea435af5156acaaccec1a505f981c9c80227633851f2810abd252a/brotli-1.2.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:2a7f1d03727130fc875448b65b127a9ec5d06d19d0148e7554384229706f9d1b", size = 1489661, upload-time = "2025-11-05T18:38:18.41Z" }, + { url = "https://files.pythonhosted.org/packages/f3/75/c1baca8b4ec6c96a03ef8230fab2a785e35297632f402ebb1e78a1e39116/brotli-1.2.0-cp311-cp311-musllinux_1_2_ppc64le.whl", hash = "sha256:9c79f57faa25d97900bfb119480806d783fba83cd09ee0b33c17623935b05fa3", size = 1599150, upload-time = "2025-11-05T18:38:19.792Z" }, + { url = "https://files.pythonhosted.org/packages/0d/1a/23fcfee1c324fd48a63d7ebf4bac3a4115bdb1b00e600f80f727d850b1ae/brotli-1.2.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:844a8ceb8483fefafc412f85c14f2aae2fb69567bf2a0de53cdb88b73e7c43ae", size = 1493505, upload-time = "2025-11-05T18:38:20.913Z" }, + { url = "https://files.pythonhosted.org/packages/36/e5/12904bbd36afeef53d45a84881a4810ae8810ad7e328a971ebbfd760a0b3/brotli-1.2.0-cp311-cp311-win32.whl", hash = "sha256:aa47441fa3026543513139cb8926a92a8e305ee9c71a6209ef7a97d91640ea03", size = 334451, upload-time = "2025-11-05T18:38:21.94Z" }, + { url = "https://files.pythonhosted.org/packages/02/8b/ecb5761b989629a4758c394b9301607a5880de61ee2ee5fe104b87149ebc/brotli-1.2.0-cp311-cp311-win_amd64.whl", hash = "sha256:022426c9e99fd65d9475dce5c195526f04bb8be8907607e27e747893f6ee3e24", size = 369035, upload-time = "2025-11-05T18:38:22.941Z" }, + { url = "https://files.pythonhosted.org/packages/11/ee/b0a11ab2315c69bb9b45a2aaed022499c9c24a205c3a49c3513b541a7967/brotli-1.2.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:35d382625778834a7f3061b15423919aa03e4f5da34ac8e02c074e4b75ab4f84", size = 861543, upload-time = "2025-11-05T18:38:24.183Z" }, + { url = "https://files.pythonhosted.org/packages/e1/2f/29c1459513cd35828e25531ebfcbf3e92a5e49f560b1777a9af7203eb46e/brotli-1.2.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:7a61c06b334bd99bc5ae84f1eeb36bfe01400264b3c352f968c6e30a10f9d08b", size = 444288, upload-time = "2025-11-05T18:38:25.139Z" }, + { url = "https://files.pythonhosted.org/packages/3d/6f/feba03130d5fceadfa3a1bb102cb14650798c848b1df2a808356f939bb16/brotli-1.2.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:acec55bb7c90f1dfc476126f9711a8e81c9af7fb617409a9ee2953115343f08d", size = 1528071, upload-time = "2025-11-05T18:38:26.081Z" }, + { url = "https://files.pythonhosted.org/packages/2b/38/f3abb554eee089bd15471057ba85f47e53a44a462cfce265d9bf7088eb09/brotli-1.2.0-cp312-cp312-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:260d3692396e1895c5034f204f0db022c056f9e2ac841593a4cf9426e2a3faca", size = 1626913, upload-time = "2025-11-05T18:38:27.284Z" }, + { url = "https://files.pythonhosted.org/packages/03/a7/03aa61fbc3c5cbf99b44d158665f9b0dd3d8059be16c460208d9e385c837/brotli-1.2.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:072e7624b1fc4d601036ab3f4f27942ef772887e876beff0301d261210bca97f", size = 1419762, upload-time = "2025-11-05T18:38:28.295Z" }, + { url = "https://files.pythonhosted.org/packages/21/1b/0374a89ee27d152a5069c356c96b93afd1b94eae83f1e004b57eb6ce2f10/brotli-1.2.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:adedc4a67e15327dfdd04884873c6d5a01d3e3b6f61406f99b1ed4865a2f6d28", size = 1484494, upload-time = "2025-11-05T18:38:29.29Z" }, + { url = "https://files.pythonhosted.org/packages/cf/57/69d4fe84a67aef4f524dcd075c6eee868d7850e85bf01d778a857d8dbe0a/brotli-1.2.0-cp312-cp312-musllinux_1_2_ppc64le.whl", hash = "sha256:7a47ce5c2288702e09dc22a44d0ee6152f2c7eda97b3c8482d826a1f3cfc7da7", size = 1593302, upload-time = "2025-11-05T18:38:30.639Z" }, + { url = "https://files.pythonhosted.org/packages/d5/3b/39e13ce78a8e9a621c5df3aeb5fd181fcc8caba8c48a194cd629771f6828/brotli-1.2.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:af43b8711a8264bb4e7d6d9a6d004c3a2019c04c01127a868709ec29962b6036", size = 1487913, upload-time = "2025-11-05T18:38:31.618Z" }, + { url = "https://files.pythonhosted.org/packages/62/28/4d00cb9bd76a6357a66fcd54b4b6d70288385584063f4b07884c1e7286ac/brotli-1.2.0-cp312-cp312-win32.whl", hash = "sha256:e99befa0b48f3cd293dafeacdd0d191804d105d279e0b387a32054c1180f3161", size = 334362, upload-time = "2025-11-05T18:38:32.939Z" }, + { url = "https://files.pythonhosted.org/packages/1c/4e/bc1dcac9498859d5e353c9b153627a3752868a9d5f05ce8dedd81a2354ab/brotli-1.2.0-cp312-cp312-win_amd64.whl", hash = "sha256:b35c13ce241abdd44cb8ca70683f20c0c079728a36a996297adb5334adfc1c44", size = 369115, upload-time = "2025-11-05T18:38:33.765Z" }, + { url = "https://files.pythonhosted.org/packages/6c/d4/4ad5432ac98c73096159d9ce7ffeb82d151c2ac84adcc6168e476bb54674/brotli-1.2.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:9e5825ba2c9998375530504578fd4d5d1059d09621a02065d1b6bfc41a8e05ab", size = 861523, upload-time = "2025-11-05T18:38:34.67Z" }, + { url = "https://files.pythonhosted.org/packages/91/9f/9cc5bd03ee68a85dc4bc89114f7067c056a3c14b3d95f171918c088bf88d/brotli-1.2.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:0cf8c3b8ba93d496b2fae778039e2f5ecc7cff99df84df337ca31d8f2252896c", size = 444289, upload-time = "2025-11-05T18:38:35.6Z" }, + { url = "https://files.pythonhosted.org/packages/2e/b6/fe84227c56a865d16a6614e2c4722864b380cb14b13f3e6bef441e73a85a/brotli-1.2.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c8565e3cdc1808b1a34714b553b262c5de5fbda202285782173ec137fd13709f", size = 1528076, upload-time = "2025-11-05T18:38:36.639Z" }, + { url = "https://files.pythonhosted.org/packages/55/de/de4ae0aaca06c790371cf6e7ee93a024f6b4bb0568727da8c3de112e726c/brotli-1.2.0-cp313-cp313-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:26e8d3ecb0ee458a9804f47f21b74845cc823fd1bb19f02272be70774f56e2a6", size = 1626880, upload-time = "2025-11-05T18:38:37.623Z" }, + { url = "https://files.pythonhosted.org/packages/5f/16/a1b22cbea436642e071adcaf8d4b350a2ad02f5e0ad0da879a1be16188a0/brotli-1.2.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:67a91c5187e1eec76a61625c77a6c8c785650f5b576ca732bd33ef58b0dff49c", size = 1419737, upload-time = "2025-11-05T18:38:38.729Z" }, + { url = "https://files.pythonhosted.org/packages/46/63/c968a97cbb3bdbf7f974ef5a6ab467a2879b82afbc5ffb65b8acbb744f95/brotli-1.2.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:4ecdb3b6dc36e6d6e14d3a1bdc6c1057c8cbf80db04031d566eb6080ce283a48", size = 1484440, upload-time = "2025-11-05T18:38:39.916Z" }, + { url = "https://files.pythonhosted.org/packages/06/9d/102c67ea5c9fc171f423e8399e585dabea29b5bc79b05572891e70013cdd/brotli-1.2.0-cp313-cp313-musllinux_1_2_ppc64le.whl", hash = "sha256:3e1b35d56856f3ed326b140d3c6d9db91740f22e14b06e840fe4bb1923439a18", size = 1593313, upload-time = "2025-11-05T18:38:41.24Z" }, + { url = "https://files.pythonhosted.org/packages/9e/4a/9526d14fa6b87bc827ba1755a8440e214ff90de03095cacd78a64abe2b7d/brotli-1.2.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:54a50a9dad16b32136b2241ddea9e4df159b41247b2ce6aac0b3276a66a8f1e5", size = 1487945, upload-time = "2025-11-05T18:38:42.277Z" }, + { url = "https://files.pythonhosted.org/packages/5b/e8/3fe1ffed70cbef83c5236166acaed7bb9c766509b157854c80e2f766b38c/brotli-1.2.0-cp313-cp313-win32.whl", hash = "sha256:1b1d6a4efedd53671c793be6dd760fcf2107da3a52331ad9ea429edf0902f27a", size = 334368, upload-time = "2025-11-05T18:38:43.345Z" }, + { url = "https://files.pythonhosted.org/packages/ff/91/e739587be970a113b37b821eae8097aac5a48e5f0eca438c22e4c7dd8648/brotli-1.2.0-cp313-cp313-win_amd64.whl", hash = "sha256:b63daa43d82f0cdabf98dee215b375b4058cce72871fd07934f179885aad16e8", size = 369116, upload-time = "2025-11-05T18:38:44.609Z" }, + { url = "https://files.pythonhosted.org/packages/17/e1/298c2ddf786bb7347a1cd71d63a347a79e5712a7c0cba9e3c3458ebd976f/brotli-1.2.0-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:6c12dad5cd04530323e723787ff762bac749a7b256a5bece32b2243dd5c27b21", size = 863080, upload-time = "2025-11-05T18:38:45.503Z" }, + { url = "https://files.pythonhosted.org/packages/84/0c/aac98e286ba66868b2b3b50338ffbd85a35c7122e9531a73a37a29763d38/brotli-1.2.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:3219bd9e69868e57183316ee19c84e03e8f8b5a1d1f2667e1aa8c2f91cb061ac", size = 445453, upload-time = "2025-11-05T18:38:46.433Z" }, + { url = "https://files.pythonhosted.org/packages/ec/f1/0ca1f3f99ae300372635ab3fe2f7a79fa335fee3d874fa7f9e68575e0e62/brotli-1.2.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:963a08f3bebd8b75ac57661045402da15991468a621f014be54e50f53a58d19e", size = 1528168, upload-time = "2025-11-05T18:38:47.371Z" }, + { url = "https://files.pythonhosted.org/packages/d6/a6/2ebfc8f766d46df8d3e65b880a2e220732395e6d7dc312c1e1244b0f074a/brotli-1.2.0-cp314-cp314-manylinux2014_ppc64le.manylinux_2_17_ppc64le.manylinux_2_28_ppc64le.whl", hash = "sha256:9322b9f8656782414b37e6af884146869d46ab85158201d82bab9abbcb971dc7", size = 1627098, upload-time = "2025-11-05T18:38:48.385Z" }, + { url = "https://files.pythonhosted.org/packages/f3/2f/0976d5b097ff8a22163b10617f76b2557f15f0f39d6a0fe1f02b1a53e92b/brotli-1.2.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:cf9cba6f5b78a2071ec6fb1e7bd39acf35071d90a81231d67e92d637776a6a63", size = 1419861, upload-time = "2025-11-05T18:38:49.372Z" }, + { url = "https://files.pythonhosted.org/packages/9c/97/d76df7176a2ce7616ff94c1fb72d307c9a30d2189fe877f3dd99af00ea5a/brotli-1.2.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:7547369c4392b47d30a3467fe8c3330b4f2e0f7730e45e3103d7d636678a808b", size = 1484594, upload-time = "2025-11-05T18:38:50.655Z" }, + { url = "https://files.pythonhosted.org/packages/d3/93/14cf0b1216f43df5609f5b272050b0abd219e0b54ea80b47cef9867b45e7/brotli-1.2.0-cp314-cp314-musllinux_1_2_ppc64le.whl", hash = "sha256:fc1530af5c3c275b8524f2e24841cbe2599d74462455e9bae5109e9ff42e9361", size = 1593455, upload-time = "2025-11-05T18:38:51.624Z" }, + { url = "https://files.pythonhosted.org/packages/b3/73/3183c9e41ca755713bdf2cc1d0810df742c09484e2e1ddd693bee53877c1/brotli-1.2.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:d2d085ded05278d1c7f65560aae97b3160aeb2ea2c0b3e26204856beccb60888", size = 1488164, upload-time = "2025-11-05T18:38:53.079Z" }, + { url = "https://files.pythonhosted.org/packages/64/6a/0c78d8f3a582859236482fd9fa86a65a60328a00983006bcf6d83b7b2253/brotli-1.2.0-cp314-cp314-win32.whl", hash = "sha256:832c115a020e463c2f67664560449a7bea26b0c1fdd690352addad6d0a08714d", size = 339280, upload-time = "2025-11-05T18:38:54.02Z" }, + { url = "https://files.pythonhosted.org/packages/f5/10/56978295c14794b2c12007b07f3e41ba26acda9257457d7085b0bb3bb90c/brotli-1.2.0-cp314-cp314-win_amd64.whl", hash = "sha256:e7c0af964e0b4e3412a0ebf341ea26ec767fa0b4cf81abb5e897c9338b5ad6a3", size = 375639, upload-time = "2025-11-05T18:38:55.67Z" }, +] + [[package]] name = "certifi" version = "2026.6.17"