Improvements - #3
Merged
Merged
Conversation
added 6 commits
September 22, 2026 11:44
/api/match and /api/generate accept arbitrary user-supplied patterns
and log text, with no bound on either size or how long matching can
take. Two independent mitigations:
- DebugRequest now caps pattern (5,000), custom_patterns (20,000) and
log_text (200,000) length via pydantic Field(max_length=...).
- Both engine calls now run off the event loop through a small
ThreadPoolExecutor with a wall-clock budget (MATCH_TIMEOUT_SECONDS,
default 3s, env-overridable), via a new _run_with_timeout() helper.
This bounds *response time*, not CPU usage - see the comment above
MATCH_TIMEOUT_SECONDS for the full reasoning, including why pygrok's
use of the third-party 'regex' package (one of its own dependencies)
already mitigates some classic catastrophic-backtracking patterns
before this timeout would ever need to fire.
Fixes a real bug this surfaces: pydantic's default validation-error
body is a list of error objects, which the frontend's
`this.errorMessage = data.detail || ...` can't render sensibly (it
would show "[object Object]"). Added a RequestValidationError handler
that flattens it to the same single-string {"detail": "..."} shape
HTTPException responses already use, so a validation failure (missing
field, oversized input, etc.) displays exactly like a 400 does.
The project had no tests at all. Adds: - tests/test_grok_engine.py: unit tests for GrokDebuggerEngine - matching (grok fields, named regex groups, strict mode, custom pattern definitions), field-name validation, type detection, dict flattening/unflattening, and pattern auto-generation. Includes explicit regression tests for the find_partial_match tokenizer bug fixed earlier (both the simple named-group case and the harder nested-parens-inside-a-group case). - tests/test_api.py: FastAPI TestClient tests for all four endpoints, including the new request size limits, the flattened validation- error shape, and the match timeout (mocked rather than relying on a genuinely catastrophic pattern, since pygrok's preference for the regex package over stdlib re makes wall-clock ReDoS timing environment-dependent - see the test's docstring). - requirements-dev.txt and a [project.optional-dependencies] dev group in pyproject.toml (ruff, bandit, pytest, httpx). - [tool.pytest.ini_options] in pyproject.toml so resolves the app package correctly regardless of invocation style.
.github/workflows/ci.yml installs requirements-dev.txt and runs 'ruff check app/ tests/', 'bandit -r app/' and 'pytest -v' on every push to main and every pull request, across Python 3.10-3.12.
- Project Structure now lists .devcontainer/, .github/workflows/, tests/, pyproject.toml and requirements-dev.txt. - New Development section: how to install dev extras and run pytest, ruff and bandit, and a pointer to the CI workflow that runs them. - New Request Limits & Timeouts section documenting the size caps and match timeout added earlier, with a pointer to the fuller reasoning in app/main.py. - New Production Notes section flagging the two frontend CDN trade-offs (Tailwind's Play CDN, Alpine's floating version/missing SRI) with concrete steps to harden them - not applied automatically, since a wrong SRI hash would break the app outright. - New License section - the repo is GPLv3 (LICENSE) but the README never said so.
Keep a Changelog style. 1.1.0-1.1.4 entries are reconstructed from git history (this project didn't keep one before); 1.1.5 documents the request limits/timeout, test suite, CI and README work from this branch.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.