Improvements

.github/workflows/ci.yml

@@ -0,0 +1,33 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.10", "3.11", "3.12"]
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+          cache: pip
+
+      - name: Install dependencies
+        run: pip install -r requirements-dev.txt
+
+      - name: Lint (ruff)
+        run: ruff check app/ tests/
+
+      - name: Security scan (bandit)
+        run: bandit -r app/
+
+      - name: Test (pytest)
+        run: pytest -v

CHANGELOG.md

@@ -0,0 +1,78 @@
+# Changelog
+
+All notable changes to this project are documented here. Format loosely
+follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Entries for
+1.1.0–1.1.4 are reconstructed from git history rather than written at the
+time of each release, so they're summaries rather than exhaustive.
+
+## [1.1.5] - 2026-09-22
+
+### Added
+- Request size limits on `pattern` (5,000 chars), `custom_patterns` (20,000
+  chars) and `log_text` (200,000 chars).
+- A wall-clock timeout (`MATCH_TIMEOUT_SECONDS`, default 3s, env-overridable)
+  around pattern matching and auto-generation, so a pathological pattern
+  can't hang a request indefinitely.
+- A pytest test suite (`tests/test_grok_engine.py`, `tests/test_api.py`)
+  covering the matching engine and all four API endpoints - the project had
+  no tests before this.
+- CI (`.github/workflows/ci.yml`): ruff, bandit and pytest run on every push
+  to `main` and every pull request, across Python 3.10-3.12.
+- `requirements-dev.txt` and a `dev` extra in `pyproject.toml` (ruff, bandit,
+  pytest, httpx) for local linting and testing.
+- README: Development, Request Limits & Timeouts, Production Notes and
+  License sections.
+
+### Fixed
+- A pydantic validation error (e.g. a request exceeding the new size limits,
+  or a missing required field) used to return a list of error objects as
+  `detail`, which the frontend rendered as `[object Object]`. Validation
+  errors now return a single human-readable string, consistent with every
+  other error response.
+
+## [1.1.4] - 2026-09-22
+
+### Fixed
+- `find_partial_match` mis-tokenized patterns that mixed a `%{...}` Grok
+  field with a `(?P<name>...)` / `(?<name>...)` regex group, silently
+  breaking Partial Match Diagnostics for any such pattern. Replaced the
+  tokenizer with a paren-balancing version that also handles nested parens
+  inside a group's body.
+- API error handlers now chain exceptions with `raise ... from e` instead of
+  discarding the original traceback.
+
+### Changed
+- Pinned a `[tool.ruff]` lint configuration (previously unconfigured, so
+  `ruff check` meant whatever ruleset happened to be run by hand, and a
+  `# noqa: BLE001` comment in `grok_engine.py` referenced a rule nothing
+  enabled).
+- `config.py` uses `Path.open()` instead of the builtin `open()`.
+
+## [1.1.3] - 2026-08-07
+
+### Changed
+- `pregenerate_pattern` uses diff-based sequence alignment across sample
+  lines instead of a positional zip, so it correctly handles optional or
+  variable-length segments between samples instead of misaligning on them.
+
+## [1.1.2] - 2026-08-07
+
+### Changed
+- Backend review pass: field-name sanitization regex fixes, improved IP
+  detection, lint cleanup, and split the inline CSS/JS out of `index.html`
+  into `app/static/`.
+- Fixes to error message display, field matching, and suggested-pattern
+  replacement.
+
+## [1.1.1] - 2026-08-06
+
+### Added
+- Flexible vs. strict match mode toggle for the pattern checker.
+
+## [1.1.0] - 2026-08-05
+
+### Added
+- Initial tagged release: live Grok/regex matching, the pattern
+  auto-generator, a JSON view of results, ECS bracket/dot notation support,
+  pattern format-consistency normalization, and version/config loading from
+  `pyproject.toml`.

README.md

@@ -29,6 +29,11 @@ This project was built to modernize and combine capabilities of Grok debugging t
 
 ```
 .
+├── .devcontainer/
+│   └── devcontainer.json      # VS Code dev container config
+├── .github/
+│   └── workflows/
+│       └── ci.yml             # Lint (ruff), security scan (bandit) & tests (pytest) on push/PR
 ├── app/
 │   ├── main.py                # FastAPI application & API routes
 │   ├── config.py              # App settings & version/feature metadata
@@ -38,7 +43,12 @@ This project was built to modernize and combine capabilities of Grok debugging t
 │   └── static/
 │       ├── css/style.css      # UI styling
 │       └── js/app.js          # Alpine.js application logic
-├── requirements.txt           # Python dependencies
+├── tests/
+│   ├── test_grok_engine.py    # Unit tests for the matching/generation engine
+│   └── test_api.py            # API-level tests (FastAPI TestClient)
+├── pyproject.toml             # Project metadata, ruff config, pytest config
+├── requirements.txt           # Runtime dependencies
+├── requirements-dev.txt       # + ruff, bandit, pytest, httpx
 └── README.md
 ```
 
@@ -72,3 +82,72 @@ uvicorn app.main:app --reload --port 8000
 ```
 
 4. Open [http://localhost:8000](http://localhost:8000) in your browser.
+
+---
+
+## Development
+
+Install the dev extras (adds ruff, bandit, pytest, httpx on top of the runtime dependencies):
+
+```bash
+pip install -r requirements-dev.txt
+```
+
+**Run the tests:**
+
+```bash
+pytest
+```
+
+**Lint:**
+
+```bash
+ruff check app/ tests/
+```
+
+The lint ruleset is pinned in `pyproject.toml` under `[tool.ruff.lint]` so `ruff check` gives the same result locally and in CI.
+
+**Security scan:**
+
+```bash
+bandit -r app/
+```
+
+All three run automatically on every push and pull request via [`.github/workflows/ci.yml`](.github/workflows/ci.yml), across Python 3.10–3.12.
+
+---
+
+## Request Limits & Timeouts
+
+`/api/match` and `/api/generate` accept arbitrary user-supplied patterns and sample text, so a few limits are enforced server-side (see `app/main.py`):
+
+| Limit | Default | Purpose |
+|---|---|---|
+| Pattern length | 5,000 chars | Bounds pathological input |
+| Custom pattern definitions length | 20,000 chars | Bounds pathological input |
+| Log text length | 200,000 chars | Bounds pathological input |
+| Match/generate timeout | 3s (`MATCH_TIMEOUT_SECONDS` env var) | Fails fast instead of hanging a request |
+
+The timeout bounds *response time*, not CPU usage — Python can't forcibly interrupt a regex match mid-flight, so it's a practical safeguard for trusted/internal use rather than a complete denial-of-service defense. See the comment above `MATCH_TIMEOUT_SECONDS` in `app/main.py` for the full reasoning, including why pygrok's use of the third-party `regex` package (installed automatically as one of pygrok's own dependencies) already mitigates some classic catastrophic-backtracking patterns on its own.
+
+---
+
+## Production Notes
+
+The frontend is intentionally dependency-light for easy self-hosting, with two trade-offs worth knowing about before deploying it beyond a trusted local/internal setting:
+
+- **Tailwind via the CDN script** (`cdn.tailwindcss.com`) is Tailwind's "Play CDN" — convenient for zero-build local use, but Tailwind's own docs note it isn't intended for production (it compiles styles in the browser on every load). For a production build, compile Tailwind via its CLI or a bundler and link the resulting stylesheet instead.
+- **Alpine.js is loaded from a floating version range** (`alpinejs@3.x.x`). Consider pinning it to an exact version (e.g. `alpinejs@3.17.1`) and adding a [Subresource Integrity](https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity) hash, e.g.:
+
+  ```bash
+  curl -s https://cdn.jsdelivr.net/npm/alpinejs@3.17.1/dist/cdn.min.js \
+    | openssl dgst -sha384 -binary | openssl base64 -A
+  ```
+
+  and add `integrity="sha384-<hash>" crossorigin="anonymous"` to the `<script>` tag. This wasn't done automatically here since a wrong hash would break the app entirely (the browser refuses to run a script whose hash doesn't match) — generate and verify it yourself before deploying.
+
+---
+
+## License
+
+GPLv3 — see [LICENSE](LICENSE).

app/main.py

@@ -1,9 +1,15 @@
+import asyncio
+import functools
 import logging
+import os
+from concurrent.futures import ThreadPoolExecutor
 
 from fastapi import FastAPI, HTTPException, Request
+from fastapi.exceptions import RequestValidationError
+from fastapi.responses import JSONResponse
 from fastapi.staticfiles import StaticFiles
 from fastapi.templating import Jinja2Templates
-from pydantic import BaseModel
+from pydantic import BaseModel, Field
 
 from app.config import settings
 from app.grok_engine import GrokDebuggerEngine
@@ -22,6 +28,97 @@
 templates = Jinja2Templates(directory="app/templates")
 app.mount("/static", StaticFiles(directory="app/static"), name="static")
 
+# Request size limits. A Grok/regex pattern or sample log pasted by mistake
+# (or crafted deliberately) can be arbitrarily large; these bound the request
+# body pydantic will accept before it ever reaches the matching engine.
+MAX_PATTERN_LENGTH = 5_000
+MAX_CUSTOM_PATTERNS_LENGTH = 20_000
+MAX_LOG_TEXT_LENGTH = 200_000
+
+# Grok/regex matching is CPU-bound and runs off the event loop with a hard
+# wall-clock budget: a pathological pattern (catastrophic backtracking) could
+# otherwise hang a request - and, since uvicorn's default worker is
+# single-threaded for the event loop, every other in-flight request - forever.
+#
+# Note pygrok prefers the third-party `regex` package over stdlib `re` when
+# it's installed (it's one of pygrok's own dependencies, so in practice it
+# normally is - see `pip show regex`), and `regex` already handles some
+# classically pathological patterns (e.g. `(a+)+` against a run of matching
+# characters) far better than stdlib `re`. This timeout is still worth
+# keeping regardless: it's a single wall-clock budget around the *entire*
+# engine call - not just one regex operation - so it also covers
+# find_partial_match's loop of progressively-longer sub-pattern compiles and
+# pregenerate_pattern's diff-based alignment, neither of which is a single
+# regex op a timeout=... kwarg could bound on its own.
+#
+# What it does NOT do is bound CPU usage: Python cannot forcibly interrupt a
+# C-level regex match mid-flight, so if the underlying call ever is slow
+# enough to hit this timeout, that worker thread keeps running in the
+# background even after asyncio.wait_for gives up on it and returns an error
+# to the caller. Combined with the size limits above, this keeps the API
+# responsive for trusted/internal use; it is not a complete DoS defense for
+# an untrusted multi-tenant deployment.
+_match_executor = ThreadPoolExecutor(max_workers=4, thread_name_prefix="grok-match")
+MATCH_TIMEOUT_SECONDS = float(os.getenv("MATCH_TIMEOUT_SECONDS", "3"))
+
+
+async def _run_with_timeout(func, *args, **kwargs):
+    """
+    Run a blocking engine call off the event loop with a wall-clock timeout.
+
+    Args:
+        func: The blocking callable to run (e.g. engine.execute_match).
+        *args: Positional arguments to pass to func.
+        **kwargs: Keyword arguments to pass to func.
+
+    Returns:
+        The return value of func.
+
+    Raises:
+        ValueError: If func does not complete within MATCH_TIMEOUT_SECONDS.
+    """
+    loop = asyncio.get_running_loop()
+    call = functools.partial(func, *args, **kwargs)
+    try:
+        return await asyncio.wait_for(
+            loop.run_in_executor(_match_executor, call),
+            timeout=MATCH_TIMEOUT_SECONDS
+        )
+    except asyncio.TimeoutError as e:
+        raise ValueError(
+            "Pattern evaluation timed out - the pattern or input is likely too "
+            "complex (possible catastrophic backtracking). Try simplifying the "
+            "pattern or reducing the sample size."
+        ) from e
+
+
+@app.exception_handler(RequestValidationError)
+async def validation_exception_handler(
+    _request: Request, exc: RequestValidationError
+) -> JSONResponse:
+    """
+    Flatten pydantic's default multi-error body into the single-string
+    `{"detail": "..."}` shape the frontend already expects from
+    HTTPException responses (see app.js's triggerMatch/autoGenerate),
+    so a request-size violation displays the same way a 400 does instead
+    of rendering "[object Object]".
+
+    Args:
+        _request: The FastAPI request that failed validation (unused;
+            required by FastAPI's exception handler signature).
+        exc: The validation error raised by pydantic.
+
+    Returns:
+        A 422 JSON response with a single human-readable `detail` string.
+    """
+    first_error = exc.errors()[0] if exc.errors() else {}
+    field = ".".join(str(p) for p in first_error.get("loc", []) if p != "body")
+    message = first_error.get("msg", "Invalid request.")
+    detail = f"{field}: {message}" if field else message
+    logger.warning(f"Request validation error: {detail}")
+    return JSONResponse(status_code=422, content={"detail": detail})
+
+
 class DebugRequest(BaseModel):
     """
     Request model for Grok debugging API endpoints.
@@ -35,9 +132,9 @@ class DebugRequest(BaseModel):
         strict_mode: If True, require a full line match (^...$).
             Otherwise, allow substring matches.
     """
-    pattern: str
-    custom_patterns: str | None = ""
-    log_text: str
+    pattern: str = Field(max_length=MAX_PATTERN_LENGTH)
+    custom_patterns: str | None = Field(default="", max_length=MAX_CUSTOM_PATTERNS_LENGTH)
+    log_text: str = Field(max_length=MAX_LOG_TEXT_LENGTH)
     naming_format: str | None = "dot"
     strict_mode: bool | None = False
 
@@ -84,15 +181,16 @@ async def match_grok(data: DebugRequest):
     try:
         # Explicitly handle empty custom_patterns
         custom_patterns = data.custom_patterns if data.custom_patterns else ""
-        matches = engine.execute_match(
+        matches = await _run_with_timeout(
+            engine.execute_match,
             data.pattern,
             custom_patterns,
             data.log_text,
             strict_mode=data.strict_mode or False
         )
         return {"success": True, "results": matches}
     except ValueError as e:
-        logger.warning(f"Validation error in match_grok: {e}")
+        logger.warning(f"match_grok request rejected: {e}")
         raise HTTPException(status_code=400, detail=str(e)) from e
     except Exception as e:
         logger.exception("Unexpected error in match_grok")
@@ -116,7 +214,8 @@ async def generate_pattern(data: DebugRequest):
         - error: Error message (if failed).
     """
     try:
-        guessed_pattern = engine.pregenerate_pattern(
+        guessed_pattern = await _run_with_timeout(
+            engine.pregenerate_pattern,
             data.log_text,
             data.naming_format or "dot"
         )