Add protocol-neutral language intelligence

.github/workflows/ci.yml

@@ -19,6 +19,9 @@ jobs:
       - name: Install latest stable Rust toolchain
         run: rustup toolchain install stable --profile minimal --component clippy,rustfmt,llvm-tools-preview
 
+      - name: Install WebAssembly target
+        run: rustup target add --toolchain stable wasm32-unknown-unknown
+
       - name: Verify crates.io package
         run: cargo +stable publish --dry-run --locked
 
@@ -46,7 +49,10 @@ jobs:
       - name: Run canonical conformance suite
         env:
           STACK_SPECIFICATION_DIR: .stack-specification
-        run: cargo +stable test --features conformance --test conformance
+        run: cargo +stable test --features conformance --test conformance --test language_intelligence_conformance
+
+      - name: Cross-build language intelligence for WebAssembly
+        run: cargo +stable build --lib --target wasm32-unknown-unknown --locked
 
       - name: Run Clippy
         run: cargo +stable clippy --all-targets --all-features -- -D warnings

Cargo.toml

@@ -25,6 +25,11 @@ name = "conformance"
 path = "tests/conformance.rs"
 required-features = ["conformance"]
 
+[[test]]
+name = "language_intelligence_conformance"
+path = "tests/language_intelligence_conformance.rs"
+required-features = ["conformance"]
+
 [dev-dependencies]
 serde_json = "1.0.151"
 

README.md

@@ -51,11 +51,19 @@ The language-independent schemas and fixtures live in [`stack-sh/specification`]
 Run the canonical suite against a local specification checkout:
 
 ```sh
-STACK_SPECIFICATION_DIR=../specification cargo test --features conformance --test conformance
+STACK_SPECIFICATION_DIR=../specification cargo test --features conformance --test conformance --test language_intelligence_conformance
 ```
 
 JSON support is development-only and does not add a runtime dependency to the compiler library. CI checks out the recorded specification revision before running the suite.
 
+## Language Intelligence
+
+The `language_intelligence` module implements the protocol-neutral [Stack language-intelligence contract](https://github.com/stack-sh/specification/blob/main/LANGUAGE_INTELLIGENCE.md). Its stateless APIs return diagnostics, semantic completion, plain-text hover information, and hierarchical document symbols for one complete source snapshot. Each response echoes the caller's document version.
+
+Completion accepts an explicit, bounded caller-owned icon catalog. The compiler never fetches a catalog or interprets its labels and documentation as markup. Source positions combine a zero-based UTF-8 byte offset with one-based Unicode-scalar line and column coordinates; inconsistent positions are rejected before analysis.
+
+Editor and transport adapters own document synchronization, cancellation, stale-result filtering, and conversion from protocol-specific coordinates such as UTF-16. Formatting remains owned by the canonical formatter. CI compares the native types with the pinned canonical fixtures and cross-builds the same dependency-free core for `wasm32-unknown-unknown`; a JavaScript or LSP binding is intentionally outside this crate.
+
 ## Lossless Source Model
 
 `parse_lossless` and `parse_lossless_bytes` expose every authored token, whitespace segment, line comment, original string spelling, CRLF sequence, and end-exclusive source span. Concatenating token text through `lossless::Document::reconstruct` reproduces the input byte-for-byte. This source-oriented API is separate from normalized IR and performs no filesystem access.
@@ -74,6 +82,7 @@ Lossless parsing succeeds for syntactically valid source even when semantic vali
 - [`docs/decisions/0004-consume-a-pinned-conformance-suite.md`](./docs/decisions/0004-consume-a-pinned-conformance-suite.md)
 - [`docs/decisions/0005-add-a-lossless-source-model.md`](./docs/decisions/0005-add-a-lossless-source-model.md)
 - [`docs/decisions/0006-add-a-source-map-sidecar.md`](./docs/decisions/0006-add-a-source-map-sidecar.md)
+- [`docs/decisions/0007-add-protocol-neutral-language-intelligence.md`](./docs/decisions/0007-add-protocol-neutral-language-intelligence.md)
 - [`docs/specs/compiler-frontend.md`](./docs/specs/compiler-frontend.md)
 
 ## License

docs/decisions/0007-add-protocol-neutral-language-intelligence.md

@@ -0,0 +1,69 @@
+# ADR-0007: Add Protocol-Neutral Language Intelligence
+
+## Status
+
+Accepted
+
+## Date
+
+2026-09-05
+
+## Context
+
+Native editors, browser editors, and future hosted tools need the same Stack diagnostics, completion, hover, and document-symbol semantics. The canonical specification defines versioned request and response data, but an editor transport such as the Language Server Protocol also introduces document synchronization, negotiated position encodings, request cancellation, and stale-result handling.
+
+Putting those transport concerns inside the compiler would make the pure frontend stateful and couple portable language semantics to one client protocol. Conversely, implementing semantic queries independently in each adapter would duplicate parsing, defaults, identifier resolution, completion ordering, and source-range behavior.
+
+Completion also needs icon identifiers that are not part of the Stack grammar. The compiler cannot discover them without taking a dependency on a theme catalog, filesystem, network, or application state.
+
+## Decision
+
+Add a public `language_intelligence` module that implements the compiler-owned parts of the canonical language-intelligence 1.0 contract:
+
+- diagnostics for UTF-8 text or bytes;
+- syntax- and scope-aware completion;
+- semantic hover with reference resolution;
+- hierarchical document symbols.
+
+Every operation is stateless and analyzes one complete source snapshot. Responses echo an opaque caller-supplied `document_version` but the compiler does not retain or order versions. Completion and hover validate that the supplied zero-based UTF-8 byte offset and one-based Unicode-scalar line and column identify the same scalar boundary.
+
+Completion accepts an explicit request-local icon catalog. The compiler bounds the number and length of entries, rejects invalid or duplicate identifiers, treats all descriptive fields as plain text, and never loads catalog data itself. Language values and document identifiers come only from canonical syntax and the parsed source.
+
+The module exposes dependency-free Rust domain types rather than JSON or protocol-specific structures. A feature-gated integration test maps those types to the exact canonical fixture responses using development-only JSON support. CI also cross-builds the same library for `wasm32-unknown-unknown`; WebAssembly bindings do not fork the semantic implementation.
+
+Canonical formatting remains a formatter responsibility. LSP, JavaScript, and other adapters own serialization, position-encoding conversion, document lifecycle and incremental state, cancellation, stale-result filtering, and presentation mapping.
+
+## Alternatives Considered
+
+### Implement an LSP server in the compiler crate
+
+- Pros: Editors could consume one ready-made process.
+- Cons: Makes a deterministic library own transport, process, document-state, and protocol-version concerns.
+- Rejected: An LSP adapter should translate to the portable compiler API rather than define its semantics.
+
+### Add JSON serialization to the runtime API
+
+- Pros: WebAssembly and process adapters could forward values directly.
+- Cons: Adds a runtime dependency and prematurely fixes a transport representation for every Rust consumer.
+- Rejected: A development-only fixture adapter proves compatibility while public native types remain transport-neutral.
+
+### Let callers provide completion callbacks
+
+- Pros: Catalog discovery could be lazy and application-specific.
+- Cons: Introduces runtime behavior, nondeterminism, and target-specific callback boundaries into the core.
+- Rejected: A bounded immutable catalog keeps each request deterministic and portable.
+
+### Implement language intelligence separately in every consumer
+
+- Pros: Each integration can optimize for its local editor framework.
+- Cons: Diagnostics, resolution, ordering, and ranges would drift between native and browser products.
+- Rejected: All consumers must share one compiler-owned semantic implementation.
+
+## Consequences
+
+- Native and WebAssembly consumers can share deterministic language semantics.
+- The compiler remains dependency-free, stateless, and free of filesystem or network access.
+- Callers must supply a complete source snapshot, exact source coordinates, a document version, and any catalog entries needed for completion.
+- Protocol adapters remain responsible for lifecycle correctness and coordinate conversion.
+- Format responses are composed with the canonical formatter rather than this compiler module.
+- Public Rust types may evolve before the crate's first stable release, but fixture compatibility remains pinned to an exact specification revision.

docs/specs/compiler-frontend.md

@@ -35,6 +35,8 @@ src/source_map.rs   Engine-facing source origins by semantic identity
 src/validation.rs   Semantic validation and normalization
 src/validation/     Focused validation unit tests
 src/ir.rs           Renderer-independent normalized model
+src/language_intelligence.rs
+                    Stateless diagnostics, completion, hover, and symbols
 src/lib.rs          Public parse and compile APIs
 tests/              Public API and conformance-oriented tests
 tests/specification-revision
@@ -53,6 +55,21 @@ pub fn compile(source: &str) -> CompileOutput;
 pub fn compile_bytes(source: &[u8]) -> CompileOutput;
 pub fn compile_with_source_map(source: &str) -> SourceMappedCompileOutput;
 pub fn compile_bytes_with_source_map(source: &[u8]) -> SourceMappedCompileOutput;
+
+pub fn diagnostics(source: &str, document_version: u64) -> DiagnosticsOutput;
+pub fn diagnostics_bytes(source: &[u8], document_version: u64) -> DiagnosticsOutput;
+pub fn completion(
+    source: &str,
+    document_version: u64,
+    position: SourcePosition,
+    catalog: &CompletionCatalog,
+) -> Result<CompletionOutput, IntelligenceError>;
+pub fn hover(
+    source: &str,
+    document_version: u64,
+    position: SourcePosition,
+) -> Result<HoverOutput, IntelligenceError>;
+pub fn document_symbols(source: &str, document_version: u64) -> DocumentSymbolsOutput;
 ```
 
 `ParseOutput` contains an AST only when decoding, lexing, and parsing succeed. `CompileOutput` contains normalized IR only when all compiler-stage errors are absent. Both outputs contain diagnostics. Semantic warnings may accompany successful IR.
@@ -61,6 +78,12 @@ pub fn compile_bytes_with_source_map(source: &[u8]) -> SourceMappedCompileOutput
 
 `SourceMappedCompileOutput` pairs successful normalized IR with a Rust-only source map. The sidecar identifies authored theme and icon values and complete layout order statements, while explicitly distinguishing omitted defaults. Compiler-stage errors suppress both IR and the source map. The normalized IR types and canonical JSON schema contain no source-map fields.
 
+The `language_intelligence` module implements portable editor semantics without adopting an editor protocol. Calls analyze one complete immutable source snapshot and echo its caller-owned document version. Diagnostics are always returned. Completion may recover useful candidates from incomplete syntax; hover and document symbols are empty when parsing cannot establish a trustworthy construct. Syntactically valid documents may still return hover and symbol structure alongside semantic errors.
+
+Completion uses a request-local `CompletionCatalog`; the compiler validates its size, identifiers, and plain-text fields but performs no catalog discovery. Request positions must be UTF-8 scalar boundaries whose byte offset, line, and Unicode-scalar column agree. Protocol adapters own position-encoding conversion, incremental document state, cancellation, and stale-result filtering.
+
+Formatting is absent from this module because canonical source edits belong to the formatter. Runtime serialization and LSP or JavaScript bindings also remain adapter responsibilities. The dependency-free library cross-builds to `wasm32-unknown-unknown`, so native and WebAssembly consumers execute the same semantic core.
+
 ## Code Style
 
 Prefer explicit domain types and exhaustive matches:
@@ -84,6 +107,8 @@ match operator {
 - Unit tests cover lexer escapes, positions, parser productions, defaults, and each implemented diagnostic.
 - Integration tests exercise public `parse` and `compile` APIs.
 - Feature-gated integration tests execute the canonical conformance suite from the recorded specification revision.
+- Language-intelligence conformance maps native types to the canonical JSON fixtures in a development-only adapter and checks every compiler-owned operation exactly.
+- CI cross-builds the library for `wasm32-unknown-unknown` to keep language-intelligence semantics portable without adding a target-specific runtime API.
 - Invalid cases assert stable diagnostic codes rather than entire prose messages.
 - Library unit-test line, function, and region coverage must each remain at or above 95 percent.
 - Every compiler change must pass formatting, tests, coverage, Clippy, and documentation builds.
@@ -129,4 +154,5 @@ match operator {
 - Theme and icon resolution
 - Layout and renderer integration
 - Multi-error syntax recovery
+- LSP, JavaScript, and other protocol or runtime bindings
 - Native Rust API stabilization and package versioning