Repository navigation
Conversation
Implement a SPATIAL flag for use in creating a spatial field.
Encodes triangles with the bounding box in the first four words, enabling efficient spatial pruning during tree traversal without reconstructing the full triangle. The remaining words contain an additional vertex and packed reconstruction metadata, allowing exact triangle recovery when needed.
The `triangulate` function takes a polygon with floating-point lat/lon coordinates, converts to integer coordinates with millimeter precision (using 2^32 scaling), performs constrained Delaunay triangulation, and encodes the resulting triangles with boundary edge information for block kd-tree spatial indexing. It handles polygons with holes correctly, preserving which triangle edges lie on the original polygon boundaries versus internal tessellation edges.
Implemented byte-wise histogram selection to find median values without comparisons, enabling efficient partitioning of spatial data during block kd-tree construction. Processes values through multiple passes, building histograms for each byte position after a common prefix, avoiding the need to sort or compare elements directly.
Implemented a `Surveyor` that will evaluate the bounding boxes of a set of triangles and determine the dimension with the maximum spread and the shared prefix for the values of dimension with the maximum spread.
Implements dimension-major bit-packing with zigzag encoding for signed i32 deltas, enabling compression of spatially-clustered triangles from 32-bit coordinates down to 4-19 bits per delta depending on spatial extent.
Implement an immutable bulk-loaded spatial index using recursive median partitioning on bounding box dimensions. Each leaf stores up to 512 triangles with delta-compressed coordinates and doc IDs. The tree provides three query types (intersects, within, contains) that use exact integer arithmetic for geometric predicates and accumulate results in bit sets for efficient deduplication across leaves. The serialized format stores compressed leaf pages followed by the tree structure (leaf and branch nodes), enabling zero-copy access through memory-mapped segments without upfront decompression.
Lossless compression for floating-point lat/lon coordinates using XOR delta encoding on IEEE 754 bit patterns with variable-length integer encoding. Designed for per-polygon random access in the document store, where each polygon compresses independently without requiring sequential decompression.
The triangulation function in `triangle.rs` is now called `delaunay_to_triangles` and it accepts the output of a Delaunay triangulation from `i_triangle` and not a GeoRust multi-polygon. The translation of user polygons to `i_triangle` polygons and subsequent triangulation will take place outside of `triangle.rs`.
Implemented a geometry document field with a minimal `Geometry` enum. Now able to add that Geometry from GeoJSON parsed from a JSON document. Geometry is triangulated if it is a polygon, otherwise it is correctly encoded as a degenerate triangle if it is a point or a line string. Write accumulated triangles to a block kd-tree on commit. Serialize the original `f64` polygon for retrieval from search. Created a query method for intersection. Query against the memory mapped block kd-tree. Return hits and original `f64` polygon. Implemented a merge of one or more block kd-trees from one or more segments during merge. Updated the block kd-tree to write to a Tantivy `WritePtr` instead of more generic Rust I/O.
Ended up using `select_nth_unstable_by_key` from the Rust standard library instead.
Read node structures using `from_le_bytes` instead of casting memory. After an inspection of columnar storage, it appears that this is the standard practice in Rust and in the Tantivy code base. Left the structure alignment for now in case it tends to align with cache boundaries.
Addressed all `todo!()` markers created when adding `Spatial` field type and `Geometry` value type to existing code paths: - Dynamic field handling: `Geometry` not supported in dynamic JSON fields, return `unimplemented!()` consistently with other complex types. - Fast field writer: Panic if geometry routed incorrectly (internal error.) - `OwnedValue` serialization: Implement `Geometry` to GeoJSON serialization and reference-to-owned conversion. - Field type: Return `None` for `get_index_record_option()` since spatial fields use BKD trees, not inverted index. - Space usage tracking: Add spatial field to `SegmentSpaceUsage` with proper integration through `SegmentReader`. - Spatial query explain: Implement `explain()` method following pattern of other binary/constant-score queries. Fixed `MultiPolygon` deserialization bug: count total points across all rings, not number of rings. Added clippy expects for legitimate too_many_arguments cases in geometric predicates.
This reverts commit 19eab16. Restore radix select in order to implement a merge solution that will not require a temporary file.
Existing code behaved as if the result of `select_nth_unstable_by_key` was either a sorted array or the product of an algorithm that gathered partition values as in the Dutch national flag problem. The existing code was written knowing that the former isn't true and the latter isn't advertised. Knowing, but not remembering. Quite the oversight.
Removed use of robust.
Change from all caps to the Rust camel case convention for enum variants.
The return value is already a copy so the clone is unnecessary.
Resolve errors encountered when a segement lacks geo data when merging and searching.
Within search only tests whether the triangle bounding box is within the search bounding box. If the triangle bounding box is not within the search bounding box then at least one line is outside the bounding box and a vertex of the triangle is on that line.
Create a mechanism for configuring the index type. Will be used to select between planar and unit sphere.
This was a new method added after the fork point. We can resolve a rebase or main merge when we are closer to landing the pull request.
Bit-packing will compress better. We are going to need compressed f64 for both the document store and the index.
Moved plane.rs to plane/ directory. Added plane/math.rs with 2D predicates: orient2d (via robust crate), segment crossing, point-in-polygon, point-to-segment distance, segment-to-segment distance, edge length. Implemented origin_inside for Plane as unconditionally false.
Added const DIMENSIONS to the Surface trait. EdgeWriter<S: Surface> and EdgeReader<S: Surface> use S::DIMENSIONS to determine vertex byte size. The writer writes S::DIMENSIONS f64 values per vertex. The reader decodes the same count. All call sites specify Sphere for now.
Spatial writers recorded global stream offsets instead of field-relative offsets. With two spatial fields the second field's directory points past its slice. Capture the base offset per field and subtract. Point geometries panicked during merge because vertices[edge_idx + 1] assumes a next vertex. Fall back to v0 for the degenerate edge with an assertion that edge_idx is zero.
read_set returns a cloned GeometrySet instead of decomposed vecs. The merger no longer reconstructs EdgeSets with empty ring_offsets and contains_hilbert_start: false. Polygons with holes survive merge with their ring boundaries intact. The third point geometry guard in collapse_siblings closes the last occurrence of the vertices[edge_idx + 1] panic from the previous commit.
EdgeReader is pure deserialization. All methods are &self. No cache, no HashMap, no LRU. Returns owned GeometrySets on every call. EdgeCache wraps one or more EdgeReaders with an LRU cache keyed by (segment, position). The query path uses segment 0 with a single reader. The merge path uses real segment indices with a shared cache and one eviction budget across all source segments.
ClippedShape carries GeometryId = (u16, u32) where the first element is the segment index and the second is the geometry position. The cell index reader fills in (0, position) on read. The cell index writer extracts the position on write. The merge stamps the real segment index. The query path passes zeros. One type everywhere. No generic. Two extra bytes per shape.
Shapes carry their source identity through the pipeline. The merge stamps the segment index onto source cells and never remaps. The caller assigns new IDs at emission time in Hilbert encounter order, writes edges first, then writes cells with the new IDs substituted.
The old interleave eagerly subdivided coarse cells when they overlapped finer cells, creating many interior-only cells that the downstream stages then recombined. The new interleave holds the coarse cell as a sponge and absorbs finer cells into it. Subdivision only fires when the combined edge count exceeds the threshold. The collapse is a new stage that groups cells by parent in Hilbert order and promotes groups up the level hierarchy when the short edge count allows. It is generic over its inner iterator. CellIndexMerge is now generic over the group iterator. The edge cache is shared across all three pipeline stages through a RefCell rather than owned by any one stage.
The geometry set on each HeapEntry is extended when cells are absorbed or distributed during splits, replacing the full recount that ran on every absorption.
EdgeCache now uses RefCell internally so callers take &EdgeCache instead of &mut EdgeCache. The cache returns GeometryEntry which holds an Rc<GeometrySet> and knows its member index. The Rc keeps vertex data alive across LRU eviction, eliminating the vertex clone at every call site. GeometryEntry.edge(index) returns the two endpoints of an edge, replacing the repeated v0/v1 pattern with bounds check across interleave, collapse, and merge.
CoarseEdge stores the edge endpoints alongside edge_index and max_level. subdivide_cell reads from CoarseEdge instead of the edge cache. flatten builds its edge unions from CoarseEdge vertices. Neither function accesses the edge cache.
The indexer asks for the segment writer's memory usage after every document, and SpatialWriter summed every accumulated geometry set on each call, so indexing a segment of n geometries cost O(n^2). Over 1.6M points with a 1 GB writer budget, throughput fell from 36k to 3k docs/s per 400k documents; with the running total it holds at 31k. The estimate now counts the heap each set holds (the set, its members, their vertex and ring offset arrays) instead of 24 bytes per vertex. That undercounted a point by about 12x, at 24 bytes against the 296 it holds, and let segments overshoot the budget.
The merger opens a segment with no geometry for a spatial field as empty bytes, and open read its footer from data[n - 8..n], so merging any such segment panicked the merge thread with "attempt to subtract with overflow" and the merge failed. EdgeReader::open already guards against short input the same way.
TopDocs is no longer a Collector by itself, so the example, and with it cargo test, did not compile.
SpatialWriter::mem_usage now counts the spare capacity of the arrays that hold the sets, which it missed by up to 10% for points, and to_geometry_set allocates every array at its exact length, so a point holds 128 bytes instead of 296. Counting one set moves to GeometrySet::mem_usage, with element sizes taken from each array's type. CellIndexReader::open and EdgeReader::open fail with DataCorruption when the footer does not locate a directory that ends where the footer begins, and get_field fails for a field that has a cell index but no edge index, or the reverse. Before, an index shorter than its footer read as empty, so a merge dropped that segment's geometry for good. The merger opens a segment without geometry for the field with the new empty() readers instead. The merger leaves a field that has no geometry in any segment out of the composites, as the segment writer does, builds the inverted doc id mapping only for segments with geometry, and logs its spatial metrics at debug level rather than printing them to stderr on every merge. Both cell index writers share write_directory, and the clippy warnings in the touched code are gone. The merge tests now check which document the geometry lands on, merging the segments in commit order, and that a field never written stays absent after the merge.
ClosestEdgeQuery missed geometries. A point query never measured its distance to the edges in a stored cell that its own cell overlapped, a polygon containing part of the query was not at distance zero, and a stored edge was only checked against the query edges in overlapping query cells, which need not hold the closest one. It now checks each stored edge against every query cell whose bounding cap could be closer than the best distance so far, counts the interiors of both geometries as distance zero, and walks the cell index with CellIndexReader::locate instead of decoding every cell under each queue cell. Within and kNN over 100k points went from about 8 ms to 30 µs per query, and the never_loop clippy error is gone. Between joins no longer stop at the first geometry found, which may lie inside the inner distance, and kNN with k = 0 no longer panics. SpatialQuery's kNN runs through the executor, so K is applied across segments rather than per segment, and hits are scored by distance. The executor returns an error instead of panicking when it has no searcher, and explains only matching documents, with their score. Contains on a multi-geometry requires every member to be contained, not just one, and segment space usage counts the edge index along with the cell index. A randomized test checks Within, Between and kNN, from points and from other geometries, against brute force distances.
The cell index stores shape counts, edge counts and edge ids as variable length integers, and edge ids are 32 bits wide, so a cell holding more than 65535 geometries and a polygon with more than 65535 vertices no longer overflow and corrupt the index. The edge index directory records the head of the set each skip block starts in, so finding a geometry's document reads at most two skip blocks instead of walking back over every member of a large multi-geometry. Both formats change, so spatial segments must be re-indexed. The edge and cell writers and the merger return I/O errors instead of panicking, and the merge edge cache holds at most 2M vertices instead of 100M. Contains and intersects use every member of a multi-geometry query, give lines and points no interior, and test candidate edges against the edges of every query index cell a covering cell overlaps, not just one, which missed crossings for query polygons with more than a few dozen vertices. Where boundaries meet only at shared vertices or along shared edges, the touching edges' midpoints and points beside shared edges decide instead of an ambiguous shared vertex: contains rejects a polygon that leaves the query through a shared vertex or surrounds a hole of the query, and intersects matches identical polygons but not ones that only share boundary. A document's geometry values make one geometry set, so contains needs all of them. The merge no longer marks a line as containing a cell's center, and GeoJSON GeometryCollection parses. Every executor stage leaves out deleted documents. A join opens each probed segment once, builds the probe query once per outer document, stops probing at the first match, and does not match a document to itself. kNN skips deleted documents before ranking, applies its boost to the score it explains, and keeps the K nearest in a heap instead of sorting all distances on every improvement. Empty distance ranges match nothing, and polygons overlapping a query point no longer decode every shape in the cell once per polygon. In the query language, a join keeps its own occur and its siblings'. The siblings only narrow the join's outer documents, so NOT and OR clauses next to a join keep their meaning. Errors in a $query(...) are the query's errors, a kNN join is rejected, a spatial predicate needs enough coordinates, and the Debug output of a join no longer panics. The unused collapse, merge and delta modules and the i_triangle dependency are gone.
PartialOrd for S1ChordAngle defers to Ord, which changes no comparison since no constructor yields a NaN length. S2CellId uses Range::contains in its debug assertions and drops casts of LOOKUP_BITS to the type it already has.
Where a candidate and the query share an edge, the interior sides of the shared edges decide whether the candidate's interior lies on the query's side, instead of testing points beside the edge, which a polygon thinner than their offset missed. A contains query of several polygons verifies candidates against the exposed boundary of their union (polygon_union.rs), so that a candidate lying across the members' shared edges, inside the union, is contained, and one around a hole the members enclose together is not. Appending a value to a document's geometry set no longer re-sums the whole set, which made indexing a document of n values cost O(n^2). The memory usage adds the incoming members' buffers and the growth of the set's member array, spare capacity included.
…queries. Polygons are made valid as ST_MakeValid does. Repeated vertices are removed and unclosed rings closed. A spike leaves its ring and stays as a line member, a ring whose vertices all lie on one great circle bounds no area, and a polygon with no area is the point or line it is. Rings bound the smaller region whichever way they run, by curvature with Kahan summation, and a ring close to a hemisphere takes its smaller side by vote, so that a self-crossing ring bounds its lobes. A field set_oriented() keeps RFC 7946's right-hand rule instead, so polygons larger than a hemisphere stay expressible; a query can choose with SpatialQuery::with_ring_orientation. Positions may carry an altitude, which is dropped, and both GeometryCollection spellings parse. The cell index builder emits cells on a face its shapes cover without edges, and its exit pass tests only edges of shapes with interiors. The merge clips edges to the cell's face with the builder's clipping and exact bounds, instead of projecting endpoints off the face. Spatial files end with a format version and magic number, so an index of another format fails to open with an error that says to re-index, and corrupt counts can no longer size huge allocations. Points and lines intersect what they meet: shared vertices and edges, a point on an edge, a line along a polygon's boundary. Where a vertex lies inside another edge, contains and intersects split the edges and decide by their pieces, as the union path does. Contains checks every ring of the query, so polygons larger than a hemisphere get no false positives. $contains reads from the field's side, as its join does: SpatialPredicate::Contains is now "contains the polygon", CoveredBy is "inside it" and has the syntax $within(lon lat, ...). The region coverer refines the coarsest cells first, as S2 does. kNN ties break by document address, scores fall with rank so that a kNN clause lifts the nearest documents, and without a searcher each segment keeps its own K. The query parser narrows a kNN or join by the clauses every match must satisfy, however they are boosted or nested, and by its optional siblings when nothing is required. The lenient parser reads spatial predicates, and skips a malformed one with an error. Query coordinates must be finite and in range, explain no longer seeks backward, scorers seek directly, and geometry values in fast or JSON fields are skipped instead of panicking. Edge caches budget bytes of heap rather than vertices, and grow to four times their largest set so that large sets sharing cells do not evict each other; join probes share the query's budget. Deprecated aliases keep set_outer and SegmentSpaceUsage::spatial, and the old space usage JSON still reads. The spatial query syntax is documented on QueryParser.
…ystem. Upstream turned the segment components into plugins (quickwit-oss#2993): postings, fast fields and the store are SegmentPlugins, and a segment's files are SegmentComponent::Custom extensions. Spatial fields join them as a built-in plugin. SpatialPlugin owns the "cells" and "edges" files: its writer takes each document's geometry, remaps document ids when the segment is sorted, and writes both composites at flush; its merge interleaves the source cells as the merger did; and it reports the space usage of both files. Segments of a schema with no spatial field have neither. The Geometry value keeps its own variants next to upstream's Custom ones, moving its in-memory type code from 13 to 14.
…ecoding them. SpatialQuery::covered_by_rect and intersects_rect find what a rectangle of longitude and latitude covers, or meets: its sides are the meridians and the parallels through its corners, as a GeoJSON bounding box has them. from_bounds makes the polygon through the corners, whose geodesic edges bow toward the nearer pole, by nearly 8 degrees at the middle of an edge 90 degrees long at 60 north, and run the short way around between corners more than 180 degrees apart. A rectangle may be of any width, cross the antimeridian, reach a pole, which has every longitude, or be the whole sphere. It is closed, and a vertex within rounding of its boundary is inside it. A point is tested by its height and against the planes of the rectangle's two meridians, with no trigonometry. An edge is tested by where its great circle is highest and lowest between its ends, and a polygon also by the points the rectangle leaves out, or by a point of the rectangle within it. S2LatLngRect gains S2's BoundaryIntersects for the edges that cross its sides. A covering finds the candidates, and a covering cell within the rectangle decides what it holds without decoding it: a geometry found only in such cells lies within the rectangle, since it is connected. A point is read from its entry in the edge index, by EdgeReader::summary, and cells are walked without decoding their edge indices, by CellIndexReader::visit_range. SpatialPredicate::Within searches the cap around its point the same way, and decides as the closest edge query did, which still serves distances between and nearest neighbors. Over three million random points, a box 170 degrees wide took 2.4 s as a polygon and takes 33 ms as a rectangle, and a 2000 km radius that holds 75 thousand takes 3.6 ms, from 53. Prepared queries add their documents to a bit set, rather than list them.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a spatial field type, indexed on the sphere with S2 cells, and the queries over it. This builds on @flatheadmill's geo work (see #2729, #2755): it is his fork's
main, merged with current upstreammain, plus a series of correctness, robustness and performance fixes on top.What it adds
SPATIALfield option, aGeometryvalue, and GeoJSON ingest for points, lines, polygons (with holes), multi-geometries andGeometryCollection. Polygons are made valid on ingest (asST_MakeValiddoes); a field can opt into RFC 7946 ring orientation withset_oriented()so polygons larger than a hemisphere stay expressible.src/spatial/— an S2 cell index plus a streaming edge index per segment, written as thecellsandedgesfiles. Spatial fields are a built-inSegmentPlugin(following feat: Extensible segment components via plugin trait #2993), so segments of a schema with no spatial field have neither file. Merges interleave the source cell indexes.SpatialQuery): intersects, contains, covered-by, within distance, between distances, kNN, latitude-longitude rectangles (covered_by_rect,intersects_rect), and spatial joins through an executor.$within,$between,$contains, …) and$query(...)for joins, documented onQueryParser.examples/geo_json.rs.Changes on top of the original work
mainand ported spatial storage onto the segment plugin system.SpatialWriter, and appending values to a document's geometry set is no longer O(n²).Notes for review
robust,num-bigint,num-rational,num-traits(exact predicates).Geometryvalue's in-memory type code moves from 13 to 14 to sit next to upstream'sCustomvariants.src/spatial/). The changes outsidesrc/spatial/andsrc/query/spatial_query.rsare small and are the best place to start.