ast: index a lookup into a collection in base data

CHANGELOG.md

@@ -106,29 +106,29 @@ collection for emptiness without asserting its type. Sets are unaffected here:
 `set[string]` describes any set of strings, the empty one included, so
 `{"foo"} == set()` still compiles.
 
-### Rule index candidates are returned in declaration order
-
-The rule index returned a ruleset's definitions in whatever order its trie happened to
-reach them. It now returns them in the order they were declared, which is what it
-documented but did not do. Numbering the rules once, rather than sorting the candidates
-of every lookup, also makes building an index and reading a lookup's result
-cheaper.
-
-Two things follow from the order. A `complete rules must not produce multiple outputs`
-error now points at the first of the conflicting definitions rather than the second:
-
-```rego
-package example
-
-p := 1 if input.x # reported here now
-
-p := 2 if input.y # reported here before
-```
-
-And partial evaluation numbers the local variables of its support rules in evaluation
-order, so `opa eval --partial` and `opa build --optimize` emit the same rules under
-different generated names, and in a different order. What a policy evaluates to is
-unaffected either way.
+### Rule indexing improvements
+
+The rule indexer now excludes rules from more kinds of expression, and builds a smaller
+trie to do it with. See [Use indexed statements](https://www.openpolicyagent.org/docs/policy-performance#use-indexed-statements)
+for what is indexed.
+
+- `startswith`, `endswith`, `strings.any_prefix_match` and `strings.any_suffix_match`
+  are indexed when the base strings are known at compile time.
+- A reference that reads a key out of the object at its ground prefix in base data
+  (`data.groups.admins.members[input.subject]`) is indexed by asking that object for the
+  key, where such a ruleset used to leave every rule a candidate.
+- References rooted at a local variable (`x := input; x.foo == "a"`) are indexed the
+  same as `input.foo == "a"`, and a chain of assignments no longer drops the constraint
+  at the end of it.
+- A rule's path through the trie stops at the last level it constrains, and a reference
+  reached by several values no longer leaves the rest of the rule unindexed.
+- Candidates come back in declaration order, which the indexer documented but did not
+  do. A `complete rules must not produce multiple outputs` error now points at the first
+  of the conflicting definitions rather than the second, and partial evaluation names
+  and orders the generated locals of its support rules differently. What a policy
+  evaluates to is unaffected.
+
+Authored by @srenatus and @tsandall
 
 ### Behavior change: `semver.is_valid` and `semver.compare` reject versions the SemVer 2.0.0 spec forbids
 

docs/docs/policy-performance.md

@@ -203,6 +203,25 @@ A bare reference used as a boolean check (without an explicit comparison) is als
 | `input.x.y`  | yes     |                               |
 | `input.x[i]` | no      | reference contains a variable |
 
+A reference into base data whose last element is the only variable in it is a lookup in the collection at the ground prefix, and is indexed on that last element:
+
+```rego
+allow if data.groups.admins.members[input.subject]
+```
+
+The collection's keys are not known when the index is built, so the index asks the collection as the lookup runs — one hash lookup, whatever the collection holds, and nothing stored per member. Reading it then, rather than at build time, also means a collection whose data has since changed, or that a `with` statement has replaced, is seen as it is now.
+
+Only an object is indexed this way. A set would answer the same question, but base data comes from JSON and holds none; the only thing an array can be asked for is a position, which tells a ruleset's rules apart only where their lengths differ. Nor is `input.subject in data.groups.admins.members` indexed — `in` asks after the collection's _values_, which is a walk for either.
+
+The collection is only asked about where every candidate is going to be evaluated anyway. A ruleset whose definitions all produce the same value lets evaluation stop at the first that holds, and asking about all of them to exclude any would be the work evaluation was about to do. Partial evaluation always asks, since its residual is built from every candidate, and so does a ruleset that cannot stop early — a partial set or object, or definitions that disagree on the value.
+
+| Expression                                    | Indexed | Notes                          |
+| --------------------------------------------- | ------- | ------------------------------ |
+| `data.groups.devs.members[input.subject]`     | yes     | asks the collection for a key  |
+| `data.groups[input.g].members[input.subject]` | no      | prefix contains a variable     |
+| `data.groups.devs.members[input.subject[i]]`  | no      | key contains a variable        |
+| `input.groups.devs.members[input.subject]`    | no      | collection is not in base data |
+
 #### Logical (`and`/`or`) statements
 
 Statements joined by the [`and` and `or` keywords](./policy-reference/keywords/logical) are indexed on the indexable statements found inside their operands, following the rules above. An `and` requires both of its operands, so each is indexed on its own and an operand with nothing indexable in it still leaves the other to narrow the rule. An `or` requires only one of its operands, so it is indexed only when every operand has something indexable: an operand that doesn't could be satisfied by any input, leaving nothing the rule can be excluded on. Combining the two multiplies the ways a rule can be reached, as in `{input.a == 1 or input.a == 2} and {input.b == 1 or input.b == 2}`, which has four; past 32 for a single rule only the conditions common to every combination are indexed.