Skip to content

Take the read lock on search/scan/ratify/checkSimilarity (GH-9) - #14

Open
chiragmangaldev3112 wants to merge 1 commit into
PhonePe:mainfrom
chiragmangaldev3112:fix/gh-9-search-read-lock
Open

chiragmangaldev3112 wants to merge 1 commit into
PhonePe:mainfrom
chiragmangaldev3112:fix/gh-9-search-read-lock

Conversation

@chiragmangaldev3112

Copy link
Copy Markdown

Summary

IndexingFacade stores only the write half of each index's ReentrantReadWriteLock (getLock() discards everything but .writeLock()). add/update/delete/replace correctly serialize against each other through it, but search(), scan(), ratify() and checkSimilarity() read the same IndexGroup with no lock at all, exactly as the issue describes.

The inverted index's nested structures are not thread-safe below the outer ConcurrentHashMap (InvertedIndex.table's nested Map/TreeMap, IndexGroup.allCriterias, idCache's PriorityQueue), and add/delete mutate them in place. A search running concurrently with a mutation can observe a torn structure.

Reproduction

A steady-state stress test: a writer thread repeatedly deletes and re-adds the same 30 criteria while 8 reader threads search the same index. On the current code, reader threads reliably end up stuck inside TreeMap.getHigherEntry/TreeSet.higher (DNFMatcher.getNextHigherId), iterating a structure being concurrently rebuilt out from under them — confirmed with real thread dumps, not just an assumption.

Fix

  • indexLocks now stores the full ReentrantReadWriteLock instead of only its write half. All eight existing write-side call sites explicitly take .writeLock(); nothing there changes behaviour.
  • IndexingFacade.readSecurely(index, block) takes the read lock for the block's whole duration, not just an initial getIndexGroup() lookup — the traversal that follows is what actually races.
  • SearchFacade.search() and MustangEngine's scan()/checkSimilarity() wrap their full body in readSecurely().
  • MustangEngine.ratify() does the same. AsyncProcessor dispatches through Guava's EventBus, which is synchronous by default, so the ratification handler's read of the IndexGroup runs on the calling thread, inside the read lock, before process() returns.
  • Utils.executeSecurely(Lock, ...) is widened from ReentrantReadWriteLock.WriteLock to the Lock interface so the same helper serializes both sides.

Left out of scope, matching what's actually broken here:

  • getRatificationResult() stays a plain, unlocked field read — it's a single reference read, not a traversal of the racy structures above.
  • replace(oldIndex, newIndex) still only takes oldIndex's lock despite reading/mutating newIndex's entry too — the pre-existing gap you flagged separately in the issue. Changing its locking order risks a new deadlock between concurrent replace() pairs and isn't part of what Search/read path is unsynchronized against concurrent index mutations (add/update/delete/replace) #9 reported; happy to take that on separately if you'd like.

Test plan

  • Added IndexingFacadeConcurrencyTest: wall-clock-bounded (not iteration-bounded, so it's not hardware-speed-dependent), with a progress counter so a real hang is distinguishable from the test merely running slower than its shutdown grace period.
  • Reliably reproduces the bug on the prior code (exception, or reader threads stuck mid-TreeMap traversal) — verified via real thread dumps, run twice.
  • Passes cleanly against this fix — run twice.
  • mvn clean verify (full suite, all modules) passes — run twice.

Fixes #9

IndexingFacade stored only the write half of each index's
ReentrantReadWriteLock (getLock() discarded everything but
writeLock()). add/update/delete/replace correctly serialize against
each other through it, but search(), scan(), ratify() and
checkSimilarity() read the same IndexGroup with no lock at all.

The inverted index's nested structures are not thread-safe below the
outer ConcurrentHashMap (InvertedIndex.table's nested Map/TreeMap,
IndexGroup.allCriterias, idCache's PriorityQueue), and are mutated in
place by add/delete. A search running concurrently with a mutation can
observe a torn structure.

Reproduced with a steady-state stress test: a writer thread repeatedly
deleting and re-adding the same 30 criteria while 8 reader threads
search the same index. On this code, reader threads reliably end up
stuck inside TreeMap.getHigherEntry/TreeSet.higher (DNFMatcher's
getNextHigherId), iterating a structure being concurrently rebuilt out
from under them.

- indexLocks now stores the full ReentrantReadWriteLock instead of
  only its write half. All eight existing write-side call sites
  explicitly take .writeLock(); nothing there changes behaviour.
- IndexingFacade.readSecurely(index, block) takes the read lock for the
  block's whole duration, not just an initial getIndexGroup() lookup.
- SearchFacade.search() and MustangEngine's scan()/checkSimilarity()
  wrap their full body in readSecurely(), since the read spans more
  than the getIndexGroup() call (the traversal that follows is what
  actually races).
- MustangEngine.ratify() does the same; AsyncProcessor dispatches
  through Guava's EventBus, which is synchronous by default, so the
  ratification handler's read of the IndexGroup runs on the calling
  thread, inside the read lock, before process() returns.
- Utils.executeSecurely(Lock, ...) is widened from
  ReentrantReadWriteLock.WriteLock to the Lock interface so the same
  helper serializes both sides.

Left out of scope, matching what's actually broken here:
- IndexingFacade.getRatificationResult() stays a plain, unlocked field
  read; it's a single reference read/write, not a traversal of the
  racy structures above.
- replace(oldIndex, newIndex) still only takes oldIndex's lock despite
  reading/mutating newIndex's entry too, a separate, pre-existing gap
  the issue itself flagged as worth tracking on its own; changing its
  locking order risks a new deadlock between concurrent replace() pairs
  and isn't part of what PhonePeGH-9 reported.

Added IndexingFacadeConcurrencyTest, a wall-clock-bounded stress test
with a writer and 8 readers against a steady-state 30-criteria index,
with a progress counter so a real hang is distinguishable from the
test just running slower than its own shutdown grace period. It
reliably reproduces the bug on the prior code (either an exception or
reader threads stuck mid-TreeMap-traversal) and passes cleanly against
this fix, run twice.

Full suite (mvn clean verify) passes, run twice.

Fixes PhonePe#9

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Search/read path is unsynchronized against concurrent index mutations (add/update/delete/replace)

1 participant