Repository navigation
Take the read lock on search/scan/ratify/checkSimilarity (GH-9) - #14
Open
chiragmangaldev3112 wants to merge 1 commit into
Open
chiragmangaldev3112 wants to merge 1 commit into
chiragmangaldev3112 wants to merge 1 commit into
Conversation
IndexingFacade stored only the write half of each index's ReentrantReadWriteLock (getLock() discarded everything but writeLock()). add/update/delete/replace correctly serialize against each other through it, but search(), scan(), ratify() and checkSimilarity() read the same IndexGroup with no lock at all. The inverted index's nested structures are not thread-safe below the outer ConcurrentHashMap (InvertedIndex.table's nested Map/TreeMap, IndexGroup.allCriterias, idCache's PriorityQueue), and are mutated in place by add/delete. A search running concurrently with a mutation can observe a torn structure. Reproduced with a steady-state stress test: a writer thread repeatedly deleting and re-adding the same 30 criteria while 8 reader threads search the same index. On this code, reader threads reliably end up stuck inside TreeMap.getHigherEntry/TreeSet.higher (DNFMatcher's getNextHigherId), iterating a structure being concurrently rebuilt out from under them. - indexLocks now stores the full ReentrantReadWriteLock instead of only its write half. All eight existing write-side call sites explicitly take .writeLock(); nothing there changes behaviour. - IndexingFacade.readSecurely(index, block) takes the read lock for the block's whole duration, not just an initial getIndexGroup() lookup. - SearchFacade.search() and MustangEngine's scan()/checkSimilarity() wrap their full body in readSecurely(), since the read spans more than the getIndexGroup() call (the traversal that follows is what actually races). - MustangEngine.ratify() does the same; AsyncProcessor dispatches through Guava's EventBus, which is synchronous by default, so the ratification handler's read of the IndexGroup runs on the calling thread, inside the read lock, before process() returns. - Utils.executeSecurely(Lock, ...) is widened from ReentrantReadWriteLock.WriteLock to the Lock interface so the same helper serializes both sides. Left out of scope, matching what's actually broken here: - IndexingFacade.getRatificationResult() stays a plain, unlocked field read; it's a single reference read/write, not a traversal of the racy structures above. - replace(oldIndex, newIndex) still only takes oldIndex's lock despite reading/mutating newIndex's entry too, a separate, pre-existing gap the issue itself flagged as worth tracking on its own; changing its locking order risks a new deadlock between concurrent replace() pairs and isn't part of what PhonePeGH-9 reported. Added IndexingFacadeConcurrencyTest, a wall-clock-bounded stress test with a writer and 8 readers against a steady-state 30-criteria index, with a progress counter so a real hang is distinguishable from the test just running slower than its own shutdown grace period. It reliably reproduces the bug on the prior code (either an exception or reader threads stuck mid-TreeMap-traversal) and passes cleanly against this fix, run twice. Full suite (mvn clean verify) passes, run twice. Fixes PhonePe#9
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
IndexingFacadestores only the write half of each index'sReentrantReadWriteLock(getLock()discards everything but.writeLock()).add/update/delete/replacecorrectly serialize against each other through it, butsearch(),scan(),ratify()andcheckSimilarity()read the sameIndexGroupwith no lock at all, exactly as the issue describes.The inverted index's nested structures are not thread-safe below the outer
ConcurrentHashMap(InvertedIndex.table's nestedMap/TreeMap,IndexGroup.allCriterias,idCache'sPriorityQueue), andadd/deletemutate them in place. A search running concurrently with a mutation can observe a torn structure.Reproduction
A steady-state stress test: a writer thread repeatedly deletes and re-adds the same 30 criteria while 8 reader threads search the same index. On the current code, reader threads reliably end up stuck inside
TreeMap.getHigherEntry/TreeSet.higher(DNFMatcher.getNextHigherId), iterating a structure being concurrently rebuilt out from under them — confirmed with real thread dumps, not just an assumption.Fix
indexLocksnow stores the fullReentrantReadWriteLockinstead of only its write half. All eight existing write-side call sites explicitly take.writeLock(); nothing there changes behaviour.IndexingFacade.readSecurely(index, block)takes the read lock for the block's whole duration, not just an initialgetIndexGroup()lookup — the traversal that follows is what actually races.SearchFacade.search()andMustangEngine'sscan()/checkSimilarity()wrap their full body inreadSecurely().MustangEngine.ratify()does the same.AsyncProcessordispatches through Guava'sEventBus, which is synchronous by default, so the ratification handler's read of theIndexGroupruns on the calling thread, inside the read lock, beforeprocess()returns.Utils.executeSecurely(Lock, ...)is widened fromReentrantReadWriteLock.WriteLockto theLockinterface so the same helper serializes both sides.Left out of scope, matching what's actually broken here:
getRatificationResult()stays a plain, unlocked field read — it's a single reference read, not a traversal of the racy structures above.replace(oldIndex, newIndex)still only takesoldIndex's lock despite reading/mutatingnewIndex's entry too — the pre-existing gap you flagged separately in the issue. Changing its locking order risks a new deadlock between concurrentreplace()pairs and isn't part of what Search/read path is unsynchronized against concurrent index mutations (add/update/delete/replace) #9 reported; happy to take that on separately if you'd like.Test plan
IndexingFacadeConcurrencyTest: wall-clock-bounded (not iteration-bounded, so it's not hardware-speed-dependent), with a progress counter so a real hang is distinguishable from the test merely running slower than its shutdown grace period.TreeMaptraversal) — verified via real thread dumps, run twice.mvn clean verify(full suite, all modules) passes — run twice.Fixes #9