fix(#55): close YoutubeDL, geoip reader, and the audio-clip retain graph - #173
Merged
Merged
Conversation
) `get_url_youtube_info` built a `yt_dlp.YoutubeDL` and never called `close()`, so every `!yt` / `!yts` (and every retry inside `get_first_valid_entry`) left the extractor's HTTP connection pools and cookie jar pinned until the garbage collector happened to run. On a long-lived bot that is a steady climb, and it matches the report that memory grows with each new external-lookup command but not when the same query is repeated (yt-dlp serves the repeat from cache). Wrapping it in `with yt_dlp.YoutubeDL(ydl_opts) as ydl:` makes cleanup deterministic. Scope: this is one identified leak on the YouTube path, not a claim that #55 is fully resolved. Much of the original 2023 report has been chipped away by the resource-management fixes in #137-146 and #160-170 (zombie ffmpeg, unclosed sockets/writers, cancellation leaks). What remains needs profiling against a live server over days; suspects are noted on the issue. Bumps VERSION to 1.8.25. Refs #55 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
#55) Follow-up to the YoutubeDL fix in the same PR, covering the other suspects noted on the issue: - `FFmpegAudioPlayer.Stop()` clears `self.callbacks` after firing them. The Play/Stop/Update callbacks close over the `AudioClip` and the `AudioManager` (and, for `!say` / `!dec`, the TTS temp-file cleanup), and the player is reachable back through those closures. While a pending `_updater` / `_read_stream` / `_reap_process` task still holds the player, that whole graph stayed pinned; now it is dropped as soon as Stop runs. - `AntiSpam` / `Advertiser`: `last_clips` was only pruned inside `SpamCheck` / `Think`, which only run while a *dominant* clip is updating, so entries could linger through quiet periods. Extracted the existing prune into `_prune_last_clips()` and also call it on every `OnStop()`. A randomized equivalence check (20k cases) confirms the prune + untouched `SpamCheck` keeps the same entry set and counted duration as the original inline loop. `OnUpdate()` now uses `.get()` and returns cleanly instead of raising `KeyError` (then logged) when the entry is already gone. - `OpenWeather` opened a `geoip2.database.Reader` (an mmap of the GeoIP DB) that was never closed, and `CommandHandler.Setup()` builds a fresh command on every reload. Added `BaseCommand.close()` (no-op) with an `OpenWeather` override, and `CommandHandler.Setup()` now calls it on the outgoing commands before dropping them. Still best-effort: worth a `tracemalloc` / `objgraph` pass on a live server to confirm clips and players are actually collected after playback. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Closed
Dolly132
added a commit
to Dolly132/torchlight
that referenced
this pull request
Sep 6, 2026
Dolly132
approved these changes
Sep 6, 2026
Dolly132
left a comment
Collaborator
There was a problem hiding this comment.
LGTM, tested and works fine without errors/exceptions
This was referenced Sep 7, 2026
Rushaway
added a commit
that referenced
this pull request
Sep 7, 2026
`OpenWeather.__init__` opened a fresh `geoip2.database.Reader` every time `CommandHandler.Setup()` ran. #173 added `close()` on the outgoing command so the previous mmap of the database no longer leaked until GC (#55), but rebuilding the reader on every `!reload` is still fragile: any failure to map the file now aborts command setup with an error traceback in the logs (#174). Move the reader into a dedicated `torchlight.GeoIP` module (only `Commands` is reloaded, so the cache survives `importlib.reload`). It is opened once per path and only reopened when the file on disk actually changes, e.g. after a GeoIP update; a failed reopen keeps serving the previous reader instead of dropping GeoIP entirely. `OpenWeather` no longer raises out of `__init__` when the database cannot be opened: it logs the error, leaves `geo_ip` unset, and `!ow` without an explicit location asks the player to name a city instead of crashing. Bumps VERSION to 1.8.26. Fixes #174
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Addresses the memory-leak report by fixing every concrete leak found on inspection. Best-effort: the remaining question ("is it fully gone?") needs a
tracemalloc/objgraphpass on a live server over days — but each item below is a real retained-resource bug.1.
YoutubeDLnever released (YouTube path)get_url_youtube_infobuilt ayt_dlp.YoutubeDLand never calledclose(). Every!yt/!yts— and every retry insideget_first_valid_entry— left the extractor's HTTP connection pools and cookie jar pinned until GC. Matches the report that memory grows with each new external-lookup command but not on a repeat (yt-dlp serves the repeat from cache).Fix:
with yt_dlp.YoutubeDL(ydl_opts) as ydl:.2. Audio-clip object graph pinned after playback
FFmpegAudioPlayer.Stop()now clearsself.callbacksafter firing them. The Play/Stop/Update callbacks close over theAudioClipand theAudioManager(and the!say/!dectemp-file cleanup), and the player is reachable back through those closures. While a pending_updater/_read_stream/_reap_processtask still held the player, that whole graph stayed alive — refcounting alone wouldn't free it. Simulated: with the clear, the graph is released as soon as Stop runs; without it, it waits on cyclic GC.3.
AntiSpam/Advertiserlast_clipsgrowthlast_clipswas only pruned insideSpamCheck/Think, which only run while a dominant clip is updating, so a burst that leaves no dominant clip behind kept its entries until the next one played. Extracted the existing prune into_prune_last_clips()and also call it on everyOnStop().OnUpdate()now uses.get()and returns cleanly instead of raisingKeyError(then logged) when the entry is already gone.A randomized equivalence check (20k cases) confirms
_prune_last_clips()+ the untouchedSpamCheckkeeps the same entry set and counted duration as the original inline loop — detection behaviour is unchanged, only the cleanup cadence.4.
geoip2.database.Readerleaked on reloadOpenWeatheropened aReader(an mmap of the GeoIP DB) that was never closed, andCommandHandler.Setup()builds a fresh command instance on every!reload. AddedBaseCommand.close()(no-op) with anOpenWeatheroverride that closes the reader, andCommandHandler.Setup()now callsclose()on the outgoing commands before dropping them.Context
Much of the original 2023 report was already chipped away by #137-146 and #160-170 (zombie ffmpeg, unclosed sockets/writers, cancellation leaks). This picks up what was left.
Test plan
ruff check/ruff format/mypyclean.yt_dlp/aiohttpnot installed in the check env —YoutubeDLhas been a context manager for years, but a live!yt+!weather+!reloadsmoke test before merge is worth it.tracemallocrun over a few days to confirm the curve is flat.Merge order
Batch with #164 (#171) and #169 (#172); sequential
VERSIONpatch bumps (this one ->1.8.25). One-lineVERSIONrebase if it doesn't merge last.Closes #55
🤖 Generated with Claude Code