Skip to content

fix(#55): close YoutubeDL, geoip reader, and the audio-clip retain graph - #173

Merged
Rushaway merged 3 commits into
masterfrom
fix/55-ytdlp-resource-leak
Sep 6, 2026
Merged

Rushaway merged 3 commits into
masterfrom
fix/55-ytdlp-resource-leak

Conversation

@Rushaway

@Rushaway Rushaway commented Sep 6, 2026

Copy link
Copy Markdown
Member

Addresses the memory-leak report by fixing every concrete leak found on inspection. Best-effort: the remaining question ("is it fully gone?") needs a tracemalloc / objgraph pass on a live server over days — but each item below is a real retained-resource bug.

1. YoutubeDL never released (YouTube path)

get_url_youtube_info built a yt_dlp.YoutubeDL and never called close(). Every !yt / !yts — and every retry inside get_first_valid_entry — left the extractor's HTTP connection pools and cookie jar pinned until GC. Matches the report that memory grows with each new external-lookup command but not on a repeat (yt-dlp serves the repeat from cache).

Fix: with yt_dlp.YoutubeDL(ydl_opts) as ydl:.

2. Audio-clip object graph pinned after playback

FFmpegAudioPlayer.Stop() now clears self.callbacks after firing them. The Play/Stop/Update callbacks close over the AudioClip and the AudioManager (and the !say / !dec temp-file cleanup), and the player is reachable back through those closures. While a pending _updater / _read_stream / _reap_process task still held the player, that whole graph stayed alive — refcounting alone wouldn't free it. Simulated: with the clear, the graph is released as soon as Stop runs; without it, it waits on cyclic GC.

3. AntiSpam / Advertiser last_clips growth

last_clips was only pruned inside SpamCheck / Think, which only run while a dominant clip is updating, so a burst that leaves no dominant clip behind kept its entries until the next one played. Extracted the existing prune into _prune_last_clips() and also call it on every OnStop(). OnUpdate() now uses .get() and returns cleanly instead of raising KeyError (then logged) when the entry is already gone.

A randomized equivalence check (20k cases) confirms _prune_last_clips() + the untouched SpamCheck keeps the same entry set and counted duration as the original inline loop — detection behaviour is unchanged, only the cleanup cadence.

4. geoip2.database.Reader leaked on reload

OpenWeather opened a Reader (an mmap of the GeoIP DB) that was never closed, and CommandHandler.Setup() builds a fresh command instance on every !reload. Added BaseCommand.close() (no-op) with an OpenWeather override that closes the reader, and CommandHandler.Setup() now calls close() on the outgoing commands before dropping them.

Context

Much of the original 2023 report was already chipped away by #137-146 and #160-170 (zombie ffmpeg, unclosed sockets/writers, cancellation leaks). This picks up what was left.

Test plan

  • ruff check / ruff format / mypy clean.
  • Retained-graph release simulated (item 2); prune equivalence fuzzed 20k cases (item 3).
  • yt_dlp / aiohttp not installed in the check env — YoutubeDL has been a context manager for years, but a live !yt + !weather + !reload smoke test before merge is worth it.
  • Live tracemalloc run over a few days to confirm the curve is flat.

Merge order

Batch with #164 (#171) and #169 (#172); sequential VERSION patch bumps (this one -> 1.8.25). One-line VERSION rebase if it doesn't merge last.

Closes #55

🤖 Generated with Claude Code

)

`get_url_youtube_info` built a `yt_dlp.YoutubeDL` and never called `close()`,
so every `!yt` / `!yts` (and every retry inside `get_first_valid_entry`) left
the extractor's HTTP connection pools and cookie jar pinned until the garbage
collector happened to run. On a long-lived bot that is a steady climb, and it
matches the report that memory grows with each new external-lookup command
but not when the same query is repeated (yt-dlp serves the repeat from cache).

Wrapping it in `with yt_dlp.YoutubeDL(ydl_opts) as ydl:` makes cleanup
deterministic.

Scope: this is one identified leak on the YouTube path, not a claim that #55
is fully resolved. Much of the original 2023 report has been chipped away by
the resource-management fixes in #137-146 and #160-170 (zombie ffmpeg,
unclosed sockets/writers, cancellation leaks). What remains needs profiling
against a live server over days; suspects are noted on the issue.

Bumps VERSION to 1.8.25.

Refs #55

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@Rushaway
Rushaway requested a review from Dolly132 September 6, 2026 10:39
#55)

Follow-up to the YoutubeDL fix in the same PR, covering the other suspects
noted on the issue:

- `FFmpegAudioPlayer.Stop()` clears `self.callbacks` after firing them. The
  Play/Stop/Update callbacks close over the `AudioClip` and the
  `AudioManager` (and, for `!say` / `!dec`, the TTS temp-file cleanup), and
  the player is reachable back through those closures. While a pending
  `_updater` / `_read_stream` / `_reap_process` task still holds the player,
  that whole graph stayed pinned; now it is dropped as soon as Stop runs.

- `AntiSpam` / `Advertiser`: `last_clips` was only pruned inside
  `SpamCheck` / `Think`, which only run while a *dominant* clip is updating,
  so entries could linger through quiet periods. Extracted the existing
  prune into `_prune_last_clips()` and also call it on every `OnStop()`.
  A randomized equivalence check (20k cases) confirms the prune +
  untouched `SpamCheck` keeps the same entry set and counted duration as
  the original inline loop. `OnUpdate()` now uses `.get()` and returns
  cleanly instead of raising `KeyError` (then logged) when the entry is
  already gone.

- `OpenWeather` opened a `geoip2.database.Reader` (an mmap of the GeoIP DB)
  that was never closed, and `CommandHandler.Setup()` builds a fresh command
  on every reload. Added `BaseCommand.close()` (no-op) with an `OpenWeather`
  override, and `CommandHandler.Setup()` now calls it on the outgoing
  commands before dropping them.

Still best-effort: worth a `tracemalloc` / `objgraph` pass on a live server
to confirm clips and players are actually collected after playback.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@Rushaway Rushaway changed the title fix(urlinfo): context-manage YoutubeDL so it releases its resources (#55) fix(#55): close YoutubeDL, geoip reader, and the audio-clip retain graph Sep 6, 2026
@Rushaway Rushaway mentioned this pull request Sep 6, 2026
Dolly132 added a commit to Dolly132/torchlight that referenced this pull request Sep 6, 2026

@Dolly132 Dolly132 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, tested and works fine without errors/exceptions

@Rushaway
Rushaway merged commit 53c6f3c into master Sep 6, 2026
2 checks passed
@Rushaway
Rushaway deleted the fix/55-ytdlp-resource-leak branch September 6, 2026 13:30
Rushaway added a commit that referenced this pull request Sep 7, 2026
`OpenWeather.__init__` opened a fresh `geoip2.database.Reader` every time
`CommandHandler.Setup()` ran. #173 added `close()` on the outgoing command so the
previous mmap of the database no longer leaked until GC (#55), but rebuilding the
reader on every `!reload` is still fragile: any failure to map the file now
aborts command setup with an error traceback in the logs (#174).

Move the reader into a dedicated `torchlight.GeoIP` module (only `Commands` is
reloaded, so the cache survives `importlib.reload`). It is opened once per path
and only reopened when the file on disk actually changes, e.g. after a GeoIP
update; a failed reopen keeps serving the previous reader instead of dropping
GeoIP entirely.

`OpenWeather` no longer raises out of `__init__` when the database cannot be
opened: it logs the error, leaves `geo_ip` unset, and `!ow` without an explicit
location asks the player to name a city instead of crashing.

Bumps VERSION to 1.8.26.

Fixes #174
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: Memory leak

2 participants