how to decrease logging to syslog via config?

[kuvaldini]

from bees --help

Logging options:
-t, --timestamps      Show timestamps in log output (default)
-T, --no-timestamps   Omit timestamps in log output
-p, --absolute-paths  Show absolute paths (default)
-P, --strip-paths     Strip $CWD from beginning of all paths in the log
-v, --verbose         Set maximum log level (0..8, default 8)

so likely can edit beesd@.service:Exec=... -v2 but no way to set via /etc/bees/UUID.conf

right?

and why default is 8? means debug logs are enabled by default? what value is optimal for long-running desktop and laptop?

[zootboy]

You can certainly set the verbosity flag from within the .conf file. Just add it to the OPTIONS="..." string, e.g.

OPTIONS="--strip-paths --thread-count 2 --verbose 6"

Although I have a related question: is there a way to get some logging of what files bees is successfully dedup-ing, but not the massive firehose of debug logging? Level 7 seems to have basically the same amount of output as the default level 8, but level 6 makes it go completely silent.

[kakra]

is there a way to get some logging of what files bees is successfully dedup-ing

You can look at the live activity:

find /run/ -name "bees.status" -mmin -5 # find the correct file here
watch cat /run/bees/bees.status # watch the found file

There can be a lot of output, and watch will truncate it. This may be better to show what you're asking for:

watch "awk '/THREADS/,EOF { print }' /run/bees/bees.status" # file name from above

[kuvaldini]

llm can help understand the contents of that status, it said my btrfs have 5gb extents of total 300gb, (and many many words of text ;) i think it can definitely do more, since most of space is already occupied.
but it did not say main thing:

which files have shared extents? how much bytes are shared between those files?
i expect some table: 100mb ext1,ext2,ext10 file1,file2,file3

e.g. having /var/lib/flatpak/ of 40gb size, most of space is occupied by "platforms" kde5,kde6,gtk{2,3,4}. i believe there is smth to share between them.

i am thinking about some tool to calculate possible duplicates. something which can be as affective as rsync's comparator.

[kakra]

i am thinking about some tool to calculate possible duplicates. something which can be as affective as rsync's comparator.

You can use compsize on btrfs to check how much data is currently shared/compressed/allocated. It doesn't tell you the potential deduplication, tho. But may use duperemove or something similar in dry-run mode to preview the potential dedup ratio. But bees works different, so the values will differ.

but it did not say main thing:

It's a live status. It will tell you which files it is currently working on, it won't log the files it already processed. So you should watch it with the awk command I suggested.

To know

which files have shared extents? how much bytes are shared between those files?
i expect some table: 100mb ext1,ext2,ext10 file1,file2,file3

you should use compsize. It works similar to du or df: You give it a list of files or directories on the cmdline and it will calculate the total, e.g.:

# sudo compsize ~/downloads ~/iso
Processed 860 files, 54505 regular extents (345273 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       99%      214G         215G         206G
none       100%      214G         214G         205G
zstd        29%      454M         1.5G         1.8G

means, the files reference 206GB (the number du would report) but actually use 214GB (so uses even more space than the actual files, probably due to bookend extents or meta data overhead).

Here's another example of my development directories which shows there are actually space savings:

# sudo compsize ~/git ~/go
Processed 1973164 files, 959674 regular extents (1975551 refs), 114719 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       95%       55G          57G          79G
none       100%       53G          53G          73G
zstd        30%      1.2G         4.0G         6.0G

However, this still won't go down to the details which files shares extents in which size with other files. This will actually be quite an effort to collect from the file system because btrfs is quite slow for doing the needed reverse lookups. But it should be possible to create a program which collects all the extent lists and then compares them between all the files. If not done well, this will be an exponential effort, so it's probably best to design it with an extent tree mapping in mind right from the start (keep a tree of extents, then attach a sorted list of file names per extent, then calculate checksums over all the file name lists and return a unique list per file name list checksum for your table). I'm not sure if this would be useful, however. Such a table will most likely grow quite large and look very noisy.

If you just want to verify if bees caught each and all possible extents, and in which files, and if it matches your expectations, you might be expecting too much from bees: bees is only best-effort, it doesn't try to get the perfect deduplication. Rather it tries to get a good deduplication without affecting file system and runtime performance too much, and it tries to avoid meta data overhead. Sometimes it's better to not dedup an 8k block, or not dedup a small extent 1000 times, because in the end it will make btrfs slow and uses more storage space due to increased overhead. bees ships with a lot of heuristics and optimization tradeoffs that other dedupe apps do not have, and it usually results in an overall better performance compared to other solutions.

But if you want to give bees a good start, you could first run duperemove over your dataset, then let bees run over it. This will probably redo some of the work from duperemove at the cost of dedup ratio but with improved performance. And it will certainly result in more storage space used because this will create additional meta data overhead.

Thus, if you want to compare results, maybe compare compsize on the dataset (after bees processed it) with a dry run of duperemove.

Hint: If you have a big dataset, duperemove will likely need huge amounts of RAM (much more than bees) and create a big hash database (which you should consider as additional space usage which counters its potential higher dedup ratio). But if you use duperemove once, then delete the hash database and let bees do the work from now on, you should get a good start with all the benefits that bees then provides.

This table from the status file can help estimate the dedup potential:

PROGRESS:
extsz   datasz  point gen_min gen_max this cycle start tm_left   next cycle ETA
----- -------- ------ ------- ------- ---------------- ------- ----------------
  max 976.891G 148442 1019839 1019840 2026-04-21 13:26       -             idle
  32M 171.151G 148442 1019839 1019840 2026-04-21 13:26       -             idle
   8M  62.724G 965634 1019838 1019839 2026-04-21 13:26      1s 2026-04-21 13:26
   2M  26.958G 182242 1019839 1019840 2026-04-21 13:26       -             idle
 512K  28.213G 186165 1019839 1019840 2026-04-21 13:26       -             idle
 128K  38.063G 000000 1019839 1019840 2026-04-21 13:26       -             idle
total   1.273T        gen_now 1019840                  updated 2026-04-21 13:26

It shows you how much data you have per extent size. Depending on your dataset, some percentage of that can be deduped, letting you estimate how much space can be saved by dedup. I'm not sure if bees currently tries to merge smaller extents back into bigger extents (I think this is sill on the roadmap to do that on purpose, currently it's more likely a side effect). But it will work on bigger extents first and tries to avoid unnecessary extent splits. Especially the small extents (up to 128k) potentially contain very common extents which bees often will not dedup to not sacrifice btrfs performance or increase space usage due to meta data rewrites. The bigger the extents the less likely there will be duplicates (that's just statistics) but OTOH, the more data can be saved per duplicate extent.

@Zygo may have some statistics on that.

[lilydjwg]

which files have shared extents? how much bytes are shared between those files?

If you want to see current status, you can use btdu.

FYI there is a rewrite of compsize called xsz that runs much faster.

[kuvaldini]

@kakra 🙏 thanks, happy to read human's message _{(am i right this is not llm gen? ;)}
i see bees does not take much resources at run-time, well-architected sw.

> btrfs-compsize /var/lib/flatpak/
Processed 803502 files, 341380 regular extents (759691 refs), 476187 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       59%       21G          36G          63G
none       100%       14G          14G          21G
zlib        11%      8.0K          72K         260K
zstd        33%      7.5G          22G          41G

looks pretty nice.

@Zygo may have some statistics on that.

@Zygo please share if so.

@lilydjwg thanks, what is useful

[kakra]

(am i right this is not llm gen? ;)

Just out of curiosity, I asked my LLM - and the answer would have been quite different. ;-) But yes, the text has been completely hand-crafted.

I hope compsize gives you a good enough insight about what you want to know, it won't go down to the extent detail level per file, tho.

Just for your curiosity, this is my whole file system on my work machine:

# sudo compsize /mnt/btrfs-pool
Processed 129305466 files, 4872777 regular extents (151350393 refs), 20475855 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       92%      1.1T         1.2T          23T
none       100%      1.0T         1.0T          21T
zstd        34%       50G         147G         1.6T
prealloc   100%       37M          37M         143M

[kuvaldini]

O_o this is wonderful. This is what i dreamed about being child. Nowadays best-effort extent-same is must have.
_{heh this is how they sell cloud storage ;)}

Yes i checked xsz and compsize, that is what i need - extenTed du. can track what is least efficient, and what is do be cleaned first. finally i just wanted to get know what is to be removed to get space on btrfs, since removing 40GB has given just 4GB freed one time... Your answers - short lecture by fact - was impressive, so as and @Zygo's extened answers in other threads.

Processed 21452463 files, 2208639 regular extents (25434199 refs), 11347798 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       84%      295G         348G         2.0T
none       100%      267G         267G         1.4T
zlib        15%       12K          80K         484K
zstd        34%       28G          80G         554G

[Zygo]

happy to read human's message (am i right this is not llm gen? ;)

Greetings, fellow human! I, too, am not a robot. ;)

i see bees does not take much resources at run-time, well-architected sw.

Architecture usually wins over micro-optimization. bees v0.11 isn't exactly slow, but there's still a lot of ways to make it faster.

Just out of curiosity, I asked my LLM - and the answer would have been quite different. ;-) But yes, the text has been completely hand-crafted.

The other day I noticed that people were feeding bees code to their LLM, and getting it to try to fix bugs...without sending a PR to me for review. Rude!

If they had sent me their PR, they could have fed my rejection comment into their LLM, and it could have fixed the real problem. Eventually.

@Zygo may have some statistics on that.

Over the years I've tried to quantify the inputs to the bees algorithm and provide guidance like "use hash table size X for filesystem size Y with extent size Z"; however, bees is a non-linear mashup of opportunistic optimisations whose behavior is easy to describe in isolation. It's difficult to predict behavior when the optimisations interact and reinforce each other. This leads to surprising data-set-specific results, like "decreasing the hash table size makes bees faster--but only on this one filesystem and specific hash table size" and "increasing the hash table size makes bees dedupe less--but increasing more makes bees dedupe more again," which occur because the size-to-performance curves are non-monotonic in places, so you sometimes get weird results around inflection points.

Now I don't try to explain it any more--I just document what it does, and let you do the math on your own data.

bees up to v0.11 doesn't have any model for unreachable blocks. That makes it underestimate dedupe cost/benefit ratios, and allows bees to create a few unreachable blocks itself. Both of these effects drag on the dedupe rate a little (about 3-6% on my test data sets, but can be higher if your data triggers bad behavior often). This effect isn't included in the description below--the description contains only things that bees models itself, and unreachable blocks aren't included in the v0.11 btrfs model.

Disclaimer: the rest is written by an LLM. I've corrected its more egregious mistakes (it's still very wordy though--apologies for that). Let me know if you like this text. If other humans approve, I'll put it in the docs as its own page.

The scan_one Algorithm, in Brief

For each btrfs data extent bees encounters during a scan:

1. Read and hash the extent, one 4 KiB block at a time. All-zero
   blocks are recognized and set aside as "punch a hole" candidates
   rather than hashed for dedupe.

2. For each non-zero block, look up the hash in the hash table.
   A hit means some previously-scanned block on the filesystem had
   the same hash.

3. For each hit, ask the kernel where that block lives (via the
   LOGICAL_INO ioctl) and try to extend the match — see how
   far forward and backward from the hit the two extents agree.
   A single matching block in the new extent can anchor a much
   longer run of adjacent duplicate blocks.

4. Pick the longest non-overlapping matched ranges and check the
   nuisance-dedupe filter: the number of bytes that would be freed
   (by dedupe plus zero-punching) must be at least the number of bytes
   that would have to be rewritten (to break the non-matching parts
   of the new extent out of the shared range so that the original
   extent can be freed). If not, the dedupe is skipped.

5. Perform the dedupes via the FILE_EXTENT_SAME ioctl.

6. Update the hash table. Hashes that produced a dedupe match
   are moved to the front of their LRU list. Hashes from the new
   extent that were not deduped are inserted at a random position
   in their bucket.

A duplicate extent is only eliminated when steps 2, 3, and 4 all
succeed. Each parameter below controls one of those steps.

When Does Dedupe Actually Happen?

Two extents A (older, already scanned) and B (newer, being scanned
now) get deduplicated only if all three of the following are true:

• Gate A — at least one hash from A is still in the hash table
  when B is scanned.
• Gate B — at least one block of B collides with a retained hash
  from A.
• Gate C — the extension pass finds enough contiguous matching
  data to pass the nuisance-dedupe filter.

Most tuning questions are really questions about one of these three
gates. The rest of this page maps each parameter onto the gate (or
gates) it controls.

Hash Table Size

The hash table holds a fixed number of (hash, physical-address) pairs,
one pair per 16 bytes of table. A 1 GB hash table holds about 64 M
pairs — one per 4 KiB block it can remember.

Effect on Gate A. Hash table size is the dominant control on how
long a scanned block's hash survives in the table. The intuition: if
your filesystem has much more unique-block data than the table can hold,
each slot is overwritten on average after a number of new inserts equal
to the number of slots, so older data drops out first.

This is why the bees rule of thumb is "1 GB of hash table per 10 TB of
unique data": it keeps the expected survival time of a hash long enough
to overlap with the scan period for most real workloads, where
duplicates tend to appear relatively near each other in time.

Under-provisioning the hash table does not prevent dedupe — it
reduces the dedupe rate, typically most on the oldest data.

Data Extent Size

btrfs extents can be up to 128 MiB for uncompressed data (128 KiB
for compressed data). Extent size affects two gates.

Effect on Gate B. Gate B asks whether any block of the new
extent hits a retained hash. If an extent has n blocks and each
block has independent probability p of having its hash retained,
then the probability of at least one hit is

            P(hit) = 1 − (1 − p)^n

Larger extents give bees more independent "shots" at finding an
anchor block. A 16-block extent (64 KiB) and a 256-block extent
(1 MiB) behave very differently, even with the same hash table.

This is why how bees works says most of the hashes
in the hash table are redundant: you don't need every block of an
extent retained, you only need one, because the extension pass in
step 3 will find the rest.

Effect on Gate C. Larger extents also raise the bar for the
nuisance filter: to pass, more of the extent has to dedupe. A small
extent is almost always either fully duplicate or fully unique; a
large extent is more likely to have only a fraction of its content
match.

Net effect: medium-to-large extents are the sweet spot for scan_one.
Very small extents don't benefit from the multi-shot retention. Very
large, partially-duplicate extents get skipped by the nuisance filter.

The 50% Must-Free Threshold

btrfs cannot free part of an extent — an extent is allocated and
freed as a whole. As long as any file anywhere in the filesystem
references any part of an extent, the whole extent stays
allocated, including blocks that no file references. Recovering
the space held by an extent therefore means eliminating every
reference to every part of it.

For the matching parts of the new extent, FILE_EXTENT_SAME
redirects their references to the duplicate data elsewhere. For
the non-matching parts, bees rewrites the data into new
allocations, so those file offsets stop pointing at the original
extent. Once no reference to the original extent remains anywhere,
btrfs frees it.

The rewritten data does not persist as a duplicate — the original
extent is freed, and the rewrite replaces it at those offsets — but
the rewrite still costs I/O, allocator churn, and extent
fragmentation, and those costs have to be paid back in recovered
free space for the dedupe to be worth doing.

bees enforces the rule:

            bytes_freed  ≥  bytes_rewritten

where bytes_freed = bytes_deduped + bytes_zero_punched and
bytes_rewritten is the size of the non-matching data that must be
rewritten to free the original extent.

For an extent with no zero blocks, this reduces to the well-known
"at least 50% of the extent must dedupe" threshold. When zero
blocks are present — for example, a mostly-empty VM disk image — the
threshold is easier to pass, because punching holes also frees space.

Effect on Gate C. This is the gate itself. Extents that match
only sparsely (say, 20% shared and 80% unique) are skipped entirely;
the hashes from the unique part are still inserted into the table,
so a future extent that shares more with them can still succeed later.

Random-Insertion-Order Hash LRU

Each bucket of the hash table is an LRU list of 256 cells. bees uses
two distinct insertion policies:

• Move to front on hit. When a hash lookup finds a cell that
  produces a successful dedupe match, that cell is moved to slot 0.
• Random position on unique insert. When a new, previously-unseen
  hash is inserted, it goes into a uniformly random slot in its
  bucket. If the bucket is full, the cell at the tail (slot 255) is
  evicted, and the cells between the insertion point and the tail are
  shifted back by one.

Effect on Gate A. This is the subtlest of the parameters.
Compared to a plain FIFO (where every cell would live exactly as long
as the bucket depth before eviction), random insertion gives a
fat-tailed survival distribution. A cell that happens to be inserted
near the front of the bucket has a much longer expected lifetime than
one inserted near the back, and that difference is amplified by the
move-to-front promotion: any hash that ever matches gets effectively
pinned near the front.

The practical consequence: bees keeps "useful" hashes (ones that match
real duplicates) much longer than "unlucky" hashes (ones that never
match), without needing an expensive explicit usage counter. It trades
a small probability of missing a rare duplicate for much better
retention of common patterns.

256 Cells Per Bucket

Each 4 KiB page of the hash table file is one bucket of 256 cells.
This size is set by the mmap page size and cell width, not a tunable
parameter, but it has two effects worth understanding.

Effect on Gate A (collision capacity). Different physical blocks
with colliding hashes can all coexist in the same bucket, up to 256 of
them. This matters for widely replicated content such as boilerplate
file headers and common fill patterns: bees can keep many addresses
for the "same" hash without any one of them crowding out the others.

Effect on Gate A (LRU depth). A 256-slot LRU is a meaningful
buffer against eviction. If buckets were shallow (say, 4 cells), only
the very-most-recent or very-popular hashes would survive, and
rare-but-real duplicates would almost always miss. The 256-deep
bucket and the random-insert policy together are what let bees find
old and rare duplicates at a useful rate.

How the Parameters Interact

A few interactions are worth singling out:

• Hash table size and extent size compound. If the hash table
  is marginal, small extents suffer disproportionately: their Gate B
  probability 1 − (1 − p)^n falls quickly with small n. Doubling
  the hash table helps small extents more than large ones.

• Extent size fights the 50% threshold. Increasing extent size
  makes Gate B easier but Gate C harder. Which effect dominates
  depends on the file's content pattern, not on btrfs:

  • Large, sequentially-written files (media files, log archives,
    installation images) tend to contain long runs where every
    block is unique within the file, but the whole sequence
    recurs whenever the file is duplicated. Match rates of 100%
    are common, so the 50% threshold is met with room to spare.
  • Files written in many small writes in random order (for
    example, databases) tend to contain a mixture of unique and
    duplicate blocks in every extent. The matched fraction can
    be non-trivial, but the rewrite cost makes the dedupe
    unprofitable, and Gate C filters the extent out.
  • VM disk images mix both patterns — contiguous regions that
    behave like media files and churn regions that behave like
    databases — and produce both outcomes depending on the extent.

  The 50% filter lets bees cherry-pick the extents that are worth
  deduping and skip the ones that would just burn time and metadata,
  without needing to know anything about file type or content
  semantics.

• Move-to-front is what keeps the hash table effective in steady
  state. Without it, even a well-sized table would slowly turn
  over and forget the data that is actually being deduped. With it,
  the table quickly learns which hashes are worth keeping and holds
  onto them, regardless of how much non-duplicate traffic is flowing
  through.

Practical Tuning Implications

• If you are missing dedupes you expect to see, the most likely cause
  is hash table size (Gate A) or extent layout (Gates B and C).

• The nuisance filter is intentional and generally should not be
  worked around: dedupes that rewrite more than they recover
  still gain some free space (the original extent is freed either
  way), but the ratio of recovered space to I/O, allocator churn,
  and metadata cost is too low to be worth doing.

• The random-insertion LRU combined with move-to-front on match
  biases the hash table toward the hashes that actually produce
  dedupes: matched hashes are effectively pinned near the front,
  and hashes that are never used drift back until they are evicted.
  Most of the cells in the table at any moment are redundant, but
  that is the cost of not knowing in advance which ones will be
  useful.

• Continuous and periodic operation are usually equivalent for
  per-extent dedupe probability. bees persists its hash table and
  scan position, so a restart does not reset the LRU and does not
  skip data, and each extent is scanned in the same order either
  way. The important exception is workloads that rapidly churn
  the same blocks (rotating logs, continuous rebuilds,
  heavily-updated databases): continuous bees will scan every
  intermediate version of those blocks and fill the LRU with
  hashes that point to extents that are already freed by the time
  the next duplicate arrives, crowding out more useful hashes.
  Running bees periodically lets short-lived data be overwritten
  before bees ever sees it; increasing the hash table size is the
  other way to compensate.

[kakra]

That... was a mouthful... Safe to say, a rather big one... :-D But I approve: It is structured well and quite easy to read. Interestingly, one thing immediately stood out to me which I always keep forgetting: bees doesn't create hashes of the full extent but rather keeps hashes of the leading (or even trailing) block of continuous runs of potentially duplicate blocks. It's what makes it possible to split and recombine extents, right? And it's probably what makes it different from other dedupers.

I wonder how other dedupers handle this because building hashes just of existing extents doesn't seem very efficient to me in regards to the dedupe hit rate. I wonder if they use CDC or CBC rolling hashes to better deal with that.

I think we now completely hijacked the original intent of this issue, I hope it's still interesting. But getting back to the original question, and it's not the first time it comes up: I think the default log level should really be decreased (less verbose) while still maintaining some level of logging for file/dedupe activity so people actually have something to watch while bees is working. Currently, decreasing verbosity just a little bit makes bees very silent, I mean: very silent.

Logging seems to very intentionally be focused on development rather than "admin entertainment". Maybe bees should give us bored admins some "progress bars" to watch. :-D

Jokes aside: I think this is really a point. Bees log levels miss a balance of seeing too much and seeing "okay, I just installed it but I don't see it doing anything in the logs". You either get the flood of everything, or it's so silent that you only see the occasional tumbleweed rolling by. From my own perspective: When I started to use bees, it has been logging so much information, even pretty scary stuff which has actually been only harmless debug logging, that it has been quite overwhelming when I really wanted to see which file it just successfully deduped. Meanwhile, I prefer its silence, just logging real warnings and problems. Watching the status file live has become my usual way of seeing if it is doing something.

That said, I had an idea of creating a "beestop" in the past which would show some nice statistics at the top and a list of recently processed files below, much like we know from top or htop. Maybe that would we a worthwhile side project, maybe even one that an LLM could do if acceptable? Essentially something to entertain the averaged bored admin, or show your matrix skills to your friends.

[Zygo]

bees doesn't create hashes of the full extent but rather keeps hashes of the leading (or even trailing) block of continuous runs of potentially duplicate blocks.

It's a random sample of the block hashes--not first, last, or contiguous. Once a block is found in common, bees switches to reading the file, and searches on either side of the match for more blocks. That's the only place where contiguity matters.

It's what makes it possible to split and recombine extents, right?

No, that's just the BeesResolve library functions in bees--that happens after bees has finished with the hashes and done block-by-block matches.

As far as I know, bees is the only deduper that tries to modify btrfs extent layout when it doesn't allow optimal dedupe. Some versions of duperemove try to read and work around btrfs extent layout, which avoids creating unreachable blocks, but it dramatically lowers dedupe rates. In bees terms, duperemove runs with min-free set to 100% because it lacks the code to do any other number.

On XFS and ZFS this is much easier: XFS doesn't have bookend extents, and ZFS dedupes on 2^N KiB slabs. The dedupers on these filesystems don't do rewrites because dedupe works without them (although in the ZFS case, this comes with a constraint, since the dedupe block size is also the clone alignment size). That said, there's still some advantage to avoiding small dedupes even on XFS--extents on XFS are far cheaper than on btrfs, but they're still not free--but that restraint comes built into the design of large-block-size dedupers like duperemove, and all file-oriented dedupers.

And it's probably what makes it different from other dedupers.

Other dedupers use a larger block size (128K or more) which means they can't find matches that don't align to their larger block size. They also use longer hashes and try to store all of them, which slows them down and consumes more memory.

bees can run with a random 4K block hash taken from every 128K extent. This allows matches at 4K granularity, but it takes the same number of hash table entries as 128K blocks. If the extents are 128M long, bees still needs only 4K to match--if it's the right 4K. That means a 1000:1 reduction in hash table size is still usable, if you're willing to completely ignore shorter extents.

The hash table entries in bees are also much smaller: 5x-20x smaller than duperemove or ZFS.

I hear ZFS is developing a sampling-hash deduper. The technique has also been used in places it's usually not visible, like SSD firmware.

bees-next uses MinHash to reduce the number of hashes inserted into the hash table, so every extent over 128 KiB gets no more than 1000 blocks. This lowers the ratio of hash count to extent size for large extents, which reduces the hash table size below where it is today without penalizing short extents. In tests this is making bees faster and smaller--faster because it's not wasting time on short matches, because unique hashes aren't put in the hash table in the first place; and smaller because the leftover space in the hash table goes to short extents which otherwise wouldn't be matched.

I wonder how other dedupers handle this because building hashes just of existing extents doesn't seem very efficient to me in regards to the dedupe hit rate. I wonder if they use CDC or CBC rolling hashes to better deal with that.

There are some dedupe systems (zpaq, git pack) that use rolling hashes; however, they're useless for filesystems, because they can find matches on non-block-aligned boundaries. On a filesystem, dedupes have to be block-aligned, and a regular hash is fine for that.

---

Anyway, back to the topic...

You either get the flood of everything, or it's so silent that you only see the occasional tumbleweed rolling by.

Yeah. Log level 7 is "debug", which in v0.11 is everything.

Log level 6 is "significant events", which includes:

• completed dedup operations
• extent rewrites (or rejections of rewrite)
• every N minutes, a status dump
• every transaction(?), a progress report

That isn't debug, but it's everything bees is supposed to do, and it does these things as quickly as it can. Just the list of deduped extents is a high-bandwidth terminal flood.

The higher layers are very quiet, reporting only errors and shutdown events.

So we need something more than levels. I see value in having options to enable debug logging by component, e.g. "system calls", "extent analysis", "dedupes", "copies", etc. bees-next has the extent rewrite planner, which writes thousands(!) of lines per extent when its debug logs are turned on. Most of these components will be off by default.

I could also see something like a "file watch" log mode, where the log reports new active filenames every second, without repeating filenames that were previously active. So you'd get a low-bandwidth report of active files: continuously active files would appear only once (or once per transid), and most short files wouldn't appear at all--but you'd see a few of them, to get an idea which part of the filesystem bees is looking at. We leave out the dedupe offset/length details, or summarize them.

Another idea is to extend the progress table, maybe making it two rows per size tier. The new rows would report extents + bytes scanned, copied, and deduped per second, and the net data size change, for each tier, and the total from all tiers. This doesn't have filenames but it does tell you if any useful work is being done with all the IO.

creating a "beestop" in the past which would show some nice statistics at the top and a list of recently processed files below, much like we know from top or htop. Maybe that would we a worthwhile side project, maybe even one that an LLM could do if acceptable?

The first thing I thought of was "ok, so just keep the last N extents in a buffer somewhere, and have a userspace tool to look up...the deleted extents...after they're deleted...oh. No filenames any more."

The second thing I thought of was to keep track of recently active files...which you can get from ls -l /proc/$bees_pid/fd/*. That lists all the files bees has opened or cached, which is a rough proxy for which files are involved in bees activity...but then I realized I stopped using that years ago, because it ends up being just a random list of files from the filesystem, with no particular weight on each one, and no indication of purpose (is the file opened for scan, copy, or dedupe?).

Inside bees, we could generate a BESSTATUS-like report file that just lists the last N (100?) open files sorted by access count. Or a JSON file that lists the filenames and counts by type of access (scan, copy, dedupe), and let an external script present the data.

I've been toying with the idea of opening a socket and running a REPL loop that would let users direct bees as it runs and subscribe to asynchronous status data (e.g. pause, resume, scan one extent, get various statistics, get scan/dedupe/copy events, end of crawl, etc.). Then if you want beestop, you write a script that talks to the socket.

[kakra]

Inside bees, we could generate a BESSTATUS-like report file that just lists the last N (100?) open files sorted by access count. Or a JSON file that lists the filenames and counts by type of access (scan, copy, dedupe), and let an external script present the data.

Well, my original idea, if I still remember it correctly, was to have some sort of event write. A user-space process could open it (a unix socket or fifo file), and then it could read all the events (maybe JSONL) and just display the important or subscribed ones. If this is not possible, a polling mode could be implemented, with some simple FTP or SMTP like protocol:

POLL stats
TOTAL:
        addr_block=12514284 addr_compressed_offset=6142762 addr_eof_e=238968 addr_from_fd=5956298 addr_from_root_fd=748590 addr_hole=315321 addr_magic=315321 addr_ms=891 addr_uncompressed=6371522
...

POLL threads
...

POLL recent files
...

I've been toying with the idea of opening a socket and running a REPL loop

Ah well, probably the same idea! :-D

It could be interesting to maybe even make a GUI to display live graphs visually. If it's a unix pipe, it could even be tunneled over SSH maybe? Such a feature could allow really interesting application ideas.

Maybe start simple with some event subscription, no mutation of the bees state yet? Or vice-versa, start with a simple implementation which allows pausing and resuming bees? Then later add event subscription?