Weird interaction with bees deduplication and compression

[MNarath1]

I have bees enabled on a system where btrfs compression is enabled in fstab problem is i run more and more into situations now where bees seems to be unable to properly deduplicate data unless i manually compress files using btrfs defrag even when bees goes over the filesystem before and supposedly deduplicates all possible data (that should already be compressed) when i run btrfs defrag with the compression flag enabled i get a lot of space back and afterwards running bee over the same filesystem again it manages to squeeze even more out.

I am now really confused if this stems from bees or if my automatic compression via fstab arguments are broken.

[MNarath1]

Here is my fstab for reference```
UUID=e4ebc2a1-2cf0-4420-a92b-9849b7468230 / btrfs subvol=root,compress=zstd:1,x-systemd.device-timeout=0 0 0 UUID=93fc9660-9bf9-4f32-ad80-d4aebad2e6d9 /boot ext4 defaults 1 2 UUID=9935-65BC /boot/efi vfat umask=0077,shortname=winnt 0 2 UUID=e4ebc2a1-2cf0-4420-a92b-9849b7468230 /home btrfs subvol=home,compress=zstd:1,x-systemd.device-timeout=0 0 0 /dev/disk/by-label/SecondaryDrive /mnt/SecondaryDrive auto nosuid,nodev,nofail,compress=zstd:1,x-gvfs-show 0 0

[MNarath1]

The filesystem that makes the most issues is the Secondary Drive line thats where i actually notice this issue

[Zygo]

Get btdu, navigate to <UNREACHABLE>, make a note of the number. Run defrag or bees, then run btdu again and see if the number changes.

If free space shows up after btrfs fi defrag, it's usually because it released a lot of unreachable space by copying the reachable blocks to new files.

[MNarath1]

Get btdu, navigate to <UNREACHABLE>, make a note of the number. Run defrag or bees, then run btdu again and see if the number changes.

If free space shows up after btrfs fi defrag, it's usually because it released a lot of unreachable space by copying the reachable blocks to new files.

Defragging did work and i managed to reclaim 50 GB of unreachable space but for some reason i do have to specifically use it with compression enabled trough otherwise this works great.
I am considering writing a script that uses btdu to defrag only files that have such unreachable extends associated periodically thinking that it probably would heavily cut down on the time needed to defrag and afterwards deduplicate the files again.

[MNarath1]

Get btdu, navigate to <UNREACHABLE>, make a note of the number. Run defrag or bees, then run btdu again and see if the number changes.

If free space shows up after btrfs fi defrag, it's usually because it released a lot of unreachable space by copying the reachable blocks to new files.

would bees be able to deduplicate the defragmented files afterwards or would i have to delete the hashtable first?

[kakra]

Since the defrag created new extents recorded in a new transaction ID, bees will pick these files up on its next cycle. Bees doesn't walk files, it walks transaction IDs and extents.