Print only warnings when data integrity problem occurs (conjunction with crontab reports to root)

[szel]

Hello!
I have found scorch while researching data integrity tools available for Linux and one of the users mentioned your script.

The script works beautifully and works as expected and thank you for creating it - great job!

Would it be possible to adjust it so it would only print information when there is actual data integrity problem ("FAILED")?

When used with crontab this would allow to setup cron job and cron by default sends e-mail to the user when there is any output on STDOUT (ie. root).

Right now the script also shows information about "CHANGED" hashes and defeats the purpose without additional scripting.

What do you think?

[Wintersdark]

How do you determine if a file has a data integrity issue vs it's been changed? I've waffled about this myself. For my media files, it's fine - if one changed, that's indicative of bitrot as I don't change them. However, the .NFO files will occasionally change with software changes, and that prompts thousands of CHANGED notifications. I don't really think there is a solution to this - the file either matches it's hash or it doesn't. At least in my case I think excluding data that can change entirely is probably the way to go. There's a way to pass regex filters to Scorch to include/exclude, but I've not figured out how to format the expressions on the command line yet. Sadly there's no examples in the docs :(

…

On Wed., Jan. 8, 2020, 12:16 a.m. szel, ***@***.***> wrote: Hello! I have found scorch while researching data integrity tools available for Linux and one of users mentioned your script. The script works beautifully and works as expected and thank you for creating it - great job! Would it be possible to adjust it so it would only print information when there is actual data integrity problem ("FAILED")? When used with crontab this would allow to setup cron job and cron by default sends e-mail to the user when there is any output on STDOUT (ie. root). Right now the script also shows information about "CHANGED" hashes and defeats the purpose without additional scripting. What do you think? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#29?email_source=notifications&email_token=ACXP6GFRATQWJEAXT5UVQ7TQ4V4UDA5CNFSM4KED4TO2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IEVYXCQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACXP6GDMPDRX3HG3U7PAW5DQ4V4UDANCNFSM4KED4TOQ> .

[Wintersdark]

Maybe store mtime along with the hash, and have an option to ignore changes if the mtime on the file is newer than in the db?

…

On Wed., Jan. 8, 2020, 12:16 a.m. szel, ***@***.***> wrote: Hello! I have found scorch while researching data integrity tools available for Linux and one of users mentioned your script. The script works beautifully and works as expected and thank you for creating it - great job! Would it be possible to adjust it so it would only print information when there is actual data integrity problem ("FAILED")? When used with crontab this would allow to setup cron job and cron by default sends e-mail to the user when there is any output on STDOUT (ie. root). Right now the script also shows information about "CHANGED" hashes and defeats the purpose without additional scripting. What do you think? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#29?email_source=notifications&email_token=ACXP6GFRATQWJEAXT5UVQ7TQ4V4UDA5CNFSM4KED4TO2YY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4IEVYXCQ>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACXP6GDMPDRX3HG3U7PAW5DQ4V4UDANCNFSM4KED4TOQ> .

[trapexit]

You need to provide more detail. How are you running the script?

There is a difference between "CHANGED" and "FAILED". "CHANGED" is a file that is no longer the same thing or metadata is different which is based on the "--diff-fields" argument. "FAILED" means the hash is different but the size (and other diff fields) are the same.

[trapexit]

If you don't want to find changed files (which could be risky) then set the "diff-fields" to an empty string.

[trapexit]

Actually. Looks like it skips to hash checks if the diff check doesn't return so those files will show as failed instead of changed. I guess I could either change the verbosity setup to make changes only print if set high enough or have another argument to ignore changes all together. Let me look.

[trapexit]

As for regex... it's just regular python regex and should work with any common regex pattern. What are you trying to do?

[trapexit]

Maybe store mtime along with the hash, and have an option to ignore changes
if the mtime on the file is newer than in the db?

mtime, inode, size, and mode are stored in the DB. That's how it determines it "changed". The better way of not having it print out changes if it is unnecessary data for you would be changing the verbosity settings so it is optional. By default it only prints changed and failed. I could change it to be just failed on default and move changed to verbose I guess. Though that would mean I'd need to move current verbose up a level.

An alternative would be to run an "update" first to refresh the metadata on changed files then run the "check". "update" is cheaper. It only does the file diff check. Not the hash check (though it does rehash updated files).

[trapexit]

That said... if you don't care about NFO files then just add -F -f ".*.nfo" and it'll only process files that don't match ending in ".nfo".

[szel]

Motivation

I was hoping to use scorch only for bitrot / corruption detection tool, cron to send STDOUT to root e-mail and normal rsync backups to another server to recover in time.

My files change on daily basis and I use rsync "mtime / size" to detect changes and external backup in combination with hardlinks to create external snapshots of data.

Use case would be the following (all commands as root user where there is nothing else but my data):

1. Initialize checksums for /home

# scorch -d /var/lib/scorch.db add /home

2. Create cron script for periodic data scrubbing

# test -d /root/cron || mkdir -p /root/cron
# cd /root/cron
# cat << EOF > ./scorch.sh
#!/bin/sh
scorch -D 'size,mtime' -d /var/lib/scorch.db check+update /home
scorch -D 'size,mtime' -d /var/lib/scorch.db append /home
scorch -D 'size,mtime' -d /var/lib/scorch.db cleanup /home
EOF
# chmod +x ./scorch.sh
# chmod o-rwx ./scorsh.sh

3. Install cron to run once per month

# crontab -e
# m h  dom mon dow   command
# run data integrity (bitrot) checks on 15th of each month at 03:00 and report to root e-mail
0  3   15 * *		/root/cron/scorch.sh

If there is anything on script STDOUT - it will be sent to root e-mail (my machine has /etc/aliases configured and e-mail working).

4. Configure external backups and keep data since last data scrub

In order to recover when bitrot is detected - we need to know "which files" are good and we need proper backups.

Here is a sample rsync script that I use:

#!/bin/bash

backup_date=`date +%Y-%m-%d`

# configure where to keep last backup and daily archives / snapshots
local_backup_dir="/mnt/backup/home/current"
local_archive_dir="/mnt/backup/home/archives"

# show commands and stop on first error
set -x
set -e

# show current date
date

# backup via ssh and rsync
ionice -c 3 rsync -ave ssh --numeric-ids --delete --rsync-path="ionice -c 3 rsync" \
root@SERVER_IP:/home/ "${local_backup_dir}/" || return_status="$?"

# ignore "vanished files"
test "$return_status" -eq "24" && return_status="0"

# show when backup finished copying data
date

backup_time=`date +%Y%m%d%H%M`

# create snapshot from current backup using hard links
cp -al "$local_backup_dir" "$local_archive_dir/$backup_date"
touch -m -t "$backup_time" "$local_archive_dir/$backup_date"

# purge archives / snapshots older than 90 days
find "$local_archive_dir" -mindepth 1 -maxdepth 1 -mtime +90 -type d | xargs -I {} rm -rf {}

# show when script has ended
date

# return correct exit code
exit "$return_status"

[szel]

I have updated my original answer.

For me any of the following would be great:

1. Have the ability of launch scorch in mode where it only looks and logs to STDOUT or STDERR when bitrot / data corruption is detected.
2. Make scorch log FAILED events on STDERR.

If you have any other questions please don't hesitate to ask :-)

[Wintersdark]

Just exclude specific extensions. Both to avoid CHANGED notifications for files that may change, and also for performance purposes - I only check X files at random each run, so I'd prefer just checking files I care about. I'm moderately familiar with regex overall and that's not a super complicated thing to google, but I've not been able to get it to work. Presumably I'm just formatting it wrong.

…

On Wed., Jan. 8, 2020, 5:17 a.m. trapexit, ***@***.***> wrote: As for regex... it's just regular python regex and should work with any common regex pattern. What are you trying to do? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#29?email_source=notifications&email_token=ACXP6GG6TE7J4CLCXXO4WULQ4W75XA5CNFSM4KED4TO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIMGFQA#issuecomment-572023488>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACXP6GD74ZTRJL3TESLJYKDQ4W75XANCNFSM4KED4TOQ> .

[trapexit]

You can use --maxdata and/or --maxactions to do more regularly checks (with --sort=random). You'd catch things more quickly.

I prefer to talk in terms of general features rather than implementations. If I understand you right you don't care about "changes"? You have lots of legitimate changes and all those changes will make seeing FAILED values?

BTW... if check+update doesn't update on FAILED. Only "CHANGED".

[trapexit]

re regex: ^.*\.(ext1|ext2|ext3)$

That will match those extensions. If you want to easily negate the regex just use -F.

[trapexit]

re changes: if you want to index the files but don't care about changes (or want them emailed at a different time) you could use different databases for them and/or use the filter. Can also use the null hash if you don't even care about bitrot checks given they change so often.

[trapexit]

What I'm getting at with these suggestions is that I'm not sure it's a good idea to ignore changes wholesale. It is better to have different checks for different usecases which I think is mostly available given the features available. Printing to stderr feels wrong because it's not an app failure. grep prints failed file opens to stderr because it's an error about the workflow. Not with the data. A FAILED hash check is not a workflow issue. That said I do have app errors printing to stdout which I should fix.