rclone

Warning

Commands have the potential to delete data, ensure there is a WORM backup in place and test all commands that write with -n first. I am not responsible for any data loss.

rclone is the technology used on the NAS machines to replicate the NAS offsite and to the S3 GDA. This page serves as a reference to the initial transfer commands, non-interactive replication, restoring, etc.

Mounts

rclone supports lots of different backends. It might be worth using a different backend for a remote NAS instead of the mounted SMB share. Backends that might work well include sftp, sshfs

Bandwidth Limits

Agreed with Dad to set the peak speed limits to 5Mbps and overnight to unlimited. This can be achieved with a --bwlimit flag:

# Disable limit at 2am, set to 5Mbps at 6am
--bwlimit "02:00,off;06:00,0.625M"

# Disable limit at 2am, set upload to 5Mbps at 6am
--bwlimit "02:00,off;06:00,off:0.625M"

Initial Transfer

Originally did a bidirectional sync locally after state drifted. Now needs to be fixed remotely. Have to be careful to not actually delete anything. Should be achievable with a bisync that checks for possible file renames and uses mtime and size to do so (doesn't involve reading every file).

Integrity Checking (Checksums/Hashes)

Ideally need some way to compute this possibly with the hasher backend, and compute checksums locally so that I don't transfer over the remote. With ZFS and duplicate drives with scrubbing bitrot should never be an issue.

S3 Retrieval

Encryption

On the S3 GDA backend, we don't want our files to be easily readable.