Setting up rclone to automatically download Google Drive content

I’ve been feeling uneasy about Google Drive since the reports about users’ data disappearing, so I had to look up how to back up your drive with rclone (https://rclone.org/). It’s surprisingly easy! For future reference, I’ve written it down here for myself (and others).

The following quick guide includes:

Steps to set up rclone with Google Drive

Sets up a script to download Google Drive to a folder on your computer

Steps to set up Cloudflare R2

Sets up a script to zip the folder and upload it to Cloudflare R2

Other thoughts / notes:

I think the rclone / Google Drive connection expires within 24 hours, but I haven’t tried that yet (since I just started). Will update this to get around the issue if it does indeed expire quickly.

To edit the rclone config any time: subl /Users/janzheng/.config/rclone/rclone.conf

R2 isn’t strictly necessary but I think is good to have a third backup option. Same goes for an external hard drive.

If your Google Drive has shortcuts pointing to things you can’t access anymore you’ll get a “dangling shortcuts” error. Add --drive-skip-dangling-shortcuts to skip shortcuts and suppress the error

rclone will pick up where it left off, so you don’t have to re-download files! Not sure if it checks for file changes, but I assume it does

rclone tends to hang sometimes (even on very small files). Not sure if it’s completely crashed or if it does something in the background. Using --vv (very verbose) doesn’t output anything; it just sits there. Quitting + restarting it seems to work.

I get my internet by hotspotting an iPhone 12 mini with Vodafone 5G in Sydney (there’s barely any fiber in the city, and NBN is slower than 5G for our area), and rclone ends up taking down the phone’s internet. I blame Vodafone.

You can easily set up automation for multiple Google Drives and other sources. Super neat!

Steps to set up rclone on a Mac:

Install clone: brew install rclone

rclone config — https://rclone.org/drive/

(n) new remote
(jan_gdrive) remote name
(18) Google Drive
(Enter) Google App ID
(Enter) Google Client Secret
(2) Read-only access
(Blank) Service account JSON
(Enter/n) Don’t edit advanced config
(Enter/y) Use web browser to login
(Enter/n) not a team drive
(Enter/y) keep name (jan_gdrive)
Done!

Commands (for “jan_gdrive”)

rclone lsd jan_gdrive: list top-level directories
clone ls jan_gdrive: list all the files in drive
rclone copy jan_gdrive:/MyFolder . — copy a MyFolder from google drive to local folder
rclone copy --update --verbose --transfers 30 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s --stats-file-name-length 0 "jan_gdrive:FOLDER" . — fancier copy of a folder to a local folder (source: https://www.youtube.com/watch?v=GvPI9ls0ahw)

Setup bash script for downloading

which rclone make sure where it’s installed
/opt/homebrew/bin/rclone (is where mine is installed)
Navigate to a proper folder, e.g. /Users/yourusername/Documents or /Downloads or wherever you want to download your files
A few setup ideas:

Create a separate folder for the Google Drive: mkdir jan_gdrive: I find this to be more convenient when you have multiple Drives to back up; this makes managing folders way neater.
Considering compressing the Google Drive folder. This is optional but I like having a single file uploaded to R2; it’s just cleaner to manage: zip -r jan_gdrive.zip ./jan_gdrive/

subl jan_gdrive.sh to create an empty file in sublime editor

#!/bin/bash

/opt/homebrew/bin/rclone copy --update --verbose --drive-skip-dangling-shortcuts --transfers 30 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s --stats-file-name-length 0 "jan_gdrive:FOLDER" ./jan_gdrive/FOLDER

# optionally zip up the folder for upload
zip -r ./jan_gdrive.zip ./jan_gdrive/

# optionally remove the original folder
# rm -r ./jan_gdrive/FOLDER

chmod +x jan_gdrive.sh give permissions to run the file
make sure the bashfile works by running it: ./jan_gdrive.sh

Setup cron

cron can perform jobs on a set schedule, like every Sunday at midnight.
get the full path of your bash file: pwd to get folder path; for me this is /Users/janzheng/Desktop/Projects/rclonedrive/jan_gdrive.sh (yes I put everything under Desktop)
open crontab: crontab -e
To run every Sunday at midnight: 0 0 * * 0 /Users/janzheng/Desktop/Projects/rclonedrive/jan_gdrive.sh
If you need some other schedule, just ask ChatGPT.

Using rclone to upload files to Cloudflare R2

If you want to put these backups online, I find Cloudflare R2 as a cheap and easy alternative to AWS.

Nagivate to R2, then create a new bucket. Call it rclone-backups

Generate an R2 API token if you don’t have it: https://developers.cloudflare.com/r2/api/s3/tokens/

Go to R2 → Manage R2 API Tokens → Create API Token → Object Read & Write
Apply to specific buckets rclone-backups and set a 1 year TTL (it’s good practice in case your keys leak)
Create your API Key
Copy your key values (you won’t be able to see Secret Access Key again)

Because you’ll be uploading a lot of data, you can’t just drag and drop your zip file into R2. You need to use the S3 Compatibility API with presigned URLs. Get your S3 API URL from the Settings tab: https://***********.r2.cloudflarestorage.com/rclone-backups

Connect rclone to your Cloudflare R2 URL

Find the location of your rclone config file… by typing rclone config file
Now edit the file to add your new backup url: subl /Users/janzheng/.config/rclone/rclone.conf. Note that you have to set no_check_bucket=true because you set specific read/write permissions for your bucket earlier.

[r2backups]
type = s3
provider = Cloudflare
access_key_id = abc123
secret_access_key = xyz456
endpoint = https://accountid.r2.cloudflarestorage.com
acl = private
no_check_bucket = true

Moving files into rclone-backups:

rclone copy ./jan_gdrive.zip r2backups:rclone-backups/jan_gdrive will copy the file into R2 storage: rclone-backups/jan_gdrive/jan_gdrive.zip

Now you can see the view the files you just uploaded: rclone tree r2backups:rclone-backups

To download these files, either go to the R2 interface or use rclone: rclone copy r2backups:rclone-backups/jan_gdrive ./mynewfolder

Modify the bash file to include uploading the zipped file. Here’s the modified jan_gdrive.sh file with a few additions:

#!/bin/bash

# Google Drive folder to back up
FOLDER=""

# Name for local copy
NAME="phaus_gdrive"

# BUCKET="rclone-backups"
BUCKET="phageaus"

# run this by typing ./phaus_gdrive.sh

echo "---->> [Download] Starting!"
/opt/homebrew/bin/rclone copy --update --verbose --drive-skip-dangling-shortcuts --transfers 30 --checkers 8 --contimeout 60s --timeout 300s --retries 3 --low-level-retries 10 --stats 1s --stats-file-name-length 0 "$NAME:$FOLDER" ./$NAME/${FOLDER:+/}

echo "<<---- [Download] Finished!"


# optionally zip up the folder for upload
echo "---->> [Compressing] Starting!"
zip -r ./$NAME.zip ./$NAME/
echo "<<---- [Compressing] Finished!"

# optionally remove the original folder
# rm -r ./$NAME/FOLDER

echo "---->> [R2 Upload] Starting!"
rclone copy ./$NAME.zip r2backups:$BUCKET/$NAME --progress --verbose --stats 10s
rclone tree r2backups:rclone-backups
echo "<<---- [R2 Upload] Finished!"

Now you should be able to have weekly Google Drive backups running!

Final thoughts

I haven’t run this with cron yet, and I’ve never actually used cron so let’s see what happens! (it’s very easy, and it works!)

I don’t have “real” internet, only 5G hotspotting, so my laptop doesn’t have connection most of the time (the phone keeps resetting itself). No idea what cron will do at midnight

I might look at some other automation service to do this instead of on my Macbook, or one day if I get a desktop

./$NAME/${FOLDER:+/} was added for more flexibility when specifying folder

If you keep running into errors, switch from --verbose to --vv for “very verbose” so you know what’s being added and what’s going on under the hood

Send me a note if there’s better ways to do this!

Backing up Google Drive to R2 with rclone

Setting up rclone to automatically download Google Drive content

Steps to set up rclone on a Mac:

Using rclone to upload files to Cloudflare R2

Final thoughts