Software Tools
Rclone
Rclone
About rclone
Many people use a web browser to transfer files between local and cloud storage. In comparison, Rclone is a robust command-line utility for synchronizing and transferring files to and from cloud storage. The command-line subcommands are similar to standard Unix file management commands, and a long list of cloud providers are supported. You can use rclone to transfer and synchronize files between HBSGrid storage and your HBS Microsoft OneDrive storage, or any other supported cloud storage provider you may use. Rclone supports transferring only files that have changed and resuming interrupted transfers.
Note: Please remember that cloud storage services, rclone, and HBSGrid storage should be used in compliance with data safety guidelines and any Data Usage Agreement or approved IRB. If you have hesitations or questions, please consult HBS IT Security's web pages on data safety levels and appropriate data storage locations (see large graphic), or contact informationsecurity@hbs.edu.
Configuring rclone
You must configure rclone for each cloud storage account you wish to use. Note that rclone configuration should be done on a login node so that rclone can use the web browser for authentication. This process is usually simple: open a terminal and run
module load rclone
to make rclone available, and then run
rclone config
to start the configuration wizard. The program will prompt you for the name and type of account you want to configure and other information needed to set up the connection. Generally you can just accept the defaults, but detailed documentation is available if you need it.
OneDrive configuration
Follow these steps to configure rclone for OneDrive:
- Open a terminal and run
rclone config
- Enter
n
at the prompt to create a new config - Enter a name of your choosing at the prompt, e.g.,
OneDrive
- At the "Storage" prompt Choose
onedrive
- Respond to subsequent prompts, accepting defaults unless you wish to change them
- A web browser window will open to retrieve credentials. Sign in to your OneDrive account if prompted
- Choose
ondedive
orsharepoint
at the prompt - Choose the drive you wish to use at the prompt
- Respond to subsequent prompts, accepting defaults unless you wish to change them
- At the main menu choose
q
to quit the rclone configuration utility
Dropbox configuration
Follow these steps to configure rclone for Dropbox:
- Open a terminal and run
rclone config
- Enter
n
at the prompt to create a new config - Enter a name of your choosing at the prompt, e.g.,
DropBox
- At the "Storage" prompt Choose
dropbox
- Respond to subsequent prompts, accepting defaults unless you wish to change them
- A web browser window will open to retrieve credentials. Sign in to your DropBox account if prompted
- Choose
Yes this is OK
at the confirmation prompt - At the main menu choose
q
to quit the rclone configuration utility
Configuring other cloud storage providers follows a similar process -- refer to the detailed documentation if you have any doubts about how to respond to the prompts.
Basic use
Once you have configured rclone, you can use it to manage files in cloud storage. The more commonly used rclone commands include:
-
rclone ls remote:path
(list objects on remote) -
rclone sync remote:path local/path
(make local/path the same as remote:path) -
rclone copy local/path remote:path
(copy local/path to remote/path).
More information
As mentioned, Rclone supports a large number of cloud storage providers, and has many
features and options. And the project provides comprehensive (yet comprehensible!)
documentation at https://rclone.org/docs/. And check out the full list of rclone
commands.