page for documenting rclone

Setting up Rclone

Rclone should already be installed in BitCurator but if you are transferring content that you need to log in to access, you'll need to do some additional set up to connect to cloud storage service. This will have to be repeated for each cloud storage service and possibly reconnected if some time has passed between the last time you've used Rclone. Below lists the general set up process with sections for specific cloud storage providers used in practice so far. If there is a cloud storage provider you need to access that's not covered below, please contact the digital archivist.

General setup

This general setup will work with DropBox

1. Open the terminal on the left-hand side of the BitCurator desktop

2. Type "rclone config" and hit enter

3. Type "new" or "n" for new remote, i.e. new cloud service
4. Enter a name for the cloud serve (e.g. dropbox)
5. Choose the number that is listed in the terminal for cloud service you named in step 4. (e.g. DropBox is (fill in blank))
6. Hit enter for questions about client id and client secret to accept the defaults
7. Type "no" and hit enter for the advanced config question
8. Type "y" and hit enter for the auto config option
9. A link will appear in the terminal, highlight and copy it. Paste the link into your internet browser.
10. On the page that pops up, choose the option to authorize Rclone access
11. Return to the terminal, if you see "got code" as part of the output above, type "y" for "this is ok" and hit enter
12. Type "q" and hit enter to quit if done setting up cloud connections.

OneDrive or Sharepoint

Choose a national cloud region for OneDrive

Choose 1 for Microsoft Cloud Global

Type of connection, 1 for basic OneDrive Personal or Business, you will get your personal one drive account and anything shared with you. 2 sharepoint root will give you sharepoints that are open to you. 3 you can choose enter the site url

Google Drive

Using Rclone

In general we use Rclone for transferring files. When possible we also use it to confirm the fixity of the files downloaded.

Copying files

The command to copy files is fairly simple, you specify that you want to copy the files, enter their location, and then their destination. For instance:

rclone copy [name of remote as set up above]:[name_of_folder_or_file (if spaces in name, you can put quotation marks around this after the colon)] [/path/to/destination/folder, i.e. processing folder, etc.]

Here is an example:

rclone copy dropbox:"Robert Birgeneau INT" /media/sf_BCShared01/processing/2022_061acc 

Extracting checksums

Some cloud providers have checksums stored in their system that you can extract and facilitate fixity checking. Some are unique to their system or some can be more standard types. Here is a general layout of the command to extract the checksums into a text file:

rclone hashsum [type of checksum] [remote source]:"folder_name or file" (same as when copying) --output-file /path/to/output/file.txt 

Here is an example for dropbox:

rclone hashsum dropbox dropbox:"Robert Birgeneau INT" /media/sf_BCShared01/processing/2022_061acc/submissionDocumentation/dropbox_checksums.txt

Here is an example for OneDrive or SharePoint:

rclone hashsum quickxor onedrive:"Robert Birgeneau INT" /media/sf_BCShared01/processing/2022_061acc/submissionDocumentation/onedrive_checksums.txt

Here is an example for Google Drive, because you can reuse md5 checksums in Archivematica, we have named the checksum file in the convention that it will recognize:

rclone hashsum md5 googledrive:"Robert Birgeneau INT" /media/sf_BCShared01/processing/2022_061acc/submissionDocumentation/checksum.md5

Confirming fixity

In order to confirm fixity, there are number of options:

Confirm the using the checksums you extracted in the steps above:

rclone checksum [checksum type] /path/to/checksum/file.txt /path/to/local_directory/of/copied_file

Confirm without local checksums/those that rclone generates:

rclone check [remote name]:[source folder] /path/to/local_copy/of/source_folder