Storing backups in Google Drive with duplicity was broken for a long time now. Luckily someone wrote a new backend that is now able to make use of the new OAuth API requirement enforced by Google.

To make use of that new backend we’ll have to run the latest version of duplicity and duply. Both of them aren’t in portage yet so in an attempt to fix that we are going to use our own local overlay that will be located in /usr/local/portage. This is not to be confused with the regular location of portage: /etc/portage.

Step 1: Create the necessary directories and set the appropriate permissions:

mkdir -p /usr/local/portage/{metadata,profiles}
echo 'LocalOverlay' > /usr/local/portage/profiles/repo_name
echo 'masters = gentoo' > /usr/local/portage/metadata/layout.conf
chown -R portage:portage /usr/local/portage

Step 2: Create the make.conf for local ebuilds

Edit or create your local.conf located in /etc/portage/repos.conf/local.conf and add the new overlay:

[LocalOverlay]
location = /usr/local/portage
masters = gentoo
auto-sync = no

Step 3: Add the ebuild to your local overlay

Because the version we are looking for is not located in portage right now we’ll have to use our own ebuild. We can just grab the one someone posted on Gentoo’s Bugtracker (Thanks!):

wget -O duplicity-0.7.03.ebuild https://bugs.gentoo.org/attachment.cgi\?id\=404618

This one’s a few weeks old already and the duplicity maintainers already released a new version so just rename it to duplicity-0.7.04.ebuild, which is the latest development version right now.

mkdir -p /usr/local/portage/app-backup/duplicity
mkdir /usr/local/portage/app-backup/duplicity/files
cp duplicity-0.7.04.ebuild /usr/local/portage/app-backup/duplicity
chown -R portage:portage /usr/local/portage
pushd /usr/local/portage/app-backup/duplicity
repoman manifest
popd

This ebuild also needs a patch (duplicity-0.6.24-skip-test.patch) to work. This patch skips some failing tests and was patched into an earlier version. We need to provide that file so emerge will be able to apply it (The ebuild is looking for that file at the wrong place).

You should be able to find that file in /usr/portage/app-backup/duplicity/files

Just copy it into our local files directory cp duplicity-0.6.24-skip-test.patch /usr/local/portage/app-backup/duplicity/files

and run ebuild duplicity-0.7.04.ebuild manifest to generate a new manifest with the new patch file included.

Now you should be able to emerge the latest version:

root@notmyhostname /usr/local/portage/app-backup/duplicity$ emerge -av1 app-backup/duplicity

These are the packages that would be merged, in order:

Calculating dependencies... done!
[ebuild  N    ~] app-backup/duplicity-0.7.04::LocalOverlay  USE="-s3 {-test}" PYTHON_TARGETS="python2_7" 0 KiB

Total: 1 package (1 new), Size of downloads: 0 KiB

Would you like to merge these packages? [Yes/No]

If duplicity isn’t in your package.accepted_keywords file yet and portage is trying to emerge an old version just edit /etc/portage/package.accepted_keywords and add the following keywords:

app-backup/duplicity ~amd64
app-backup/duply ~amd64

Now that we also added duply we can also install the latest version of duply via emerge (Version 1.10).

Step 4: Install PyDrive

Right now there’s no version of PyDrive listed in the portage tree so we’ll have to install it via pip.

Run pip install PyDrive to install the latest version.

Step 5: Create Google API credentials

The whole process is explained on the duplicity man page (man duplicity) but either Google changed their interface or the man page isn’t very detailed because the process is a little bit different now.

First log into your Google Account and access:

https://console.developers.google.com

Click on “Create Project” and wait for Google to process/create the Project. Once it’s done click on the project and then on “APIs & auth” in the sidebar. You’ll see a bunch of different APIs listed there but we only need “Drive API” which is located in the section called “Google Apps APIs”. Make sure to click on “Enable API” on the top.

Once it’s enabled navigate to “Credentials” also located in the sidebar and click on “Add Credentials”. Select “OAuth Client ID”, then “Other” and then “Create”. In the next step you’ll be able to obtain your “Client ID” and “Client Secret”. We are going to make use of these credentials later.

Step 6: Make duplicity use the new backend

In our duply config file (usually located in ~/.duply/<server name>/config) I was using the following target until the API broke:

TARGET='gdocs://username:password@example.com/backup-incoming/notmyhostna.me'

In the latest version of duplicity the default backend for gdocs is now pydrive and gdocs is an alias to that, we don’t need to change anything here. The username and password will be ignored by duplicity because we are going to supply the credentials another way.

Create a file called gdrive in ~/.duply/<server name>/ with the following content:

client_config_backend: settings
client_config:
   client_id: xxx.apps.googleusercontent.com
   client_secret: yyyy
save_credentials: True
save_credentials_backend: file
save_credentials_file: gdrive.cache
get_refresh_token: True

Make sure to change client_id and client_secret to the values you created earlier and keep the identation like that, otherwise you’ll run into errors later on.

The gdrive.cache file will be created by duplicity after logging in for the first time.

Step 7: Run duply / duplicity

To pass the name of the file containing our credentials to duplicity we could use a environment variable called GOOGLE_DRIVE_SETTINGS like that:

GOOGLE_DRIVE_SETTINGS=gdrive duply <server name> status

If you don’t want to use the variable like that it’s possible to add export GOOGLE_DRIVE_SETTINGS=gdrive to the top of duply’s configuration file.

This was suggested on the duplicity-talk mailing list and is already present in the new configuration files for duply 1.10.

Running duply <server name> status> for the first time will start the authentication process with Google by displaying a link to start the OAuth authentication flow. Click on that link, log into your account and copy the string Google presents to you after being logged in successfully. Paste the string into your terminal and press return to finish the login process.

The whole process looks like this:

GOOGLE_DRIVE_SETTINGS=./gdrive duply notmyhostname status
Start duply v1.10, time is 2015-08-15 17:34:56.
...
--- Start running command STATUS at 17:34:56.933 ---
Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth...

Enter verification code: yyyyy
Authentication successful.
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Sun Apr 12 00:00:04 2015

This should now be the duply output you are already familiar with!

That’s it!