Syncing Files on the Raspberry Pi

Submitted by Evan Boldt on Thu, 08/15/2013 - 14:08

Introduction

Syncing files automatically from a Raspberry Pi is a good idea to make sure that nothing gets lost the Pi breaks. The problem is that due to the unusual ARM architecture, most of the common methods for syncing are not available. Dropbox does not have an ARM package. Google Drive never had a Linux client. There are ways around these limitations either by use of lower quality third party applications to interface with the major sync solutions or by use of open source and self hosted solutions.

Sparkleshare

Sparkleshare is a self-hosted solution. Sparkleshare uses Git on its backend, so it's particularly good for text files. You can also use a git host like Bitbucket. It provides an easy to use graphical interface and is very easy to set up

On the Raspberry Pi:

pi@raspberrypi:~# sudo apt-get install sparkleshare

Building Sparkleshare (unnecessary if using the package from apt-get):

#install all mono stuff, which is probably (definitely) overkill
pi@raspberrypi:~# sudo apt-get install libmono-2.0-dev mono-complete gtk-sharp2 libwebkit-cil-dev libnotify0.4-cil libnotify-cil-dev
pi@raspberrypi:~# mkdir apps ; cd apps
pi@raspberrypi:~# wget <getrecentlink>.tar.gz
pi@raspberrypi:~# tar -xzf sparkleshare-linux-1.1.0-tar.gz
pi@raspberrypi:~# cd sparkleshare-1.1.0/
pi@raspberrypi:~# ./configure
pi@raspberrypi:~# make
pi@raspberrypi:~# sudo make install

On your "host" computer (unnecessary if using third party Git host)

root@evan-ubuntu:~# sudo su && cd
# Fetch the Dazzle script
root@evan-ubuntu:~# curl https://raw.github.com/hbons/Dazzle/master/dazzle.sh \
  --output /usr/bin/dazzle && chmod +x /usr/bin/dazzle
# Run the initial setup
root@evan-ubuntu:~# dazzle setup
# Link SparkleShare clients using their Client ID found in the status menu
root@desktop:~# dazzle link
Paste your Client ID (found in the status icon menu) below and press <ENTER>. Client ID: ssh-rsa AAAAB3NzaC1yc2EAAAADAQ... ... ...TnTL4SK1+7gYKT raspberrypi # Create a new project. Add as many as you need root@evan-ubuntu:~# dazzle create raspberry # raspbery project or directory name
Project "raspberry" was successfully created. To link up a SparkleShare client, enter the following details into the "Add Hosted Project..." dialog: Address: ssh://storage@evan-ubuntu.local Remote Path: /home/storage/raspberry

Setup on the client side uses a GUI:

pi@raspberrypi:~# DISPLAY=:0 sparkleshare start

Sparkleshare is a way to sync files to a Raspberry Pi which uses Git to manage changes.   

Grive

Grive is the unnofficial Google Drive client for Linux. Google has no official linux client, but in this specific case that is actually a good thing, in a weird way, because the open source version can be built from source and can run on the Raspberry Pi's ARM processor.

First, try installing from the repos. In a lot of debian archives, grive is now available. At the point of writting this though, it is not currenlty in the Raspbian repo, but is in other Debian repos.

pi@raspberrypi ~ $ sudo apt-get install grive 

Building Grive from Source

If that package does not work, you can build it from source, but it takes a really long time to build. 

This will add quite a few packages to your Raspbian - a couple hundred MB worth, which takes more than a couple minutes to install. The build also takes about 20 minutes.

sudo apt-get install cmake git build-essential libgcrypt11-dev libjson0-dev libcurl4-openssl-dev libexpat1-dev libboost-filesystem-dev libboost-program-options-dev binutils-dev libqt4-core libqt4-dev libboost-test-dev libboost-filesystem-dev libboost-program-options-dev libboost-system-dev libyajl-dev

pi@raspberrypi $ mkdir ~/apps ; cd ~/apps pi@raspberrypi ~/apps $ git clone git://github.com/Grive/grive.git
pi@raspberrypi ~/apps $ cd grive
pi@raspberrypi ~/apps/grive $ cmake .
-- Found libgcrypt: -L/lib/arm-linux-gnueabihf -lgcrypt
-- Found JSON-C: /usr/lib/arm-linux-gnueabihf/libjson.so
-- Boost version: 1.49.0
-- Found the following Boost libraries:
--   program_options
--   filesystem
--   unit_test_framework
--   system
-- Found libbfd: /usr/lib/libbfd.so
-- Found libiberty: /usr/lib/libiberty.a
-- Boost version: 1.49.0
-- Found the following Boost libraries:
--   program_options
-- Boost version: 1.49.0
-- Configuring done
-- Generating done
-- Build files have been written to: /home/pi/apps/grive

If you do not add these scope resolution operators, the compiler does not seem to know to use the Boost library. It might also have to do with the sizes of variables on the ARM processor versus normal.

pi@raspberrypi ~/apps/grive $ nano libgrive/src/drive/State.cc
void State::Write( const fs::path& filename ) const
{
        Json last_sync ;

        last_sync.Add( "sec", Json((boost::uint64_t) m_last_sync.Sec() ) );
        last_sync.Add( "nsec", Json((boost::uint64_t) m_last_sync.NanoSec() ) );

        //last_sync.Add( "sec", Json(m_last_sync.Sec() ) );
        //last_sync.Add( "nsec",        Json(m_last_sync.NanoSec() ) );

        Json result ;
        result.Add( "last_sync", last_sync ) ;

        //result.Add( "change_stamp", Json(m_cstamp) ) ;
        result.Add( "change_stamp", Json((boost::uint64_t) m_cstamp) ) ;

        std::ofstream fs( filename.string().c_str() ) ;
        fs << result ;
}

Now that you've patched it a little bit, you can actually build it. This takes a while.

pi@raspberrypi ~/apps/grive $ make
[  1%] Building CXX object libgrive/CMakeFiles/grive.dir/src/json/JsonParser.cc.o
/tmp/ccN9aaS9.s: Assembler messages:
/tmp/ccN9aaS9.s:1258: Warning: swp{b} use is deprecated for this architecture
[  3%] Building CXX object libgrive/CMakeFiles/grive.dir/src/json/ValBuilder.cc.o
/tmp/cc0qtdOv.s: Assembler messages:
/tmp/cc0qtdOv.s:1317: Warning: swp{b} use is deprecated for this architecture
[  5%] Building CXX object libgrive/CMakeFiles/grive.dir/src/json/JsonWriter.cc.o
[  7%] Building CXX object libgrive/CMakeFiles/grive.dir/src/json/ValResponse.cc.o
[  8%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/File.cc.o
/tmp/ccCrhkFO.s: Assembler messages:
/tmp/ccCrhkFO.s:1317: Warning: swp{b} use is deprecated for this architecture
[ 10%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/Config.cc.o
[ 12%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/StdStream.cc.o
[ 14%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/Exception.cc.o
/tmp/cca93GMZ.s: Assembler messages:
/tmp/cca93GMZ.s:1298: Warning: swp{b} use is deprecated for this architecture
[ 16%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/OS.cc.o
/tmp/cchdH7ge.s: Assembler messages:
/tmp/cchdH7ge.s:1258: Warning: swp{b} use is deprecated for this architecture
[ 17%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/DateTime.cc.o
/tmp/ccnsT4HV.s: Assembler messages:
/tmp/ccnsT4HV.s:1258: Warning: swp{b} use is deprecated for this architecture
[ 19%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/StringStream.cc.o
[ 21%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/SignalHandler.cc.o
[ 23%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/Crypt.cc.o
/tmp/ccx344KH.s: Assembler messages:
/tmp/ccx344KH.s:1258: Warning: swp{b} use is deprecated for this architecture
[ 25%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/MemMap.cc.o
[ 26%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/log/CompositeLog.cc.o
[ 28%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/log/DefaultLog.cc.o
[ 30%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/log/CommonLog.cc.o
[ 32%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/log/Log.cc.o
[ 33%] Building CXX object libgrive/CMakeFiles/grive.dir/src/xml/String.cc.o
[ 35%] Building CXX object libgrive/CMakeFiles/grive.dir/src/xml/NodeSet.cc.o
/tmp/ccshU8Rr.s: Assembler messages:
/tmp/ccshU8Rr.s:1284: Warning: swp{b} use is deprecated for this architecture
[ 37%] Building CXX object libgrive/CMakeFiles/grive.dir/src/xml/Node.cc.o
/tmp/cc03tDy6.s: Assembler messages:
/tmp/cc03tDy6.s:1284: Warning: swp{b} use is deprecated for this architecture
[ 39%] Building CXX object libgrive/CMakeFiles/grive.dir/src/xml/TreeBuilder.cc.o
/tmp/ccEaKl2M.s: Assembler messages:
/tmp/ccEaKl2M.s:1258: Warning: swp{b} use is deprecated for this architecture
[ 41%] Building CXX object libgrive/CMakeFiles/grive.dir/src/bfd/Debug.cc.o
[ 42%] Building CXX object libgrive/CMakeFiles/grive.dir/src/bfd/SymbolInfo.cc.o
[ 44%] Building CXX object libgrive/CMakeFiles/grive.dir/src/bfd/Backtrace.cc.o
Linking CXX static library libgrive.a
[ 82%] Built target grive
Scanning dependencies of target btest
[ 83%] Building CXX object libgrive/CMakeFiles/btest.dir/test/btest/UnitTest.cc.o
[ 85%] Building CXX object libgrive/CMakeFiles/btest.dir/test/btest/ValTest.cc.o
/tmp/ccK8TOeR.s: Assembler messages:
/tmp/ccK8TOeR.s:1331: Warning: swp{b} use is deprecated for this architecture
[ 87%] Building CXX object libgrive/CMakeFiles/btest.dir/test/btest/JsonValTest.cc.o
/tmp/ccOAUEHb.s: Assembler messages:
/tmp/ccOAUEHb.s:1331: Warning: swp{b} use is deprecated for this architecture
Linking CXX executable btest
[ 87%] Built target btest
Scanning dependencies of target grive_executable
Linking CXX executable grive
[ 89%] Built target grive_executable
[ 91%] Generating src/moc_MainWnd.cxx
[ 92%] Generating ui_MainWindow.h
Scanning dependencies of target bgrive_executable
[ 94%] Building CXX object bgrive/CMakeFiles/bgrive_executable.dir/src/DriveModel.cc.o
/tmp/ccgDURnf.s: Assembler messages:
/tmp/ccgDURnf.s:1731: Warning: swp{b} use is deprecated for this architecture
[ 96%] Building CXX object bgrive/CMakeFiles/bgrive_executable.dir/src/MainWnd.cc.o
[ 98%] Building CXX object bgrive/CMakeFiles/bgrive_executable.dir/src/main.cc.o
[100%] Building CXX object bgrive/CMakeFiles/bgrive_executable.dir/src/moc_MainWnd.cxx.o
Linking CXX executable bgrive
[100%] Built target bgrive_executable

Well, now that that's over, the install process seems to need an extra doc, so we'll just add an empty one, which might not be necessary.

pi@raspberrypi ~/apps/grive $ mkdir /home/pi/apps/grive/bgrive/doc/
pi@raspberrypi ~/apps/grive $ touch /home/pi/apps/grive/bgrive/doc/grive.1
pi@raspberrypi ~/apps/grive $ sudo make install

Usage

First, set up a folder to sync your stuff to. Then you open a long authentication link in a browser and log in to your Google account. It will then give you a long code to copy back to the program. It is important that your current directory is the directory you wish to sync with your Google Drive before you do this.

pi@raspberrypi ~ $ mkdir ~/gdrive/ ; cd ~/gdrive
pi@raspberrypi ~/gdrive $ grive -a ----------------------- Please go to this URL and get an authentication code: https://accounts.google.com/o/oauth2/auth?scope=............ ----------------------- Please input the authentication code here:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Reading local directories Synchronizing folders Reading remote server file list Synchronizing files
... ... ... ...

Now, to set up automatic sync, it unfortunately has to be done this somewhat simplistic way:

pi@raspberrypi ~ $ mkdir ~/bin/
pi@raspberrypi ~ $ echo "#! /bin/bash > cd ~/gdrive/ > grive" > ~/bin/gsync
pi@raspberrypi ~ $ chmod +x ~/bin/gysnc
pi@raspberrypi ~ $ crontab -e
*/10 * * * * /home/pi/bin/gsync

Now, your Google Drive will sync to your to your Raspberry Pi every 10 minutes. There might be a way to do this every time a file changes on the Raspberry Pi using inotifywait, but it would not sync changes from Google Drive until there is a change locally.

dropfuse

Dropfuse is not one of the best or most secure options, but Dropbox is a really nice service. Dropfuse is a third party script that relies on the "share link" feature. It does not actually connect to your entire Dropbox account. It will only connect to a single shared folder. Log onto dropbox.com, navigate to the folder you want sync'd and click "share link". You'll find a link like this:

https://www.dropbox.com/sh/AAAAAAAAAAAAAAA/BBBBBBBBBB

Modify the link yourself to this format:

https://www.dropbox.com/s/AAAAAAAAAAAAAAA

The install process is relatively easy.

pi@raspberrypi ~ $ sudo apt-get install python-pyquery
pi@raspberrypi ~ $ sudo usermod -a -G fuse pi
pi@raspberrypi ~ $ mkdir apps
pi@raspberrypi ~ $ mkdir Dropbox
pi@raspberrypi ~ $ cd apps
pi@raspberrypi ~/apps $ git clone git://github.com/arekzb/dropfuse.git Cloning into 'dropfuse'... remote: Counting objects: 38, done. remote: Compressing objects: 100% (21/21), done. remote: Total 38 (delta 19), reused 34 (delta 17) Receiving objects: 100% (38/38), 14.16 KiB, done. Resolving deltas: 100% (19/19), done.
pi@raspberrypi ~/apps $ cd dropfuse
sudo python dropfuse.py https://www.dropbox.com/s/c6ecc2plwconh5x ~/Dropbox &

 To stop the service, do this:

pi@raspberrypi ~ $ fusermount -u Dropbox

BitTorrent Sync

A word of warning, btsync causes lots of networking issues. On an Ubuntu machine, it can cause 10% of your dns queries to timeout. The symptoms are less severe on a Windows computer, but still problematic for seemingly anything but web browsing, which seems unaffected. I cannot recommend BitTorrent Sync as a result, but maybe they will have patched it by the time you are reading this, so here goes.

In concept, this might be the best of any of the options. It is distributed and peer to peer. It deals with binary files well. You can share over the internet without DNS or public IPs. It is extremely simple to set up, despite what all is going on under the hood.

wget http://btsync.s3-website-us-east-1.amazonaws.com/btsync_arm.tar.gz
tar xzf btsync_arm.tar.gz
rm btsync_arm.tar.gz LICENSE.TXT
sudo ln -s /lib/arm-linux-gnueabihf/ld-linux.so.3 /lib/ld-linux.so.3
./btsync

Then just take a browser and visit http://raspberrypi.local:8888/.