Syncing Files on the Raspberry Pi
Syncing Files on the Raspberry PiIntroduction
Syncing files automatically from a Raspberry Pi is a good idea to make sure that nothing gets lost the Pi breaks. The problem is that due to the unusual ARM architecture, most of the common methods for syncing are not available. Dropbox does not have an ARM package. Google Drive never had a Linux client. There are ways around these limitations either by use of lower quality third party applications to interface with the major sync solutions or by use of open source and self hosted solutions.
Sparkleshare
Sparkleshare is a self-hosted solution. Sparkleshare uses Git on its backend, so it's particularly good for text files. You can also use a git host like Bitbucket. It provides an easy to use graphical interface and is very easy to set up
On the Raspberry Pi:
pi@raspberrypi:~# sudo apt-get install sparkleshare
Building Sparkleshare (unnecessary if using the package from apt-get):
#install all mono stuff, which is probably (definitely) overkill pi@raspberrypi:~# sudo apt-get install libmono-2.0-dev mono-complete gtk-sharp2 libwebkit-cil-dev libnotify0.4-cil libnotify-cil-dev pi@raspberrypi:~# mkdir apps ; cd apps pi@raspberrypi:~# wget <getrecentlink>.tar.gz pi@raspberrypi:~# tar -xzf sparkleshare-linux-1.1.0-tar.gz pi@raspberrypi:~# cd sparkleshare-1.1.0/ pi@raspberrypi:~# ./configure pi@raspberrypi:~# make pi@raspberrypi:~# sudo make install
On your "host" computer (unnecessary if using third party Git host)
root@evan-ubuntu:~# sudo su && cd # Fetch the Dazzle script root@evan-ubuntu:~# curl https://raw.github.com/hbons/Dazzle/master/dazzle.sh \ --output /usr/bin/dazzle && chmod +x /usr/bin/dazzle # Run the initial setup root@evan-ubuntu:~# dazzle setup # Link SparkleShare clients using their Client ID found in the status menu root@desktop:~# dazzle link
Paste your Client ID (found in the status icon menu) below and press <ENTER>. Client ID: ssh-rsa AAAAB3NzaC1yc2EAAAADAQ... ... ...TnTL4SK1+7gYKT raspberrypi # Create a new project. Add as many as you need root@evan-ubuntu:~# dazzle create raspberry # raspbery project or directory name
Project "raspberry" was successfully created. To link up a SparkleShare client, enter the following details into the "Add Hosted Project..." dialog: Address: ssh://storage@evan-ubuntu.local Remote Path: /home/storage/raspberry
Setup on the client side uses a GUI:
pi@raspberrypi:~# DISPLAY=:0 sparkleshare start
Grive
Grive is the unnofficial Google Drive client for Linux. Google has no official linux client, but in this specific case that is actually a good thing, in a weird way, because the open source version can be built from source and can run on the Raspberry Pi's ARM processor.
First, try installing from the repos. In a lot of debian archives, grive is now available. At the point of writting this though, it is not currenlty in the Raspbian repo, but is in other Debian repos.
pi@raspberrypi ~ $ sudo apt-get install grive
Building Grive from Source
If that package does not work, you can build it from source, but it takes a really long time to build.
This will add quite a few packages to your Raspbian - a couple hundred MB worth, which takes more than a couple minutes to install. The build also takes about 20 minutes.
sudo apt-get install cmake git build-essential libgcrypt11-dev libjson0-dev libcurl4-openssl-dev libexpat1-dev libboost-filesystem-dev libboost-program-options-dev binutils-dev libqt4-core libqt4-dev libboost-test-dev libboost-filesystem-dev libboost-program-options-dev libboost-system-dev libyajl-dev
pi@raspberrypi $ mkdir ~/apps ; cd ~/apps pi@raspberrypi ~/apps $ git clone git://github.com/Grive/grive.git
pi@raspberrypi ~/apps $ cd grive
pi@raspberrypi ~/apps/grive $ cmake .
-- Found libgcrypt: -L/lib/arm-linux-gnueabihf -lgcrypt -- Found JSON-C: /usr/lib/arm-linux-gnueabihf/libjson.so -- Boost version: 1.49.0 -- Found the following Boost libraries: -- program_options -- filesystem -- unit_test_framework -- system -- Found libbfd: /usr/lib/libbfd.so -- Found libiberty: /usr/lib/libiberty.a -- Boost version: 1.49.0 -- Found the following Boost libraries: -- program_options -- Boost version: 1.49.0 -- Configuring done -- Generating done -- Build files have been written to: /home/pi/apps/grive
If you do not add these scope resolution operators, the compiler does not seem to know to use the Boost library. It might also have to do with the sizes of variables on the ARM processor versus normal.
pi@raspberrypi ~/apps/grive $ nano libgrive/src/drive/State.cc
void State::Write( const fs::path& filename ) const
{
Json last_sync ;
last_sync.Add( "sec", Json((boost::uint64_t) m_last_sync.Sec() ) );
last_sync.Add( "nsec", Json((boost::uint64_t) m_last_sync.NanoSec() ) );
//last_sync.Add( "sec", Json(m_last_sync.Sec() ) );
//last_sync.Add( "nsec", Json(m_last_sync.NanoSec() ) );
Json result ;
result.Add( "last_sync", last_sync ) ;
//result.Add( "change_stamp", Json(m_cstamp) ) ;
result.Add( "change_stamp", Json((boost::uint64_t) m_cstamp) ) ;
std::ofstream fs( filename.string().c_str() ) ;
fs << result ;
}
Now that you've patched it a little bit, you can actually build it. This takes a while.
pi@raspberrypi ~/apps/grive $ make
[ 1%] Building CXX object libgrive/CMakeFiles/grive.dir/src/json/JsonParser.cc.o
/tmp/ccN9aaS9.s: Assembler messages:
/tmp/ccN9aaS9.s:1258: Warning: swp{b} use is deprecated for this architecture
[ 3%] Building CXX object libgrive/CMakeFiles/grive.dir/src/json/ValBuilder.cc.o
/tmp/cc0qtdOv.s: Assembler messages:
/tmp/cc0qtdOv.s:1317: Warning: swp{b} use is deprecated for this architecture
[ 5%] Building CXX object libgrive/CMakeFiles/grive.dir/src/json/JsonWriter.cc.o
[ 7%] Building CXX object libgrive/CMakeFiles/grive.dir/src/json/ValResponse.cc.o
[ 8%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/File.cc.o
/tmp/ccCrhkFO.s: Assembler messages:
/tmp/ccCrhkFO.s:1317: Warning: swp{b} use is deprecated for this architecture
[ 10%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/Config.cc.o
[ 12%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/StdStream.cc.o
[ 14%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/Exception.cc.o
/tmp/cca93GMZ.s: Assembler messages:
/tmp/cca93GMZ.s:1298: Warning: swp{b} use is deprecated for this architecture
[ 16%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/OS.cc.o
/tmp/cchdH7ge.s: Assembler messages:
/tmp/cchdH7ge.s:1258: Warning: swp{b} use is deprecated for this architecture
[ 17%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/DateTime.cc.o
/tmp/ccnsT4HV.s: Assembler messages:
/tmp/ccnsT4HV.s:1258: Warning: swp{b} use is deprecated for this architecture
[ 19%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/StringStream.cc.o
[ 21%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/SignalHandler.cc.o
[ 23%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/Crypt.cc.o
/tmp/ccx344KH.s: Assembler messages:
/tmp/ccx344KH.s:1258: Warning: swp{b} use is deprecated for this architecture
[ 25%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/MemMap.cc.o
[ 26%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/log/CompositeLog.cc.o
[ 28%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/log/DefaultLog.cc.o
[ 30%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/log/CommonLog.cc.o
[ 32%] Building CXX object libgrive/CMakeFiles/grive.dir/src/util/log/Log.cc.o
[ 33%] Building CXX object libgrive/CMakeFiles/grive.dir/src/xml/String.cc.o
[ 35%] Building CXX object libgrive/CMakeFiles/grive.dir/src/xml/NodeSet.cc.o
/tmp/ccshU8Rr.s: Assembler messages:
/tmp/ccshU8Rr.s:1284: Warning: swp{b} use is deprecated for this architecture
[ 37%] Building CXX object libgrive/CMakeFiles/grive.dir/src/xml/Node.cc.o
/tmp/cc03tDy6.s: Assembler messages:
/tmp/cc03tDy6.s:1284: Warning: swp{b} use is deprecated for this architecture
[ 39%] Building CXX object libgrive/CMakeFiles/grive.dir/src/xml/TreeBuilder.cc.o
/tmp/ccEaKl2M.s: Assembler messages:
/tmp/ccEaKl2M.s:1258: Warning: swp{b} use is deprecated for this architecture
[ 41%] Building CXX object libgrive/CMakeFiles/grive.dir/src/bfd/Debug.cc.o
[ 42%] Building CXX object libgrive/CMakeFiles/grive.dir/src/bfd/SymbolInfo.cc.o
[ 44%] Building CXX object libgrive/CMakeFiles/grive.dir/src/bfd/Backtrace.cc.o
Linking CXX static library libgrive.a
[ 82%] Built target grive
Scanning dependencies of target btest
[ 83%] Building CXX object libgrive/CMakeFiles/btest.dir/test/btest/UnitTest.cc.o
[ 85%] Building CXX object libgrive/CMakeFiles/btest.dir/test/btest/ValTest.cc.o
/tmp/ccK8TOeR.s: Assembler messages:
/tmp/ccK8TOeR.s:1331: Warning: swp{b} use is deprecated for this architecture
[ 87%] Building CXX object libgrive/CMakeFiles/btest.dir/test/btest/JsonValTest.cc.o
/tmp/ccOAUEHb.s: Assembler messages:
/tmp/ccOAUEHb.s:1331: Warning: swp{b} use is deprecated for this architecture
Linking CXX executable btest
[ 87%] Built target btest
Scanning dependencies of target grive_executable
Linking CXX executable grive
[ 89%] Built target grive_executable
[ 91%] Generating src/moc_MainWnd.cxx
[ 92%] Generating ui_MainWindow.h
Scanning dependencies of target bgrive_executable
[ 94%] Building CXX object bgrive/CMakeFiles/bgrive_executable.dir/src/DriveModel.cc.o
/tmp/ccgDURnf.s: Assembler messages:
/tmp/ccgDURnf.s:1731: Warning: swp{b} use is deprecated for this architecture
[ 96%] Building CXX object bgrive/CMakeFiles/bgrive_executable.dir/src/MainWnd.cc.o
[ 98%] Building CXX object bgrive/CMakeFiles/bgrive_executable.dir/src/main.cc.o
[100%] Building CXX object bgrive/CMakeFiles/bgrive_executable.dir/src/moc_MainWnd.cxx.o
Linking CXX executable bgrive
[100%] Built target bgrive_executable
Well, now that that's over, the install process seems to need an extra doc, so we'll just add an empty one, which might not be necessary.
pi@raspberrypi ~/apps/grive $ mkdir /home/pi/apps/grive/bgrive/doc/ pi@raspberrypi ~/apps/grive $ touch /home/pi/apps/grive/bgrive/doc/grive.1
pi@raspberrypi ~/apps/grive $ sudo make install
Usage
First, set up a folder to sync your stuff to. Then you open a long authentication link in a browser and log in to your Google account. It will then give you a long code to copy back to the program. It is important that your current directory is the directory you wish to sync with your Google Drive before you do this.
pi@raspberrypi ~ $ mkdir ~/gdrive/ ; cd ~/gdrive
pi@raspberrypi ~/gdrive $ grive -a ----------------------- Please go to this URL and get an authentication code: https://accounts.google.com/o/oauth2/auth?scope=............ ----------------------- Please input the authentication code here:
XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Reading local directories Synchronizing folders Reading remote server file list Synchronizing files
... ... ... ...
Now, to set up automatic sync, it unfortunately has to be done this somewhat simplistic way:
pi@raspberrypi ~ $ mkdir ~/bin/
pi@raspberrypi ~ $ echo "#! /bin/bash > cd ~/gdrive/ > grive" > ~/bin/gsync
pi@raspberrypi ~ $ chmod +x ~/bin/gysnc
pi@raspberrypi ~ $ crontab -e
*/10 * * * * /home/pi/bin/gsync
Now, your Google Drive will sync to your to your Raspberry Pi every 10 minutes. There might be a way to do this every time a file changes on the Raspberry Pi using inotifywait, but it would not sync changes from Google Drive until there is a change locally.
dropfuse
Dropfuse is not one of the best or most secure options, but Dropbox is a really nice service. Dropfuse is a third party script that relies on the "share link" feature. It does not actually connect to your entire Dropbox account. It will only connect to a single shared folder. Log onto dropbox.com, navigate to the folder you want sync'd and click "share link". You'll find a link like this:
https://www.dropbox.com/sh/AAAAAAAAAAAAAAA/BBBBBBBBBB
Modify the link yourself to this format:
https://www.dropbox.com/s/AAAAAAAAAAAAAAA
The install process is relatively easy.
pi@raspberrypi ~ $ sudo apt-get install python-pyquery
pi@raspberrypi ~ $ sudo usermod -a -G fuse pi
pi@raspberrypi ~ $ mkdir apps
pi@raspberrypi ~ $ mkdir Dropbox
pi@raspberrypi ~ $ cd apps
pi@raspberrypi ~/apps $ git clone git://github.com/arekzb/dropfuse.git Cloning into 'dropfuse'... remote: Counting objects: 38, done. remote: Compressing objects: 100% (21/21), done. remote: Total 38 (delta 19), reused 34 (delta 17) Receiving objects: 100% (38/38), 14.16 KiB, done. Resolving deltas: 100% (19/19), done.
pi@raspberrypi ~/apps $ cd dropfuse
sudo python dropfuse.py https://www.dropbox.com/s/c6ecc2plwconh5x ~/Dropbox &
To stop the service, do this:
pi@raspberrypi ~ $ fusermount -u Dropbox
BitTorrent Sync
A word of warning, btsync causes lots of networking issues. On an Ubuntu machine, it can cause 10% of your dns queries to timeout. The symptoms are less severe on a Windows computer, but still problematic for seemingly anything but web browsing, which seems unaffected. I cannot recommend BitTorrent Sync as a result, but maybe they will have patched it by the time you are reading this, so here goes.
In concept, this might be the best of any of the options. It is distributed and peer to peer. It deals with binary files well. You can share over the internet without DNS or public IPs. It is extremely simple to set up, despite what all is going on under the hood.
wget http://btsync.s3-website-us-east-1.amazonaws.com/btsync_arm.tar.gz tar xzf btsync_arm.tar.gz rm btsync_arm.tar.gz LICENSE.TXT sudo ln -s /lib/arm-linux-gnueabihf/ld-linux.so.3 /lib/ld-linux.so.3 ./btsync
Then just take a browser and visit http://raspberrypi.local:8888/.