For something I was working on needed to keep two folders in sync with eachother. Especially, one of the folders was the write&read folder and the other was only a read folder. That is the latter will be untouched by the user the only thing that write to the folder should be the sync program. After looking at the idiosyncrasies of rsync, i settled to use Unison[1].
Unison is a file synchronization tool that works, pretty unchanged on both *nix, Linux, OS X and Windows. Yes I too had to rub my eyes when I saw Windows, but thats what is cool about it. I am taking their word for this, as I have not tested it on Windows. Me running the ubiquitous Linux Ubuntu, the installation was a breeze. All I had to do was ...
$ apt-get install unison
Yeah, thats it. With just that I was ready to roll. Now I had to figure out what I needed to do to sync two folders, well as it turns out this too was fairly simple command except that you would prompted all the way through all the files that jumping across. Let me show you, I have this folder.
~/mypictures » ls -l
total 4856
-rw-r--r-- 1 root root 377576 2012-05-18 12:05 batman_2_versions-1366x768.jpg
-rw-r--r-- 1 root root 1148650 2012-05-18 12:05 Contestant_Kit.pdf
-rw-r--r-- 1 root root 105693 2012-05-18 12:05 dojo.jpg
-rw-r--r-- 1 root root 307543 2012-05-18 12:05 Hadoop Distributed File System HDFS.png
-rw-r--r-- 1 root root 85028 2012-05-18 12:05 icons.tar.bz2
-rw-r--r-- 1 root root 339323 2012-05-18 12:05 into_the_curtain.jpg
-rw-r--r-- 1 root root 1188 2012-05-18 12:05 logo.jpg
-rw-r--r-- 1 root root 337922 2012-05-18 12:05 PPT_Template6_2011.ppt
-rw-r--r-- 1 root root 339377 2012-05-18 12:05 Rain.jpg
-rw-r--r-- 1 root root 1552278 2012-05-18 12:05 thinkstats.pdf
-rw-r--r-- 1 root root 132634 2012-05-18 12:05 truly_abstract.svg
-rw-r--r-- 1 root root 101980 2012-05-18 12:05 tryme.psd
-rw-r--r-- 1 root root 122514 2012-05-18 12:05 Wallpaper_Values_04.jpg
and just wanted to share to share it across to another folder ~/mypicsbacked. All I had to do is run the following command
$ unison ~/mypictures ~/mypicsbacked
But once I pressed the RETURN on that command, I get this...
Contacting server...
Connected [//samuel-work//home/sam/mypicsbacked -> //samuel-work//home/sam/mypictures]
Looking for changes
Warning: No archive files were found for these roots, whose canonical names are:
/home/sam/mypictures
/home/sam/mypicsbacked
This can happen either
because this is the first time you have synchronized these roots,
or because you have upgraded Unison to a new version with a different
archive format.
Update detection may take a while on this run if the replicas are
large.
Unison will assume that the 'last synchronized state' of both replicas
was completely empty. This means that any files that are different
will be reported as conflicts, and any files that exist only on one
replica will be judged as new and propagated to the other replica.
If the two replicas are identical, then no changes will be reported.
If you see this message repeatedly, it may be because one of your machines
is getting its address from DHCP, which is causing its host name to change
between synchronizations. See the documentation for the UNISONLOCALHOSTNAME
environment variable for advice on how to correct this.
Donations to the Unison project are gratefully accepted:
Press return to continue.[<spc>]
This message is shown becasue this is the first time I am synchronizing these folders, as the reading through this will suggest and never I bothered about all the dhcp crap as this synchronization was inside the same machine. Yeah thats right, unison can take care of remote synchronization as well, worked pretty well, but scoping that out off this article. All I did was press the spacebar. Then you should see some thing like this.
Reconciling changes Distributed File System HDFS.png
mypictures mypicsbacked
file ----> Contestant_Kit.pdf [f]
I guess that f was for "force", but I was wrong after trying about a bit I figured that "?" will give a short help. And it turns out that "f" stands for "Follow unison's recommendations", so I did. And ended up with this,
file ----> Contestant_Kit.pdf [f] f
file ----> Hadoop Distributed File System HDFS.png [f] f
file ----> PPT_Template6_2011.ppt [f] f
file ----> Rain.jpg [f] f
file ----> Wallpaper_Values_04.jpg [f] f
file ----> batman_2_versions-1366x768.jpg [f] f
file ----> dojo.jpg [f] f
file ----> icons.tar.bz2 [f] f
file ----> into_the_curtain.jpg [f] f
file ----> logo.jpg [f] f
file ----> thinkstats.pdf [f] f
file ----> truly_abstract.svg [f] f
file ----> tryme.psd [f] f
Proceed with propagating updates? []
Now I just pressed the "?" again...
Proceed with propagating updates? [] ?
Commands:
y or g Yes: proceed with updates as selected above
n No: go through selections again
q exit unison without propagating any changes
And then keyed in a "y" to complete the process, of syncing.
Proceed with propagating updates? [] y
Propagating updates
UNISON 2.32.52 started propagating changes at 12:16:46 on 18 May 2012
[BGN] Copying Contestant_Kit.pdf from /home/sam/mypictures to /home/sam/mypicsbacked
[END] Copying Contestant_Kit.pdf
[BGN] Copying Hadoop Distributed File System HDFS.png from /home/sam/mypictures to /home/sam/mypicsbacked
[END] Copying Hadoop Distributed File System HDFS.png
[BGN] Copying PPT_Template6_2011.ppt from /home/sam/mypictures to /home/sam/mypicsbacked
[END] Copying PPT_Template6_2011.ppt
[BGN] Copying Rain.jpg from /home/sam/mypictures to /home/sam/mypicsbacked
[END] Copying Rain.jpg
[BGN] Copying Wallpaper_Values_04.jpg from /home/sam/mypictures to /home/sam/mypicsbacked
[END] Copying Wallpaper_Values_04.jpg
[BGN] Copying batman_2_versions-1366x768.jpg from /home/sam/mypictures to /home/sam/mypicsbacked
[END] Copying batman_2_versions-1366x768.jpg
[BGN] Copying dojo.jpg from /home/sam/mypictures to /home/sam/mypicsbacked
[END] Copying dojo.jpg
[BGN] Copying icons.tar.bz2 from /home/sam/mypictures to /home/sam/mypicsbacked
[END] Copying icons.tar.bz2
[BGN] Copying into_the_curtain.jpg from /home/sam/mypictures to /home/sam/mypicsbacked
[END] Copying into_the_curtain.jpg
[BGN] Copying logo.jpg from /home/sam/mypictures to /home/sam/mypicsbacked
[END] Copying logo.jpg
[BGN] Copying thinkstats.pdf from /home/sam/mypictures to /home/sam/mypicsbacked
[END] Copying thinkstats.pdf
[BGN] Copying truly_abstract.svg from /home/sam/mypictures to /home/sam/mypicsbacked
[END] Copying truly_abstract.svg
[BGN] Copying tryme.psd from /home/sam/mypictures to /home/sam/mypicsbacked
[END] Copying tryme.psd
UNISON 2.32.52 finished propagating changes at 12:16:46 on 18 May 2012
Saving synchronizer state
Synchronization complete at 12:16:46 (13 items transferred, 0 skipped, 0 failed)
All this to and fro of prompts and answers between unison and me really did not auger well for the solution that I intended to figure out. So I had to figure out a silent way of doing this. I had not look no further that "-help" option and found exactly what I needed.
-batch batch mode: ask no questions at all
So my final command turned out to be
$ unison -batch ~/mypictures ~/mypicsbacked
This was only half the problem solved. I had to figure out how do I run this every time something changed in ~/mypictures how to I get that to the ~/mypicsbacked. I was looking for some kinda notifier on file changes to the source directory that, could trigger a process to unison. It turned out there is something called "incrond"[2], a very similar utility to cron but works using the inotify module, a process can be triggered whenever there are any filesystem changes on a specific directory. The definition for what needs to run for what filesystem event have to be defined in an incrontab.
So I went ahead and installed incron
# apt-get install incron
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
incron
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 115 kB of archives.
After this operation, 385 kB of additional disk space will be used.
Fetched 115 kB in 1s (77.6 kB/s)
Selecting previously deselected package incron.
(Reading database ... 163836 files and directories currently installed.)
Unpacking incron (from .../incron_0.5.9-5_amd64.deb) ...
Processing triggers for ureadahead ...
Processing triggers for man-db ...
Setting up incron (0.5.9-5) ...
Adding group `incron' (GID 129) ...
Done.
* Starting File system events scheduler
And immediately tried incrontab, but it turns out that I cannot run this command unless I have authorized users to run this command. I looked at the /etc/incron.conf, and apperently there is a file /etc/incron.allow where I had to list the user who are allowed to run the incrontab or rather the user who are allowed to use incron. So I went ahead and edited /etc/incron.allow and myself.
Restart the incron daemon and ran the incrontab command.
# service incron restart
$ incrontab -e
There was nothing in the file, to help me out. I figured out that you need to specify the folder that needs looking into, and what changes need to monitored, and the command that need to run. Thats pretty simple.
~/mypictures - The folder that needs to watched
create, delete and modify - The actions that need to watched for the folder
unison -batch ~/mypictures ~/mypicsbacked - The command that needs to run every event listed above on the originating folder
So my incron command looked something like this...
/home/sam/mypictures IN_CREATE,IN_DELETE,IN_MODIFY /usr/bin/unison -batch /home/sam/mypictures /home/san/mypicsbacked
The IN_CREATE, IN_DELETE and IN_MODIFY are the inotity masks for the signify, create, delete and modify operations respectively. Also I choose to give absolute paths to avoid any ambiquity. Now to check my setup works well, I copy a file in the origin directory into itself with a different name.
$ cd ~/mypictures
$ cp batman_2_versions-1366x768.jpg batman_2_versions-1.jpg
And evidently this should appear on the synced destination folder, ~/mypicsbacked
$ cd ~/mypicsbacked
$ ls
batman_2_versions-1366x768.jpg dojo.jpg into_the_curtain.jpg Rain.jpg tryme.psd
batman_2_versions-1.jpg Hadoop Distributed File System HDFS.png logo.jpg thinkstats.pdf Wallpaper_Values_04.jpg
Contestant_Kit.pdf icons.tar.bz2 PPT_Template6_2011.ppt truly_abstract.svg
Well, you can see that worked. Now I assure you this is just the beginning of using a great tool, would be exploring to see how to I get the unison to work with S3, and also explore its myriad of option in the coming days. But I have scratched my initial itch.
This setup worked great for me. I do not give any assurance that it should work for you. You might encounter your own pains on the way to achieving what I have done.