My Universe Logo

My Universe Blog » Entries Tagged as synctory

Synctory Stepping towards Release

Posted by Jesco Freund at June 19, 2011 8:36 p.m.

Synctory (or libsynctory) is the project I'm currently spending most of my leisure time on. Basically, it implements algorightms similar to librsync. Originally, I developed libsynctory as part of a backup software project, but later decided to continue it as a separate library. The first code was written during my last summer holiday in Denmark, and since then, a lot (of the code) has changed.

Now, after almost one year of rather less steady development, libsynctory is stepping towards release maturity. Today I finally managed to commit a performance test to evaluate how fast libsynctory actually works. Though I immediately discovered a memory leak bug which is not resolved yet, the performance seems to be somewhat satisfactory for a first approach (average runtime in seconds):

+-------------+---------+---------+---------+---------+
|             | small   | big     | huge    | giant   |
+-------------+---------+---------+---------+---------+
| fingerprint |    0.01 |    6.67 |   33.94 |   67.94 |
+-------------+---------+---------+---------+---------+
| diff        |    0.04 |   31.99 |  144.99 |  303.08 |
+-------------+---------+---------+---------+---------+
| synth       |    0.02 |   14.11 |   81.01 |  152.82 |
+-------------+---------+---------+---------+---------+

The test was taken by a constant chunk size (512 bytes) for all four file sizes (0.5 MiB, 0.5 GiB, 2.5 GiB, 5.0 GiB). For me, the results illustrate two main conclusions: First, the algorithms of libsynctory seem to scale linearly with the file size (however I would have been surprised if they didn't), and secondly, bigger files require larger chunk sizes to allow a runtime-friendly processing. Therefore, I think the decision to allow individual chunk size adjustmens was a good one.

The way forward seems to be clear by now – some bugs to fix, some man pages to write, and then I'll hopefully be able to produce a first release of libsynctory. As (almost) always in open source projects, any help is appreciated, particularly I am looking for testers on Linux and OpenSolaris. So if you have some spare time, don't hesitate to contact me!

No comments | Defined tags for this entry: development, open source, synctory

First alpha of libsynctory released

Posted by Jesco Freund at Jan. 5, 2011 5:02 p.m.

Some minutes ago, I released the first alpha (ever!) of libsynctory. It is a replacement for librsync, but makes use of different algorithms. Therefore, processing might be slower and fingerprint chunks are definitely larger than the ones produced by librsync. However, libsynctory's fingerprints are much more collision safe, so it is more adequate for environments, where high reliability is estimated more than performance or network load (like e. g. backup applications).

Currently, I consider libsynctory to be “feature-complete”, meaning it offers all the functionality required to serve its intended purpose. It can create a fingerprint of a given file. Using this fingerprint only (and without resorting to the original file), it can calculate the difference between the original file and another file. Using this difference and the original file, the other file can be restored.

However, libsynctory does neither offer any documentation yet (shame on me), nor does it provide a decent interface for error handling (or even backtracing). Furthermore, it has not yet been reviewed for thread safety, so use it with care (if you intend to do so). If you would like to test libsynctory, just grab the source archive from the project page and do the following:

tar -xjvf libsynctory-0.1.0-a1-Source.tar.bz2
cd libsynctory-0.1.0-a1-Source
cmake .
make
sudo make install

This will install the header files into /usr/local/include and the library itself into /usr/local/lib. Within the build directory, you will find some test binaries (in src/test). You can use them for very simple single file operations, just to see how libsynctory works.

No comments | Defined tags for this entry: backtory, C, development, synctory

Backtory is moving forward

Posted by Jesco Freund at July 27, 2010 7:35 p.m.

The going is tough, but at least there is any going – I think this describes best my progress with Backtory. During the last two weeks, we've been on holiday in Denmark. I used some of the rather rainy days to code on Backtory, and here's what I got done so far:

  • Libsynctory is in a working state now, meaning it does what is expected of it. However, it's not yet nice – no proper documentation, no thread safety, no error tracing.
  • Libyar got a bit further. The container file layout is finalized and properly documented. An awful lot of macros and some init voodoo for thread saftey are ready, but the lib itself is of no use yet.
  • The Backtory application itself hasn't been touched by me for some weeks now – I concentrated efforts on the two above-mentionned libs. However, I have made up my mind about the application (again ;-)). There will be a “cheap” CLI application, most probably implemented in Python. The more complex thing (daemon, configuration via network and stuff) will become eBacktory, of which I'll take care later.

So first priority for me is to get libyar ready to use, and to finish libsynctory into a state that could be called “release-ready”. When both libs are done, I'll first finalize a CLI for YAR files before taking care of the Backtory implementation itself. As I see tough times ahead concerning my professional life, I dare not give any forecast when I'll be able to spare the hours so urgently needed to finish any of the mentionned tasks. But as I already statet – the going is tough, but at least there is any going…

No comments | Defined tags for this entry: backtory, C, development, synctory, yar

Page 1 of 1