seek & destroy

In my last entry I wrote a bit about optimizing my little project. One other significant optimization I found was inode-sorting, from an idea I got from some old postings on the mutt mailing list.

The idea is as follows: some file systems, in particular ext3, support hashed b-trees to speed-up lookups in large directories (paper). That's nice for finding particular files. However, as a side-effect, when you scan full directories (as mu does when indexing), you might get the entries back in a rather chaotic order. If you then try to open the files in that order, you suffer from long seek times, and consequently, bad performance.

The solution is to sort the dir entries by their inode (in ascending order), and then open the corresponding files in that order. This is what mu (mu-index) does by default, starting with version 0.3. You can turn it off with --tune-sort-inodes=0, but there is usually little need for that, as the overhead of sorting is negligible.

So, what difference does it make? Answer: it depends on how the files are laid out; if you already get your files back in their 'natural order', there won't be much difference - this is what happens on my main machine. But, on another (old) machine where the files are not in that order, the improvements are substantial: I found that indexing 1500 message in 25 seconds without inode-sorting, goes down to 15 seconds with inode-sorting; a nice 40% improvement.

Note(1): this works for ext3 directories with dir_index enabled; there's a HOWTO. There are other file systems that have similar features, but I haven't tested those. Note(2): This optimization is not very useful for flash-based file systems, as they don't really care in what order you open files.