[Libtorrent-devel] Re: IO performance problems

[Raimund Berger]

Josef Drexler <josef at ttdpatch.net> writes:

> On Aug 18, 2008, at 11:33 PM, ghost at tetridome.com wrote:
>> Checking the memory usage of the torrent shows that rtorrent is
>> using very little memory, about 20 MiB (~0.4 MiB per upload slot),
>> and thus isn't able to use much of cache for transfers.
>
> rtorrent doesn't cache anything at all, or use any memory for the
> transfers. It simply asks the kernel to put a bunch of pages in the
> filecache and has it transfer the data directly from there to the
> network stack.
>
> By design, it leaves the caching, (most of the) read-ahead and read
> merging decisions to the kernel/OS.
>
> Basically, if this doesn't work for you, you can either try to tweak
> the kernel settings, or use a different torrent client. Transferring
> directly from the filecache is the distinguishing feature of
> rtorrent, so this won't change.
>
> You can try to play with rtorrent's read-ahead settings though. That
> will help especially if peers regularly request complete chunks,
> instead of just random pieces from them, because then rtorrent will
> tell the kernel to always page in the entire chunk too.

It might be also worth noticing that the bt piece selection algorithm,
while optimizing bandwidth distribution among peers, is about the worst
regarding disk IO throughput.

Since pieces/chunks may be requested by other peers more or less
randomly, there's afaik really no obvious application level caching
strategy the client could perform apart from maybe keeping rare pieces
in memory.

But in any case bittorrent will put substantially more load on the IO
subsystem to reach a given link saturation than e.g. a web server
handing out files sequentially.

Another thing maybe worth trying is putting the torrents into a tmpfs
partition and back that up with sufficient swap space. I've never tried
that with bt myself but rather only with sequential read/write apps
where I wanted to minimize disk access. It may or may not bring
improvements, really depending on the usage patterns.