Note: This report is very outdated now. Consult this page for a more recent report.

This is the second report on memory consumption

And this second report has been updated

It has been updated at the "Initial load of a folder" section

And this second report is now outdated

That's because new memory reductions have happened.

Memory usage of tinymail

A few details

These tests where done with

export G_SLICE=always-malloc
export G_DEBUG=gc-friendly

as explained here. It means that memory allocators, that reduce for example the heap admin memory consumption, where not utilised. The massif tool of valgrind was the primary tool for measuring the memory consumption itself. Massif does not measure mmaps. The mmaps are measured by simply looking at the filesize of the files being mmapped.

Note: The memory analysis of the initial download of a folder (which is a section by itself in this report, check below) was not done by first disabling the memory allocators this way. This memory analysis report, however, is primarily focused at normal usage on a folder that was already received. The "initial download of a folder" section should be considered as a "nice to know" addition of this memory usage report or analysis.

Library size

The image size of the shared object files of the framework add an initial memory consumption to a client. Luckily most systems use mmap for this. When statically linking with the framework, this size will be different. Read about it here.

Help keeping this report updated

By reading this page about how to help memory testing.

Data

Other than the image size, a client, constructed using the tinymail framework, consumes memory per header being displayed in its summary view of a folder.

Let's do a little data dictionary here. The message header in tinymail means all the headers attached to an E-mail. The summary of a folder is all the information a user needs to get an overview of the contents of a folder. It contains the header values to, from, subject, cc, information about attachments and some message flags like whether or not the message was read or not. It also contains the message uid which is a unique identifier per message in the folder.

Some folders are 20,000 such headers in size. A full E-mail message of course consumes a lot more than one summary item. It would be plain insanity to load all the messages in memory. That would consume gigabytes of memory.

Reproducing these tests

Using the test tool in tests/memory you can reproduce these tests. The IMAP account that was being used is available for everybody at IMAP server mail.tinymail.org, username tinymailunittest and password unittest. It stays available as long as nobody abuses it.

Initial load of a folder (this test loads 30,000 headers)

Note: valgrind graphs in this sections where not done by first disabling the memory allocators (gslice and gmemchunk) as described above. Feel free to reproduce the same graphs with the memory allocators disabled if you really need the analysis of that.

When the folder is being loaded from the IMAP service, the framework periodically uses the mmap techniques (by dumping the received data and reloading the mmap). Without those techniques, each summary item needs to be fully stored in memory until all is received. Its memory consumption would look like this (this is a normal Camel's behaviour, it is NOT tinymail's embedded Camel's (camel-lite's) behaviour):

This is not tinymail's behaviour. It is the behaviour of the default Camel as shipped with Evolution

This, however, is how tinymail consumes memory during the initial download of a IMAP folder. The NNTP one's memory consumption looks a lot like this one too. Exchange will soon be supported (more info here) and that support will also get a patch applied that will let memory consumption during initial download of a message folder be like this too.

This is tinymail's embedded camel-lite behaviour This graph is outdated, tinymail's camel-lite consumes ~20% less for this feature today

The memory test itself needs to get all the headers before you perform the test. Else you will be measuring the memory consumption explained in this section (the memory consumption of an initial download of a bunch of folders). Loading and using "already-received" headers, however, is what we are really measuring in this report. Therefore you must tell the test-tool to get headers using the -o switch (--online) and you reuse cached headers using the -c switch (--cache). It goes like this:

### Getting the headers
$prefix/bin/memory-test -o
### Loading the headers
valgrind --tool=massif $prefix/bin/memory-test -c /tmp/tinymail.0
### Resetting for the next test
rm -rf /tmp/tinymail.0

Normal operation (with mmap)

Information about mmap can be found here and here.

These values measured using valgrind and tested using real IMAP mailboxes that have been filled up with nothing but pure spam. Spam is interesting for testing because most mailboxes contain mostly spam and because spam often uses larger subject, from, cc and to fields than normal E-mail. So spam is probably a worst-case scenario. Which makes it very suitable for testing. They aren't guaranteed but they do give a clear indication of the amount of memory tinymail actually uses.

What do these values mean?! The "mmap size" is the size of the file that is mapped using the mmap POSIX syscall. The "allocated with malloc, gslice or memchunk" is the total amount of memory valgrind detected being allocated by the test application.

An mmap doesn't at all consume the amount of memory that is being mapped. Realistically an mmap consumes a lot less, but it's depending on the total amount of available memory on your system. To get a realistic overview of how much memory an mmap uses, you need to enter kernel development and kernel concepts.

Amount of headers	on-disk mmap size	allocated with malloc, gslice or memchunk
50,453 headers	18,039 Kb	15,900 Kb
45,396 headers	15,857 Kb	15,000 Kb
30,264 headers	10,570 Kb	10,200 Kb
15,132 headers	5,400 Kb	4,700 Kb
3,000 headers	978 Kb	1,200 Kb
2,000 headers	641 Kb	500 Kb
400 headers	91 Kb	400 Kb
200 headers	45 Kb	250 Kb

Let's put this into a graph:

Valgrind graphs of Oct 19 2006

Important note: these graphs are not done using GSlice. GSlice does make a difference in amount of memory used (especially heap-admins differs quite a lot). GSlice, however, would not deallocate everything always immediately (which makes it less nice for memory debugging and memory leak hunting).

This is the second memory report. The first showed that tinymail used significantly more memory to achieve the same results. This memory reduction is of course the result of intensively trying to find unused features and unneeded memory allocations. The graphs of the first report are still available below (they are just not visible anymore, as this is current, today).

The eight smaller first folders (1, 10, 100, 200, 400, 800, 2000 and 3000 headers). This graph also shows the library size overhead in non mmaped memory being used. As you can see it's less than 100K (as that amount of memory never goes lower than that, and the graph immediately starts at ~100kb). You can also see that the account setup and folder-list consumes another ~ 100K. You can find the code of the test being used here.

All fourteen folders (1, 10, 100, 200, 400, 800, 2000, 3000, 5000, 15000, 20000, 30000, 40000 and 50000 headers)