OpenBSD's MALLOC_STATS and memory leak detection

The last few years I have been working on and off on the malloc(3) implementation in OpenBSD. OpenBSD's malloc is a bit of a different beast compared to most other implementations: it has built-in, always-on randomization and it returns pages to the operating system when it no longer needs them. Both these features help a lot when debugging memory management problems in programs and make various forms of heap based attacks much harder. You can find more details in the sheets of the presentation I gave on EuroBSDCon 2009.

Additionally, there are a few features that can be switched on runtime that implement even more strict checks, at the cost of performance and/or increased memory usage. You can read more about that in the man page of malloc(3).

Most recently, I did work on improving the speed of the allocation of small chunks. Working on that, I really got into the mood to return to the old goal of adding a feature that is very useful to developers: memory leak detection. Once I realized I could use a compiler feature to determine the address of the code malloc has been called from, I got on a roll. In this page I'd like to show how to use this new feature.

Enable MALLOC_STATS

By default, the code of malloc does not collect instrumentation data. To enable the features I'm discussing here, start by defining MALLOC_STATS in /usr/src/lib/libc/stdlib/malloc.c and rebuilding at least libc.
  $ cd /usr/src/lib/libc
  $ vi stdlib/malloc.c
  $ make obj
  $ make depend
  $ make
  # make install

Looking at malloc.out

If a program is run with MALLOC_OPTIONS=D and a writeable file called malloc.out exists in the current working directory, malloc will dump a report into that file when the program exits.

I made a little leaky program to illustrate some points.

  $ cat x.c
  #include <stdlib.h>
  #include <stdio.h>
  
  main() 
  {
          void *p;
          int i;
  
          for (i = 0; i < 3; i++)
                  printf("%p\n", p = malloc(10240));
          free(p);
          for (i = 0; i < 5; i++)
                  printf("%p\n", malloc(1000));
  }
  $ cp /dev/null malloc.out
  $ cc -g x.c
  $ MALLOC_OPTIONS=D a.out
  0x16adf2000
  0x165cc4000
  0x16046c000
  0x16046e800
  0x16046ec00
  0x16046e400
  0x16046f000
  0x16046f400
  $ cat malloc.out
  Malloc dir of a.out at 0x16d176010
  Regions slots 512
  Finds 1/0 1.000000
  Inserts 5/0 1.000000
  Deletes 1/0
  Cheap reallocs 0/0
  Regions slots free 508
  Free chunk structs:
  10) chunk  0x16046e000          0x0 1024 3/8
  Free pages cached: 1
   0) free at 0x16046c000: 1
  slot)  hash d  type         page            f size [free/n]
    85) #  85 0 pages  0x169b0c000  0x164c8c7d0 65536
   154) # 154 0 chunk  0x16046e000          0x0 1024 3/8
   192) # 192 0 pages  0x16adf2000  0x120000bb8 10240
   1a9) # 1a9 0 pages  0x165cc4000  0x120000bb8 10240 
  In use 135168
  Guarded 0
  Leak report
             f     sum      #    avg
           0x0    5120      5   1024
   0x120000bb8   20480      2  10240
   0x164c8c7d0   65536      1  65536  

   $
For this discussion, the interesting parts are the slot list and the leak report. The slot list lists the slot number, the hash of the entry, the d parameter used to check hash collisons, and then the type of allocation: chunk or pages, the address of the page(s), the address of the code that allocated the memory f, the size of the allocation and for chunks, a summary of the bitmap. Allocation smaller than half a pagesize are stored in a page together, a bitmap specifing which chunks are still free is maintened by malloc.

From the slot list above we can see that three page regions are allocated, one of size 65536 and two of size 10240. One page of chunks of size 1024 was allocated, of which 3 chunks are still free. You now have enough information to deduce the page size of this machine :-)

The 0x0 in the f column needs some explanation. The address of the code that allocated a chunk is only recorded if the chunk ended up as the first chunk in a page. This is to save space: storing an extra pointer for each chunk adds considerable overhead. Due to randomization, another run of the program might see a chunk allocated to chunk index 0, with an address recorded. Since chunks tend to be allocated in bunches, in a lot of cases you'll end up with chunk index 0 filled for a chunk page.

The Leak report shows a summary of all leaks with the same value of f. We can conclude that this program has 5 leaks of size 1024, two leaks of size 10240 and one leak of size 65536.

So where did my program leak?

So far we only know the addresses of the code that leaked memory. We can use a debugger to get more information about the code addresses. Note that I'm not running with any MALLOC_OPTIONS and that calling malloc_dump(fd) from gdb prepares the malloc_leaks table.
  $ gdb ./a.out
  GNU gdb 6.3
  Copyright 2004 Free Software Foundation, Inc.
  GDB is free software, covered by the GNU General Public License, and you are
  welcome to change it and/or distribute copies of it under certain conditions.
  Type "show copying" to see the conditions.
  There is absolutely no warranty for GDB.  Type "show warranty" for details.
  This GDB was configured as "alpha-unknown-openbsd4.9"...
  (gdb) br exit
  Breakpoint 1 at 0x120030ef4
  (gdb) run
  Starting program: /home/otto/a.out 
  Breakpoint 1 at 0x160531648: file /usr/src/lib/libc/stdlib/exit.c, line 57.
  0x16efa2000
  0x160d42000
  0x16b606000
  0x16b608c00
  0x16b608400
  0x16b608800
  0x16b609000
  0x16b608000
  
  Breakpoint 1, exit (status=12) at /usr/src/lib/libc/stdlib/exit.c:57
  57              __cxa_finalize(NULL);
  (gdb) call malloc_dump(2)
  Malloc dir of a.out at 0x165448450
  Regions slots 512
  Finds 1/0 1.000000
  Inserts 5/0 1.000000
  Deletes 1/0
  Cheap reallocs 0/0
  Regions slots free 508
  Free chunk structs:
  10) chunk  0x16b608000  0x120000c28 1024 3/8
  Free pages cached: 1
   0) free at 0x16b606000: 1
  slot)  hash d  type         page            f size [free/n]
     7) #   7 0 chunk  0x16b608000  0x120000c28 1024 3/8
    ba) #  ba 0 pages  0x16efa2000  0x120000bb8 10240
   12a) # 12a 0 pages  0x1692c2000  0x16052a7d0 65536
   1ea) # 1ea 0 pages  0x160d42000  0x120000bb8 10240
  In use 135168
  Guarded 0
  Leak report
             f     sum      #    avg
           0x0    4096      4   1024
   0x120000bb8   20480      2  10240
   0x120000c28    1024      1   1024
   0x16052a7d0   65536      1  65536
  
  (gdb) set print pretty on
  (gdb) print *malloc_leaks@4

  $2 = {{
      f = 0, 
      total_size = 4096, 
      count = 4
    }, {
      f = 0x120000bb8 <main+56>, 
      total_size = 20480, 
      count = 2
    }, {
      f = 0x120000c28 <main+168>, 
      total_size = 1024, 
      count = 1
    }, {
      f = 0x16052a7d0 <__smakebuf+128>, 
      total_size = 65536, 
      count = 1
    }}
  (gdb) list *0x120000bb8
  0x120000bb8 is in main (x.c:10).
  5       {
  6               void *p;
  7               int i;
  8       
  9               for (i = 0; i < 3; i++)
  10                      printf("%p\n", p = malloc(10240));
  11              free(p);
  12              for (i = 0; i < 5; i++)
  13                      printf("%p\n", malloc(1000));
  14      }
  (gdb) list *0x120000c28
  0x120000c28 is in main (x.c:13).
  8       
  9               for (i = 0; i < 3; i++)
  10                      printf("%p\n", p = malloc(10240));
  11              free(p);
  12              for (i = 0; i < 5; i++)
  13                      printf("%p\n", malloc(1000));
  14     
  (gdb) list *0x16052a7d0
  0x16052a7d0 is in __smakebuf (/usr/src/lib/libc/stdio/makebuf.c:62).
  57                      fp->_bf._base = fp->_p = fp->_nbuf;
  58                      fp->_bf._size = 1;
  59                      return;
  60              }
  61              flags = __swhatbuf(fp, &size, &couldbetty);
  62              if ((p = malloc(size)) == NULL) {
  63                      fp->_flags |= __SNBF;
  64                      fp->_bf._base = fp->_p = fp->_nbuf;
  65                      fp->_bf._size = 1;
  66                      return;
  (gdb) quit
  The program is running.  Exit anyway? (y or n) y
  $
Running in gdb and setting breakpoints at strategic locations, you can get memory reports at other interesting moments during the lifetime of your program. Note that it is also possible to call malloc_dump(fd) from within your program itself.


Please email comments to Otto Moerbeek

Powered by OpenBSD