[Pkg-icecast-devel] Bug#566210: icecast2: Memory leak eats up all memory until OOM

Harald Oest bugreports at ixsys.de
Fri Jan 22 00:33:12 UTC 2010 

Package: icecast2
Version: 2.3.2-2
Severity: important


We are using icecast2 as an on-demand streaming server offering about 120 different mp3 streams. Streams are changing very frequently.
Ices is used as streaming source client. Additionally apache2 and mysql5-server are installed and running, providing a HTTP based API
to request on-demand streams.

Hardware: Intel i7 CPU, 4 phys + 4 virt cores, 8 GB RAM + 4 GB swap
 
After some hours in production the server became unresponsive for several minutes, showing a system load of about 100. All memory
including the whole swap space was in use. 

/var/log/syslog:

Jan 21 09:23:12 istream1 kernel: [168616.389704] icecast2 invoked oom-killer: gfp_mask=0x1201d2, order=0, oomkilladj=0
Jan 21 09:23:12 istream1 kernel: [168616.389750] Pid: 6192, comm: icecast2 Not tainted 2.6.26-2-amd64 #1
Jan 21 09:23:12 istream1 kernel: [168616.389775] 
Jan 21 09:23:12 istream1 kernel: [168616.389775] Call Trace:
Jan 21 09:23:12 istream1 kernel: [168616.389826]  [<ffffffff802738a0>] oom_kill_process+0x57/0x1dc
Jan 21 09:23:12 istream1 kernel: [168616.389855]  [<ffffffff8023b531>] __capable+0x9/0x1c
Jan 21 09:23:12 istream1 kernel: [168616.389880]  [<ffffffff80273bcb>] badness+0x188/0x1c7
Jan 21 09:23:12 istream1 kernel: [168616.389906]  [<ffffffff80273dff>] out_of_memory+0x1f5/0x28e
Jan 21 09:23:12 istream1 kernel: [168616.389935]  [<ffffffff80276b50>] __alloc_pages_internal+0x31d/0x3bf
Jan 21 09:23:12 istream1 kernel: [168616.389967]  [<ffffffff80278806>] __do_page_cache_readahead+0x79/0x183
Jan 21 09:23:12 istream1 kernel: [168616.389998]  [<ffffffff802730b5>] filemap_fault+0x15d/0x33c
Jan 21 09:23:12 istream1 kernel: [168616.390027]  [<ffffffff8027e530>] __do_fault+0x50/0x3e6
Jan 21 09:23:12 istream1 kernel: [168616.390054]  [<ffffffff8024fb3f>] futex_wake+0x74/0xfa
Jan 21 09:23:12 istream1 kernel: [168616.390081]  [<ffffffff80281893>] handle_mm_fault+0x3f4/0x867
Jan 21 09:23:12 istream1 kernel: [168616.390107]  [<ffffffff80250734>] do_futex+0xa6/0x777
Jan 21 09:23:12 istream1 kernel: [168616.390133]  [<ffffffff803b14af>] sys_recvfrom+0xff/0x119
Jan 21 09:23:12 istream1 kernel: [168616.390160]  [<ffffffff8024612b>] bit_waitqueue+0x10/0x97
Jan 21 09:23:12 istream1 kernel: [168616.390188]  [<ffffffff80221fbc>] do_page_fault+0x5d8/0x9c8
Jan 21 09:23:12 istream1 kernel: [168616.390219]  [<ffffffff8042a6a9>] error_exit+0x0/0x60
Jan 21 09:23:12 istream1 kernel: [168616.390249] 
Jan 21 09:23:12 istream1 kernel: [168616.390266] Mem-info:
Jan 21 09:23:12 istream1 kernel: [168616.390285] Node 0 DMA per-cpu:
Jan 21 09:23:12 istream1 kernel: [168616.390306] CPU    0: hi:    0, btch:   1 usd:   0
Jan 21 09:23:12 istream1 kernel: [168616.390329] CPU    1: hi:    0, btch:   1 usd:   0
Jan 21 09:23:12 istream1 kernel: [168616.390352] CPU    2: hi:    0, btch:   1 usd:   0
Jan 21 09:23:12 istream1 kernel: [168616.390375] CPU    3: hi:    0, btch:   1 usd:   0
Jan 21 09:23:12 istream1 kernel: [168616.390398] CPU    4: hi:    0, btch:   1 usd:   0
Jan 21 09:23:12 istream1 kernel: [168616.390421] CPU    5: hi:    0, btch:   1 usd:   0
Jan 21 09:23:12 istream1 kernel: [168616.393041] CPU    6: hi:    0, btch:   1 usd:   0
Jan 21 09:23:12 istream1 kernel: [168616.393064] CPU    7: hi:    0, btch:   1 usd:   0
Jan 21 09:23:12 istream1 kernel: [168616.393086] Node 0 DMA32 per-cpu:
Jan 21 09:23:12 istream1 kernel: [168616.393108] CPU    0: hi:  186, btch:  31 usd:  91
Jan 21 09:23:12 istream1 kernel: [168616.393131] CPU    1: hi:  186, btch:  31 usd:  27
Jan 21 09:23:12 istream1 kernel: [168616.393154] CPU    2: hi:  186, btch:  31 usd:  48
Jan 21 09:23:12 istream1 kernel: [168616.393177] CPU    3: hi:  186, btch:  31 usd: 105
Jan 21 09:23:12 istream1 kernel: [168616.393200] CPU    4: hi:  186, btch:  31 usd: 148
Jan 21 09:23:12 istream1 kernel: [168616.393223] CPU    5: hi:  186, btch:  31 usd:  80
Jan 21 09:23:12 istream1 kernel: [168616.393246] CPU    6: hi:  186, btch:  31 usd: 181
Jan 21 09:23:12 istream1 kernel: [168616.393312] CPU    7: hi:  186, btch:  31 usd:  70
Jan 21 09:23:12 istream1 kernel: [168616.393335] Node 0 Normal per-cpu:
Jan 21 09:23:12 istream1 kernel: [168616.393335] Node 0 Normal per-cpu:
Jan 21 09:23:12 istream1 kernel: [168616.393356] CPU    0: hi:  186, btch:  31 usd:  99
Jan 21 09:23:12 istream1 kernel: [168616.393379] CPU    1: hi:  186, btch:  31 usd: 108
Jan 21 09:23:12 istream1 kernel: [168616.393402] CPU    2: hi:  186, btch:  31 usd: 159
Jan 21 09:23:12 istream1 kernel: [168616.393425] CPU    3: hi:  186, btch:  31 usd: 185
Jan 21 09:23:12 istream1 kernel: [168616.393447] CPU    4: hi:  186, btch:  31 usd: 178
Jan 21 09:23:12 istream1 kernel: [168616.393470] CPU    5: hi:  186, btch:  31 usd: 106
Jan 21 09:23:12 istream1 kernel: [168616.393493] CPU    6: hi:  186, btch:  31 usd: 169
Jan 21 09:23:12 istream1 kernel: [168616.393516] CPU    7: hi:  186, btch:  31 usd: 183
Jan 21 09:23:12 istream1 kernel: [168616.395622] Active:1041731 inactive:971416 dirty:0 writeback:0 unstable:0
Jan 21 09:23:12 istream1 kernel: [168616.395623]  free:10818 slab:4404 mapped:18 pagetables:8080 bounce:0
Jan 21 09:23:13 istream1 kernel: [168616.395673] Node 0 DMA free:11680kB min:12kB low:12kB high:16kB active:0kB inactive:0kB present:10756kB pages_scanned:0 all_unreclaimable? yes
Jan 21 09:23:13 istream1 kernel: [168616.395724] lowmem_reserve[]: 0 2991 8041 8041
Jan 21 09:23:13 istream1 kernel: [168616.395754] Node 0 DMA32 free:24264kB min:4264kB low:5328kB high:6396kB active:1450652kB inactive:1447976kB present:3063520kB pages_scanned:5038855 all_unreclaimable? yes
Jan 21 09:23:13 istream1 kernel: [168616.395822] lowmem_reserve[]: 0 0 5050 5050
Jan 21 09:23:13 istream1 kernel: [168616.395850] Node 0 Normal free:7328kB min:7200kB low:9000kB high:10800kB active:2716272kB inactive:2437688kB present:5171200kB pages_scanned:8942966 all_unreclaimable? yes
Jan 21 09:23:13 istream1 kernel: [168616.395919] lowmem_reserve[]: 0 0 0 0
Jan 21 09:23:13 istream1 kernel: [168616.395946] Node 0 DMA: 6*4kB 7*8kB 5*16kB 6*32kB 5*64kB 2*128kB 2*256kB 2*512kB 1*1024kB 0*2048kB 2*4096kB = 11680kB
Jan 21 09:23:13 istream1 kernel: [168616.396010] Node 0 DMA32: 236*4kB 157*8kB 311*16kB 50*32kB 34*64kB 24*128kB 12*256kB 0*512kB 1*1024kB 1*2048kB 1*4096kB = 24264kB
Jan 21 09:23:13 istream1 kernel: [168616.396076] Node 0 Normal: 148*4kB 4*8kB 5*16kB 1*32kB 3*64kB 2*128kB 2*256kB 1*512kB 1*1024kB 0*2048kB 1*4096kB = 7328kB
Jan 21 09:23:13 istream1 kernel: [168616.396141] 118 total pagecache pages
Jan 21 09:23:13 istream1 kernel: [168616.396162] Swap cache: add 3615167, delete 3615164, find 932862/944525
Jan 21 09:23:13 istream1 kernel: [168616.396188] Free swap  = 0kB
Jan 21 09:23:13 istream1 kernel: [168616.396207] Total swap = 4200888kB
Jan 21 09:23:13 istream1 kernel: [168616.421112] 2097152 pages of RAM
Jan 21 09:23:13 istream1 kernel: [168616.421112] 48651 reserved pages
Jan 21 09:23:13 istream1 kernel: [168616.421112] 3168 pages shared
Jan 21 09:23:13 istream1 kernel: [168616.421112] 3 pages swap cached
Jan 21 09:23:13 istream1 kernel: [168616.421112] Out of memory: kill process 5232 (icecast2) score 3087713 or a child
Jan 21 09:23:13 istream1 kernel: [168616.421112] Killed process 5232 (icecast2)

This happened several times. After googling i found this thread "http://icecast.imux.net/viewtopic.php?t=7001" which describes the same
problem. I started then to reproduce the scenario on an identical server using some test scripts to trigger a high load. "top" showed the icecast2
process slowly eating up the mem. I'm sure that the very frequent mounting and unmounting of streaming sources (as we do in our on-demand setup)
accelerates the allocation and not deallocation of memory by icecast2, but the forum post mentioned above uses a somewhat lowlevel setup with only a
few "static" streams, resulting in OOM too.

Icecast2 compiled from source (SVN, trunk version, http://svn.xiph.org/icecast/trunk/icecast) at the same machine 
(all required libs, headers etc. installed from lenny packages) runs rock solid, utilising just about 320M while serving 120 streams.

Cheers

Harald  


-- System Information:
Debian Release: 5.0.3
  APT prefers stable
  APT policy: (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 2.6.26-2-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/bash

More information about the Pkg-icecast-devel mailing list