Posted by Alex Greg on 01/08/05 23:58

Hi,

Recently I migrated the front-end of our bulletin board (running
phpBB, patched against the recently highlight vulnerability) to a pair
of servers running Fedora Core 3. I compiled Apache 1.3.33 and PHP
4.3.10 from source. The MySQL database is running on a separate machine.

This morning, the servers slowed down to the point that they were
almost completely unresponsive. After a while I managed to SSH into
the machines and saw that the httpd processes had grown to 35MB each
and were using up all the swap and RAM on the machines (they have 1GB
RAM, 1GB swap). I then set MaxClients to 30 and restarted Apache, but
the slowdown happened again:

top - 18:18:51 up 32 days, 4:51, 1 user, load average: 15.22, 41.62, 39.75
Tasks: 90 total, 1 running, 89 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.2% us, 0.0% sy, 0.0% ni, 99.7% id, 0.2% wa, 0.0% hi, 0.0% si
Mem: 1033484k total, 898000k used, 135484k free, 1268k buffers
Swap: 1052248k total, 896880k used, 155368k free, 10896k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
19305 nobody 16 0 41892 35m 5732 S 0.0 3.5 0:38.49 /www/bin/httpd
19219 nobody 18 0 41940 29m 5732 S 0.0 3.0 0:20.87 /www/bin/httpd
19290 nobody 18 0 41940 29m 5732 S 0.0 3.0 0:42.64 /www/bin/httpd
19217 nobody 18 0 48780 29m 5736 S 0.0 2.9 0:51.25 /www/bin/httpd
19252 nobody 16 0 41916 29m 5732 S 0.0 2.9 0:45.81 /www/bin/httpd
19298 nobody 18 0 41940 29m 5732 S 0.0 2.9 0:16.61 /www/bin/httpd
19296 nobody 18 0 44052 25m 5732 S 0.0 2.6 0:42.79 /www/bin/httpd
19294 nobody 18 0 41796 25m 5732 S 0.0 2.5 0:11.85 /www/bin/httpd
19215 nobody 17 0 41892 25m 5732 S 0.0 2.5 0:22.84 /www/bin/httpd
19218 nobody 17 0 41940 25m 5732 S 0.0 2.5 0:17.84 /www/bin/httpd
19221 nobody 16 0 41900 24m 5732 S 0.0 2.5 0:45.56 /www/bin/httpd
19299 nobody 17 0 41796 24m 5732 S 0.0 2.5 0:11.63 /www/bin/httpd
19302 nobody 17 0 41796 24m 5732 S 0.0 2.5 0:13.46 /www/bin/httpd
19220 nobody 16 0 42300 24m 5732 S 0.0 2.4 0:24.97 /www/bin/httpd
19292 nobody 16 0 41796 21m 5732 S 0.0 2.1 0:37.17 /www/bin/httpd
19303 nobody 16 0 43736 19m 5732 S 0.0 1.9 0:45.86 /www/bin/httpd
19300 nobody 18 0 42184 16m 5732 S 0.0 1.6 0:31.80 /www/bin/httpd
19222 nobody 18 0 42264 14m 5732 S 0.0 1.4 0:16.17 /www/bin/httpd
19291 nobody 16 0 41904 13m 5732 S 0.0 1.4 0:12.57 /www/bin/httpd
19293 nobody 16 0 41796 12m 5732 S 0.0 1.3 0:33.04 /www/bin/httpd
19295 nobody 18 0 44040 11m 5732 S 0.0 1.1 0:35.78 /www/bin/httpd
19224 nobody 17 0 42292 8600 5732 S 0.0 0.8 0:15.22 /www/bin/httpd
19304 nobody 16 0 43804 7908 5732 S 0.0 0.8 0:37.06 /www/bin/httpd
19301 nobody 16 0 43736 7316 5732 S 0.0 0.7 0:40.19 /www/bin/httpd
19537 nobody 18 0 21112 7020 5732 S 0.0 0.7 0:05.79 /www/bin/httpd
19250 nobody 16 0 42272 5712 5732 S 0.0 0.6 0:15.20 /www/bin/httpd
19223 nobody 16 0 41940 5248 5732 S 0.0 0.5 0:45.97 /www/bin/httpd
19216 nobody 16 0 41892 4828 5732 S 0.0 0.5 0:42.77 /www/bin/httpd
19287 nobody 18 0 41892 4312 5732 S 0.0 0.4 0:38.22 /www/bin/httpd
19288 nobody 18 0 41892 4096 5732 S 0.0 0.4 0:08.98 /www/bin/httpd

I then checked the error log and found this:

*** glibc detected *** double free or corruption: 0x097ec7e8 ***
*** glibc detected *** double free or corruption: 0x097ec7e8 ***
*** glibc detected *** double free or corruption: 0x097e6d58 ***
*** glibc detected *** double free or corruption: 0x096ec7b8 ***
*** glibc detected *** double free or corruption: 0x09723188 ***
*** glibc detected *** double free or corruption: 0x097ed3b8 ***
*** glibc detected *** double free or corruption: 0x097e8870 ***
*** glibc detected *** double free or corruption: 0x097ffe88 ***
*** glibc detected *** double free or corruption: 0x096ee320 ***
*** glibc detected *** double free or corruption: 0x08f147e8 ***
*** glibc detected *** double free or corruption: 0x08f147f8 ***
*** glibc detected *** double free or corruption: 0x08f147f8 ***

and the same on the other server:

*** glibc detected *** double free or corruption: 0x09957ac0 ***
*** glibc detected *** double free or corruption: 0x09a50710 ***
*** glibc detected *** double free or corruption: 0x09a622e0 ***
*** glibc detected *** double free or corruption: 0x09978768 ***
*** glibc detected *** double free or corruption: 0x099df900 ***
*** glibc detected *** double free or corruption: 0x0993ca08 ***
*** glibc detected *** double free or corruption: 0x0992b6d8 ***
*** glibc detected *** double free or corruption: 0x0992b6d8 ***
*** glibc detected *** double free or corruption: 0x0992b5f8 ***

This bulletin board application has been running fine for a month or
so on a pair of Fedora Core 1 servers running PHP 4.3.9 (which I've
recently upgraded to 4.3.10), and they didn't have the above problem.

Any advice would be much appreciated.

Regards,


-- Alex

[Back to original message]