[Modules] [mod_gnutls 0.5.1] Apache began to hang at maximum CPU usage when using HTTPS

[Steven Chamberlain]

Hi,

For the last few months I've had no problems at with mod_gnutls 0.5.1 on 
Apache 2.2.9.  I reported some time ago that mod_gnutls was crashing 
Apache on startup (bug#82) but after rewriting much of my Apache 
configuration from scratch I could no longer reproduce that problem.

Today, however, I caught my server load average at 40.  I had a few 
apache2 worker threads trying to use maximum CPU.  After restarting 
Apache it became obvious that HTTPS sessions were triggering this.  (The 
server tends to only serve via HTTP, and it's usually only me that 
requests pages from it via HTTPS.)

The client (Firefox 2.0.0.14), would take a very long time (a minute or 
two) to load the page.  After that, the apache2 worker thread would 
continue to run, with maximum CPU usage.  For example, in the output of 
'top' I would see numerous process like this:

     PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
   23018 www-data  20   0  247m  16m 3928 S   35  2.2   6:44.41 apache2

I recently upgraded to Apache 2.2.9 from 2.2.8, so I tried downgrading 
but I continued to experience the problem.  I also started recompiling 
mod_gnutls and related libraries (gnutls, apr_memcache, apr).  That had 
no effect either.

I eventually thought to look in /var/cache.  This is what I saw:

   interlock:/var/cache# du -h mod_gnutls.*
   36K     mod_gnutls.dir
   5.5M    mod_gnutls.pag

   interlock:/var/cache# ls -al mod_gnutls.*
   -rw-r--r-- 1 www-data root     364544 2008-06-15 22:23 mod_gnutls.dir
   -rw-r--r-- 1 www-data root 3053067264 2008-06-16 02:07 mod_gnutls.pag

I'm not sure what to make of the difference in reported file size here 
(5.5MB vs. 3.1GB, although the partition on which the file resided has 
only 1GB capacity).  There is no evidence of filesystem corruption (the 
machine hasn't been powered off or restarted, dmesg is clean, and there 
are no other apparent problems).

After simply moving those two files to another location and restarting 
apache2, everything seems to be working again just fine.

I'm truly baffled by this.  I've no idea why this happened, and I doubt 
I could ever make this happen again if I tried.  Is there anything 
regarding these two files which I may have overlooked?

Thanks,
-- 
Steven Chamberlain
steven at pyro.eu.org