[Issues] [mod_gnutls 0000106]: mod_gnutls >= 0.5.6 always crashes while accessing Horde's pages on FreeBSD

Mantis Bug Tracker issues at outoforder.cc
Wed Aug 18 18:06:38 EDT 2010


A NOTE has been added to this issue. 
====================================================================== 
http://issues.outoforder.cc/view.php?id=106 
====================================================================== 
Reported By:                tinlans
Assigned To:                
====================================================================== 
Project:                    mod_gnutls
Issue ID:                   106
Category:                   Apache Integration
Reproducibility:            always
Severity:                   crash
Priority:                   normal
Status:                     new
Apache Version:             2.2.16 (MPM=worker) 
====================================================================== 
Date Submitted:             2010-08-05 13:19 EDT
Last Modified:              2010-08-18 18:06 EDT
====================================================================== 
Summary:                    mod_gnutls >= 0.5.6 always crashes while accessing
Horde's pages on FreeBSD
Description: 
OS: FreeBSD 8.1-STABLE amd64
Apache: apache-worker-2.2.16
Apache Info: Apache/2.2.16 (FreeBSD) DAV/2 PHP/5.2.14 with Suhosin-Patch
SVN/1.6.12 mod_gnutls/0.5.7
APR: apr-devrandom-db46-ldap24-mysql55-1.4.2.1.3.9_1
gnutls: gnutls-devel-2.11.0_1

Horde is a well-known PHP-based Horde Application Framework.
It's often used for providing webmail service via Horde/IMP.
A large set of PEAR packages are used by it.

Everything works properly on mod_gnutls-0.5.5.
Since mod_gnutls-0.5.6, to access my web's Horde pages causes Apache
crashed.
This problem is also occured on mod_gnutls-0.5.7.

Here is the backtrace info of my crash dump:
http://issues.outoforder.cc/view.php?id=0  0x0000000807177b22 in
_gnutls_recv_int () from
/usr/local/lib/libgnutls.so.43
http://issues.outoforder.cc/view.php?id=1  0x0000000807057706 in
gnutls_io_input_read (ctxt=0x81041e028, 
    buf=0x81041e070 "\r\nokie: default_imp_view=imp;
ITHorde=83d0j2gu59itsc4pn2irurdg36;
auth_key=155e3ae9df0ace4698270f0c1dfd02f9;
imp_key=f7235d305b37b51f6c6d9380c0302a02\r\n\r\nb51f6c6d9380c0302a02\r\n\r\nb37b51f6c6d9380c0302a02"...,
len=0x7fffff7fcd68) at gnutls_io.c:227
http://issues.outoforder.cc/view.php?id=2  0x0000000807057ba2 in
mgs_filter_input (f=0x810422120, bb=0x810712740,
mode=Variable "mode" is not available.
) at gnutls_io.c:323
http://issues.outoforder.cc/view.php?id=3  0x000000000042ab98 in
ap_rgetline_core (s=0x8107110d0, n=8192,
read=0x7fffff7fce70, r=0x8107110a0, fold=0, bb=0x810712740) at
protocol.c:231
http://issues.outoforder.cc/view.php?id=4  0x000000000042b4d5 in ap_read_request
(conn=0x81041a2b8) at
protocol.c:596
http://issues.outoforder.cc/view.php?id=5  0x0000000000440ce3 in
ap_process_http_connection (c=0x81041a2b8) at
http_core.c:183
http://issues.outoforder.cc/view.php?id=6  0x000000000043d562 in
ap_run_process_connection (c=0x81041a2b8) at
connection.c:43
http://issues.outoforder.cc/view.php?id=7  0x0000000000448992 in worker_thread
(thd=0x80827b880, dummy=Variable
"dummy" is not available.
) at worker.c:544
http://issues.outoforder.cc/view.php?id=8  0x00000008011b7511 in pthread_getprio
() from /lib/libthr.so.3
http://issues.outoforder.cc/view.php?id=9  0x0000000000000000 in ?? ()
Error accessing memory address 0x7fffff7fd000: Bad address.
====================================================================== 

---------------------------------------------------------------------- 
 (0000213) nmav (manager) - 2010-08-17 13:06
 http://issues.outoforder.cc/view.php?id=106#c213 
---------------------------------------------------------------------- 
Hi. Does the attached patch solve your issue? 

---------------------------------------------------------------------- 
 (0000214) tinlans (reporter) - 2010-08-17 15:32
 http://issues.outoforder.cc/view.php?id=106#c214 
---------------------------------------------------------------------- 
This patch solves a lot of "segmentation fault" issues,
but I also find some new issues.

Some contents of a page cannot show in Horde/IMP randomly.
For example, icons randomly displayed as red "X", sub-pages (or partial
pages) in a table (or a frame) randomly displayed as "connection
refused".
I cannot find any error messages in httpd-error.log. 

---------------------------------------------------------------------- 
 (0000215) nmav (manager) - 2010-08-17 15:35
 http://issues.outoforder.cc/view.php?id=106#c215 
---------------------------------------------------------------------- 
By "a lot" do you mean that some segmentation faults remain? 

---------------------------------------------------------------------- 
 (0000216) tinlans (reporter) - 2010-08-17 15:40
 http://issues.outoforder.cc/view.php?id=106#c216 
---------------------------------------------------------------------- 
Yes, some segmentation faults remain.
They're very hard to be repeated so that I cannot catch it by gdb. 

---------------------------------------------------------------------- 
 (0000217) nmav (manager) - 2010-08-17 15:44
 http://issues.outoforder.cc/view.php?id=106#c217 
---------------------------------------------------------------------- 
This is strange. I've updated the patch in patch2.txt to try (it applies to
clean 0.5.7). If the problem persists, could you try increasing your ulimit
-c, to allow the server to create core file, and check that with gdb?

Thanks. 

---------------------------------------------------------------------- 
 (0000218) tinlans (reporter) - 2010-08-17 15:57
 http://issues.outoforder.cc/view.php?id=106#c218 
---------------------------------------------------------------------- 
Yes, I can configure it to allow to dump core files.
I've been applied the patch2.txt.

In a rare case, I could find this error (it's also hard to be repeated):
Program terminated with signal 11, Segmentation fault.
(gdb) where
http://issues.outoforder.cc/view.php?id=0  0x0000000000000000 in ?? ()
http://issues.outoforder.cc/view.php?id=1  0x000000080705aba2 in mgs_hook_fixups
(r=0x0) at gnutls_hooks.c:752
Cannot access memory at address 0x800000000868
(gdb) frame 1
http://issues.outoforder.cc/view.php?id=1  0x000000080705aba2 in mgs_hook_fixups
(r=0x0) at gnutls_hooks.c:752
752         apr_table_setn(env, "SSL_PROTOCOL",
(gdb) list
747         apr_table_setn(env, "SSL_VERSION_LIBRARY",
748                        "GnuTLS/" LIBGNUTLS_VERSION);
749         apr_table_setn(env, "SSL_VERSION_INTERFACE",
750                        "mod_gnutls/" MOD_GNUTLS_VERSION);
751
752         apr_table_setn(env, "SSL_PROTOCOL",
753                       
gnutls_protocol_get_name(gnutls_protocol_get_version
754                                                 (ctxt->session)));
755
756         /* should have been called SSL_CIPHERSUITE instead */
(gdb) print env
$1 = (apr_table_t *) 0x0 

---------------------------------------------------------------------- 
 (0000219) nmav (manager) - 2010-08-18 11:07
 http://issues.outoforder.cc/view.php?id=106#c219 
---------------------------------------------------------------------- 
I've understood where the problem is for this case, but it shouldn't
happen. Which apache do you use? Does patch3 solve the issue? 

---------------------------------------------------------------------- 
 (0000220) tinlans (reporter) - 2010-08-18 14:24
 http://issues.outoforder.cc/view.php?id=106#c220 
---------------------------------------------------------------------- 
Yes, patch3 solves this issue.
There are not any segmentation faults occurred,
but some icons and partial pages still failed to show randomly.
Nothing can be found in Apache's log files.
Perhaps some related handler gives up some jobs and exits/returns
directly.

I'm using apache-worker-2.2.16.
It's the latest version in the ports tree of FreeBSD.

Since it's configured as MPM=worker, thread-safetey issues may be
introduced.
Most of apache modules work perfectly with MPM=prefork (default),
and a few apache modules doesn't work with MPM=worker.
I cannot prove that all of these issues are casued by thread-safety
issues,
but most of these problems were presented when these modules were using
some APR functions,
especially of APR::Pool and APR::Table series.

 

---------------------------------------------------------------------- 
 (0000221) nmav (manager) - 2010-08-18 15:16
 http://issues.outoforder.cc/view.php?id=106#c221 
---------------------------------------------------------------------- 
Thank you. Is it a multicore system? Could it be that the memory allocator
used by apache in freebsd fails in some cases when in multi-threaded mode?
(by fails I mean that it might return NULL)

 

---------------------------------------------------------------------- 
 (0000222) nmav (manager) - 2010-08-18 15:15
 http://issues.outoforder.cc/view.php?id=106#c222 
---------------------------------------------------------------------- 
btw. I've released 0.5.8 that includes this patch. 

---------------------------------------------------------------------- 
 (0000223) tinlans (reporter) - 2010-08-18 18:06
 http://issues.outoforder.cc/view.php?id=106#c223 
---------------------------------------------------------------------- 
Thanks.

Now it seems to work properly on my machine.
I forgot to kill the httpd processes what I were ran durning debug.
Perhaps it's the reason why I cannot open partial contents of a web page.
You can just ignore that problem.

The latest patch seems to solve all issues.

Yes, my system is a multicore system (CPU is Intel QX9650).
FreeBSD's kernel is configured as SMP.

My knowledge about Apache's API is actually out-of-date (Apache 1.x).
I'm not sure the behavior of the memory allocator used by Apache in
FreeBSD,
but I guess that might be platform-independent.

NULL-related issues may be caused by multiple initializations (I guess).
MPM worker is a model that prefork some processes,
and each process contains many threads.
Perhaps a global resource in the same process may be initialized to NULL,
setup by a API of the memory allocator, and then re-initialize to NULL by
other threads. 

Issue History 
Date Modified    Username       Field                    Change               
====================================================================== 
2010-08-05 13:19 tinlans        New Issue                                    
2010-08-05 13:19 tinlans        Apache Version            => 2.2.16 (MPM=worker)
2010-08-17 13:06 nmav           File Added: patch.txt                        
2010-08-17 13:06 nmav           Note Added: 0000213                          
2010-08-17 15:32 tinlans        Note Added: 0000214                          
2010-08-17 15:35 nmav           Note Added: 0000215                          
2010-08-17 15:40 tinlans        Note Added: 0000216                          
2010-08-17 15:42 nmav           File Added: patch2.txt                       
2010-08-17 15:44 nmav           Note Added: 0000217                          
2010-08-17 15:57 tinlans        Note Added: 0000218                          
2010-08-18 11:05 nmav           File Deleted: patch.txt                      
2010-08-18 11:06 nmav           File Added: patch3.txt                       
2010-08-18 11:06 nmav           File Deleted: patch2.txt                     
2010-08-18 11:07 nmav           Note Added: 0000219                          
2010-08-18 14:24 tinlans        Note Added: 0000220                          
2010-08-18 14:24 tinlans        Note Edited: 0000220                         
2010-08-18 15:15 nmav           Note Added: 0000221                          
2010-08-18 15:15 nmav           Note Added: 0000222                          
2010-08-18 15:16 nmav           Note Edited: 0000221                         
2010-08-18 18:06 tinlans        Note Added: 0000223                          
======================================================================




More information about the Issues mailing list