[Modules] apr_memcache Errors on heavy access

[Juhani Connolly]

I've written an apache module that among other things enters a visitors user 
agent into a database and records the index in a flat file(or just records 
the index if it already exists). To speed this process up I'm using 
aprmemcache to store indexes with the user agent as the key.
At moderate access rates this works perfectly fine, and I've had it running 
in the background for weeks on a site with about 500,000 daily accesses, but 
when I run it full pelt with a tool like ab for 50,000 requests, it will go 
at about 5000-8000 requests initially but most times towards the end I get 
the following error in my logs(from my module, though the errno being 
reported is the return from apr_memcache_getp):
[Tue Feb 17 19:38:07 2009] [notice] (99)Cannot assign requested address: 
[access] Couldn't retrieve agent from mc

After this occurs, apr_memcache_getp won't get anything and index requests 
end up going to mysql slowing the system to a crawl.

Taking a quick peek at the source which I had some trouble following, I 
could see that pretty much any error will cause the server connected to be 
marked as bad. What would the correct action be to take here? Can I just go 
and open up the connection again(is this even safe when I'm running 
multi-threaded? )?

Rest is optional reading that might be useful if anyone could spare some of 
their valuable time to help.
Thanks,
 Juhani Connolly


---------------------------------------------------------------------------------------------------

More specifics:
Using worker mpm

Currently I set up my memcache server  list from the post_config hook.
memcache creation(error code/basic string manipulations like fetching the 
server address/port stripped out for conciseness)
static int mna_hook_post_config(apr_pool_t *p, apr_pool_t *plog, apr_pool_t 
*ptemp, server_rec *s)
{
 // do other stuff, make sure we only run this once etc...
    .
    .
    .
 //**************************
 mna_srv_conf* svr = ap_get_module_config(s->module_config, &access_module);
 int thread_limit = 0;
 ap_mpm_query(AP_MPMQ_HARD_LIMIT_THREADS, &thread_limit);
 cache_config = apr_pstrdup(p, svr->servers); // memcache server info loaded 
from config file

 rv = apr_memcache_create(p, srv_count, 0, &(svr->mc));

 // init memcache
  while(getServer(cache_config))        // get the parameters
 {
  rv = apr_memcache_server_create(p, host_str, port, 0, 1, thread_limit, 
600, &st);    // I've tried using other settings for min and smax and the 
problem persists
  rv = apr_memcache_add_server(svr->mc, st);
  }
}

memcache usage:

char * GetAgentInd(request_rec *r, mna_srv_conf* svr, char * agent)
{
 char *agentIndex;
 char *mcAgent = apr_psprintf(r->pool, "NA_%s", agent);
 char *p = mcAgent;

 apr_status_t rv = apr_memcache_getp(svr->mc, r->pool, mcAgent, &agentIndex, 
&size, NULL);
  if(rv != APR_NOTFOUND && rv != APR_SUCCESS)
    ap_log_error(APLOG_MARK, APLOG_NOTICE, rv, r->server, "[access] Couldn't 
retrieve agent from mc");

After the fail case where we load from mysql we try to write to memcache
 rv = apr_memcache_set(svr->mc, mcAgent, agentIndex, strlen(agentIndex), 
86400, 0);

After the aforementioned 99(EADDRNOTAVAIL) error occurs this will start to 
fail with(because the connection is presumably down)
(70015)Could not find specified socket in poll list.