Apache stuck indefinitely waiting for PSOL

[GoogleCodeExporter]

I just installed mod_pagespeed on my centos 7 and got tons of httpd errors log 
in 1 minute. An example line of error:
<code>
[Tue Feb 10 11:05:14.311755 2015] [pagespeed:warn] [pid 21132:tid 
139634310850304] [mod_pagespeed 1.9.32.3-4448 @21132] Waiting for completion of 
URL http://exampledomain.com/example-slug/ for 45.001 sec
</code>

ALL requests got error, include image requests too.


My server hardware specs:
* Intel(R) Xeon(R) CPU E3-1246 v3 @ 3.50GHz, 8 cores
* 32 GB DDR3 RAM
* 2 x 2 TB SATA 6 Gb/s 7200 rpm HDD (Software-RAID 1) Class Enterprise

Software specs:
Operating system: CentOS Linux 7.0.1406
Kernel: Linux 3.10.0-123.20.1.el7.x86_64 on x86_64

Server Version: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 
PHP/5.6.5 mod_perl/2.0.9dev Perl/v5.16.3
Server MPM: event


What version of the product are you using (please check X-Mod-Pagespeed
header)?
mod-pagespeed-stable-1.9.32.3-4448.x86_64

URL of broken page:
I removed module after 1 minute of terror. If a google developer want to learn 
more of server information, mail me an ip address that i can give permission to 
look mod_info print.

Original issue reported on code.google.com by unsalkor...@gmail.com on 10 Feb 2015 at 9:35

[Cyriltra]

Hi,
same issue with Debian 7.8 64bits / Apache 2.2.22.
I had to disable the module to prevent crash (got 2 in less than a month)

[crowell]

@Cyriltra did you have IPRO enabled? it is enabled by default on 1.9.32.x

[Cyriltra]

@crowell yes i left it enable (by default). I didn't try to disable it as another user said before it still not fix the issue for him. So I didn't dare to take that risk as I was in rush to fix the issue.
I'll give a try this weekend

[jeffkaufman]

This looks very suspicious:

void ApacheFetch::Wait() {
  ScopedMutex lock(mutex_.get());
  while (!done_) { ... }
}
...
void ApacheFetch::HandleDone(bool success) {
  ScopedMutex lock(mutex_.get());
  done_ = true;
}

[oschaaf]

@jeffkaufman fwiw, looks like something that could very well explain the behaviour described here to me indeed.

[jeffkaufman]

Looking more and talking to @jmarantz that's suspicious but actually things working properly. In the snipped code theres a Condvar:

mutex_(server_context->thread_system()->NewMutex()),
condvar_(mutex_->NewCondvar()),
...
void ApacheFetch::Wait() {
  ScopedMutex lock(mutex_.get());
  while (!done_) {
    condvar_->TimedWait(blocking_fetch_timeout_ms_);
  }
}
...
void ApacheFetch::HandleDone(bool success) {
  ScopedMutex lock(mutex_.get());
  done_ = true;
  condvar_->Signal();
}

I believe, but I haven't verified, that the lock is released while in a timed wait on the condvar, and then when you signal the condvar it wakes up and grabs the lock again.

[jeffkaufman]

Verfied about the condvar. TimedWait() delegates to pthread_cond_timedwait.

[oschaaf]

InstawebHandler::~InstawebHandler() {
  WaitForFetch();
}

I might have an unnatural focus on shutting down :-)
But what happens during when Apache recycles worker processes (assuming it does that)? Will ApacheFetch::Done be called somehow? If not, these wait calls might not ever stop waiting.

[jeffkaufman]

@oschaaf All our uses of ApacheFetch should look like:

1. MakeFetch()
2. SomeFetcher->Fetch(fetch)
3. WaitForFetch()

Which I think means WaitForFetch() in the destructor should never be called.

[morlovich]

It should also quick-exit because done is already true.

On Fri, Apr 10, 2015 at 9:12 AM, Jeff Kaufman notifications@github.com
wrote:

@oschaaf https://github.com/oschaaf All our uses of ApacheFetch should
look like:

1. MakeFetch()
2. SomeFetcher->Fetch(fetch)
3. WaitForFetch()

Which I think means WaitForFetch() in the destructor should never be
called.

—
Reply to this email directly or view it on GitHub
#1048 (comment)
.