net/http: application hangs after a while #28441
Comments
|
SIGABRT: abort goroutine 0 [idle]: goroutine 1 [runnable]: goroutine 5 [select, 1969 minutes]: goroutine 9 [chan receive, 2046 minutes]: goroutine 10 [chan receive, 2046 minutes]: goroutine 11 [chan receive, 2046 minutes]: goroutine 12 [runnable]: goroutine 21 [select, 1971 minutes]: goroutine 16 [chan receive, 2044 minutes]: goroutine 23 [IO wait, 1976 minutes]: goroutine 24 [sleep, 1976 minutes]: goroutine 25 [select, 1969 minutes]: goroutine 26 [select, 1969 minutes]: goroutine 27 [select, 1971 minutes]: goroutine 330 [select]: goroutine 29 [chan receive, 1985 minutes]: goroutine 220 [runnable]: goroutine 213 [sleep, 1975 minutes]: goroutine 221 [select]: goroutine 219 [select, 1969 minutes]: goroutine 215 [sleep, 1985 minutes]: goroutine 218 [sleep, 1985 minutes]: goroutine 222 [select]: goroutine 225 [runnable]: goroutine 329 [select]: goroutine 354 [runnable]: goroutine 318914 [runnable]: goroutine 318899 [runnable]: goroutine 248 [runnable]: goroutine 333 [runnable]: goroutine 295 [running]: goroutine 292 [select]: goroutine 318916 [runnable]: goroutine 318915 [runnable]: goroutine 318913 [runnable]: goroutine 318900 [select]: goroutine 318897 [runnable]: goroutine 318917 [runnable]: goroutine 318898 [select]: trap 0x0 |
|
Hi @dannin, Please provide us with detailed information of how exactly you tested the application and a sample source code of the application so that we can debug and reproduce the issue. Also, does the issue occur only in arm cpus ? What happens if you run it on amd64 ? |
agnivade
added
the
WaitingForInfo
|
@agnivade Basically the service replies to http request with regular json. One of the handlers does a exec command. We don’t have any mutexs in any of those code paths. As you can see from the stack trace, all of our go routines are in some wait / sleep. Go routine 0 sticks out to me and 318913. |
|
@dannin - Without a way to reproduce, it will be very hard to debug much. If you can find a way to extract sensitive parts of the application and still make it reproducible, that would be very helpful. I will page @bradfitz to see if we can do anything here. Also, have you seen this comment of mine ?
It would be interesting to see if this happens only on arm. That would atleast eliminate some possibilities. |
agnivade
changed the title
agnivade
added
the
NeedsInvestigation
|
It has happened about 10-20 times so far in the past 6 months. We only run it on arm. |
|
Actually to add, just realized, majority of the request are just proxied through. proxy := SitemasterProxy{reverseProxy: httputil.NewSingleHostReverseProxy(proxyUrl)} only one of the handlers does an exec.Command, which you can see in go routine 318913 |
|
Also we set GOMAXPROCS to 2, or board has one CPU. |
|
Historically bug reports like these are related to poor error handling or data races. Are you able to run the various vet/check/lint programs over your code, they may help spot ineffective error handling. While it is not available on arm, if you can build and run your application on a platform supported by the race detector this may help find data race issues in your application. |
|
Could this be related to #28204? I'm seeing hangs using the HTTP reverse proxy as well. I traced down that issue, but I think there's another (or that issue doesn't fully fix the problem). |
|
@dannin - We would need something more just the stack trace to debug this better. I understand that you don't run in on amd64. But does this repro too in amd64 ? I understand that there are no mutexes in the code paths, but could you run the code under race detector and see if there are any race issues ? Also, could you run the code using |
agnivade
added
WaitingForInfo
|
Hey @agnivade - @prashantv managed to isolate a minimal repro of #28204 (see the ticket for the code). That issue could be the underlying cause for this issue since it uses the ReverseProxy. |
|
Possibly. I will let @bradfitz weigh in. |
|
Timed out in state WaitingForInfo. Closing. (I am just a bot, though. Please speak up if this is a mistake or you have the requested information.) |
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (
go version)?1.10.1
Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?linux arm
1cpu
What did you do?
Application is left on with repeated http requests
If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.
What did you expect to see?
Application not to hang
What did you see instead?
Application becomes unresponsive
The text was updated successfully, but these errors were encountered: