Microsoft.AspNet.Server.Kestrel.FunctionalTests failures on CoreCLR on Linux #265

cesarblum · 2015-10-19T18:58:43Z

info: Exec
info:   program: /usr/bin/env
info:   commandline: bash -c ". \"/home/cesars/src/aspnet/universe/KestrelHttpServer/packages/KoreBuild/build/dnvm.sh\"; dnvm run default -runtime coreclr test"
info:   workingdir: test/Microsoft.AspNet.Server.Kestrel.FunctionalTests
xUnit.net DNX Runner (64-bit DNXCore 5.0)
  Discovering: Microsoft.AspNet.Server.Kestrel.FunctionalTests
  Discovered:  Microsoft.AspNet.Server.Kestrel.FunctionalTests
  Starting:    Microsoft.AspNet.Server.Kestrel.FunctionalTests
      System.Threading.Tasks.TaskCanceledException : A task was canceled.
      Stack Trace:
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
           at Microsoft.AspNet.Server.Kestrel.FunctionalTests.ResponseTests.<LargeDownload>d__0.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
    Microsoft.AspNet.Server.Kestrel.FunctionalTests.ThreadCountTests.ZeroToTenThreads [SKIP]
      https://github.com/aspnet/KestrelHttpServer/issues/232
      System.Threading.Tasks.TaskCanceledException : A task was canceled.
      Stack Trace:
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
           at Microsoft.AspNet.Server.Kestrel.FunctionalTests.AddressRegistrationTests.<RegisterAddresses_Success>d__0.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      System.Threading.Tasks.TaskCanceledException : A task was canceled.
      Stack Trace:
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
           at Microsoft.AspNet.Server.Kestrel.FunctionalTests.AddressRegistrationTests.<RegisterAddresses_Success>d__0.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      System.Threading.Tasks.TaskCanceledException : A task was canceled.
      Stack Trace:
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
           at Microsoft.AspNet.Server.Kestrel.FunctionalTests.AddressRegistrationTests.<RegisterAddresses_Success>d__0.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      System.Threading.Tasks.TaskCanceledException : A task was canceled.
      Stack Trace:
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
           at Microsoft.AspNet.Server.Kestrel.FunctionalTests.AddressRegistrationTests.<RegisterAddresses_Success>d__0.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      System.Threading.Tasks.TaskCanceledException : A task was canceled.
      Stack Trace:
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
           at Microsoft.AspNet.Server.Kestrel.FunctionalTests.AddressRegistrationTests.<RegisterAddresses_Success>d__0.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      System.Threading.Tasks.TaskCanceledException : A task was canceled.
      Stack Trace:
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
           at Microsoft.AspNet.Server.Kestrel.FunctionalTests.AddressRegistrationTests.<RegisterAddresses_Success>d__0.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      System.Threading.Tasks.TaskCanceledException : A task was canceled.
      Stack Trace:
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
           at Microsoft.AspNet.Server.Kestrel.FunctionalTests.AddressRegistrationTests.<RegisterAddresses_Success>d__0.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      System.Threading.Tasks.TaskCanceledException : A task was canceled.
      Stack Trace:
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
           at Microsoft.AspNet.Server.Kestrel.FunctionalTests.AddressRegistrationTests.<RegisterAddresses_Success>d__0.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
      Assert.Equal() Failure
                ↓ (pos 0)
      Expected: 0
      Actual:   1048576
                ↑ (pos 0)
      Stack Trace:
           at Microsoft.AspNet.Server.Kestrel.FunctionalTests.RequestTests.<LargeUpload>d__0.MoveNext()
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
        --- End of stack trace from previous location where exception was thrown ---
           at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
           at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
  Finished:    Microsoft.AspNet.Server.Kestrel.FunctionalTests
=== TEST EXECUTION SUMMARY ===
   Microsoft.AspNet.Server.Kestrel.FunctionalTests  Total: 11, Errors: 0, Failed: 10, Skipped: 1, Time: 901.973s

The text was updated successfully, but these errors were encountered:

cesarblum · 2015-10-19T19:38:15Z

This affects aspnet/Universe#304.

halter73 · 2015-10-28T23:57:51Z

@CesarBS I was actually abut to close this, since the functional tests are now passing on my linux box. Are you seeing something different?

cesarblum · 2015-10-29T21:58:42Z

Have a look at the latest CI build. There are some Kestrel test failures there.

muratg · 2015-11-25T18:48:03Z

@halter73 this is one of the high pri fundamentals bugs that Eilon was talking about. Please look into this on Mac and Linux runs on the CI.

halter73 · 2015-11-30T19:15:30Z

Both the KestrelTests and the Kestrel.FunctionalTests projects have been running on Travis over a week with both coreclr and mono. The last 5 Travis builds of the dev branch have passed.

https://travis-ci.org/aspnet/KestrelHttpServer/builds/93789119#L2317

I'm about to manually run the build on OS X.

halter73 · 2015-11-30T23:25:13Z

CoreClr tests are looking good on Yosemite. I also ran the tests on a non-travis Trusty VM and things are good there.

muratg · 2015-12-01T17:25:17Z

@halter73 All pass on aspnetci as well?

halter73 · 2015-12-01T18:24:34Z

Ubuntu-Universe and Mac-Universe haven't run since 11 days ago.

The last Mac run timed out before reaching the Kestrel tests. The last Ubuntu run timed out during the Mono Kestrel tests.

We have skipped some hanging tests on Mono since the last runs, so I'm rerunning both configurations now. I'll report back once I get new results.

halter73 · 2015-12-09T20:46:33Z

I ran the full Kestrel build with both CoreCLR and Mono tests on our Mac agent twice, and it passed both tests.

I did the same on our Ubuntu agent, and the build hung both times after showing the (successful) result of the Mono test run. We saw this issue a while back, but I hadn't seen it recently on travis until @CesarBS showed me the exact same type of failure happened earlier today.

Previously @CesarBS was able to (apparently) fix this issue by skipping a lot of functional tests on Mono.

halter73 · 2015-12-10T01:22:13Z

Full Kestrel builds are once again passing on both the Mac and Ubuntu agents!

muratg · 2015-12-10T04:54:56Z

Cool, thanks!

halter73 · 2015-12-11T20:49:31Z

At least the last two Ubuntu-Universe failures are due to it hanging in the middle of the Mono Kestrel tests.

After running Kestrel's build.sh in a loop until failure on the ubuntu agent, I am now once again seeing the Mono test hangs. Apparently I didn't catch everything. :(

benaadams · 2015-12-13T18:29:29Z

#473 may help with diagnosis; its helped me track down a few issues that didn't exhibit causes because some of the exceptions were getting masked in testing (though still exhibiting the symptoms of going wrong).

muratg · 2015-12-14T12:35:02Z

@halter73 I believe @CesarBS enabled CoreCLR only agents... do we have 100% in those runs?

halter73 · 2015-12-14T23:05:41Z

The Kestrel repo successfully builds and all the tests pass for the CoreCLR only runs.

This test failed on the last Ubuntu CoreClr run:

Test collection for E2ETests.PublishAndRunTests_OnX86 (1): E2ETests (1)
PublishAndRunTests_OnX86.NonWindowsOS(serverType: Kestrel, runtimeFla…e: x86, applicationBaseUrl: "http://localhost:5035/", noSource: False)
System.OperationCanceledException : Failed to connect, retry canceled.

I'm not sure if this is an issue with Kestrel or publish. I can look into it if it fails again on the next run.

muratg · 2015-12-14T23:47:24Z

Sounds good, thanks.

halter73 · 2015-12-17T20:31:44Z

I seems the the E2ETests.PublishAndRunTests_OnX86 failure is caused by a TargetInvocationException from ConsoleLoggerProvider.CreateLoggerImplementation. @CesarBS is tracking the fix.

cesarblum assigned halter73 Oct 28, 2015

muratg added Fundamentals 1 - Ready labels Nov 23, 2015

muratg added this to the 1.0.0-rc2 milestone Nov 23, 2015

halter73 added 2 - Working and removed 1 - Ready labels Nov 30, 2015

halter73 closed this as completed Nov 30, 2015

halter73 added 3 - Done and removed 2 - Working labels Nov 30, 2015

halter73 reopened this Dec 1, 2015

halter73 added 2 - Working and removed 3 - Done labels Dec 1, 2015

halter73 mentioned this issue Dec 9, 2015

Skip more tests on Mono to prevent hangs #465

Merged

halter73 mentioned this issue Dec 17, 2015

Microsoft.AspNet.Server.Kestrel.FunctionalTests failures on Mono on Linux #503

Closed

halter73 closed this as completed Dec 17, 2015

halter73 added 3 - Done and removed 2 - Working labels Dec 17, 2015

Microsoft.AspNet.Server.Kestrel.FunctionalTests failures on CoreCLR on Linux #265

Microsoft.AspNet.Server.Kestrel.FunctionalTests failures on CoreCLR on Linux #265

Comments

cesarblum commented Oct 19, 2015

cesarblum commented Oct 19, 2015

Uh oh!

halter73 commented Oct 28, 2015

Uh oh!

cesarblum commented Oct 29, 2015

Uh oh!

muratg commented Nov 25, 2015

Uh oh!

halter73 commented Nov 30, 2015

Uh oh!

halter73 commented Nov 30, 2015

Uh oh!

muratg commented Dec 1, 2015

Uh oh!

halter73 commented Dec 1, 2015

Uh oh!

halter73 commented Dec 9, 2015

Uh oh!

halter73 commented Dec 10, 2015

Uh oh!

muratg commented Dec 10, 2015

Uh oh!

halter73 commented Dec 11, 2015

Uh oh!

benaadams commented Dec 13, 2015

Uh oh!

muratg commented Dec 14, 2015

Uh oh!

halter73 commented Dec 14, 2015

Uh oh!

muratg commented Dec 14, 2015

Uh oh!

halter73 commented Dec 17, 2015

Uh oh!