Skip to content
This repository was archived by the owner on Dec 18, 2018. It is now read-only.

Limit size of memory buffer when reading request (#304) #912

Merged
merged 37 commits into from
Jun 14, 2016

Conversation

mikeharder
Copy link
Contributor

@mikeharder mikeharder commented Jun 4, 2016

Creating PR to start getting feedback and run tests on Travis/AppVeyor.

Addresses #304.

Tasks

  • Implement changes in Connection, SocketInput, and KestrelServerOptions
  • Unit tests
  • Functional tests for HTTP
  • Fix bug where Connection.OnRead() and IConnectionControl.Pause() can both call _socket.ReadStop() and throw.
    • Occasionally repros during functional test which sets max buffer to the exact same size as the uploaded data. So Connection can see the buffer become full and EOF around the same time.
    • Made UvStreamHandle.ReadStop() idempotent
  • Fix bug where UvStreamHandle.ReadStart() can throw a UvException in some cases (e.g. socket is no longer connected).
  • Change functional test from Socket to NetworkStream, so the same test can be reused with SslStream.
  • Test HTTPS
  • Functional tests sometimes fail with Block being garbage collected instead of returned to pool assertion.
    • Happens very occasionally on Windows, but almost always on Mac and Linux
    • Only happens when testing SSL with Content-Length. Non-SSL and non-Content-Length doesn't repro.
    • Owner: @CesarBS
  • Functional tests fail on Mac and Linux with unexpected client behavior, writing either too many or too few bytes before being paused
    • Too few bytes appears to be caused by a deadlock in xunit when VM only has a single core.
    • Too many bytes caused by larger OS buffer on Linux (10MB) compared to Windows (700kb).
    • Still getting too few bytes on OSX Travis, may need to increase timeout.
  • Test performance of TechEmpower scenarios
  • Test performance of large uploads. Worst-case is when the server is reading slightly slower than the client is sending, which will cause frequent calls to Pause()/Resume().
  • Test real-world application (e.g. MVC) that writes uploaded file to a (simulated) slow disk.
  • Change type from int? to long? to support buffers larger than Int32.MaxValue. Future-proof without breaking change.
  • Add sentence to doc comment explaining that null disables the buffer length checking.
  • Decide final name of option: MaxRequestBufferSize
  • Decide default value for MaxRequestBufferSize. Reasonable options:
    • null: No limit. Least likely to cause issues since execution flow is nearly identical to before this change. However, few customers will change the default, so we are least likely to find issues since the code is not being executed very often, and customers may be surprised if they increase request limit in nginx/IIS but don't change MaxInputBufferLength and see Kestrel using a lot of memory.
    • 1 MB: Matches default limit in nginx. Most requests will be under the limit, unless they are large uploads.
  • Decide threshold to call resume. Reasonable options:
    • MaxInputBufferLength - 1: If the client is sending faster than the server can consume, Pause()/Resume() may be called as often as every packet, which might degrade perf. However, it might also be the most efficient way to do things, since the buffers are kept full and the client is never blocked very long. Another advantage is the client is less likely to get a false-positive timeout.
    • MaxInputBufferLength * 0.5: Blocks the client until the server has drained at least half the buffer. The client will be paused less often, but each pause will be for a longer time.
  • Add timeouts to async calls in functional test, so test fails fast if an async call fails to return
  • Consider adding unit tests for new code in Connection.

Performance

No observable change in Plaintext or Json RPS.

Notes

@mikeharder mikeharder force-pushed the mikeharder/max-input-buffer-length branch from 69016d9 to fd30294 Compare June 4, 2016 00:44
}
set
{
if (value < -1 || value == 0)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't use magic values. Use int? instead.

@halter73
Copy link
Member

halter73 commented Jun 6, 2016

Can you include the before and after plaintext benchmark results the description?

// and server, which allow the client to send more than maxInputBufferLength before getting
// paused. We assume the combined buffers are smaller than the difference between
// data.Length and maxInputBufferLength.
Assert.InRange(bytesWritten, maxInputBufferLength.Value - maxSendSize + 1, data.Length - 1);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This assertion is failing on AppVeyor and Travis. It seems that bytesWritten is usually (but not always) 0. E.g. On AppVeyor:

Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.MaxInputBufferLengthTests.LargeUpload(maxInputBufferLength: 16384, sendContentLengthHeader: True, expectPause: True) [FAIL] 
  Assert.InRange() Failure 
  Range:  (12289 - 10485759) 
  Actual: 0 
  Stack Trace: 
    MaxInputBufferLengthTests.cs(79,0): at Microsoft.AspNetCore.Server.Kestrel.FunctionalTests.MaxInputBufferLengthTests.<LargeUpload>d__0.MoveNext() 
    --- End of stack trace from previous location where exception was thrown --- 
       at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) 
       at System.Runtime.CompilerSer

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will try a larger timeout, but 100ms is reliable on my dev machine.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently, the only issue we're seeing on AppVeyor and Travis is the Block being garbage collected instead of returned to pool assertion, which @CesarBS is investigating. We'll need to either fix the root cause or disable this assertion for 1.0.0, to unblock the tests for my change.

@mikeharder mikeharder force-pushed the mikeharder/max-input-buffer-length branch from 74570f3 to f2a4f77 Compare June 6, 2016 22:46
@@ -321,5 +341,73 @@ private static void ReturnBlocks(MemoryPoolBlock block, MemoryPoolBlock end)
returnBlock.Pool.Return(returnBlock);
}
}

private class BufferLengthConnectionController
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I still don't think this component should pause and resume. Leave that up to the caller. This couples too many pieces together (especially knowing the changes that are coming). If this is the easiest thing to do for 1.0 then it's fine but this really shouldn't know that much about pausing and resuming the connection and the fact that it has to be posted to the libuv thread.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This design was also problematic when using a connection filter, since this involves two instances of SocketInput. Refactoring will be pushed shortly.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, we'll have to redo this when the refactoring happens.

@mikeharder mikeharder force-pushed the mikeharder/max-input-buffer-length branch 2 times, most recently from e59aabc to c8f756c Compare June 9, 2016 00:05

public static IEnumerable<object[]> LargeUploadData
{
get
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please include some rationale for the test data you're creating here i.e. why those values and what you're trying to achieve with each one (or each group).

I'm fine with having the test like this (since we already follow this pattern in other places), but I'm not a fan of this approach. I prefer separate, descriptive tests (e.g. UploadExpectsPause, BufferLengthControlWorksWithSsl, etc.) that stand as a spec of what the code is supposed to do.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will add a comment for each buffer size explaining why it's worth testing.

I do think the buffer sizes (and whether we expect the client to pause) should be passed as theory data. Creating a new test for each buffer size would be a lot of extra work.

For the 4 combinations of sendContentLengthHeader and ssl, it would be reasonable to create 4 tests like:

  1. LargeUploadWithContentLength
  2. LargeUploadWithoutContentLength
  3. LargeUploadWithContentLengthOverSsl
  4. LargeUploadWithoutContentLengthOverSsl

But I still think passing these as boolean parameters is cleaner.

@mikeharder mikeharder force-pushed the mikeharder/max-input-buffer-length branch from 5597da5 to e928169 Compare June 9, 2016 21:45
@@ -189,10 +197,16 @@ public MemoryPoolIterator ConsumingStart()
{
if (!consumed.IsDefault)
{
var lengthConsumed = new MemoryPoolIterator(_head).GetLength(consumed);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only compute this if _bufferLengthControl is not null.

- Prevents breaking tests which call SocketInput.ctor()
- Add Moq to KestrelTests
Add comments explaining why each size is tested.
- HTTP requires "\r\n", and WriteLine() uses "\n" on Mac/Linux
… paused until around 10MB after the server sends backpressure.
- Blocking causes deadlock on single-core machines
- Also rename "Length" to "Size" in the implementation classes
…loaded the expected number of bytes.

- If test fails, it may now wait forever instead of throwing a nice assert.
- However, this change should make the test more robust on slow machines.
@mikeharder mikeharder force-pushed the mikeharder/max-input-buffer-length branch from 4766327 to ed94ef8 Compare June 14, 2016 01:42
@mikeharder mikeharder merged commit 5ecb1f5 into dev Jun 14, 2016
@mikeharder mikeharder deleted the mikeharder/max-input-buffer-length branch June 14, 2016 01:58
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants