Skip to content

Use System.Text.Json's IAsyncEnumerable support #31894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 19, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
292 changes: 146 additions & 146 deletions eng/Version.Details.xml

Large diffs are not rendered by default.

150 changes: 75 additions & 75 deletions eng/Versions.props

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions global.json
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
{
"sdk": {
"version": "6.0.100-preview.3.21168.19"
"version": "6.0.100-preview.4.21216.8"
},
"tools": {
"dotnet": "6.0.100-preview.3.21168.19",
"dotnet": "6.0.100-preview.4.21216.8",
"runtimes": {
"dotnet/x64": [
"2.1.25",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
<TargetFramework>$(DefaultNetCoreTargetFramework)</TargetFramework>
<RuntimeIdentifier>browser-wasm</RuntimeIdentifier>
<SelfContained>true</SelfContained>
<UseMonoRuntime>true</UseMonoRuntime>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this really related to this PR, or just something we need for another reason?

Could you clarify why we need it (and didn't before)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change was ported from the PR in main: https://github.com/dotnet/aspnetcore/pull/31748/files. The targeting packs in the runtime were recently renamed and this is now required if you do not reference the BlazorWebAssembly SDK (which this project does not)

</PropertyGroup>
<ItemGroup>
<Reference Include="Microsoft.AspNetCore.Metadata" />
Expand Down
36 changes: 2 additions & 34 deletions src/Mvc/Mvc.Core/src/Infrastructure/ObjectResultExecutor.cs
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,6 @@ namespace Microsoft.AspNetCore.Mvc.Infrastructure
/// </summary>
public class ObjectResultExecutor : IActionResultExecutor<ObjectResult>
{
private readonly AsyncEnumerableReader _asyncEnumerableReaderFactory;

/// <summary>
/// Creates a new <see cref="ObjectResultExecutor"/>.
/// </summary>
Expand Down Expand Up @@ -54,8 +52,6 @@ public ObjectResultExecutor(
FormatterSelector = formatterSelector;
WriterFactory = writerFactory.CreateWriter;
Logger = loggerFactory.CreateLogger<ObjectResultExecutor>();
var options = mvcOptions?.Value ?? throw new ArgumentNullException(nameof(mvcOptions));
_asyncEnumerableReaderFactory = new AsyncEnumerableReader(options);
}

/// <summary>
Expand Down Expand Up @@ -103,23 +99,9 @@ public virtual Task ExecuteAsync(ActionContext context, ObjectResult result)
}

var value = result.Value;

if (value != null && _asyncEnumerableReaderFactory.TryGetReader(value.GetType(), out var reader))
{
return ExecuteAsyncEnumerable(context, result, value, reader);
}

return ExecuteAsyncCore(context, result, objectType, value);
}

private async Task ExecuteAsyncEnumerable(ActionContext context, ObjectResult result, object asyncEnumerable, Func<object, Task<ICollection>> reader)
{
Log.BufferingAsyncEnumerable(Logger, asyncEnumerable);

var enumerated = await reader(asyncEnumerable);
await ExecuteAsyncCore(context, result, enumerated.GetType(), enumerated);
}

Copy link
Member

@javiercn javiercn Apr 19, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By doing this, are we not removing support for this on other formatters? (JSON.NET, XML)

Nevermind, I see that this has been pushed down to our two other formatters. That said, this might break people creating their own formatter?

Would it make sense for this to remain on the base class and have STJ formatter signal that it doesn't need it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was one of the options I considered. The current implementation is also a bit of a whack-a-mole in that it only works for all the result types that we account for. Custom result types would have the same issue (e.g. if a user makes a XmlResult or a ProtobufResult). Given the choice of either handling it at per-formatter vs per-result, the latter seems like a better choice.

I can make an announcement since this is a breaking behavior change.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Result types derived from ObjectResult would have gotten this for free right?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the ordinary case, yes. It's possible that there's also custom result executors that would negate it though.

private Task ExecuteAsyncCore(ActionContext context, ObjectResult result, Type? objectType, object? value)
{
var formatterContext = new OutputFormatterWriteContext(
Expand Down Expand Up @@ -166,21 +148,7 @@ private static void InferContentTypes(ActionContext context, ObjectResult result
}
}

private static class Log
{
private static readonly Action<ILogger, string?, Exception?> _bufferingAsyncEnumerable = LoggerMessage.Define<string?>(
LogLevel.Debug,
new EventId(1, "BufferingAsyncEnumerable"),
"Buffering IAsyncEnumerable instance of type '{Type}'.",
skipEnabledCheck: true);

public static void BufferingAsyncEnumerable(ILogger logger, object asyncEnumerable)
{
if (logger.IsEnabled(LogLevel.Debug))
{
_bufferingAsyncEnumerable(logger, asyncEnumerable.GetType().FullName, null);
}
}
}
// Removed Log.
// new EventId(1, "BufferingAsyncEnumerable")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍🏾

}
}
Original file line number Diff line number Diff line change
Expand Up @@ -69,12 +69,6 @@ public async Task ExecuteAsync(ActionContext context, JsonResult result)
Log.JsonResultExecuting(_logger, result.Value);

var value = result.Value;
if (value != null && _asyncEnumerableReaderFactory.TryGetReader(value.GetType(), out var reader))
{
Log.BufferingAsyncEnumerable(_logger, value);
value = await reader(value);
}

var objectType = value?.GetType() ?? typeof(object);

// Keep this code in sync with SystemTextJsonOutputFormatter
Expand Down Expand Up @@ -147,11 +141,7 @@ private static class Log
"Executing JsonResult, writing value of type '{Type}'.",
skipEnabledCheck: true);

private static readonly Action<ILogger, string?, Exception?> _bufferingAsyncEnumerable = LoggerMessage.Define<string?>(
LogLevel.Debug,
new EventId(2, "BufferingAsyncEnumerable"),
"Buffering IAsyncEnumerable instance of type '{Type}'.",
skipEnabledCheck: true);
// EventId 2 BufferingAsyncEnumerable

public static void JsonResultExecuting(ILogger logger, object? value)
{
Expand All @@ -161,14 +151,6 @@ public static void JsonResultExecuting(ILogger logger, object? value)
_jsonResultExecuting(logger, type, null);
}
}

public static void BufferingAsyncEnumerable(ILogger logger, object asyncEnumerable)
{
if (logger.IsEnabled(LogLevel.Debug))
{
_bufferingAsyncEnumerable(logger, asyncEnumerable.GetType().FullName, null);
}
}
}
}
}
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
// Copyright (c) .NET Foundation. All rights reserved.
// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Text;
using System.Text.Json;
using System.Text.Json.Serialization;
Expand Down Expand Up @@ -81,6 +83,36 @@ public async Task WriteResponseBodyAsync_WithNonUtf8Encoding_FormattingErrorsAre
await Assert.ThrowsAsync<TimeZoneNotFoundException>(() => formatter.WriteResponseBodyAsync(outputFormatterContext, Encoding.GetEncoding("utf-16")));
}

[Fact]
public async Task WriteResponseBodyAsync_ForLargeAsyncEnumerable()
{
// Arrange
var expected = new MemoryStream();
await JsonSerializer.SerializeAsync(expected, LargeAsync(), new JsonSerializerOptions(JsonSerializerDefaults.Web));
var formatter = GetOutputFormatter();
var mediaType = MediaTypeHeaderValue.Parse("application/json; charset=utf-8");
var encoding = CreateOrGetSupportedEncoding(formatter, "utf-8", isDefaultEncoding: true);

var body = new MemoryStream();
var actionContext = GetActionContext(mediaType, body);

var asyncEnumerable = LargeAsync();
var outputFormatterContext = new OutputFormatterWriteContext(
actionContext.HttpContext,
new TestHttpResponseStreamWriterFactory().CreateWriter,
asyncEnumerable.GetType(),
asyncEnumerable)
{
ContentType = new StringSegment(mediaType.ToString()),
};

// Act
await formatter.WriteResponseBodyAsync(outputFormatterContext, Encoding.GetEncoding("utf-8"));

// Assert
Assert.Equal(expected.ToArray(), body.ToArray());
}

private class Person
{
public string Name { get; set; }
Expand Down Expand Up @@ -108,5 +140,15 @@ public override void Write(Utf8JsonWriter writer, ThrowingFormatterModel value,
throw new TimeZoneNotFoundException();
}
}

private static async IAsyncEnumerable<int> LargeAsync()
{
await Task.Yield();
// MvcOptions.MaxIAsyncEnumerableBufferLimit is 8192. Pick some value larger than that.
foreach (var i in Enumerable.Range(0, 9000))
{
yield return i;
}
}
}
}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
// Copyright (c) .NET Foundation. All rights reserved.
// Copyright (c) .NET Foundation. All rights reserved.
// Licensed under the Apache License, Version 2.0. See License.txt in the project root for license information.

using System;
Expand Down Expand Up @@ -190,6 +190,19 @@ public async Task Reader_ThrowsIfBufferLimitIsReached()
Assert.Equal(expected, ex.Message);
}

[Fact]
public async Task Reader_ThrowsIfIAsyncEnumerableThrows()
{
// Arrange
var enumerable = ThrowingAsyncEnumerable();
var options = new MvcOptions();
var readerFactory = new AsyncEnumerableReader(options);

// Act & Assert
Assert.True(readerFactory.TryGetReader(enumerable.GetType(), out var reader));
await Assert.ThrowsAsync<TimeZoneNotFoundException>(() => reader(enumerable));
}

public static async IAsyncEnumerable<string> TestEnumerable(int count = 3)
{
await Task.Yield();
Expand Down Expand Up @@ -225,5 +238,18 @@ public IAsyncEnumerator<string> GetAsyncEnumerator(CancellationToken cancellatio
IAsyncEnumerator<object> IAsyncEnumerable<object>.GetAsyncEnumerator(CancellationToken cancellationToken)
=> GetAsyncEnumerator(cancellationToken);
}

private static async IAsyncEnumerable<string> ThrowingAsyncEnumerable()
{
await Task.Yield();
for (var i = 0; i < 10; i++)
{
yield return $"Hello {i}";
if (i == 5)
{
throw new TimeZoneNotFoundException();
}
}
}
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -333,7 +333,7 @@ public async Task ExecuteAsync_WithNullValue()
public async Task ExecuteAsync_SerializesAsyncEnumerables()
{
// Arrange
var expected = Encoding.UTF8.GetBytes(JsonSerializer.Serialize(new[] { "Hello", "world" }));
var expected = JsonSerializer.Serialize(new[] { "Hello", "world" });

var context = GetActionContext();
var result = new JsonResult(TestAsyncEnumerable());
Expand All @@ -344,7 +344,7 @@ public async Task ExecuteAsync_SerializesAsyncEnumerables()

// Assert
var written = GetWrittenBytes(context.HttpContext);
Assert.Equal(expected, written);
Assert.Equal(expected, Encoding.UTF8.GetString(written));
}

[Fact]
Expand Down
100 changes: 0 additions & 100 deletions src/Mvc/Mvc.Core/test/Infrastructure/ObjectResultExecutorTest.cs
Original file line number Diff line number Diff line change
Expand Up @@ -459,106 +459,6 @@ public async Task ObjectResult_NullValue()
Assert.Null(formatterContext.Object);
}

[Fact]
public async Task ObjectResult_ReadsAsyncEnumerables()
{
// Arrange
var executor = CreateExecutor();
var result = new ObjectResult(AsyncEnumerable());
var formatter = new TestJsonOutputFormatter();
result.Formatters.Add(formatter);

var actionContext = new ActionContext()
{
HttpContext = GetHttpContext(),
};

// Act
await executor.ExecuteAsync(actionContext, result);

// Assert
var formatterContext = formatter.LastOutputFormatterContext;
Assert.Equal(typeof(List<string>), formatterContext.ObjectType);
var value = Assert.IsType<List<string>>(formatterContext.Object);
Assert.Equal(new[] { "Hello 0", "Hello 1", "Hello 2", "Hello 3", }, value);
}

[Fact]
public async Task ObjectResult_Throws_IfEnumerableThrows()
{
// Arrange
var executor = CreateExecutor();
var result = new ObjectResult(AsyncEnumerable(throwError: true));
var formatter = new TestJsonOutputFormatter();
result.Formatters.Add(formatter);

var actionContext = new ActionContext()
{
HttpContext = GetHttpContext(),
};

// Act & Assert
await Assert.ThrowsAsync<TimeZoneNotFoundException>(() => executor.ExecuteAsync(actionContext, result));
}

[Fact]
public async Task ObjectResult_AsyncEnumeration_AtLimit()
{
// Arrange
var count = 24;
var executor = CreateExecutor(options: new MvcOptions { MaxIAsyncEnumerableBufferLimit = count });
var result = new ObjectResult(AsyncEnumerable(count: count));
var formatter = new TestJsonOutputFormatter();
result.Formatters.Add(formatter);

var actionContext = new ActionContext()
{
HttpContext = GetHttpContext(),
};

// Act
await executor.ExecuteAsync(actionContext, result);

// Assert
var formatterContext = formatter.LastOutputFormatterContext;
var value = Assert.IsType<List<string>>(formatterContext.Object);
Assert.Equal(24, value.Count);
}

[Theory]
[InlineData(25)]
[InlineData(1024)]
public async Task ObjectResult_Throws_IfEnumerationExceedsLimit(int count)
{
// Arrange
var executor = CreateExecutor(options: new MvcOptions { MaxIAsyncEnumerableBufferLimit = 24 });
var result = new ObjectResult(AsyncEnumerable(count: count));
var formatter = new TestJsonOutputFormatter();
result.Formatters.Add(formatter);

var actionContext = new ActionContext()
{
HttpContext = GetHttpContext(),
};

// Act & Assert
var ex = await Assert.ThrowsAsync<InvalidOperationException>(() => executor.ExecuteAsync(actionContext, result));
}

private static async IAsyncEnumerable<string> AsyncEnumerable(int count = 4, bool throwError = false)
{
await Task.Yield();
for (var i = 0; i < count; i++)
{
yield return $"Hello {i}";
}

if (throwError)
{
throw new TimeZoneNotFoundException();
}
}

private static IServiceCollection CreateServices()
{
var services = new ServiceCollection();
Expand Down
Loading