Secondary paging (#1100)

* JSON:API spec compliance: do not unescape brackets in response
From https://jsonapi.org/format/1.1/#appendix-query-details-square-brackets:
> According to the query parameter serialization rules above, a compliant implementation will percent-encode these square brackets.

* Updated existing IResourceDefinition tests to capture all relevant callbacks

* Retrieve total resource count on secondary/relationship endpoints using inverse relationship
Bugfix: links.next was not set on full page at relationship endpoint

* Rename flags enum to plural

* Clarified documentation; fixed broken link

* Check off roadmap entry

Author: Bart Koelman

ROADMAP.md

@@ -22,10 +22,10 @@ The need for breaking changes has blocked several efforts in the v4.x release, s
 - [x] Support System.Text.Json [#664](https://github.com/json-api-dotnet/JsonApiDotNetCore/issues/664) [#999](https://github.com/json-api-dotnet/JsonApiDotNetCore/issues/999) [1077](https://github.com/json-api-dotnet/JsonApiDotNetCore/issues/1077) [1078](https://github.com/json-api-dotnet/JsonApiDotNetCore/issues/1078)
 - [x] Optimize IIdentifiable to ResourceObject conversion [#1028](https://github.com/json-api-dotnet/JsonApiDotNetCore/issues/1028) [#1024](https://github.com/json-api-dotnet/JsonApiDotNetCore/issues/1024) [#233](https://github.com/json-api-dotnet/JsonApiDotNetCore/issues/233)
 - [x] Nullable reference types [#1029](https://github.com/json-api-dotnet/JsonApiDotNetCore/issues/1029)
+- [x] Improved paging links [#1010](https://github.com/json-api-dotnet/JsonApiDotNetCore/issues/1010)
 
 Aside from the list above, we have interest in the following topics. It's too soon yet to decide whether they'll make it into v5.x or in a later major version.
 
-- Improved paging links [#1010](https://github.com/json-api-dotnet/JsonApiDotNetCore/issues/1010)
 - Auto-generated controllers [#732](https://github.com/json-api-dotnet/JsonApiDotNetCore/issues/732) [#365](https://github.com/json-api-dotnet/JsonApiDotNetCore/issues/365)
 - Configuration validation [#170](https://github.com/json-api-dotnet/JsonApiDotNetCore/issues/170)
 - Optimistic concurrency [#1004](https://github.com/json-api-dotnet/JsonApiDotNetCore/issues/1004)

benchmarks/Serialization/ResourceSerializationBenchmarks.cs

@@ -129,7 +129,7 @@ protected override IEvaluatedIncludeCache CreateEvaluatedIncludeCache(IResourceG
             RelationshipAttribute multi4 = resourceAType.GetRelationshipByPropertyName(nameof(OutgoingResource.Multi4));
             RelationshipAttribute multi5 = resourceAType.GetRelationshipByPropertyName(nameof(OutgoingResource.Multi5));
 
-            ImmutableArray<ResourceFieldAttribute> chain = ArrayFactory.Create<ResourceFieldAttribute>(single2, single3, multi4, multi5).ToImmutableArray();
+            ImmutableArray<ResourceFieldAttribute> chain = ImmutableArray.Create<ResourceFieldAttribute>(single2, single3, multi4, multi5);
             IEnumerable<ResourceFieldChainExpression> chains = new ResourceFieldChainExpression(chain).AsEnumerable();
 
             var converter = new IncludeChainConverter();

docs/usage/options.md

@@ -39,6 +39,9 @@ options.MaximumPageNumber = new PageNumber(50);
 options.IncludeTotalResourceCount = true;
 ```
 
+To retrieve the total number of resources on secondary and relationship endpoints, the reverse of the relationship must to be available. For example, in `GET /customers/1/orders`, both the relationships `[HasMany] Customer.Orders` and `[HasOne] Order.Customer` must be defined.
+If `IncludeTotalResourceCount` is set to `false` (or the inverse relationship is unavailable on a non-primary endpoint), best-effort paging links are returned instead. This means no `last` link and the `next` link only occurs when the current page is full.
+
 ## Relative Links
 
 All links are absolute by default. However, you can configure relative links.

docs/usage/resources/nullability.md

@@ -2,7 +2,7 @@
 
 Properties on a resource class can be declared as nullable or non-nullable. This affects both ASP.NET ModelState validation and the way Entity Framework Core generates database columns.
 
-ModelState validation is enabled by default since v5.0. In earlier versions, it can be enabled in [options](~/usage/options.md#enable-modelstate-validation).
+ModelState validation is enabled by default since v5.0. In earlier versions, it can be enabled in [options](~/usage/options.md#modelstate-validation).
 
 # Value types
 

src/JsonApiDotNetCore/CollectionExtensions.cs

@@ -76,6 +76,13 @@ public static IEnumerable<T> EmptyIfNull<T>(this IEnumerable<T>? source)
             return source ?? Enumerable.Empty<T>();
         }
 
+        public static IEnumerable<T> WhereNotNull<T>(this IEnumerable<T?> source)
+        {
+#pragma warning disable AV1250 // Evaluate LINQ query before returning it
+            return source.Where(element => element is not null)!;
+#pragma warning restore AV1250 // Evaluate LINQ query before returning it
+        }
+
         public static void AddRange<T>(this ICollection<T> source, IEnumerable<T> itemsToAdd)
         {
             ArgumentGuard.NotNull(source, nameof(source));

src/JsonApiDotNetCore/Queries/Expressions/LogicalExpression.cs

@@ -34,6 +34,15 @@ public LogicalExpression(LogicalOperator @operator, IImmutableList<FilterExpress
             Terms = terms;
         }
 
+        public static FilterExpression? Compose(LogicalOperator @operator, params FilterExpression?[] filters)
+        {
+            ArgumentGuard.NotNull(filters, nameof(filters));
+
+            ImmutableArray<FilterExpression> terms = filters.WhereNotNull().ToImmutableArray();
+
+            return terms.Length > 1 ? new LogicalExpression(@operator, terms) : terms.FirstOrDefault();
+        }
+
         public override TResult Accept<TArgument, TResult>(QueryExpressionVisitor<TArgument, TResult> visitor, TArgument argument)
         {
             return visitor.VisitLogical(this, argument);

src/JsonApiDotNetCore/Queries/IQueryLayerComposer.cs

@@ -12,9 +12,14 @@ namespace JsonApiDotNetCore.Queries
     public interface IQueryLayerComposer
     {
         /// <summary>
-        /// Builds a top-level filter from constraints, used to determine total resource count.
+        /// Builds a filter from constraints, used to determine total resource count on a primary collection endpoint.
         /// </summary>
-        FilterExpression? GetTopFilterFromConstraints(ResourceType primaryResourceType);
+        FilterExpression? GetPrimaryFilterFromConstraints(ResourceType primaryResourceType);
+
+        /// <summary>
+        /// Builds a filter from constraints, used to determine total resource count on a secondary collection endpoint.
+        /// </summary>
+        FilterExpression? GetSecondaryFilterFromConstraints<TId>(TId primaryId, HasManyAttribute hasManyRelationship);
 
         /// <summary>
         /// Collects constraints and builds a <see cref="QueryLayer" /> out of them, used to retrieve the actual resources.

src/JsonApiDotNetCore/Queries/Internal/QueryLayerComposer.cs

@@ -46,7 +46,7 @@ public QueryLayerComposer(IEnumerable<IQueryConstraintProvider> constraintProvid
         }
 
         /// <inheritdoc />
-        public FilterExpression? GetTopFilterFromConstraints(ResourceType primaryResourceType)
+        public FilterExpression? GetPrimaryFilterFromConstraints(ResourceType primaryResourceType)
         {
             ExpressionInScope[] constraints = _constraintProviders.SelectMany(provider => provider.GetConstraints()).ToArray();
 
@@ -65,6 +65,75 @@ public QueryLayerComposer(IEnumerable<IQueryConstraintProvider> constraintProvid
             return GetFilter(filtersInTopScope, primaryResourceType);
         }
 
+        /// <inheritdoc />
+        public FilterExpression? GetSecondaryFilterFromConstraints<TId>(TId primaryId, HasManyAttribute hasManyRelationship)
+        {
+            ArgumentGuard.NotNull(hasManyRelationship, nameof(hasManyRelationship));
+
+            if (hasManyRelationship.InverseNavigationProperty == null)
+            {
+                return null;
+            }
+
+            RelationshipAttribute? inverseRelationship =
+                hasManyRelationship.RightType.FindRelationshipByPropertyName(hasManyRelationship.InverseNavigationProperty.Name);
+
+            if (inverseRelationship == null)
+            {
+                return null;
+            }
+
+            ExpressionInScope[] constraints = _constraintProviders.SelectMany(provider => provider.GetConstraints()).ToArray();
+
+            var secondaryScope = new ResourceFieldChainExpression(hasManyRelationship);
+
+            // @formatter:wrap_chained_method_calls chop_always
+            // @formatter:keep_existing_linebreaks true
+
+            FilterExpression[] filtersInSecondaryScope = constraints
+                .Where(constraint => secondaryScope.Equals(constraint.Scope))
+                .Select(constraint => constraint.Expression)
+                .OfType<FilterExpression>()
+                .ToArray();
+
+            // @formatter:keep_existing_linebreaks restore
+            // @formatter:wrap_chained_method_calls restore
+
+            FilterExpression? primaryFilter = GetFilter(Array.Empty<QueryExpression>(), hasManyRelationship.LeftType);
+            FilterExpression? secondaryFilter = GetFilter(filtersInSecondaryScope, hasManyRelationship.RightType);
+
+            FilterExpression inverseFilter = GetInverseRelationshipFilter(primaryId, hasManyRelationship, inverseRelationship);
+
+            return LogicalExpression.Compose(LogicalOperator.And, inverseFilter, primaryFilter, secondaryFilter);
+        }
+
+        private static FilterExpression GetInverseRelationshipFilter<TId>(TId primaryId, HasManyAttribute relationship,
+            RelationshipAttribute inverseRelationship)
+        {
+            return inverseRelationship is HasManyAttribute hasManyInverseRelationship
+                ? GetInverseHasManyRelationshipFilter(primaryId, relationship, hasManyInverseRelationship)
+                : GetInverseHasOneRelationshipFilter(primaryId, relationship, (HasOneAttribute)inverseRelationship);
+        }
+
+        private static FilterExpression GetInverseHasOneRelationshipFilter<TId>(TId primaryId, HasManyAttribute relationship,
+            HasOneAttribute inverseRelationship)
+        {
+            AttrAttribute idAttribute = GetIdAttribute(relationship.LeftType);
+            var idChain = new ResourceFieldChainExpression(ImmutableArray.Create<ResourceFieldAttribute>(inverseRelationship, idAttribute));
+
+            return new ComparisonExpression(ComparisonOperator.Equals, idChain, new LiteralConstantExpression(primaryId!.ToString()!));
+        }
+
+        private static FilterExpression GetInverseHasManyRelationshipFilter<TId>(TId primaryId, HasManyAttribute relationship,
+            HasManyAttribute inverseRelationship)
+        {
+            AttrAttribute idAttribute = GetIdAttribute(relationship.LeftType);
+            var idChain = new ResourceFieldChainExpression(ImmutableArray.Create<ResourceFieldAttribute>(idAttribute));
+            var idComparison = new ComparisonExpression(ComparisonOperator.Equals, idChain, new LiteralConstantExpression(primaryId!.ToString()!));
+
+            return new HasExpression(new ResourceFieldChainExpression(inverseRelationship), idComparison);
+        }
+
         /// <inheritdoc />
         public QueryLayer ComposeFromConstraints(ResourceType requestResourceType)
         {
@@ -309,7 +378,7 @@ private IncludeExpression RewriteIncludeForSecondaryEndpoint(IncludeExpression?
                 filter = new AnyExpression(idChain, constants);
             }
 
-            return filter == null ? existingFilter : existingFilter == null ? filter : new LogicalExpression(LogicalOperator.And, filter, existingFilter);
+            return LogicalExpression.Compose(LogicalOperator.And, filter, existingFilter);
         }
 
         /// <inheritdoc />
@@ -419,8 +488,8 @@ protected virtual IImmutableSet<IncludeElementExpression> GetIncludeElements(IIm
             ArgumentGuard.NotNull(expressionsInScope, nameof(expressionsInScope));
             ArgumentGuard.NotNull(resourceType, nameof(resourceType));
 
-            ImmutableArray<FilterExpression> filters = expressionsInScope.OfType<FilterExpression>().ToImmutableArray();
-            FilterExpression? filter = filters.Length > 1 ? new LogicalExpression(LogicalOperator.And, filters) : filters.FirstOrDefault();
+            FilterExpression[] filters = expressionsInScope.OfType<FilterExpression>().ToArray();
+            FilterExpression? filter = LogicalExpression.Compose(LogicalOperator.And, filters);
 
             return _resourceDefinitionAccessor.OnApplyFilter(resourceType, filter);
         }

src/JsonApiDotNetCore/Repositories/EntityFrameworkCoreRepository.cs

@@ -85,11 +85,11 @@ public virtual async Task<IReadOnlyCollection<TResource>> GetAsync(QueryLayer qu
         }
 
         /// <inheritdoc />
-        public virtual async Task<int> CountAsync(FilterExpression? topFilter, CancellationToken cancellationToken)
+        public virtual async Task<int> CountAsync(FilterExpression? filter, CancellationToken cancellationToken)
         {
             _traceWriter.LogMethodStart(new
             {
-                topFilter
+                filter
             });
 
             using (CodeTimingSessionManager.Current.Measure("Repository - Count resources"))
@@ -98,7 +98,7 @@ public virtual async Task<int> CountAsync(FilterExpression? topFilter, Cancellat
 
                 var layer = new QueryLayer(resourceType)
                 {
-                    Filter = topFilter
+                    Filter = filter
                 };
 
                 IQueryable<TResource> query = ApplyQueryLayer(layer);

src/JsonApiDotNetCore/Repositories/IResourceReadRepository.cs

@@ -27,8 +27,8 @@ public interface IResourceReadRepository<TResource, in TId>
         Task<IReadOnlyCollection<TResource>> GetAsync(QueryLayer queryLayer, CancellationToken cancellationToken);
 
         /// <summary>
-        /// Executes a read query using the specified top-level filter and returns the top-level count of matching resources.
+        /// Executes a read query using the specified filter and returns the count of matching resources.
         /// </summary>
-        Task<int> CountAsync(FilterExpression? topFilter, CancellationToken cancellationToken);
+        Task<int> CountAsync(FilterExpression? filter, CancellationToken cancellationToken);
     }
 }

src/JsonApiDotNetCore/Repositories/IResourceRepositoryAccessor.cs

@@ -27,8 +27,7 @@ Task<IReadOnlyCollection<TResource>> GetAsync<TResource>(QueryLayer queryLayer,
         /// <summary>
         /// Invokes <see cref="IResourceReadRepository{TResource,TId}.CountAsync" /> for the specified resource type.
         /// </summary>
-        Task<int> CountAsync<TResource>(FilterExpression? topFilter, CancellationToken cancellationToken)
-            where TResource : class, IIdentifiable;
+        Task<int> CountAsync(ResourceType resourceType, FilterExpression? filter, CancellationToken cancellationToken);
 
         /// <summary>
         /// Invokes <see cref="IResourceWriteRepository{TResource,TId}.GetForCreateAsync" />.

src/JsonApiDotNetCore/Repositories/ResourceRepositoryAccessor.cs

@@ -50,11 +50,10 @@ public async Task<IReadOnlyCollection<IIdentifiable>> GetAsync(ResourceType reso
         }
 
         /// <inheritdoc />
-        public async Task<int> CountAsync<TResource>(FilterExpression? topFilter, CancellationToken cancellationToken)
-            where TResource : class, IIdentifiable
+        public async Task<int> CountAsync(ResourceType resourceType, FilterExpression? filter, CancellationToken cancellationToken)
         {
-            dynamic repository = ResolveReadRepository(typeof(TResource));
-            return (int)await repository.CountAsync(topFilter, cancellationToken);
+            dynamic repository = ResolveReadRepository(resourceType);
+            return (int)await repository.CountAsync(filter, cancellationToken);
         }
 
         /// <inheritdoc />

src/JsonApiDotNetCore/Serialization/Response/LinkBuilder.cs

@@ -114,6 +114,7 @@ private bool ShouldIncludeTopLevelLink(LinkTypes linkType, ResourceType? resourc
 
         private string GetLinkForTopLevelSelf()
         {
+            // Note: in tests, this does not properly escape special characters due to WebApplicationFactory short-circuiting.
             return _options.UseRelativeLinks ? HttpContext.Request.GetEncodedPathAndQuery() : HttpContext.Request.GetEncodedUrl();
         }
 
@@ -223,13 +224,7 @@ private string GetQueryStringInPaginationLink(int pageOffset, string? pageSizeVa
                 parameters[PageNumberParameterName] = pageOffset.ToString();
             }
 
-            string queryStringValue = QueryString.Create(parameters).Value ?? string.Empty;
-            return DecodeSpecialCharacters(queryStringValue);
-        }
-
-        private static string DecodeSpecialCharacters(string uri)
-        {
-            return uri.Replace("%5B", "[").Replace("%5D", "]").Replace("%27", "'").Replace("%3A", ":");
+            return QueryString.Create(parameters).Value ?? string.Empty;
         }
 
         /// <inheritdoc />