Tracking Search API Calls

CHANGELOG.md

@@ -5,6 +5,38 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 
 ## [Unreleased 3.x]
 ### Added
+
+- Add search API tracker ([#18601](https://github.com/opensearch-project/OpenSearch/pull/18601))
+- Add support for linux riscv64 platform ([#18156](https://github.com/opensearch-project/OpenSearch/pull/18156))
+- [Rule based auto-tagging] Add get rule API ([#17336](https://github.com/opensearch-project/OpenSearch/pull/17336))
+- [Rule based auto-tagging] Add Delete Rule API ([#18184](https://github.com/opensearch-project/OpenSearch/pull/18184))
+- Implement parallel shard refresh behind cluster settings ([#17782](https://github.com/opensearch-project/OpenSearch/pull/17782))
+- Bump OpenSearch Core main branch to 3.0.0 ([#18039](https://github.com/opensearch-project/OpenSearch/pull/18039))
+- [Rule based Auto-tagging] Add wlm `ActionFilter` ([#17791](https://github.com/opensearch-project/OpenSearch/pull/17791))
+- Update API of Message in index to add the timestamp for lag calculation in ingestion polling ([#17977](https://github.com/opensearch-project/OpenSearch/pull/17977/))
+- Add Warm Disk Threshold Allocation Decider for Warm shards ([#18082](https://github.com/opensearch-project/OpenSearch/pull/18082))
+- Add composite directory factory ([#17988](https://github.com/opensearch-project/OpenSearch/pull/17988))
+- Add pull-based ingestion error metrics and make internal queue size configurable ([#18088](https://github.com/opensearch-project/OpenSearch/pull/18088))
+- Adding support for derive source feature and implementing it for various type of field mappers ([#17759](https://github.com/opensearch-project/OpenSearch/pull/17759))
+- [Security Manager Replacement] Enhance Java Agent to intercept newByteChannel ([#17989](https://github.com/opensearch-project/OpenSearch/pull/17989))
+- Enabled Async Shard Batch Fetch by default ([#18139](https://github.com/opensearch-project/OpenSearch/pull/18139))
+- Allow to get the search request from the QueryCoordinatorContext ([#17818](https://github.com/opensearch-project/OpenSearch/pull/17818))
+- Reject close index requests, while remote store migration is in progress.([#18327](https://github.com/opensearch-project/OpenSearch/pull/18327))
+- Improve sort-query performance by retaining the default `totalHitsThreshold` for approximated `match_all` queries ([#18189](https://github.com/opensearch-project/OpenSearch/pull/18189))
+- Enable testing for ExtensiblePlugins using classpath plugins ([#16908](https://github.com/opensearch-project/OpenSearch/pull/16908))
+- Introduce system generated ingest pipeline ([#17817](https://github.com/opensearch-project/OpenSearch/pull/17817)))
+- Apply cluster state metadata and routing table diff when building cluster state from remote([#18256](https://github.com/opensearch-project/OpenSearch/pull/18256))
+- Support create mode in pull-based ingestion and add retries for transient failures ([#18250](https://github.com/opensearch-project/OpenSearch/pull/18250)))
+- Decouple the init of Crypto Plugin and KeyProvider in CryptoRegistry ([18270](https://github.com/opensearch-project/OpenSearch/pull18270)))
+- Support cluster write block in pull-based ingestion ([#18280](https://github.com/opensearch-project/OpenSearch/pull/18280)))
+- Use QueryCoordinatorContext for the rewrite in validate API. ([#18272](https://github.com/opensearch-project/OpenSearch/pull/18272))
+- Upgrade crypto kms plugin dependencies for AWS SDK v2.x. ([#18268](https://github.com/opensearch-project/OpenSearch/pull/18268))
+- Add support for `matched_fields` with the unified highlighter ([#18164](https://github.com/opensearch-project/OpenSearch/issues/18164))
+- [repository-s3] Add support for SSE-KMS and S3 bucket owner verification ([#18312](https://github.com/opensearch-project/OpenSearch/pull/18312))
+- Optimize gRPC perf by passing by reference ([#18303](https://github.com/opensearch-project/OpenSearch/pull/18303))
+- Added File Cache Stats - Involves Block level as well as full file level stats ([#17538](https://github.com/opensearch-project/OpenSearch/issues/17479))
+- Added File Cache Pinning ([#17617](https://github.com/opensearch-project/OpenSearch/issues/13648))
+- Support consumer reset in Resume API for pull-based ingestion. This PR includes a breaking change for the experimental pull-based ingestion feature. ([#18332](https://github.com/opensearch-project/OpenSearch/pull/18332))
 - Add support for Warm Indices Write Block on Flood Watermark breach ([#18375](https://github.com/opensearch-project/OpenSearch/pull/18375))
 - Ability to run Code Coverage with Gradle and produce the jacoco reports locally ([#18509](https://github.com/opensearch-project/OpenSearch/issues/18509))
 - Add NodeResourceUsageStats to ClusterInfo ([#18480](https://github.com/opensearch-project/OpenSearch/issues/18472))

libs/core/src/main/java/org/opensearch/core/rest/RestStatus.java

@@ -536,6 +536,17 @@ public int getStatusFamilyCode() {
         return status / 100;
     }
 
+    public String getErrorType() {
+        int family = getStatusFamilyCode();
+        if (family <= 3) {
+            return "success";
+        } else if (family == 4) {
+            return "user_error";
+        } else {
+            return "system_failure";
+        }
+    }
+
     public static RestStatus readFrom(StreamInput in) throws IOException {
         return RestStatus.valueOf(in.readString());
     }

server/src/internalClusterTest/java/org/opensearch/nodestats/NodeStatsIT.java

@@ -13,6 +13,7 @@
 import org.opensearch.action.admin.cluster.node.stats.NodeStats;
 import org.opensearch.action.admin.cluster.node.stats.NodesStatsResponse;
 import org.opensearch.action.admin.indices.stats.CommonStatsFlags;
+import org.opensearch.action.admin.indices.stats.DocStatusStats;
 import org.opensearch.action.bulk.BulkItemResponse;
 import org.opensearch.action.bulk.BulkRequest;
 import org.opensearch.action.bulk.BulkResponse;
@@ -32,7 +33,6 @@
 import org.opensearch.index.IndexNotFoundException;
 import org.opensearch.index.engine.DocumentMissingException;
 import org.opensearch.index.engine.VersionConflictEngineException;
-import org.opensearch.index.shard.IndexingStats.Stats.DocStatusStats;
 import org.opensearch.indices.NodeIndicesStats;
 import org.opensearch.test.OpenSearchIntegTestCase;
 import org.opensearch.test.OpenSearchIntegTestCase.ClusterScope;
@@ -47,7 +47,7 @@
 import java.util.LinkedHashMap;
 import java.util.List;
 import java.util.Map;
-import java.util.concurrent.atomic.AtomicLong;
+import java.util.concurrent.atomic.LongAdder;
 
 import static java.util.Collections.singletonMap;
 import static org.opensearch.test.hamcrest.OpenSearchAssertions.assertAcked;
@@ -281,15 +281,14 @@ public void testNodeIndicesStatsDocStatsWithAggregations() {
                 .getNodes()
                 .get(0)
                 .getIndices()
-                .getIndexing()
-                .getTotal()
+                .getStatusCounterStats()
                 .getDocStatusStats();
 
             assertTrue(
                 Arrays.equals(
                     docStatusStats.getDocStatusCounter(),
                     expectedDocStatusStats.getDocStatusCounter(),
-                    Comparator.comparingLong(AtomicLong::longValue)
+                    Comparator.comparingLong(LongAdder::longValue)
                 )
             );
         }
@@ -543,15 +542,14 @@ private void assertDocStatusStats() {
             .getNodes()
             .get(0)
             .getIndices()
-            .getIndexing()
-            .getTotal()
+            .getStatusCounterStats()
             .getDocStatusStats();
 
         assertTrue(
             Arrays.equals(
                 docStatusStats.getDocStatusCounter(),
                 expectedDocStatusStats.getDocStatusCounter(),
-                Comparator.comparingLong(AtomicLong::longValue)
+                Comparator.comparingLong(LongAdder::longValue)
             )
         );
     }

server/src/main/java/org/opensearch/action/admin/indices/stats/DocStatusStats.java

@@ -0,0 +1,161 @@
+/*
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+/*
+ * Licensed to Elasticsearch under one or more contributor
+ * license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright
+ * ownership. Elasticsearch licenses this file to you under
+ * the Apache License, Version 2.0 (the "License"); you may
+ * not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+/*
+ * Modifications Copyright OpenSearch Contributors. See
+ * GitHub history for details.
+ */
+
+package org.opensearch.action.admin.indices.stats;
+
+import org.opensearch.common.annotation.PublicApi;
+import org.opensearch.core.common.io.stream.StreamInput;
+import org.opensearch.core.common.io.stream.StreamOutput;
+import org.opensearch.core.common.io.stream.Writeable;
+import org.opensearch.core.rest.RestStatus;
+import org.opensearch.core.xcontent.ToXContentFragment;
+import org.opensearch.core.xcontent.XContentBuilder;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.concurrent.atomic.LongAdder;
+
+/**
+ * Tracks item level rest category class codes during indexing
+ *
+ * @opensearch.api
+ */
+@PublicApi(since = "1.0.0")
+public class DocStatusStats implements Writeable, ToXContentFragment {
+    final LongAdder[] docStatusCounter;
+
+    public DocStatusStats() {
+        docStatusCounter = new LongAdder[5];
+        for (int i = 0; i < docStatusCounter.length; ++i) {
+            docStatusCounter[i] = new LongAdder();
+        }
+    }
+
+    public DocStatusStats(StreamInput in) throws IOException {
+        docStatusCounter = in.readArray(i -> {
+            LongAdder adder = new LongAdder();
+            adder.add(i.readLong());
+            return adder;
+
+        }, LongAdder[]::new);
+
+        assert docStatusCounter.length == 5 : "Length of incoming array should be 5! Got " + docStatusCounter.length;
+    }
+
+    /**
+     * Increment counter for status
+     *
+     * @param status {@link RestStatus}
+     */
+    public void inc(final RestStatus status) {
+        add(status, 1L);
+    }
+
+    /**
+     * Increment counter for status by count
+     *
+     * @param status {@link RestStatus}
+     * @param delta The value to add
+     */
+    void add(final RestStatus status, final long delta) {
+        docStatusCounter[status.getStatusFamilyCode() - 1].add(delta);
+    }
+
+    /**
+     * Accumulate stats from the passed Object
+     *
+     * @param stats Instance storing {@link DocStatusStats}
+     */
+    public void add(final DocStatusStats stats) {
+        if (null == stats) {
+            return;
+        }
+
+        for (int i = 0; i < docStatusCounter.length; ++i) {
+            docStatusCounter[i].add(stats.docStatusCounter[i].longValue());
+        }
+    }
+
+    public LongAdder[] getDocStatusCounter() {
+        return docStatusCounter;
+    }
+
+    @Override
+    public XContentBuilder toXContent(XContentBuilder builder, Params params) throws IOException {
+        builder.startObject(Fields.DOC_STATUS);
+
+        Map<String, Long> errorTypeCounts = new HashMap<>();
+
+        for (int i = 0; i < docStatusCounter.length; ++i) {
+            long count = docStatusCounter[i].longValue();
+
+            if (count > 0) {
+                RestStatus familyStatus = RestStatus.fromCode((i + 1) * 100);
+                String errorType = familyStatus.getErrorType();
+                errorTypeCounts.put(errorType, errorTypeCounts.getOrDefault(errorType, (long) 0) + count);
+            }
+        }
+
+        String successType = RestStatus.ACCEPTED.getErrorType();
+        String userFailureType = RestStatus.BAD_REQUEST.getErrorType();
+        String systemErrorType = RestStatus.INTERNAL_SERVER_ERROR.getErrorType();
+        builder.field(successType, errorTypeCounts.getOrDefault(successType, (long) 0));
+        builder.field(userFailureType, errorTypeCounts.getOrDefault(userFailureType, (long) 0));
+        builder.field(systemErrorType, errorTypeCounts.getOrDefault(systemErrorType, (long) 0));
+
+        return builder.endObject();
+    }
+
+    @Override
+    public void writeTo(StreamOutput out) throws IOException {
+        out.writeArray((o, v) -> o.writeLong(v.longValue()), docStatusCounter);
+    }
+
+    /**
+     * For this function, I just want to retrieve a point in time snapshot of the DocStatusStats.
+     */
+    public DocStatusStats getSnapshot() {
+        DocStatusStats curSnapshot = new DocStatusStats();
+        curSnapshot.add(this);
+        return curSnapshot;
+    }
+
+    /**
+     * Fields for parsing and toXContent
+     *
+     * @opensearch.internal
+     */
+    static final class Fields {
+        static final String DOC_STATUS = "doc_status";
+    }
+}