diff --git a/deploy-manage/deploy/cloud-enterprise/customize-deployment.md b/deploy-manage/deploy/cloud-enterprise/customize-deployment.md index 0d8e2fd3a..9be26ecc7 100644 --- a/deploy-manage/deploy/cloud-enterprise/customize-deployment.md +++ b/deploy-manage/deploy/cloud-enterprise/customize-deployment.md @@ -79,10 +79,10 @@ When you select **Save changes** on the **Edit deployment** page, the orchestrat * **Grow and shrink**: The orchestrator creates new instances with the new configuration, then migrates the data, and eventually deletes the original ones. This strategy is automatically selected when adding or removing master-eligible instances. * **Rolling grow and shrink**: Similar to grow and shrink, but creating one instance at a time. This strategy can take a lot longer than grow and shrink. -The `Extended maintenance` optional flag will make ECE to [stop routing requests](../../maintenance/ece/start-stop-routing-requests.md) to all instances during the plan execution. The cluster will be unavailable for external connections while the configuration changes are in progress. +The `Extended maintenance` optional flag will make ECE to [stop routing requests](../../maintenance/start-stop-routing-requests.md) to all instances during the plan execution. The cluster will be unavailable for external connections while the configuration changes are in progress. ::::{note} -If you enable the **Extended maintenance** optional flag, ECE will [stop routing requests](../../maintenance/ece/start-stop-routing-requests.md) to all instances during the plan execution, making the cluster unavailable for external connections while configuration changes are in progress. +If you enable the **Extended maintenance** optional flag, ECE will [stop routing requests](../../maintenance/start-stop-routing-requests.md) to all instances during the plan execution, making the cluster unavailable for external connections while configuration changes are in progress. This option introduces downtime and is rarely needed. Use it only when you need to block all traffic to the cluster during the update. :::: diff --git a/deploy-manage/deploy/elastic-cloud/manage-deployments.md b/deploy-manage/deploy/elastic-cloud/manage-deployments.md index 841880490..1a4e9f69d 100644 --- a/deploy-manage/deploy/elastic-cloud/manage-deployments.md +++ b/deploy-manage/deploy/elastic-cloud/manage-deployments.md @@ -18,7 +18,7 @@ mapped_pages: * Ensure the health of your deployment over time * [Keep track of your deployment's activity](keep-track-of-deployment-activity.md) or [Enable logging and monitoring](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) of the deployment performance. - * Perform maintenance operations to ensure the health of your deployment, such as [restarting your deployment](../../maintenance/start-stop-services/restart-cloud-hosted-deployment.md) or [stopping routing](../../maintenance/ece/start-stop-routing-requests.md). + * Perform maintenance operations to ensure the health of your deployment, such as [restarting your deployment](../../maintenance/start-stop-services/restart-cloud-hosted-deployment.md) or [stopping routing](../../maintenance/start-stop-routing-requests.md). * Manage the lifecycle of your deployment: diff --git a/deploy-manage/distributed-architecture.md b/deploy-manage/distributed-architecture.md index e96e95dff..112109bdc 100644 --- a/deploy-manage/distributed-architecture.md +++ b/deploy-manage/distributed-architecture.md @@ -22,5 +22,5 @@ The topics in this section provides information about the architecture of {{es}} * [Shard allocation, relocation, and recovery](distributed-architecture/shard-allocation-relocation-recovery.md): Learn how {{es}} allocates and balances shards across nodes. * [Shard allocation awareness](distributed-architecture/shard-allocation-relocation-recovery/shard-allocation-awareness.md): Learn how to use custom node attributes to distribute shards across different racks or availability zones. * [Disocvery and cluster formation](distributed-architecture/discovery-cluster-formation.md): Learn about the cluster formation process including voting, adding nodes and publishing the cluster state. -* [Shard request cache](elasticsearch://reference/elasticsearch/configuration-reference/shard-request-cache-settings.md): Learn how {{es}} caches search requests to improve performance. +* [Shard request cache](/deploy-manage/distributed-architecture/shard-request-cache.md): Learn how {{es}} caches search requests to improve performance. * [Kibana task management](distributed-architecture/kibana-tasks-management.md): Learn how {{kib}} runs background tasks and distribute work across multiple {{kib}} instances to be persistent and scale with your deployment. \ No newline at end of file diff --git a/raw-migrated-files/elasticsearch/elasticsearch-reference/shard-request-cache.md b/deploy-manage/distributed-architecture/shard-request-cache.md similarity index 91% rename from raw-migrated-files/elasticsearch/elasticsearch-reference/shard-request-cache.md rename to deploy-manage/distributed-architecture/shard-request-cache.md index e80002fb2..d513a4609 100644 --- a/raw-migrated-files/elasticsearch/elasticsearch-reference/shard-request-cache.md +++ b/deploy-manage/distributed-architecture/shard-request-cache.md @@ -1,3 +1,14 @@ +--- +mapped_pages: +- https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-request-cache.html +applies_to: + deployment: + ece: + ess: + eck: + self: +--- + # The shard request cache [shard-request-cache] When a search request is run against an index or against many indices, each involved shard executes the search locally and returns its local results to the *coordinating node*, which combines these shard-level results into a global result set. @@ -7,7 +18,7 @@ The shard-level request cache module caches the local results on each shard. Thi You can control the size and expiration of the cache at the node level using the [shard request cache settings](elasticsearch://reference/elasticsearch/configuration-reference/shard-request-cache-settings.md). ::::{important} -By default, the requests cache will only cache the results of search requests where `size=0`, so it will not cache `hits`, but it will cache `hits.total`, [aggregations](../../../explore-analyze/query-filter/aggregations.md), and [suggestions](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html). +By default, the requests cache will only cache the results of search requests where `size=0`, so it will not cache `hits`, but it will cache `hits.total`, [aggregations](/explore-analyze/query-filter/aggregations.md), and [suggestions](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html). Most queries that use `now` (see [Date Math](elasticsearch://reference/elasticsearch/rest-apis/common-options.md#date-math)) cannot be cached. @@ -15,8 +26,6 @@ Scripted queries that use the API calls which are non-deterministic, such as `Ma :::: - - ## Cache invalidation [_cache_invalidation] The cache is smart — it keeps the same *near real-time* promise as uncached search. @@ -82,8 +91,6 @@ A hash of the whole JSON body is used as the cache key. This means that if the J Most JSON libraries support a *canonical* mode which ensures that JSON keys are always emitted in the same order. This canonical mode can be used in the application to ensure that a request is always serialized in the same way. :::: - - ## Monitoring cache usage [_monitoring_cache_usage] The size of the cache (in bytes) and the number of evictions can be viewed by index, with the [`indices-stats`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-stats) API: diff --git a/deploy-manage/maintenance.md b/deploy-manage/maintenance.md index 7236e6f63..aff0ce5cf 100644 --- a/deploy-manage/maintenance.md +++ b/deploy-manage/maintenance.md @@ -14,6 +14,7 @@ This section outlines the key tasks and processes required to maintain a healthy The topics covered include: -* **[ECE Maintenance](maintenance/ece.md)**: Explains the procedures for maintaining both the host infrastructure and {{es}} deployments within Elastic Cloud Enterprise (ECE). -* **[Start and Stop services](maintenance/start-stop-services.md)**: Provides step-by-step instructions on how to safely start and stop your {{es}} deployment or {{kib}} instance, particularly when performing actions that require a restart. -* **[Add and remove {{es}} nodes](maintenance/add-and-remove-elasticsearch-nodes.md)**: Guides you through the process of enrolling new nodes or safely removing existing ones from a self-managed {{es}} cluster to optimize resource utilization and cluster performance. +* **[](maintenance/ece.md)**: Explains the procedures for maintaining both the host infrastructure and {{es}} deployments within Elastic Cloud Enterprise (ECE). +* **[](maintenance/start-stop-services.md)**: Provides step-by-step instructions on how to safely start and stop your {{es}} deployment or {{kib}} instance, particularly when performing actions that require a restart. +* **[](maintenance/start-stop-routing-requests.md)**: Start or stop routing requests to an {{ech}} or {{ece}} deployment, or to all instances on an allocator in {{ece}}. +* **[](maintenance/add-and-remove-elasticsearch-nodes.md)**: Guides you through the process of enrolling new nodes or safely removing existing ones from a self-managed {{es}} cluster to optimize resource utilization and cluster performance. diff --git a/deploy-manage/maintenance/ece.md b/deploy-manage/maintenance/ece.md index 5ec8df745..56deb2ccb 100644 --- a/deploy-manage/maintenance/ece.md +++ b/deploy-manage/maintenance/ece.md @@ -10,7 +10,7 @@ Elastic Cloud Enterprise (ECE), being a self-managed Elastic Stack deployment pl ## Deployment maintenance and host infrastructure maintenance [ece-deployment-host-infra-maintenance] -[Deployment maintenance](ece/deployments-maintenance.md) focuses on managing individual {{es}} and {{kib}} instances within ECE. This includes actions such as [pausing instances](ece/pause-instance.md), [stopping request routing to nodes](ece/start-stop-routing-requests.md), and [moving instances between allocators](ece/move-nodes-instances-from-allocators.md) to optimize resource usage or prepare for maintenance. These tasks help maintain service availability and performance without affecting the underlying infrastructure. +[Deployment maintenance](ece/deployments-maintenance.md) focuses on managing individual {{es}} and {{kib}} instances within ECE. This includes actions such as [pausing instances](ece/pause-instance.md), [stopping request routing to nodes](/deploy-manage/maintenance/start-stop-routing-requests.md), and [moving instances between allocators](ece/move-nodes-instances-from-allocators.md) to optimize resource usage or prepare for maintenance. These tasks help maintain service availability and performance without affecting the underlying infrastructure. [ECE host infrastructure maintenance](ece/perform-ece-hosts-maintenance.md) involves managing virtual machines that host ECE itself. This includes tasks like applying operating system patches, upgrading software, or decommissioning hosts. Infrastructure maintenance often requires more careful planning, as it can impact multiple deployments running on the affected hosts. Methods such as placing allocators into [maintenance mode](ece/enable-maintenance-mode.md) and redistributing workloads provide a smooth transition during maintenance operations. diff --git a/deploy-manage/maintenance/ece/deployments-maintenance.md b/deploy-manage/maintenance/ece/deployments-maintenance.md index ab61c7311..57cf42cbd 100644 --- a/deploy-manage/maintenance/ece/deployments-maintenance.md +++ b/deploy-manage/maintenance/ece/deployments-maintenance.md @@ -12,7 +12,7 @@ In some circumstances, you might need to temporarily restrict access to a node s These actions act as a maintenance mode for cluster node. Performing these actions can stop the cluster from becoming unresponsive so that you can resolve operational issues much more effectively. -* [**Stop routing to the instance**](start-stop-routing-requests.md): Block requests from being routed to the cluster node. This is a less invasive action than pausing the cluster. +* [**Stop routing to the instance**](/deploy-manage/maintenance/start-stop-routing-requests.md): Block requests from being routed to the cluster node. This is a less invasive action than pausing the cluster. * [**Pause an instance**](pause-instance.md): Suspend the node immediately by stopping the container that the node runs on without completing existing requests. This is a more aggressive action to regain control of an unresponsive node. As an alternative, to quickly add capacity to a deployment if it is unhealthy or at capacity, you can also [override the resource limit for a deployment](../../deploy/cloud-enterprise/resource-overrides.md). diff --git a/deploy-manage/maintenance/ece/enable-maintenance-mode.md b/deploy-manage/maintenance/ece/enable-maintenance-mode.md index 3f57ab6d0..2595b6a18 100644 --- a/deploy-manage/maintenance/ece/enable-maintenance-mode.md +++ b/deploy-manage/maintenance/ece/enable-maintenance-mode.md @@ -22,6 +22,6 @@ After the allocator enters maintenance mode, no new Elasticsearch nodes or Kiban If you want to make the allocator fully active again, select **Disable Maintenance Mode**. Confirm the action. ::::{tip} -If you need the existing instances to stop routing requests, refer to the [stop routing request documentation](start-stop-routing-requests.md) to learn more. +If you need the existing instances to stop routing requests, refer to the [stop routing request documentation](../start-stop-routing-requests.md) to learn more. :::: diff --git a/deploy-manage/maintenance/ece/start-stop-routing-requests.md b/deploy-manage/maintenance/start-stop-routing-requests.md similarity index 55% rename from deploy-manage/maintenance/ece/start-stop-routing-requests.md rename to deploy-manage/maintenance/start-stop-routing-requests.md index 9130380f3..6df9fa377 100644 --- a/deploy-manage/maintenance/ece/start-stop-routing-requests.md +++ b/deploy-manage/maintenance/start-stop-routing-requests.md @@ -5,25 +5,26 @@ mapped_urls: applies_to: deployment: ece: + ess: --- # Start and stop routing requests [maintenance-mode-routing] -The {{ecloud}} proxy service routes traffic from external sources to the deployment, between deployments, and between products within a deployment. For example, it routes API requests from your local machine to your deployment, CCR and CCS requests between your deployments, and communications between {{kib}} and {{es}}. It does not direct the TCP traffic between {{es}} nodes, nor does it manage requests starting within {{es}} outwards to external sources such as to snapshot repositories. +The cloud proxy service routes traffic from external sources to the deployment, between deployments, and between products within a deployment. For example, it routes API requests from your local machine to your deployment, CCR and CCS requests between your deployments, and communications between {{kib}} and {{es}}. It does not direct the TCP traffic between {{es}} nodes, nor does it manage requests starting within {{es}} outwards to external sources such as to snapshot repositories. -The {{ecloud}} proxy routes HTTP requests to its deployment’s individual product instances through the product’s endpoint. By default, instances are enabled to route HTTP traffic and will report no special messaging. +The cloud proxy routes HTTP requests to its deployment’s individual product instances through the product’s endpoint. By default, instances are enabled to route HTTP traffic and will report no special messaging. It might be helpful to temporarily block upstream requests in order to protect some or all instances or products within your deployment. For example, you might stop request routing in the following cases: * If another team within your company starts streaming new data into your production {{integrations-server}} without previous load testing, both it and {{es}} might experience performance issues. You might consider stopping routing requests on all {{integrations-server}} instances in order to protect your downstream {{es}} instance. -* If {{es}} is being overwhelmed by upstream requests, it might experience increased response times or even become unresponsive. This might impact your ability to resize components in your deployment and increase the duration of pending plans or increase the chance of plan changes failing. Because every {{es}} node is an [implicit coordinating node](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md), you should stop routing requests across all {{es}} nodes to completely block upstream traffic. +* If {{es}} is being overwhelmed by upstream requests, it might experience increased response times or even become unresponsive. This might impact your ability to resize components in your deployment and increase the duration of pending plans or increase the chance of plan changes failing. Because every {{es}} node is an [implicit coordinating node](/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#node-roles-list), you should stop routing requests across all {{es}} nodes to completely block upstream traffic. ## Considerations [request-routing-considerations] -* {{ecloud}} will automatically set and remove routing blocks during plan changes. Elastic recommends avoiding manually overriding these settings for a deployment while its plans are pending. -* The [{{es}} API console](../../../explore-analyze/query-filter/tools/console.md) bypasses {{ecloud}} proxy routing blocks against {{es}} to enable administrative tasks while plan changes are pending. You should generally default traffic to the {{es}} endpoint. However, if you enable **Stop routing requests** across all {{es}} nodes, you need to use this UI to administer your cluster. -* While {{es}} has **Stop routing requests** set across all nodes, other products with the deployment may become unhealthy. This is because {{es}} is a prerequisite for those other products, such as {{kib}}. In {{kib}}, this results in a [**Kibana server is not ready yet**](../../../troubleshoot/kibana/error-server-not-ready.md) message. -* Enabling **Stop routing requests** does not affect your [billing](../../../deploy-manage/cloud-organization/billing.md). If needed, you can stop charges for a deployment by [deleting the deployment](../../../deploy-manage/uninstall/delete-a-cloud-deployment.md). +* ECE and ECH will automatically set and remove routing blocks during plan changes. Elastic recommends avoiding manually overriding these settings for a deployment while its plans are pending. +* The [{{es}} API console](/explore-analyze/query-filter/tools/console.md) bypasses cloud proxy routing blocks against {{es}} to enable administrative tasks while plan changes are pending. You should generally default traffic to the {{es}} endpoint. However, if you enable **Stop routing requests** across all {{es}} nodes, you need to use this UI to administer your cluster. +* While {{es}} has **Stop routing requests** set across all nodes, other products with the deployment may become unhealthy. This is because {{es}} is a prerequisite for those other products, such as {{kib}}. In {{kib}}, this results in a [**Kibana server is not ready yet**](/troubleshoot/kibana/error-server-not-ready.md) message. +* In {{ech}}, enabling **Stop routing requests** does not affect your [billing](/deploy-manage/cloud-organization/billing.md). If needed, you can stop charges for a deployment by [deleting the deployment](/deploy-manage/uninstall/delete-a-cloud-deployment.md). ## Stop routing requests [stop-routing-requests] @@ -31,7 +32,16 @@ To block HTTP requests for an instance, select **Stop routing requests** under f The instance will then report **Not routing requests**. It will complete existing requested traffic, but not be sent new requests. -You can stop routing requests to disable incoming requests to particular instances. You can also massively disable all allocator instances routing with the [allocator-toggle-routing-requests.sh](https://download.elastic.co/cloud/allocator-toggle-routing-requests.sh) script. The script runs with the following parameters in the form environment variables: +## Restart routing requests [restart-routing-requests] + +To unblock HTTP requests for an instance, select **Start routing requests** under from instance’s menu. + +## Toggle routing to all instances on an allocator +```{applies_to} +deployment: + ece: +``` +In {{ece}}, in addition to stopping routing requests for particular instances, you can also massively disable routing to all instances on a specified allocator with the [allocator-toggle-routing-requests.sh](https://download.elastic.co/cloud/allocator-toggle-routing-requests.sh) script. The script runs with the following parameters in the form environment variables: * `API_URL` Url of the administration API. * `AUTH_HEADER` Curl format string representing the authentication header. @@ -49,7 +59,3 @@ The same script can be used to enable traffic again: ```bash AUTH_HEADER="Authorization: ApiKey $(cat ~/api.key)" API_URL="https://adminconsole:12443" ALLOCATOR_ID="192.168.44.10" ENABLE_TRAFFIC=true ./allocator-toggle-routing-requests.sh ``` - -## Restart routing requests [restart-routing-requests] - -To unblock HTTP requests for an instance, select **Start routing requests** under from instance’s menu. diff --git a/deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md b/deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md index 2308a5817..aab632556 100644 --- a/deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md +++ b/deploy-manage/monitor/access-performance-metrics-on-elastic-cloud.md @@ -104,7 +104,7 @@ It is not uncommon for performance issues on {{ech}} to be caused by an undersiz To help diagnose high CPU usage you can also use the {{es}} [nodes hot threads API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-hot-threads), which identifies the threads on each node that have the highest CPU usage or that have been executing for a longer than normal period of time. ::::{tip} -Got an overwhelmed cluster that needs to be upsized? [Try enabling maintenance mode first](/deploy-manage/maintenance/ece/start-stop-routing-requests.md). It will likely help with configuration changes. +Got an overwhelmed cluster that needs to be upsized? [Try enabling maintenance mode first](/deploy-manage/maintenance/start-stop-routing-requests.md). It will likely help with configuration changes. :::: diff --git a/deploy-manage/production-guidance/optimize-performance/search-speed.md b/deploy-manage/production-guidance/optimize-performance/search-speed.md index d68f9ddd1..cb9319f08 100644 --- a/deploy-manage/production-guidance/optimize-performance/search-speed.md +++ b/deploy-manage/production-guidance/optimize-performance/search-speed.md @@ -308,7 +308,7 @@ Loading data into the filesystem cache eagerly on too many indices or too many f ## Use `preference` to optimize cache utilization [preference-cache-optimization] -There are multiple caches that can help with search performance, such as the [filesystem cache](https://en.wikipedia.org/wiki/Page_cache), the [request cache](elasticsearch://reference/elasticsearch/configuration-reference/shard-request-cache-settings.md) or the [query cache](elasticsearch://reference/elasticsearch/configuration-reference/node-query-cache-settings.md). Yet all these caches are maintained at the node level, meaning that if you run the same request twice in a row, have 1 replica or more and use [round-robin](https://en.wikipedia.org/wiki/Round-robin_DNS), the default routing algorithm, then those two requests will go to different shard copies, preventing node-level caches from helping. +There are multiple caches that can help with search performance, such as the [filesystem cache](https://en.wikipedia.org/wiki/Page_cache), the [request cache](/deploy-manage/distributed-architecture/shard-request-cache.md) or the [query cache](elasticsearch://reference/elasticsearch/configuration-reference/node-query-cache-settings.md). Yet all these caches are maintained at the node level, meaning that if you run the same request twice in a row, have 1 replica or more and use [round-robin](https://en.wikipedia.org/wiki/Round-robin_DNS), the default routing algorithm, then those two requests will go to different shard copies, preventing node-level caches from helping. Since it is common for users of a search application to run similar requests one after another, for instance in order to analyze a narrower subset of the index, using a preference value that identifies the current user or session could help optimize usage of the caches. diff --git a/deploy-manage/toc.yml b/deploy-manage/toc.yml index 0c5489f36..5caf0fcc8 100644 --- a/deploy-manage/toc.yml +++ b/deploy-manage/toc.yml @@ -318,6 +318,7 @@ toc: - file: distributed-architecture/shard-allocation-relocation-recovery/index-level-shard-allocation.md children: - file: distributed-architecture/shard-allocation-relocation-recovery/delaying-allocation-when-node-leaves.md + - file: distributed-architecture/shard-request-cache.md - file: distributed-architecture/discovery-cluster-formation.md children: - file: distributed-architecture/discovery-cluster-formation/discovery-hosts-providers.md @@ -742,7 +743,6 @@ toc: children: - file: maintenance/ece/deployments-maintenance.md children: - - file: maintenance/ece/start-stop-routing-requests.md - file: maintenance/ece/pause-instance.md - file: maintenance/ece/maintenance-activities.md children: @@ -758,6 +758,7 @@ toc: - file: maintenance/start-stop-services/restart-cloud-hosted-deployment.md - file: maintenance/start-stop-services/restart-an-ece-deployment.md - file: maintenance/start-stop-services/full-cluster-restart-rolling-restart-procedures.md + - file: maintenance/start-stop-routing-requests.md - file: maintenance/add-and-remove-elasticsearch-nodes.md - file: upgrade.md children: diff --git a/explore-analyze/query-filter/aggregations.md b/explore-analyze/query-filter/aggregations.md index 4789c0b0c..9fa5985b7 100644 --- a/explore-analyze/query-filter/aggregations.md +++ b/explore-analyze/query-filter/aggregations.md @@ -288,7 +288,7 @@ Scripts calculate field values dynamically, which adds a little overhead to the ## Aggregation caches [agg-caches] -For faster responses, {{es}} caches the results of frequently run aggregations in the [shard request cache](elasticsearch://reference/elasticsearch/configuration-reference/shard-request-cache-settings.md). To get cached results, use the same [`preference` string](elasticsearch://reference/elasticsearch/rest-apis/search-shard-routing.md#shard-and-node-preference) for each search. If you don’t need search hits, [set `size` to `0`](#return-only-agg-results) to avoid filling the cache. +For faster responses, {{es}} caches the results of frequently run aggregations in the [shard request cache](/deploy-manage/distributed-architecture/shard-request-cache.md). To get cached results, use the same [`preference` string](elasticsearch://reference/elasticsearch/rest-apis/search-shard-routing.md#shard-and-node-preference) for each search. If you don’t need search hits, [set `size` to `0`](#return-only-agg-results) to avoid filling the cache. {{es}} routes searches with the same preference string to the same shards. If the shards' data doesn’t change between searches, the shards return cached aggregation results. diff --git a/manage-data/data-store/aliases.md b/manage-data/data-store/aliases.md index 043961a36..3f8d171a9 100644 --- a/manage-data/data-store/aliases.md +++ b/manage-data/data-store/aliases.md @@ -316,7 +316,7 @@ Filters are only applied when using the [Query DSL](../../explore-analyze/query- ## Routing [alias-routing] -Use the `routing` option to [route](elasticsearch://reference/elasticsearch/mapping-reference/mapping-routing-field.md) requests for an alias to a specific shard. This lets you take advantage of [shard caches](elasticsearch://reference/elasticsearch/configuration-reference/shard-request-cache-settings.md) to speed up searches. Data stream aliases do not support routing options. +Use the `routing` option to [route](elasticsearch://reference/elasticsearch/mapping-reference/mapping-routing-field.md) requests for an alias to a specific shard. This lets you take advantage of [shard caches](/deploy-manage/distributed-architecture/shard-request-cache.md) to speed up searches. Data stream aliases do not support routing options. ```console POST _aliases diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece-upgrade.md b/raw-migrated-files/cloud/cloud-enterprise/ece-upgrade.md deleted file mode 100644 index 78dc45d8d..000000000 --- a/raw-migrated-files/cloud/cloud-enterprise/ece-upgrade.md +++ /dev/null @@ -1,127 +0,0 @@ -# Upgrade your installation [ece-upgrade] - -Periodically, you might need to upgrade an Elastic Cloud Enterprise installation as new versions with additional features become available. The upgrade process updates all hosts that are part of an Elastic Cloud Enterprise installation to the latest version of ECE, with little or no downtime for managed deployments. - -Before initiating the ECE upgrade process, consult the [OS & Docker supportability matrix](https://www.elastic.co/support/matrix) to make sure that the Operating System (OS) and Docker versions you run are compatible with the ECE version you’re about to upgrade to. We recommend that Docker and the operating system be at the target version before starting the ECE upgrade procedure. - -::::{note} -During the upgrade window, there might be a short time you are on a mix of versions which is not declared explicitly in the support matrix. For example, if you are on ECE 3.5 with Docker version 20.10 on Ubuntu 20.04, and plan to upgrade to ECE 3.7 on same OS, you will need to upgrade Docker to version 24.0 first. In this case, **only during your upgrade window**, we support the mixed OS/Docker versions. In general, this won’t be a problem, however, should anything become a blocker for the upgrade, please [reach out to support for help](../../../troubleshoot/deployments/cloud-enterprise/ask-for-help.md). -:::: - - - -## The upgrade version matrix [ece-upgrade-version-matrix] - -The following table shows the recommended upgrade paths from older {{ece}} versions to 3.8.1. - -| Upgrade from | Recommended upgrade path to 3.8.1 | -| --- | --- | -| Any previous 3.x version (for example, 3.0.0) | Upgrade directly to 3.8.1 | -| 2.13 | Upgrade directly to 3.8.1 | -| 2.5-2.12 | 1. Upgrade to 2.13.4
2. Upgrade to 3.8.1
| -| 2.0-2.4 | 1. Upgrade to 2.5.1
2. Upgrade to 2.13.4
3. Upgrade to 3.8.1
| - - -## The upgrade process [ece-upgrade-overview] - -Upgrading Elastic Cloud Enterprise works by replacing the [containers](/reference/glossary/index.md#glossary-container) that ECE itself requires to run on each host. Upgrading ECE does not touch any of the containers that run Elasticsearch clusters and Kibana instances. Each container that needs to be upgraded is renamed and stopped, followed by the creation of a new container with an upgraded instance of the ECE software and its dependencies. When the upgrade process has completed successfully, it cleans up after itself and removes the old containers. - -The upgrade process creates a `frc-upgraders-monitor` container on the host where you initiate the process that performs the following actions: - -* Back up the ZooKeeper transaction log to `HOST_STORAGE_PATH/RUNNER_ID/services/zookeeper/data/backup`, where `HOST_STORAGE_PATH` and `RUNNER_ID` are specific to an ECE installation. -* Configure Elastic Cloud Enterprise to perform the individual container upgrades by creating a `frc-upgraders-upgrader` container on each host that is part of the installation. -* Monitor the upgrade process to ensure that all `frc-upgraders-upgrader` containers perform their part of the upgrade as expected and report their status. -* After all hosts have been upgraded successfully, clean up temporary artifacts created during the upgrade process, and remove the old containers. - -The entire process is designed to be failsafe. Containers get upgraded sequentially and the upgrade status of each container is tracked. The upgrade process also monitors that each new container is viable and continues to run as expected. If there is an issue with any part of the upgrade, [the entire process is rolled back](../../../troubleshoot/deployments/cloud-enterprise/common-issues.md#ece-troubleshooting-upgrade). - - -## Recommendations [ece-upgrade-recommendations] - -Before starting the upgrade process, check which of the following recommendations may apply for your setup: - -* Upgrading to 2.12.x or 2.13.0 is not recommended as it can cause issues and you may lose access to the admin console. You are strongly advised to upgrade to 2.13.1 and later. -* If you are upgrading to ECE versions 2.10, 2.11, or 2.12, refer to the ECE version 2.12 [upgrade steps](https://www.elastic.co/guide/en/cloud-enterprise/2.12/ece-upgrade.html#ece-upgrade-system-deployments) for guidance about certain default ECE visualizations not working. -* We strongly recommend that you routinely update your ECE installation to the most current version so that any bugs and security issues are fixed promptly. If you need to upgrade but are currently experiencing any issues with your platform, note that as long as ZooKeeper is running and healthy you should be able to upgrade (you can use the [get runners API](https://www.elastic.co/docs/api/doc/cloud-enterprise/operation/operation-get-runners) to easily verify the health of the runners on the [ECE allocators](../../../deploy-manage/deploy/cloud-enterprise/ece-architecture.md#ece-architecture-allocators)). That is, healthy system deployments are not required in order to perform an upgrade successfully. -* Before upgrading to Elastic Cloud Enterprise 3.0, refer to the [lists of removals](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-3-0-removals.html) to find out about features and API endpoints that are no longer supported. -* We strongly recommend that you do not attempt to perform certain actions during the upgrade process, such as: - - * Creating or changing Elasticsearch clusters and Kibana instances - * Adding new hosts to an installation or removing existing hosts - -* As a precaution, we recommend that taking current snapshots of the Elasticsearch clusters. -* We recommend that you take a backup snapshot of the `security` [system deployment](../../../deploy-manage/deploy/cloud-enterprise/system-deployments-configuration.md). This cluster stores [role-based access control configurations](../../../deploy-manage/users-roles/cloud-enterprise-orchestrator/manage-users-roles.md), and a snapshot will allow you to restore those in case the upgrade fails. - - -## Requirements [ece-upgrade-prereqs] - -Before starting the upgrade process, verify that your setup meets the following requirements: - -* **XFS with quotas enabled on all allocators.** You must use XFS and have quotas enabled on all allocators, otherwise disk usage won’t display correctly. To enable XFS quotas, modify the entry for the XFS volume in the `/etc/fstab file` to add `pquota` and `prjquota`. The default filesystem path used by Elastic Cloud Enterprise is `/mnt/data`. -* **Supported Docker / Podman version** Make sure that you run a supported Docker or Podman version for the version of ECE that you are going to upgrade to. An overview of compatible versions can be found in the [support matrix](https://www.elastic.co/support/matrix#matrix_os&#elastic-cloud-enterprise) and install instructions are available under [Installing Elastic Cloud Enterprise](../../../deploy-manage/deploy/cloud-enterprise/install.md). -* **Required user, roles and groups** To run the script to upgrade Elastic Cloud Enterprise, login as the user used to run Elastic Cloud Enterprise (by default called `elastic` with UID/GID 1000). Initiate the upgrade process by running the `elastic-cloud-enterprise.sh` script with the `upgrade` action on a single host. The host that the script is run on must be a host that holds the director role. You do not need to run the script on additional hosts. -* **Available disk space** Each host in the Elastic Cloud Enterprise installation must have at least 5 GB of disk space available to ensure that the upgrade process can complete successfully. -* **Proxies and load balancing** To avoid any downtime for Elastic Cloud Enterprise, the installation must include more than one proxy and must use a load balancer as recommended. If only a single proxy is configured or if the installation is not using a load balancer, some downtime is expected when the containers on the proxies are upgraded. Each container upgrade typically takes five to ten seconds, times the number of containers on a typical host. -* **For *offline* or *air-gapped* installations** Additional steps are required to upgrade Elastic Cloud Enterprise. After downloading the installation script for the new version, pull and load the required container images and push them to a private Docker registry. To learn more about pulling and loading Docker images, check [Install ECE offline](../../../deploy-manage/deploy/cloud-enterprise/air-gapped-install.md). -* **Verify if you can upgrade directly** If you are upgrading to ECE 3.0 or a higher version, you need to upgrade to ECE 2.13.1 or later. Refer to the ECE version 2.13 [upgrade instructions](https://www.elastic.co/guide/en/cloud-enterprise/2.13/ece-upgrade.html) for details. - -::::{warning} -Don’t manually upgrade your system deployments if you are on ECE version 2.7.0 or a later version, as it can cause issues and you may lose access to the Cloud UI. Note that the only exception to that rule is when you’re upgrading to ECE 3.6.0 and your system deployments are at a version lower than 7.17.0. -:::: - - -* **If you are on an ECE version below 2.7, verify if the system deployments need to be upgraded** - - * If there are system deployments below version 6.8 or an ECE installation below version 2.7.0, refer to the ECE version 2.7 documentation for the steps to [upgrade your system deployments](https://www.elastic.co/guide/en/cloud-enterprise/2.7/ece-upgrade.html#ece-upgrade-system-deployments). Starting with ECE 2.7, all system deployments are upgraded automatically to their latest supported Elasticsearch version as part of the ECE upgrade process. This means that each system deployment can be upgraded to different major versions. For example, the security cluster could be upgraded to 8.x, while the admin and logging-and-metrics clusters could be upgraded to the latest 7.17.x version. - * When upgrading from an ECE version between 2.0 and 2.4 included, we recommend to upgrade to ECE 2.5 first, then to manually upgrade system deployments to version 6.8. - -* If you are upgrading to ECE 3.6.0, you need to ensure that your system deployments are at version 7.17 because the upgrade process will attempt to upgrade the security cluster to 8.5 or higher. Starting with ECE 3.6.1, the upgrade process automatically upgrades system deployments to the minimum required version. If the Elastic Cloud Enterprise platform was upgraded successfully and yet one or more system deployments were not upgraded to a higher Elastic Stack version during the very last phase of the Elastic Cloud Enterprise upgrade, you can re-run the `elastic-cloud-enterprise.sh upgrade --cloud-enterprise-version ` command to retry system deployment upgrade only. -* **Check the security cluster’s zone count** Due to internal limitations in ECE, the built-in security cluster cannot be scaled to two zones during the ECE upgrade procedure. If the zone count is set to 2 zones, scale the cluster to 3 or 1 zone(s) before upgrading ECE. -* **Outdated cluster versions** If the ECE installation has clusters using version 5.5 or earlier, upgrading to version 5.6 is mandatory before a major upgrade. - - -### Certificate rotation [ece-upgrade-certificates] - -If your ECE installation is still using the default, auto-generated certificates, we recommend performing one of the following steps to avoid trust errors related to the proxy server certificate after the upgrade. The proxy server certificate is used when connecting to Kibana and Elasticsearch clusters. During the upgrade, the ECE certificate authority generates a new certificate. As with any server certificate rotation, you must add an exception for the new proxy server certificate, unless the certificate authority is present in the trust store of the system or browser. You can perform either of these steps before or after the upgrade: - -* Recommended: [Add your organization’s own certificate](../../../deploy-manage/security/secure-your-elastic-cloud-enterprise-installation/manage-security-certificates.md) to Elastic Cloud Enterprise. The upgrade process ensures that the certificates you add do not change, which avoids the trust errors. -* Add the default CA certificate to the trust store of your system or of your browser. Only the server certificate changes during upgrade, but the CA certificate remains the same. Adding the CA certificate to your trust store alone is sufficient to avoid the trust errors. -* Apply valid license. It is required to have an `Enterprise resource unit`-compatible license applied before upgrading to ECE 2.7+. The most reliable way to check if your license is compatible is to use the Elastic Cloud Enterprise API and check the value of the license version field: - - ```sh - curl -X GET -u admin:PASSWORD -k https://COORDINATOR_HOST:12443/api/v1/platform/license - { - "license": { - "version": 4, - // other fields - } - } - ``` - - If the version is not 4 or higher, you must request an updated license from [Elastic Support](../../../troubleshoot/deployments/cloud-enterprise/ask-for-help.md). Once you receive your new license, make sure Elastic Cloud Enterprise is upgraded to at least version 2.5.0, and then upload the new license in the **Settings** page under the **Platform** menu. - - -In versions from 2.6 to 2.10 included, some or all platform certificates are generated with a 398-day expiration. Installations that ran on these versions, even temporarily, must have their certificates rotated manually before expiry. For details, check [our KB article](https://ela.st/ece-certificate-rotation). - - -## Perform the upgrade [ece-upgrade-steps] - -To upgrade an Elastic Cloud Enterprise installation, download the latest installation script. Login as the user used to run Elastic Cloud Enterprise (by default called `elastic` with UID/GID 1000), and run the script with the `upgrade` action on a single host that holds the director role: - -::::{important} -If you are using SELinux, make sure you also use `--selinux`. - -:::: - - -```sh -bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) upgrade -``` - -You can follow along while each container for Elastic Cloud Enterprise is upgraded on the hosts that are part of the installation. - -By default, ECE updates to the most current available version. If you want to upgrade to a specific ECE version, use the `--cloud-enterprise-version` option: - -```sh -bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) upgrade --user admin --pass $PASSWORD --cloud-enterprise-version 3.0.0 -``` diff --git a/raw-migrated-files/cloud/cloud-enterprise/ece_re_running_the_ece_upgrade.md b/raw-migrated-files/cloud/cloud-enterprise/ece_re_running_the_ece_upgrade.md deleted file mode 100644 index 8f38d75e5..000000000 --- a/raw-migrated-files/cloud/cloud-enterprise/ece_re_running_the_ece_upgrade.md +++ /dev/null @@ -1,8 +0,0 @@ -# Re-running the ECE upgrade [ece_re_running_the_ece_upgrade] - -By default, re-running the `elastic-cloud-enterprise.sh upgrade` command resumes the previous ECE upgrade process. However, if your previous ECE upgrade attempt got stuck (for example, due to infrastructure problems, a host going offline, or similar) and re-attempting the upgrade still results in the upgrade process being blocked, run `elastic-cloud-enterprise.sh upgrade --force-upgrade` to ensure that any existing upgrade state gets cleared before starting the new ECE upgrade process. - -The `--force-upgrade` parameter is also helpful in situations where the Elastic Cloud Enterprise platform is already upgraded to the desired target version but there are containers still running at the old version. These rare situations can also be caused by infrastructure issues, for example, runners temporarily going offline and not being "seen" by the upgrade process. Running `elastic-cloud-enterprise.sh upgrade --force-upgrade` with the same target version makes the Elastic Cloud Enterprise upgrader perform the upgrade procedure anyway, thereby covering any containers that failed to upgrade previously. - -If the Elastic Cloud Enterprise platform was upgraded successfully and yet one or more system deployments were not upgraded to a higher Elastic Stack version during the very last phase of the Elastic Cloud Enterprise upgrade, we recommend running the Elastic Cloud Enterprise upgrader again without the `--force-upgrade` parameter. The Elastic Cloud Enterprise upgrader will recognize that the platform is already at the desired target version and will apply upgrade plans to system deployments. - diff --git a/raw-migrated-files/cloud/cloud/ec-maintenance-mode-routing.md b/raw-migrated-files/cloud/cloud/ec-maintenance-mode-routing.md deleted file mode 100644 index f516081e8..000000000 --- a/raw-migrated-files/cloud/cloud/ec-maintenance-mode-routing.md +++ /dev/null @@ -1,30 +0,0 @@ -# Request routing [ec-maintenance-mode-routing] - -The {{ecloud}} proxy service routes traffic from external sources to the deployment, between deployments, and between products within a deployment. For example, it routes API requests from your local machine to your deployment, CCR and CCS requests between your deployments, and communications between {{kib}} and {{es}}. It does not direct the TCP traffic between {{es}} nodes, nor does it manage requests starting within {{es}} outwards to external sources such as to snapshot repositories. - -The {{ecloud}} proxy routes HTTP requests to its deployment’s individual product instances through the product’s endpoint. By default, instances are enabled to route HTTP traffic and will report no special messaging. - -It might be helpful to temporarily block upstream requests in order to protect some or all instances or products within your deployment. For example, you might stop request routing in the following cases: - -* If another team within your company starts streaming new data into your production {{integrations-server}} without previous load testing, both it and {{es}} might experience performance issues. You might consider stopping routing requests on all {{integrations-server}} instances in order to protect your downstream {{es}} instance. -* If {{es}} is being overwhelmed by upstream requests, it might experience increased response times or even become unresponsive. This might impact your ability to resize components in your deployment and increase the duration of pending plans or increase the chance of plan changes failing. Because every {{es}} node is an [implicit coordinating node](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md), you should stop routing requests across all {{es}} nodes to completely block upstream traffic. - - -## Considerations [ec_considerations] - -* {{ecloud}} will automatically set and remove routing blocks during plan changes. Elastic recommends avoiding manually overriding these settings for a deployment while its plans are pending. -* The [{{es}} API console](cloud://reference/cloud-hosted/ec-api-console.md) bypasses {{ecloud}} proxy routing blocks against {{es}} to enable administrative tasks while plan changes are pending. You should generally default traffic to the {{es}} endpoint. However, if you enable **Stop routing requests** across all {{es}} nodes, you need to use this UI to administer your cluster. -* While {{es}} has **Stop routing requests** set across all nodes, other products with the deployment may become unhealthy. This is because {{es}} is a prerequisite for those other products, such as {{kib}}. In {{kib}}, this results in a [**Kibana server is not ready yet**](/troubleshoot/kibana/error-server-not-ready.md) message. -* Enabling **Stop routing requests** does not affect your [billing](../../../deploy-manage/cloud-organization/billing.md). If needed, you can stop charges for a deployment by [deleting the deployment](../../../deploy-manage/uninstall/delete-a-cloud-deployment.md). - - -## Stop routing requests [ec_stop_routing_requests] - -To block HTTP requests for an instance, select **Stop routing requests** under from instance’s menu. - -The instance will then report **Not routing requests**. It will complete existing requested traffic, but not be sent new requests. - - -## Restart routing requests [ec_restart_routing_requests] - -To unblock HTTP requests for an instance, select **Start routing requests** under from instance’s menu. diff --git a/raw-migrated-files/docs-content/serverless/elasticsearch-http-apis.md b/raw-migrated-files/docs-content/serverless/elasticsearch-http-apis.md deleted file mode 100644 index 5482dfc18..000000000 --- a/raw-migrated-files/docs-content/serverless/elasticsearch-http-apis.md +++ /dev/null @@ -1,33 +0,0 @@ -# REST APIs [elasticsearch-http-apis] - - -## API references [elasticsearch-api-references-links] - -The following APIs are available for {{es-serverless}} users. These links will take you to the autogenerated API reference documentation. - -[Elasticsearch Serverless APIs →](https://www.elastic.co/docs/api/doc/elasticsearch-serverless) -: Use these APIs to index, manage, search, and analyze your data in {{es-serverless}}. - - ::::{tip} - Learn how to [connect to your {{es-serverless}} endpoint](../../../solutions/search/get-started.md). - - :::: - - -[Kibana Serverless APIs →](https://www.elastic.co/docs/api/doc/serverless) -: Use these APIs to manage resources such as connectors, data views, and saved objects for your {{serverless-full}} project. - -[{{serverless-full}} APIs →](https://www.elastic.co/docs/api/doc/elastic-cloud-serverless) -: Use these APIs to manage your {{serverless-full}} projects. - - -## Additional API information [additional-api-details] - -[{{es}} API conventions](https://www.elastic.co/guide/en/serverless/current/elasticsearch-api-conventions.html) -: Reference information about headers and request body conventions for {{es-serverless}} REST APIs. - -[Management API conventions](https://www.elastic.co/guide/en/serverless/current/elasticsearch-kibana-api-conventions.html) -: Reference information about request header conventions for {{serverless-full}} REST APIs. - - - diff --git a/raw-migrated-files/toc.yml b/raw-migrated-files/toc.yml index b76cc2851..46fe536ca 100644 --- a/raw-migrated-files/toc.yml +++ b/raw-migrated-files/toc.yml @@ -16,20 +16,16 @@ toc: - file: cloud-on-k8s/cloud-on-k8s/k8s-securing-stack.md - file: cloud/cloud-enterprise/index.md children: - - file: cloud/cloud-enterprise/ece_re_running_the_ece_upgrade.md - file: cloud/cloud-enterprise/ece-securing-clusters.md - file: cloud/cloud-enterprise/ece-securing-ece.md - - file: cloud/cloud-enterprise/ece-upgrade.md - file: cloud/cloud/index.md children: - file: cloud/cloud/ec-faq-technical.md - - file: cloud/cloud/ec-maintenance-mode-routing.md - file: cloud/cloud/ec-monitoring-setup.md - file: cloud/cloud/ec-planning.md - file: cloud/cloud/ec-security.md - file: docs-content/serverless/index.md children: - - file: docs-content/serverless/elasticsearch-http-apis.md - file: docs-content/serverless/observability-ai-assistant.md - file: docs-content/serverless/observability-apm-get-started.md - file: docs-content/serverless/observability-ecs-application-logs.md @@ -43,7 +39,6 @@ toc: - file: elasticsearch/elasticsearch-reference/secure-cluster.md - file: elasticsearch/elasticsearch-reference/security-files.md - file: elasticsearch/elasticsearch-reference/security-limitations.md - - file: elasticsearch/elasticsearch-reference/shard-request-cache.md - file: kibana/kibana/index.md children: - file: kibana/kibana/reporting-production-considerations.md