Skip to content

More raw cleanup #885

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 11 commits into from
Mar 21, 2025
Original file line number Diff line number Diff line change
Expand Up @@ -79,10 +79,10 @@ When you select **Save changes** on the **Edit deployment** page, the orchestrat
* **Grow and shrink**: The orchestrator creates new instances with the new configuration, then migrates the data, and eventually deletes the original ones. This strategy is automatically selected when adding or removing master-eligible instances.
* **Rolling grow and shrink**: Similar to grow and shrink, but creating one instance at a time. This strategy can take a lot longer than grow and shrink.

The `Extended maintenance` optional flag will make ECE to [stop routing requests](../../maintenance/ece/start-stop-routing-requests.md) to all instances during the plan execution. The cluster will be unavailable for external connections while the configuration changes are in progress.
The `Extended maintenance` optional flag will make ECE to [stop routing requests](../../maintenance/start-stop-routing-requests.md) to all instances during the plan execution. The cluster will be unavailable for external connections while the configuration changes are in progress.

::::{note}
If you enable the **Extended maintenance** optional flag, ECE will [stop routing requests](../../maintenance/ece/start-stop-routing-requests.md) to all instances during the plan execution, making the cluster unavailable for external connections while configuration changes are in progress.
If you enable the **Extended maintenance** optional flag, ECE will [stop routing requests](../../maintenance/start-stop-routing-requests.md) to all instances during the plan execution, making the cluster unavailable for external connections while configuration changes are in progress.

This option introduces downtime and is rarely needed. Use it only when you need to block all traffic to the cluster during the update.
::::
Expand Down
2 changes: 1 addition & 1 deletion deploy-manage/deploy/elastic-cloud/manage-deployments.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ mapped_pages:
* Ensure the health of your deployment over time

* [Keep track of your deployment's activity](keep-track-of-deployment-activity.md) or [Enable logging and monitoring](../../monitor/stack-monitoring/ece-ech-stack-monitoring.md) of the deployment performance.
* Perform maintenance operations to ensure the health of your deployment, such as [restarting your deployment](../../maintenance/start-stop-services/restart-cloud-hosted-deployment.md) or [stopping routing](../../maintenance/ece/start-stop-routing-requests.md).
* Perform maintenance operations to ensure the health of your deployment, such as [restarting your deployment](../../maintenance/start-stop-services/restart-cloud-hosted-deployment.md) or [stopping routing](../../maintenance/start-stop-routing-requests.md).

* Manage the lifecycle of your deployment:

Expand Down
2 changes: 1 addition & 1 deletion deploy-manage/distributed-architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,5 +22,5 @@ The topics in this section provides information about the architecture of {{es}}
* [Shard allocation, relocation, and recovery](distributed-architecture/shard-allocation-relocation-recovery.md): Learn how {{es}} allocates and balances shards across nodes.
* [Shard allocation awareness](distributed-architecture/shard-allocation-relocation-recovery/shard-allocation-awareness.md): Learn how to use custom node attributes to distribute shards across different racks or availability zones.
* [Disocvery and cluster formation](distributed-architecture/discovery-cluster-formation.md): Learn about the cluster formation process including voting, adding nodes and publishing the cluster state.
* [Shard request cache](elasticsearch://reference/elasticsearch/configuration-reference/shard-request-cache-settings.md): Learn how {{es}} caches search requests to improve performance.
* [Shard request cache](/deploy-manage/distributed-architecture/shard-request-cache.md): Learn how {{es}} caches search requests to improve performance.
* [Kibana task management](distributed-architecture/kibana-tasks-management.md): Learn how {{kib}} runs background tasks and distribute work across multiple {{kib}} instances to be persistent and scale with your deployment.
Original file line number Diff line number Diff line change
@@ -1,3 +1,14 @@
---
mapped_pages:
- https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-request-cache.html
applies_to:
deployment:
ece:
ess:
eck:
self:
---

# The shard request cache [shard-request-cache]

When a search request is run against an index or against many indices, each involved shard executes the search locally and returns its local results to the *coordinating node*, which combines these shard-level results into a global result set.
Expand All @@ -7,16 +18,14 @@ The shard-level request cache module caches the local results on each shard. Thi
You can control the size and expiration of the cache at the node level using the [shard request cache settings](elasticsearch://reference/elasticsearch/configuration-reference/shard-request-cache-settings.md).

::::{important}
By default, the requests cache will only cache the results of search requests where `size=0`, so it will not cache `hits`, but it will cache `hits.total`, [aggregations](../../../explore-analyze/query-filter/aggregations.md), and [suggestions](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html).
By default, the requests cache will only cache the results of search requests where `size=0`, so it will not cache `hits`, but it will cache `hits.total`, [aggregations](/explore-analyze/query-filter/aggregations.md), and [suggestions](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-suggesters.html).

Most queries that use `now` (see [Date Math](elasticsearch://reference/elasticsearch/rest-apis/common-options.md#date-math)) cannot be cached.

Scripted queries that use the API calls which are non-deterministic, such as `Math.random()` or `new Date()` are not cached.

::::



## Cache invalidation [_cache_invalidation]

The cache is smart — it keeps the same *near real-time* promise as uncached search.
Expand Down Expand Up @@ -82,8 +91,6 @@ A hash of the whole JSON body is used as the cache key. This means that if the J
Most JSON libraries support a *canonical* mode which ensures that JSON keys are always emitted in the same order. This canonical mode can be used in the application to ensure that a request is always serialized in the same way.
::::



## Monitoring cache usage [_monitoring_cache_usage]

The size of the cache (in bytes) and the number of evictions can be viewed by index, with the [`indices-stats`](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-indices-stats) API:
Expand Down
7 changes: 4 additions & 3 deletions deploy-manage/maintenance.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ This section outlines the key tasks and processes required to maintain a healthy

The topics covered include:

* **[ECE Maintenance](maintenance/ece.md)**: Explains the procedures for maintaining both the host infrastructure and {{es}} deployments within Elastic Cloud Enterprise (ECE).
* **[Start and Stop services](maintenance/start-stop-services.md)**: Provides step-by-step instructions on how to safely start and stop your {{es}} deployment or {{kib}} instance, particularly when performing actions that require a restart.
* **[Add and remove {{es}} nodes](maintenance/add-and-remove-elasticsearch-nodes.md)**: Guides you through the process of enrolling new nodes or safely removing existing ones from a self-managed {{es}} cluster to optimize resource utilization and cluster performance.
* **[](maintenance/ece.md)**: Explains the procedures for maintaining both the host infrastructure and {{es}} deployments within Elastic Cloud Enterprise (ECE).
* **[](maintenance/start-stop-services.md)**: Provides step-by-step instructions on how to safely start and stop your {{es}} deployment or {{kib}} instance, particularly when performing actions that require a restart.
* **[](maintenance/start-stop-routing-requests.md)**: Start or stop routing requests to an {{ech}} or {{ece}} deployment, or to all instances on an allocator in {{ece}}.
* **[](maintenance/add-and-remove-elasticsearch-nodes.md)**: Guides you through the process of enrolling new nodes or safely removing existing ones from a self-managed {{es}} cluster to optimize resource utilization and cluster performance.
2 changes: 1 addition & 1 deletion deploy-manage/maintenance/ece.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Elastic Cloud Enterprise (ECE), being a self-managed Elastic Stack deployment pl

## Deployment maintenance and host infrastructure maintenance [ece-deployment-host-infra-maintenance]

[Deployment maintenance](ece/deployments-maintenance.md) focuses on managing individual {{es}} and {{kib}} instances within ECE. This includes actions such as [pausing instances](ece/pause-instance.md), [stopping request routing to nodes](ece/start-stop-routing-requests.md), and [moving instances between allocators](ece/move-nodes-instances-from-allocators.md) to optimize resource usage or prepare for maintenance. These tasks help maintain service availability and performance without affecting the underlying infrastructure.
[Deployment maintenance](ece/deployments-maintenance.md) focuses on managing individual {{es}} and {{kib}} instances within ECE. This includes actions such as [pausing instances](ece/pause-instance.md), [stopping request routing to nodes](/deploy-manage/maintenance/start-stop-routing-requests.md), and [moving instances between allocators](ece/move-nodes-instances-from-allocators.md) to optimize resource usage or prepare for maintenance. These tasks help maintain service availability and performance without affecting the underlying infrastructure.

[ECE host infrastructure maintenance](ece/perform-ece-hosts-maintenance.md) involves managing virtual machines that host ECE itself. This includes tasks like applying operating system patches, upgrading software, or decommissioning hosts. Infrastructure maintenance often requires more careful planning, as it can impact multiple deployments running on the affected hosts. Methods such as placing allocators into [maintenance mode](ece/enable-maintenance-mode.md) and redistributing workloads provide a smooth transition during maintenance operations.

Expand Down
2 changes: 1 addition & 1 deletion deploy-manage/maintenance/ece/deployments-maintenance.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ In some circumstances, you might need to temporarily restrict access to a node s

These actions act as a maintenance mode for cluster node. Performing these actions can stop the cluster from becoming unresponsive so that you can resolve operational issues much more effectively.

* [**Stop routing to the instance**](start-stop-routing-requests.md): Block requests from being routed to the cluster node. This is a less invasive action than pausing the cluster.
* [**Stop routing to the instance**](/deploy-manage/maintenance/start-stop-routing-requests.md): Block requests from being routed to the cluster node. This is a less invasive action than pausing the cluster.
* [**Pause an instance**](pause-instance.md): Suspend the node immediately by stopping the container that the node runs on without completing existing requests. This is a more aggressive action to regain control of an unresponsive node.

As an alternative, to quickly add capacity to a deployment if it is unhealthy or at capacity, you can also [override the resource limit for a deployment](../../deploy/cloud-enterprise/resource-overrides.md).
2 changes: 1 addition & 1 deletion deploy-manage/maintenance/ece/enable-maintenance-mode.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,6 @@ After the allocator enters maintenance mode, no new Elasticsearch nodes or Kiban
If you want to make the allocator fully active again, select **Disable Maintenance Mode**. Confirm the action.

::::{tip}
If you need the existing instances to stop routing requests, refer to the [stop routing request documentation](start-stop-routing-requests.md) to learn more.
If you need the existing instances to stop routing requests, refer to the [stop routing request documentation](../start-stop-routing-requests.md) to learn more.

::::
Original file line number Diff line number Diff line change
Expand Up @@ -5,33 +5,43 @@ mapped_urls:
applies_to:
deployment:
ece:
ess:
---

# Start and stop routing requests [maintenance-mode-routing]

The {{ecloud}} proxy service routes traffic from external sources to the deployment, between deployments, and between products within a deployment. For example, it routes API requests from your local machine to your deployment, CCR and CCS requests between your deployments, and communications between {{kib}} and {{es}}. It does not direct the TCP traffic between {{es}} nodes, nor does it manage requests starting within {{es}} outwards to external sources such as to snapshot repositories.
The cloud proxy service routes traffic from external sources to the deployment, between deployments, and between products within a deployment. For example, it routes API requests from your local machine to your deployment, CCR and CCS requests between your deployments, and communications between {{kib}} and {{es}}. It does not direct the TCP traffic between {{es}} nodes, nor does it manage requests starting within {{es}} outwards to external sources such as to snapshot repositories.

The {{ecloud}} proxy routes HTTP requests to its deployment’s individual product instances through the product’s endpoint. By default, instances are enabled to route HTTP traffic and will report no special messaging.
The cloud proxy routes HTTP requests to its deployment’s individual product instances through the product’s endpoint. By default, instances are enabled to route HTTP traffic and will report no special messaging.

It might be helpful to temporarily block upstream requests in order to protect some or all instances or products within your deployment. For example, you might stop request routing in the following cases:

* If another team within your company starts streaming new data into your production {{integrations-server}} without previous load testing, both it and {{es}} might experience performance issues. You might consider stopping routing requests on all {{integrations-server}} instances in order to protect your downstream {{es}} instance.
* If {{es}} is being overwhelmed by upstream requests, it might experience increased response times or even become unresponsive. This might impact your ability to resize components in your deployment and increase the duration of pending plans or increase the chance of plan changes failing. Because every {{es}} node is an [implicit coordinating node](elasticsearch://reference/elasticsearch/configuration-reference/node-settings.md), you should stop routing requests across all {{es}} nodes to completely block upstream traffic.
* If {{es}} is being overwhelmed by upstream requests, it might experience increased response times or even become unresponsive. This might impact your ability to resize components in your deployment and increase the duration of pending plans or increase the chance of plan changes failing. Because every {{es}} node is an [implicit coordinating node](/deploy-manage/distributed-architecture/clusters-nodes-shards/node-roles.md#node-roles-list), you should stop routing requests across all {{es}} nodes to completely block upstream traffic.

## Considerations [request-routing-considerations]

* {{ecloud}} will automatically set and remove routing blocks during plan changes. Elastic recommends avoiding manually overriding these settings for a deployment while its plans are pending.
* The [{{es}} API console](../../../explore-analyze/query-filter/tools/console.md) bypasses {{ecloud}} proxy routing blocks against {{es}} to enable administrative tasks while plan changes are pending. You should generally default traffic to the {{es}} endpoint. However, if you enable **Stop routing requests** across all {{es}} nodes, you need to use this UI to administer your cluster.
* While {{es}} has **Stop routing requests** set across all nodes, other products with the deployment may become unhealthy. This is because {{es}} is a prerequisite for those other products, such as {{kib}}. In {{kib}}, this results in a [**Kibana server is not ready yet**](../../../troubleshoot/kibana/error-server-not-ready.md) message.
* Enabling **Stop routing requests** does not affect your [billing](../../../deploy-manage/cloud-organization/billing.md). If needed, you can stop charges for a deployment by [deleting the deployment](../../../deploy-manage/uninstall/delete-a-cloud-deployment.md).
* ECE and ECH will automatically set and remove routing blocks during plan changes. Elastic recommends avoiding manually overriding these settings for a deployment while its plans are pending.
* The [{{es}} API console](/explore-analyze/query-filter/tools/console.md) bypasses cloud proxy routing blocks against {{es}} to enable administrative tasks while plan changes are pending. You should generally default traffic to the {{es}} endpoint. However, if you enable **Stop routing requests** across all {{es}} nodes, you need to use this UI to administer your cluster.
* While {{es}} has **Stop routing requests** set across all nodes, other products with the deployment may become unhealthy. This is because {{es}} is a prerequisite for those other products, such as {{kib}}. In {{kib}}, this results in a [**Kibana server is not ready yet**](/troubleshoot/kibana/error-server-not-ready.md) message.
* In {{ech}}, enabling **Stop routing requests** does not affect your [billing](/deploy-manage/cloud-organization/billing.md). If needed, you can stop charges for a deployment by [deleting the deployment](/deploy-manage/uninstall/delete-a-cloud-deployment.md).

## Stop routing requests [stop-routing-requests]

To block HTTP requests for an instance, select **Stop routing requests** under from instance’s menu.

The instance will then report **Not routing requests**. It will complete existing requested traffic, but not be sent new requests.

You can stop routing requests to disable incoming requests to particular instances. You can also massively disable all allocator instances routing with the [allocator-toggle-routing-requests.sh](https://download.elastic.co/cloud/allocator-toggle-routing-requests.sh) script. The script runs with the following parameters in the form environment variables:
## Restart routing requests [restart-routing-requests]

To unblock HTTP requests for an instance, select **Start routing requests** under from instance’s menu.

## Toggle routing to all instances on an allocator
```{applies_to}
deployment:
ece:
```
In {{ece}}, in addition to stopping routing requests for particular instances, you can also massively disable routing to all instances on a specified allocator with the [allocator-toggle-routing-requests.sh](https://download.elastic.co/cloud/allocator-toggle-routing-requests.sh) script. The script runs with the following parameters in the form environment variables:

* `API_URL` Url of the administration API.
* `AUTH_HEADER` Curl format string representing the authentication header.
Expand All @@ -49,7 +59,3 @@ The same script can be used to enable traffic again:
```bash
AUTH_HEADER="Authorization: ApiKey $(cat ~/api.key)" API_URL="https://adminconsole:12443" ALLOCATOR_ID="192.168.44.10" ENABLE_TRAFFIC=true ./allocator-toggle-routing-requests.sh
```

## Restart routing requests [restart-routing-requests]

To unblock HTTP requests for an instance, select **Start routing requests** under from instance’s menu.
Original file line number Diff line number Diff line change
Expand Up @@ -104,7 +104,7 @@ It is not uncommon for performance issues on {{ech}} to be caused by an undersiz
To help diagnose high CPU usage you can also use the {{es}} [nodes hot threads API](https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-nodes-hot-threads), which identifies the threads on each node that have the highest CPU usage or that have been executing for a longer than normal period of time.

::::{tip}
Got an overwhelmed cluster that needs to be upsized? [Try enabling maintenance mode first](/deploy-manage/maintenance/ece/start-stop-routing-requests.md). It will likely help with configuration changes.
Got an overwhelmed cluster that needs to be upsized? [Try enabling maintenance mode first](/deploy-manage/maintenance/start-stop-routing-requests.md). It will likely help with configuration changes.
::::


Expand Down
Loading
Loading