Kibana Dev Tools query not working when kibana is meshed

[tmyhu]

What is the issue?

When both Kibana and Elasticsearch are meshed with Linkerd, any queries in the Kibana Dev Tools return all documents ignoring the actual query. If Kibana is not meshed everything works as expected.

How can it be reproduced?

Deploy Elasticsearch and Kibana via the elastic operator (ECK), populate an index with some documents and try to query a subset via Kibana Dev Tools e.g.

GET /my-index/_search
{
    "query": {
        "match": {
          "category": "concerts"
        }
    }
}

Logs, error output, etc

No errors anywhere. With slowlogs enabled in Elasticsearch, we can see that the source field which should contain the query is empty:

phq-api8-es-data-0 elasticsearch {"@timestamp":"2025-09-17T23:36:09.328Z", "log.level":"TRACE",  "elasticsearch.slowlog.id":null,"elasticsearch.slowlog.message":"[events-api.events.0.staging][4]","elasticsearch.slowlog.search_type":"QUERY_THEN_FETCH","elasticsearch.slowlog.source":"{}","elasticsearch.slowlog.stats":"[]","elasticsearch.slowlog.took":"574.3micros","elasticsearch.slowlog.took_millis":0,"elasticsearch.slowlog.total_hits":"10001+ hits","elasticsearch.slowlog.total_shards":5 , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.index_search_slowlog","process.thread.name":"elasticsearch[phq-api8-es-data-0][search][T#1]","log.logger":"index.search.slowlog.query","trace.id":"7b9005701ee8385f48f2736ddbc4722d","elasticsearch.cluster.uuid":"u1ceFvjSSpmqYVb0PREPAw","elasticsearch.node.id":"Y5bK6wz0QwWKjIStwsVL-A","elasticsearch.node.name":"phq-api8-es-data-0","elasticsearch.cluster.name":"phq-api8"}

As comparison, the same query via curl is logged like this:

phq-api8-es-data-1 elasticsearch {"@timestamp":"2025-09-17T23:36:39.882Z", "log.level":"TRACE",  "elasticsearch.slowlog.id":null,"elasticsearch.slowlog.message":"[events-api.events.0.staging][0]","elasticsearch.slowlog.search_type":"QUERY_THEN_FETCH","elasticsearch.slowlog.source":"{\\\"query\\\":{\\\"term\\\":{\\\"category\\\":{\\\"value\\\":\\\"concerts\\\"}}}}","elasticsearch.slowlog.stats":"[]","elasticsearch.slowlog.took":"1ms","elasticsearch.slowlog.took_millis":1,"elasticsearch.slowlog.total_hits":"3556 hits","elasticsearch.slowlog.total_shards":5 , "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.index_search_slowlog","process.thread.name":"elasticsearch[phq-api8-es-data-1][search][T#2]","log.logger":"index.search.slowlog.query","elasticsearch.cluster.uuid":"u1ceFvjSSpmqYVb0PREPAw","elasticsearch.node.id":"uNWd_oM8Sbau9gZ9kd9hcw","elasticsearch.node.name":"phq-api8-es-data-1","elasticsearch.cluster.name":"phq-api8"}

output of linkerd check -o short

linkerd-identity
----------------
‼ issuer cert is valid for at least 60 days
    issuer certificate will expire on 2025-09-19T06:13:22Z
    see https://linkerd.io/2/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints

linkerd-version
---------------
‼ cli is up-to-date
    is running version 25.8.5 but the latest edge version is 25.9.2
    see https://linkerd.io/2/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
    is running version 25.8.5 but the latest edge version is 25.9.2
    see https://linkerd.io/2/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
    some proxies are not running the current version:
	* linkerd-destination-f455c657b-7j6fr (edge-25.8.5)
	* linkerd-destination-f455c657b-fff6h (edge-25.8.5)
	* linkerd-destination-f455c657b-ttvll (edge-25.8.5)
	* linkerd-identity-6bf9c9c667-hkbrt (edge-25.8.5)
	* linkerd-identity-6bf9c9c667-hxdxz (edge-25.8.5)
	* linkerd-identity-6bf9c9c667-tlws7 (edge-25.8.5)
	* linkerd-proxy-injector-7d7fbcd9b5-5sqgd (edge-25.8.5)
	* linkerd-proxy-injector-7d7fbcd9b5-6xt67 (edge-25.8.5)
	* linkerd-proxy-injector-7d7fbcd9b5-zmjhm (edge-25.8.5)
    see https://linkerd.io/2/checks/#l5d-cp-proxy-version for hints

linkerd-multicluster
--------------------
‼ multicluster extension proxies are up-to-date
    some proxies are not running the current version:
	* controller-eks-7689b4fd7d-dvxf9 (edge-25.8.5)
	* linkerd-gateway-6d48ffd9fd-mb9dp (edge-25.8.5)
	* linkerd-gateway-6d48ffd9fd-spfzb (edge-25.8.5)
	* linkerd-gateway-6d48ffd9fd-trlsz (edge-25.8.5)
	* linkerd-local-service-mirror-74f8958b5f-2f2mk (edge-25.8.5)
    see https://linkerd.io/2/checks/#l5d-multicluster-proxy-cp-version for hints

linkerd-viz
-----------
‼ viz extension proxies are up-to-date
    some proxies are not running the current version:
	* metrics-api-68656bb5f9-cp25q (edge-25.8.5)
	* prometheus-b66d6b4f6-bf9x5 (edge-25.8.5)
	* tap-7c88886d45-jtjxh (edge-25.8.5)
	* tap-injector-6f59b48b9c-zqpgn (edge-25.8.5)
	* web-d5d5cfb5b-lkccw (edge-25.8.5)
    see https://linkerd.io/2/checks/#l5d-viz-proxy-cp-version for hints

Status check results are √

Environment

• Kubernetes version: 1.32.8
• Cluster environment: kops
• Linkerd version: 25.8.5
• Elasticsearch/Kibana versions tested: 7.17.29 and 8.19.2

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

no

[alpeb]

The problem is the GET request with a body, that although not strictly forbidden by the HTTP spec, is badly supported by libraries, including the ones we depend on. Is it possible to configure Kibana to issue POST requests instead?

[tmyhu]

Thanks, I can confirm that the query works as expected if I use POST instead of GET in the Dev Tools console ie:

POST /my-index/_search
{
    "query": {
        "match": {
          "category": "concerts"
        }
    }
}

While that is great to know, it's not ideal since any documentation related to running queries in Kibana Dev Tools usually uses GET as method. For example, the official API doc for a search operation uses GET for Console even though the primary method is POST.

If there is no workaround via a linkerd config tweak I would probably rather leave Kibana unmeshed.

When you say this is badly supported, would you still consider it a bug that can be fixed or does this fall under unsupported behaviour? Note that a GET request with the same body via curl from inside the kibana pod returns the expected result. 🤷‍♂️

[kflynn]

The GET via curl from inside the Kibana pod happens over localhost, so it's effectively not meshed. It's a little odd to me that Kibana relies on GET-with-body, but I think this is a fine thing for us to track as a feature request.