[BUG] _cat/indices/<non_existing_index> is very slow when ignore_unavailable set to true

[lukas-vlcek]

Describe the bug

When _cat/indices/<index> is used for non existing index and ignore_unavailable=true is used then it takes very long time to get the response. In my demonstration setup (see below) it takes about 30 seconds.

curl 'localhost:9200/_cat/indices/foo?v=false&h=index,status&expand_wildcards=all&ignore_unavailable=true'

Related component

Other

To Reproduce

1. Setup docker-compose configuration with single OpenSearch node and Dashboards. The following is my config file:

services:
  opensearch-node1: # This is also the hostname of the container within the Docker network (i.e. https://opensearch-node1/)
    image: opensearchproject/opensearch:3.1.0
    container_name: opensearch-node1
    environment:
      - cluster.name=opensearch-cluster # Name the cluster
      - node.name=opensearch-node1 # Name the node that will run in this container
      - discovery.type=single-node
      - bootstrap.memory_lock=true # Disable JVM heap memory swapping
      - "OPENSEARCH_JAVA_OPTS=-Xms512m -Xmx512m" # Set min and max JVM heap sizes to at least 50% of system RAM
      - "DISABLE_INSTALL_DEMO_CONFIG=true"
      - "DISABLE_SECURITY_PLUGIN=true"
      - "plugins.query.datasources.encryption.masterkey=5e3dc78f06ef582dedbb5154" # Leftover from different example, shouldn't be relevant 
    ulimits:
      memlock:
        soft: -1 # Set memlock to unlimited (no soft or hard limit)
        hard: -1
      nofile:
        soft: 65536 # Maximum number of open files for the opensearch user - set to at least 65536
        hard: 65536
    volumes:
      - opensearch-data1:/usr/share/opensearch/data # Creates volume called opensearch-data1 and mounts it to the container
    ports:
      - 9200:9200 # REST API
      - 9600:9600 # Performance Analyzer
    networks:
      - opensearch-net # All of the containers will join the same Docker bridge network
  opensearch-dashboards:
    image: opensearchproject/opensearch-dashboards:3.1.0 # Make sure the version of opensearch-dashboards matches the version of opensearch installed on other nodes
    container_name: opensearch-dashboards
    ports:
      - 5601:5601 # Map host port 5601 to container port 5601
    expose:
      - "5601" # Expose port 5601 for web access to OpenSearch Dashboards
    environment:
      - data_source.enabled=true
      - 'OPENSEARCH_HOSTS=["http://opensearch-node1:9200"]' # Define the OpenSearch nodes that OpenSearch Dashboards will query
      - "DISABLE_SECURITY_DASHBOARDS_PLUGIN=true"
    networks:
      - opensearch-net

volumes:
  opensearch-data1:

networks:
  opensearch-net:

2. Once system is up and running use either Dashobards DevConsole or CLI and compare response time of the following two requests:

# Takes ~30 seconds
curl 'localhost:9200/_cat/indices/foo?v=false&h=index,status&expand_wildcards=all&ignore_unavailable=true'

# Response returned immediately
curl 'localhost:9200/_cat/indices/foo?v=false&h=index,status&expand_wildcards=all&ignore_unavailable=false'
{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [foo]","index":"foo","resource.id":"foo","resource.type":"index_or_alias","index_uuid":"_na_"}],"type":"index_not_found_exception","reason":"no such index [foo]","index":"foo","resource.id":"foo","resource.type":"index_or_alias","index_uuid":"_na_"},"status":404}

Expected behavior

I am not sure why the request takes so long to process but I would expect that it should be very fast. The index foo does not exist in the cluster so what is the request waiting for?

BTW, when DevConsole is used instead of CLI then it takes even longer to get expected response.

GET /_cat/indices/foo?v=false&h=index,status&ignore_unavailable=true

Can it be because DevConsole is setting up some timeout that is larger than the default one?

Additional Details

Plugins
OOTB, see the docker-compose.yml setup above.

Screenshots
n/a

Host/Environment (please complete the following information):

• The docker engine runs on Fedora 41
• OpenSearch 3.1.0 but I noticed the same issue with older releases as well

Additional context
I am trying to get list of indices from the cluster when index name pattern can be provided (which means that non-existing-indices can be part of the template) and I thought that _cat/indices/<pattern> would be good approach but I will need to look for different API alternatives.

[owaiskazi19]

Thanks @lukas-vlcek for reporting the bug. I was able to reproduce it on our end as well