qdrant_client-1.6.9: "Can't init WAL" when creating simple collection

[mfirry]

I'm getting error processing request: Can't init WAL: Os { code: 22, kind: InvalidInput, message: "Invalid argument" } just by trying to create a new collection as suggested on the docs

I'm just doing client.create_collection("2db5aa02", vectors_config) where vectors_config is models.VectorParams(size=100, distance=models.Distance.COSINE).

Using qdrant_client-1.6.9 and using locally docker.io/qdrant/qdrant:latest

[joein]

Hi, @mfirry

First of all, I've tried to reproduce the issue with your code, but unsuccessfully, the code is correct

Could you provide more logs and the exact actions you were doing?

Could you also provide info about your system (OS, docker version, anything specific for your machine you could think of)

[mfirry]

Sure. First of all I'm using podman version 4.7.2 on 13-inch, M1, 2020 MacBook Pro.

Here's the output when running podman run -p 6333:6333 -p 6334:6334 -v /Users/c305453/Desktop/fair:/qdrant/storage qdrant/qdrant.

podman run -p 6333:6333 -p 6334:6334 -v /Users/pippo/Desktop/yolo:/qdrant/storage qdrant/qdrant
           _                 _
  __ _  __| |_ __ __ _ _ __ | |_
 / _` |/ _` | '__/ _` | '_ \| __|
| (_| | (_| | | | (_| | | | | |_
 \__, |\__,_|_|  \__,_|_| |_|\__|
    |_|

Access web UI at http://localhost:6333/dashboard

2023-11-29T15:32:09.803043Z  INFO storage::content_manager::consensus::persistent: Loading raft state from ./storage/raft_state.json
2023-11-29T15:32:09.821274Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/16a28419", skipping
2023-11-29T15:32:09.822043Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/2db5aa02", skipping
2023-11-29T15:32:09.822691Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/39c8cc60", skipping
2023-11-29T15:32:09.823327Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/e948fc28", skipping
2023-11-29T15:32:09.823992Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/714fb1a6", skipping
2023-11-29T15:32:09.824637Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/b19d3d1a", skipping
2023-11-29T15:32:09.825243Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/3c72528e", skipping
2023-11-29T15:32:09.825855Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/ab34b016", skipping
2023-11-29T15:32:09.826475Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/0fc9f42f", skipping
2023-11-29T15:32:09.827347Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/9c0f037f", skipping
2023-11-29T15:32:09.828000Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/e5df4689", skipping
2023-11-29T15:32:09.828598Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/a35b40dd", skipping
2023-11-29T15:32:09.829225Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/2d633f4e", skipping
2023-11-29T15:32:09.830093Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/pippo", skipping
2023-11-29T15:32:09.830713Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/4fc159ce", skipping
2023-11-29T15:32:09.831532Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/ad1a62ba", skipping
2023-11-29T15:32:09.832158Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/ab1b9ab6", skipping
2023-11-29T15:32:09.832811Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/45bea5ec", skipping
2023-11-29T15:32:09.835791Z  INFO qdrant: Distributed mode disabled
2023-11-29T15:32:09.836063Z  INFO qdrant: Telemetry reporting enabled, id: 62b848fc-8d43-43cb-99af-58021037f06d
2023-11-29T15:32:09.839002Z  INFO qdrant::actix: TLS disabled for REST API
2023-11-29T15:32:09.839726Z  INFO qdrant::actix: Qdrant HTTP listening on 6333
2023-11-29T15:32:09.839813Z  INFO actix_server::builder: Starting 1 workers
2023-11-29T15:32:09.839985Z  INFO actix_server::server: Actix runtime found; starting in Actix runtime
2023-11-29T15:32:09.841451Z  INFO qdrant::tonic: Qdrant gRPC listening on 6334
2023-11-29T15:32:09.841539Z  INFO qdrant::tonic: TLS disabled for gRPC API

Even if I just use curl like

$ curl -X PUT http://localhost:6333/collections/pippo -d @qdrant.json
{"status":{"error":"Payload error: Content type error"},"time":0.0}%

and in the log I just see
2023-11-29T15:33:35.178298Z INFO actix_web::middleware::logger: 192.168.127.1 "PUT /collections/pippo HTTP/1.1" 400 67 "-" "curl/8.1.2" 0.000259

Using python, like I wrote earlier I do

from qdrant_client import QdrantClient
from qdrant_client.http import models

client = QdrantClient("localhost", port=6333)
vectors_config=models.VectorParams(size=100, distance=models.Distance.COSINE)
client.create_collection("pippo", vectors_config)

And I get

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/pippo/Library/Caches/pypoetry/virtualenvs/test1-2gxCvkVu-py3.11/lib/python3.11/site-packages/qdrant_client/qdrant_client.py", line 1411, in create_collection
    return self._client.create_collection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pippo/Library/Caches/pypoetry/virtualenvs/test1-2gxCvkVu-py3.11/lib/python3.11/site-packages/qdrant_client/qdrant_remote.py", line 1800, in create_collection
    result: Optional[bool] = self.http.collections_api.create_collection(
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pippo/Library/Caches/pypoetry/virtualenvs/test1-2gxCvkVu-py3.11/lib/python3.11/site-packages/qdrant_client/http/api/collections_api.py", line 1024, in create_collection
    return self._build_for_create_collection(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pippo/Library/Caches/pypoetry/virtualenvs/test1-2gxCvkVu-py3.11/lib/python3.11/site-packages/qdrant_client/http/api/collections_api.py", line 82, in _build_for_create_collection
    return self.api_client.request(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pippo/Library/Caches/pypoetry/virtualenvs/test1-2gxCvkVu-py3.11/lib/python3.11/site-packages/qdrant_client/http/api_client.py", line 74, in request
    return self.send(request, type_)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/pippo/Library/Caches/pypoetry/virtualenvs/test1-2gxCvkVu-py3.11/lib/python3.11/site-packages/qdrant_client/http/api_client.py", line 97, in send
    raise UnexpectedResponse.for_response(response)
qdrant_client.http.exceptions.UnexpectedResponse: Unexpected Response: 500 (Internal Server Error)
Raw response content:
b'{"status":{"error":"Service internal error: Can\'t init WAL: Os { code: 22, kind: InvalidInput, message: \\"Invalid argument\\" }"},"time":0.356285809}'

and on the logs I see

2023-11-29T15:35:45.158805Z  WARN qdrant::actix::helpers: error processing request: Can't init WAL: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }
2023-11-29T15:35:45.160151Z  INFO actix_web::middleware::logger: 192.168.127.1 "PUT /collections/pippo HTTP/1.1" 500 148 "-" "python-httpx/0.25.2" 0.357267

[joein]

Could you also show the output of ?

cd qdrant_storage/collections
ls -1 | xargs ls -1

[mfirry]

Nothing. I think I might know why.
You suggest to link the volume with :/qdrant/storage:z but this won't work for me. (see containers/podman-compose#509 (comment))

I get Error: preparing container bb505c451dab093b589785592d633b678c84fe07dfcf0eefc35962d6c27871f3 for attach: lsetxattr /Users/pippo/Desktop/fair/aliases: operation not supported when I do it.

So I read somewhere that I had to remove the z which might be the cause (somehow).

[joein]

I don't think that :z should cause this kind of problems

An interesting thing is that qdrant sees some collections directories in your qdrant_storage/collections on start-up according to the log

2023-11-29T15:32:09.821274Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/16a28419", skipping
2023-11-29T15:32:09.822043Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/2db5aa02", skipping
2023-11-29T15:32:09.822691Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/39c8cc60", skipping
2023-11-29T15:32:09.823327Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/e948fc28", skipping
2023-11-29T15:32:09.823992Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/714fb1a6", skipping
2023-11-29T15:32:09.824637Z  WARN storage::content_manager::toc: Collection config is not found in the collection directory: "./storage/collections/b19d3d1a", skipping

But according to you, collections are empty

I guess, Qdrant used to work, but after some actions and reboot it stopped working, am I right?

It seems like there were some problems with volumes or with write permissions

[joein]

I will try to reproduce with podman tomorrow

[mfirry]

I'll trash the current image/container and try again from scratch. Thanks for your time.

[mfirry]

Oh my. I ls-ed the wrong folder.
Here it comes:

$ ls collections
0fc9f42f 16a28419 2d633f4e 2db5aa02 39c8cc60 3c72528e 45bea5ec 4fc159ce 714fb1a6 9c0f037f a35b40dd ab1b9ab6 ab34b016 ad1a62ba b19d3d1a e5df4689 e948fc28 pippo

and for example

$ ls collections/pippo/0
segments wal

[joein]

seems a little bit empty :(

for me

collections % ls
2345rsd	2345t	234df	wefg654

collections % cd 2345rsd

2345rsd % ls
0		config.json	version.info

0 % ls
replica_state.json	segments		shard_config.json	wal

0 % cd wal

wal % ls -lah
total 72
drwxr-xr-x  5 user  staff   160B Nov 29 18:16 .
drwxr-xr-x  6 user  staff   192B Nov 29 18:16 ..
-rw-r--r--  1 user  staff    15B Nov 29 18:16 first-index
-rw-r--r--  1 user  staff    32M Nov 29 18:16 open-1
-rw-r--r--  1 user  staff    32M Nov 29 18:16 open-2

Do you have only directories in there or some files exist as well ?

(I am just gathering more information so I could work on it tomorrow, if we won't solve it now)

[mfirry]

$  tree collections
collections
|-- 0fc9f42f
|-- 16a28419
|-- 2d633f4e
|   `-- 0
|       |-- segments
|       |   |-- 02405a1e-43a0-4aab-a914-d4700f447b87
|       |   |   |-- 000004.log
|       |   |   |-- CURRENT
|       |   |   |-- IDENTITY
|       |   |   |-- LOCK
|       |   |   |-- LOG
|       |   |   |-- MANIFEST-000005
|       |   |   |-- OPTIONS-000013
|       |   |   |-- OPTIONS-000015
|       |   |   |-- payload_index
|       |   |   |   |-- 000004.log
|       |   |   |   |-- CURRENT
|       |   |   |   |-- IDENTITY
|       |   |   |   |-- LOCK
|       |   |   |   |-- LOG
|       |   |   |   |-- MANIFEST-000005
|       |   |   |   |-- OPTIONS-000007
|       |   |   |   `-- config.json
|       |   |   |-- segment.json
|       |   |   `-- version.info
|       |   `-- 916eeb55-a94f-409a-9146-ad99434330e4
|       |       |-- 000004.log
|       |       |-- CURRENT
|       |       |-- IDENTITY
|       |       |-- LOCK
|       |       |-- LOG
|       |       |-- MANIFEST-000005
|       |       |-- OPTIONS-000013
|       |       |-- OPTIONS-000015
|       |       |-- payload_index
|       |       |   |-- 000004.log
|       |       |   |-- CURRENT
|       |       |   |-- IDENTITY
|       |       |   |-- LOCK
|       |       |   |-- LOG
|       |       |   |-- MANIFEST-000005
|       |       |   |-- OPTIONS-000007
|       |       |   `-- config.json
|       |       |-- segment.json
|       |       `-- version.info
|       `-- wal
|           `-- tmp-open-1
|-- 2db5aa02
|   `-- 0
|       |-- segments
|       |   |-- 556d3817-428e-4f8f-a9ac-448dd7ebcda6
|       |   |   |-- 000004.log
|       |   |   |-- CURRENT
|       |   |   |-- IDENTITY
|       |   |   |-- LOCK
|       |   |   |-- LOG
|       |   |   |-- MANIFEST-000005
|       |   |   |-- OPTIONS-000013
|       |   |   |-- OPTIONS-000015
|       |   |   |-- payload_index
|       |   |   |   |-- 000004.log
|       |   |   |   |-- CURRENT
|       |   |   |   |-- IDENTITY
|       |   |   |   |-- LOCK
|       |   |   |   |-- LOG
|       |   |   |   |-- MANIFEST-000005
|       |   |   |   |-- OPTIONS-000007
|       |   |   |   `-- config.json
|       |   |   |-- segment.json
|       |   |   `-- version.info
|       |   `-- 8d69ec9b-e4f1-48a0-8ae4-50f9fd488e56
|       |       |-- 000004.log
|       |       |-- CURRENT
|       |       |-- IDENTITY
|       |       |-- LOCK
|       |       |-- LOG
|       |       |-- MANIFEST-000005
|       |       |-- OPTIONS-000013
|       |       |-- OPTIONS-000015
|       |       |-- payload_index
|       |       |   |-- 000004.log
|       |       |   |-- CURRENT
|       |       |   |-- IDENTITY
|       |       |   |-- LOCK
|       |       |   |-- LOG
|       |       |   |-- MANIFEST-000005
|       |       |   |-- OPTIONS-000007
|       |       |   `-- config.json
|       |       |-- segment.json
|       |       `-- version.info
|       `-- wal
|           `-- tmp-open-1

[mfirry]

hi @joein did you perhaps find the time to try podman?

[joein]

Hi @mfirry

Not yet :( will try to do it at the weekends

[joein]

This problem is reproducible with podman, however with docker everything works as intended 🙃

When I tried to create a collection in Qdrant which is running inside a podman container, it failed with:

2023-12-03T19:55:46.531009Z DEBUG wal: Wal { path: "./storage/collections/first/0/wal" }: opening    
2023-12-03T19:55:46.573062Z  WARN qdrant::actix::helpers: error processing request: Can't init WAL: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }    

When I tried to create a collection in Qdrant which is running inside a docker container, it succeeded with

2023-12-03T19:54:37.243586Z DEBUG wal: Wal { path: "./storage/collections/first/0/wal" }: opening
2023-12-03T19:54:37.245455Z DEBUG wal::segment: Segment { path: "./storage/collections/first/0/wal/open-1", flush_offset: 0, entries: 0, space: (8/33554432) }: created
2023-12-03T19:54:37.245577Z  INFO wal: Wal { path: "./storage/collections/first/0/wal", segment-count: 1, entries: [0, 0)  }: opened
2023-12-03T19:54:37.246843Z DEBUG wal::segment: Segment { path: "./storage/collections/first/0/wal/open-2", flush_offset: 0, entries: 0, space: (8/33554432) }: created
2023-12-03T19:54:37.254079Z DEBUG collection::collection: Changing shard first:0 replica state from Some(Initializing) to Active
2023-12-03T19:54:37.254095Z DEBUG collection::shards::replica_set: Changing local shard first:0 state from Some(Initializing) to Active

[bashofmann]

I get the same error with the unprivileged image:

podman run -p 6333:6333 -p 6334:6334 -v ~/tmp/qdrant_storage:/qdrant/storage --userns=keep-id:uid=1000,gid=1000 qdrant/qdrant:latest-unprivileged

I found this issue here containers/podman#17560, so I tried the --userns argument.

As soon as I try to add data:

2023-12-06T11:37:28.884705Z  WARN qdrant::actix::helpers: error processing request: Can't init WAL: Os { code: 22, kind: InvalidInput, message: "Invalid argument" }

I don't think it's "only" a simple permission issue. The volume actually contains data:

qdrant@7fb40fb4ef7b:/qdrant/storage$ ls -la
total 4
drwxr-xr-x. 5 qdrant nogroup 160 Dec  6 11:37 .
drwxr-xr-x. 1 qdrant qdrant   65 Dec  6 11:37 ..
drwxr-xr-x. 3 qdrant nogroup  96 Dec  6 11:37 aliases
drwxr-xr-x. 3 qdrant nogroup  96 Dec  6 11:37 collections
-rw-r--r--. 1 qdrant nogroup 299 Dec  6 11:37 raft_state.json

So qdrant seems to be able to write to it.