pip3 search finds only libraries with whitespace/dash after search term

[meridsa]

Environment

• pip version: 19.0.3
• Python version: 3.7.1
• OS: macOS Mojave 10.14.3
• Terminal: Default apple Terminal: 2.9.1

Description
Searching a term with pip3 search [foo] will only return libraries or descriptions with [foo] followed by whitespace or dash. Whilst any prefixes or upper/lower combination will be returned. I am wondering if this is a result of the terminal adding a whitespace after any command being entered.

Expected behavior
Returning all libraries that include the term in anyway in the description or name.

How to Reproduce
Search any term you know is included/vital in a library but does not end in the term.
Ex: pip3 search wx -> fails to return wxPython

This issue was raised in pip repository, was told this was the more appropriate repo for raising this issue.

[yeraydiazdiaz]

For reference: pypa/pip#6309

[di]

Seems like the actual issue here is that they query is for a package for which the package name on PyPI has mixed case. This is reproducible for pip search django (Django) and pip search sqlalchemy (SQLAlchemy) but not for pip search tensorflow, pip search twine or pip search pip, etc.

[yeraydiazdiaz]

I've had a look at this and it seems pip search is partially to blame for this behaviour.

The search command adds makes an XMLRPC request with a spec passing the term as the name and summary fields along with an or parameter (https://github.com/pypa/pip/blob/master/src/pip/_internal/commands/search.py#L65).

In warehouse the procedure is using a bool query with a should clause which does return an flavor of logical OR but ranks the results by score.

Seems the fact that "derived" packages tend to include the original framework's name in the summary makes them rank higher in the results and, if there's enough of them, push the upstream package out of the top 100 results.

Given the last commit for the line is 5 years old pip has been including the summary field for quite some time. Is this something that's different from the legacy implementation of the XMLRPC search? Is it worth recreating that behaviour?

/cc @pfmoore @pradyunsg

[stephen-dexda]

Behaviour seems to be just that pip search only finds "whole word" matches, where a "word" is a string delimited by spaces or hyphens (but not e.g. underscores - pip search prometheus will not find prometheus_client).

[McSinyx]

Coming from the pip's issue, is it sensible for warehouse to provide a wildcard search API, e.g. using regex or wildcard search term? I imagine that then pip search can use this new query for the name field and maybe stick with the old match for summary.

[socketpair]

$ pip3 search uring
io-uring (0.0.1)               - This is a light-weight python wrapper around "io_uring" linux library.
pycopy-cpython-ure (0.2)       - Pycopy module ure ported to CPython
micropython-cpython-ure (0.1)  - MicroPython module ure ported to CPython

$ pip3 search liburing
liburing (2020.7.13)  - This is a Python + CFFI wrapper around Liburing C library, which is a helper to setup and tear-down io_uring
                        instances.

You know what? I discovered liburing PIP package only using google! Because neither pip website nor pip search found it!

Much more interesting thing, is that website returns 14 results for the same query. WHY.

[socketpair]

Another question is about pycopy-cpython-ure. It DOES NOT contain uring. What's happening ?!

[woodruffw]

Triaging: PyPI no longer supports the XMLRPC API that backs (backed?) pip search, so this is no longer directly applicable.

xref pypa/pip#5216 for some relevant discussion about the future of pip search itself.

xref #6236 as well.

[brainwane]

I second the reference to the search meta issue #6236 -- I think it would be good if further work in Python packaging user experience research included some serious investigation into how people discover/search/browse packages/projects, and hope the comments in this issue inform that work.