MCP for rankllm

[sahel-sh]

[lilyjge]

I can work on this!

[lilyjge]

With how retrieval functionalities already exist with MCPyserini, we have several options on how to do this:

• Copying over the existing MCPyserini controller logic + new RankLLM tools in a distinct MCP RankLLM
  • Pros: Separates completely from MCPyserini and we can tweak certain methods to fit RankLLM as needed, e.g. returning output of the search method in RankLLM format directly
  • Cons: code duplication and any updates that might be useful for retrieval on MCPyserini side would have to be copied over manually
• Importing MCPyserini's controller methods and redeclaring all the tools + new RankLLM tools in MCP RankLLM
  • Pros: Easy integration with MCPyserini's retrieval capabilities, not duplicating the logic code
  • Cons: Harder to customize existing MCPyserini tools, tool declaration is still duplicated, susceptible to API changes on MCPyserini
• Integrating RankLLM into MCPyserini
  • Pros: No code duplication, everything happens in one place
  • Cons: Pyserini having RankLLM as a dependency
• Separate MCP repo
  • Pros: Clean, same pros as above
  • Cons: A whole extra repo to deal with, needs refactoring
• MCP completely in RankLLM
  • Pros: Maintains current pattern of RankLLM using Pyserini, not the other way around, no code duplication
  • Cons: Pyserini is more widely used

RankLLM's current strategy for integration with Pyserini is probably closest to 'Importing MCPyserini's controller methods and redeclaring all the tools + new RankLLM tools in MCP RankLLM', with how RankLLM imports and uses Pyserini's searchers. However, MCP differs with that a lot of the usage/server code exists already in MCPyserini as well... but perhaps that duplication is unavoidable?

Bonus:

• Interestingly, RankLLM's pyserini_retriever.py seems to share similarities with MCPyserini's search method. Could we extract this logic and put it into Pyserini's util or search or somewhere, and MCPyserini and RankLLM can both wrap it? This is very similar to what was mentioned before, except with a bonus layer of abstraction that makes it 'cleaner', since we could have two MCP servers both wrapping the same logic instead of one server wrapping another's logic, e.g. MCPyserini would import from pyserini.util.tools (random placeholder name) and RankLLM would import from rank_llm.retrieve.PyseriniRetriever which imports from pyserini.util.tools
  • Pros: Clean(?)
  • Cons: Refactoring, a lot.

[sahel-sh]

There are two different topics here, and they should be addressed separately.
1- How is rankllm using pyserini under the hood:
There is the pyserini_retriever now, and the interactive mode using service_retriever (based on Anserini REST which no longer exists).
Here we should add support for MCP integration with pyserini. i.e, be able to send requests to an already running Pyserini MCP server.

2- rankllm functionality as an API
This is where we do retrieval first as a client of anserini rest API, then do the reranking.
Now we want an MCP server for this part, that is in turn a client of Pyserini MCP.

@lilyjge all your suggested options have the assumption that rankllm needs to have support for pyserini MCP. This is not needed, rankllm only needs to use the pyserini MCP as a client. (Similar to using the Anserini REST API as a client)

[sahel-sh]

Interestingly, RankLLM's pyserini_retriever.py seems to share similarities with MCPyserini's search method. Could we extract this logic and put it into Pyserini's util or search or somewhere, and MCPyserini

Nice observation! The pyserini_retriever indeed seems like a super set of the search tool defined for the pyserini MCP.
+1 to importing this logic into Pyserini, so that both rankllm and the MCPyserini can use it.