Skip to content

model: add perplexity-ai/pplx-embed-v2-late - #5606

Merged
Samoed merged 1 commit into
embeddings-benchmark:mainfrom
whybe-choi:model/pplx-embed-v2-late
Oct 8, 2026
Merged

Samoed merged 1 commit into
embeddings-benchmark:mainfrom
whybe-choi:model/pplx-embed-v2-late

Conversation

@whybe-choi

@whybe-choi whybe-choi commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Closes #5602

Checklists

  • I have filled out the ModelMeta object to the extent possible
  • I have ensured that my model can be loaded using
    • mteb.get_model(model_name, revision) and
    • mteb.get_model_meta(model_name, revision)
  • I have tested the implementation works on a representative set of tasks.
  • The model is public, i.e., is available either as an API or the weights are publicly available to download
  • I reproduced results from the original paper (if applicable) on at least one benchmark, and I am including the results in the PR description.

Results

  • perplexity-ai/pplx-embed-v2-late-0.6b
Task Reported Reprod Diff
Finance (en) 68.7 67.2 -1.5
Finance (fr) 51.1 50.5 -0.6
Industrial 53.1 51.9 -1.2
Pharmaceuticals 66.6 66.5 -0.1
Computer Science 76.4 75.0 -1.4
HR 65.2 64.4 -0.8
Energy 69.7 69.8 +0.1
Physics 47.7 47.5 -0.2
Average 62.3 61.6 -0.7
  • perplexity-ai/pplx-embed-v2-late-9b
Task Reported Reprod Diff
Finance (en) 71.8 71.6 -0.2
Finance (fr) 53.9 53.9 +0.0
Industrial 56.4 56.2 -0.2
Pharmaceuticals 69.2 68.9 -0.3
Computer Science 79.8 79.5 -0.3
HR 68.0 67.7 -0.3
Energy 71.6 71.7 +0.1
Physics 50.5 50.1 -0.4
Average 65.2 65.0 -0.2

@Samoed

Samoed commented Oct 8, 2026

Copy link
Copy Markdown
Member

Cc @NohTow

Comment thread uv.lock

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How did you generate lock? We have an issue that it's not worked with current uv version #5482. I use compiled version from draft ai pr astral-sh/uv-dev#2107 that's what created in my issue astral-sh/uv#21954

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran a plain uv lock (no flags, no cache clearing) with the stock releaseuv 0.12.0 (x86_64-unknown-linux-gnu), after adding the extra and its entry in [tool.uv] conflicts.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm, strange. We had an issue with endless lock

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It took about 20 minutes. As noted in #5482, that was with the existing uv.lock in place, not a cold resolution, which is probably why it finished.

@whybe-choi
whybe-choi marked this pull request as ready for review October 8, 2026 08:49
@Samoed

Samoed commented Oct 8, 2026

Copy link
Copy Markdown
Member

@whybe-choi or @NohTow can you reproduce results of this configuration?

@whybe-choi

Copy link
Copy Markdown
Contributor Author

I've already posted the reproduction results in the PR description above

@Samoed Samoed left a comment •

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, didn't see update. Looks good now, thank you for addition! Can you submit results?

@Samoed
Samoed merged commit 0e233f7 into embeddings-benchmark:main Oct 8, 2026
16 checks passed
@Samoed Samoed added the new model Questions related to adding a new model to the benchmark label Oct 8, 2026
@whybe-choi

Copy link
Copy Markdown
Contributor Author

Sure, I will submit the results! I've also already finished evaluating the models on MIRACL-VISION and other Korean VDR tasks alongside ViDoRe v3, so I will submit those together as well.

@Samoed

Samoed commented Oct 10, 2026 •

Copy link
Copy Markdown
Member

A small note. During the investigation from the Perplexity team it turned out that the difference in scores comes from using the FastPlaid index.

Results for 0.6b model:

Task FastPlaid FP16 brute force
Computer Science 76.29 75.08
Energy 69.71 69.49
Finance EN 68.66 67.58
Finance FR 51.06 51.28
HR 65.12 63.85
Industrial 53.14 52.71
Pharmaceuticals 66.50 66.46
Physics 47.57 46.69
Average 62.26 61.64

Brobably more diff comes from fp32 search

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

new model Questions related to adding a new model to the benchmark

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add model: perplexity-ai/pplx-embed-v2-late

2 participants