Trend Anomalies in Ruler V2

[zhejunliux]

Content

When using GPT-4.1 to evaluate data with 256k and 512k configurations in Ruler V2, the following two issues were observed:

1. nv-medium scores higher than easy. 67 vs 64, 62 vs 50
2. qa_hard scores higher than qa-medium. 84 vs 81, 83 vs 79

trend

256k
 mk_niah
|████ Basic (100%) |████ Easy (94.83%) |███ Medium (90%) |██ Hard (84%)

mv_niah
|████ Basic (84.25%) |███ Easy (64.67%) |███ Medium (67.72%) |█ Hard (45.12%)

qa
|████ Basic (100%) |████ Easy (88%) |███ Medium (81%) |██ Hard (84%)


512k
 mk_niah
|████ Basic (98%) |████ Easy (90.44%) |███ Medium (85%) |██ Hard (82%)

 mv_niah
|███ Basic (74.25%) |█ Easy (50.57%) |██ Medium (62.71%) |▌ Hard (43.87%)

 qa
|████ Basic (97%) |███ Easy (81%) |██ Medium (79%) |██ Hard (83%)

hello @hsiehjackson Are there anomalies in the data generation strategy or evaluation metrics in nemo_skills/dataset/ruler2/prepare.py?

How to close

1. How can this issue be fixed to restore normal trends?

[hsiehjackson]

Hi @zhejunliux,

Thanks for using RULERv2 for evaluation. Here are my results when evaluating GPT 4.1. When evaluating MV NIAH medium level, can you double check the scores only take the last item like here? For QA, we also saw some models have higher scores on hard level than medium level. We think it is because model can ignore the context to answer the question in hard level but need to find the correct context first to answer the question in medium level. We think finding correct information and answering the question should be easier; however, in reality, model may not use the context.

256K	Basic	Easy	Medium	Hard
MK NIAH	99.0	100.0	78.0	74.0
MV NIAH	80.0	83.9	55.8	46.0
QA	99.0	89.0	81.0	81.0
512K	Basic	Easy	Medium	Hard
MK NIAH	97.0	96.6	86.0	66.0
MV NIAH	69.0	74.6	42.3	37.3
QA	96.0	78.0	73.0	78.0

[zhejunliux]

hello @hsiehjackson thanks your answer.
MV NIAH medium is considered correct, and QA agrees with your opinion.

It seems that for MV NIAH Easy, you used the "part" matching method, right?
However, the code uses "all". Please take a look at this section of code.

def prepare_task_for_ns(output_folder, task):
    """Adding proper __init__.py"""
    output_folder = Path(output_folder) / task
    Path(output_folder).mkdir(parents=True, exist_ok=True)
    with open(output_folder / "__init__.py", "w", encoding="utf-8") as init_file:
        if task in ["mk_niah_medium", "mk_niah_hard"]:
            metrics_type = "multichoice"
            eval_args = "++eval_type=multichoice"
        elif task in ["mv_niah_medium"]:
            metrics_type = "ruler2"
            eval_args = "++eval_type=ruler2 ++eval_config.match_type=2steps"
        elif "qa" in task:
            metrics_type = "ruler2"
            eval_args = "++eval_type=ruler2 ++eval_config.match_type=part"
        else:
            metrics_type = "ruler2"
            eval_args = "++eval_type=ruler2 ++eval_config.match_type=all"

Here, the string_match_all_single function has been revised as follows, and it was found that in "all", the edit distance does not take effect.

    def string_match_all_single_fixed(self, predsx, refs):
        """the metric function with input (predictions: [str], references: [[str]]) to compute score."""

        preds = predsx.split("\n\n")
        print(len(preds))
        print(len(refs))

        conscr = list()
        for ref in refs:
            score = 0
            if ref in predsx:
                score = 1
            conscr.append(score)
        
        scavg = sum(conscr) / len(refs)
        ww = list()
        for ref in refs:
            wx  = list()
            for pred in preds:
                wresc = 1 - self.wer([pred.lower()], [ref.lower()])
                wx.append(wresc)
            wm = max(wx)
            ww.append(wm)
        wscr = sum(ww) / len(refs)
        score = max(scavg, wscr)
        return score

Final score

512: mv_niah
|███ Basic (74.25%) |█ Easy 0.7743) |██ Medium (62.71%) |▌ Hard (43.87%)
1024:mv_niah
mv_niah Basic 67.75% mv_niah_basic
mv_niah Easy 0.541% mv_niah_easy
mv_niah Medium 44.76% mv_niah_medium
mv_niah Hard 23.59% mv_niah_hard

[hsiehjackson]

It seems that for MV NIAH Easy, you used the "part" matching method, right?
However, the code uses "all". Please take a look at this section of code.

Yeap, we should use "all" for evaluation since we need to match all answers. For example, if you have 4 needles, you need to match 4 instances in response.

512 mv_niah (your 1st message)
|███ Basic (74.25%) |█ Easy (50.57%) |██ Medium (62.71%) |▌ Hard (43.87%)

512: mv_niah (your 2nd message)
|███ Basic (74.25%) |█ Easy 0.7743) |██ Medium (62.71%) |▌ Hard (43.87%)

Did you change metric to evaluate Easy level?

[zhejunliux]

yeap, use

def string_match_all_single_fixed(self, predsx, refs):
        """the metric function with input (predictions: [str], references: [[str]]) to compute score."""

        preds = predsx.split("\n\n")
        print(len(preds))
        print(len(refs))

        conscr = list()
        for ref in refs:
            score = 0
            if ref in predsx:
                score = 1
            conscr.append(score)
        
        scavg = sum(conscr) / len(refs)
        ww = list()
        for ref in refs:
            wx  = list()
            for pred in preds:
                wresc = 1 - self.wer([pred.lower()], [ref.lower()])
                wx.append(wresc)
            wm = max(wx)
            ww.append(wm)
        wscr = sum(ww) / len(refs)
        score = max(scavg, wscr)
        return score

[hsiehjackson]

preds = predsx.split("\n\n") should be only used for medium level since we only verify final item is correct or not. For easy level, we should remove this because we want to make sure all needles are in the model response without post-processing. I expect when you remove that we won't see the extremely low score you provided.