Bug Fix: False Negatives in LeetCode and AoPS

[545999961]

I would like to express my sincere appreciation for the authors and contributors of this benchmark. The benchmark provides a valuable contribution to the community by providing a rigorous and standardized evaluation framework. Your work greatly facilitates research progress in this important area.

After going through the datasets, I had a couple of observations and questions regarding the LeetCode and AoPS datasets, which I hope could be helpful for further refinement:

• LeetCode

  • It appears that the inclusion of the CodeSearchNet corpus within the LeetCode task may inadvertently introduce certain false negatives.
  • Would it be feasible or beneficial to consider utilizing a LLM to identify the topical distribution within the CodeSearchNet corpus, followed by applying the filtering approach proposed in the paper? Such an approach may help to refine the dataset construction, thereby potentially improving the overall robustness of the benchmark on this task.

• AoPS

  • The paper mentions that multiple topics are filtered when constructing excluded_ids for AoPS.
  • However, it seems that, in the current implementation, only the first topic is filtered while the remaining topics may be omitted.
  • Might it be possible to incorporate the entirety of the relevant topic list into excluded_ids? This could enhance the thoroughness of the exclusion documents and strengthen the robustness of the evaluation results.

Once again, thank you for your significant contributions and efforts. I look forward to any insights you might share regarding these points and hope these suggestions prove useful in ongoing and future work.