Merged
Conversation
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR fixes lint issues in the CLI module by standardizing quote usage and formatting. The changes focus on ensuring consistent code style by converting single quotes to double quotes and improving argument formatting.
- Standardized all string literals to use double quotes instead of single quotes
- Added trailing commas to function arguments and parameter lists for consistency
- Consolidated multi-line argument definitions to single lines where appropriate
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
ofermend
added a commit
that referenced
this pull request
Sep 5, 2025
* Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Added support to process queries in parallel across all connectors. * Version bumped version to 0.1.7 (#110) fixed lint issues and bumped version to 0.1.7 * Merge conflict 1 (#112) * Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> * After rebase1 (#116) * Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * initial * fixed issues from copilot review * Added support to process queries in parallel across all connectors. * Updated CitationMetric * Version bumped version to 0.1.7 (#110) fixed lint issues and bumped version to 0.1.7 * Merge conflict 1 (#112) * Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> * bugfix --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> Co-authored-by: Vishal Naik <naik.vishalishwar@gmail.com> * Fixed lint issues (#127) * Add LLM Judges for Anthropic and Together * Remove Bad kwargs for Anthropic Models * Update structured output method for TogetherModel * Add keyword handling for gemini models * Fix PR Comments * Changed version to 0.2.1 (#132) * some improvements, esp for Together.AI models, version bump etc * updated * bugfix in unit tests * minor updates * Add anthropic and together requirements, config example for llama_index * added TRANSFORMER VERBOSITY override --------- Co-authored-by: david-oplatka <david.oplatka@vectara.com> * added CLI options so that users don't have to clone the repo (#134) * added CLI options so that users don't have to clone the repo * version bump * removing unused imports * fixed lint issues (#135) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Vishal Naik <naik.vishalishwar@gmail.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> Co-authored-by: david-oplatka <david.oplatka@vectara.com>
ofermend
added a commit
that referenced
this pull request
Oct 27, 2025
* Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Added support to process queries in parallel across all connectors. * Version bumped version to 0.1.7 (#110) fixed lint issues and bumped version to 0.1.7 * Merge conflict 1 (#112) * Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> * After rebase1 (#116) * Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * initial * fixed issues from copilot review * Added support to process queries in parallel across all connectors. * Updated CitationMetric * Version bumped version to 0.1.7 (#110) fixed lint issues and bumped version to 0.1.7 * Merge conflict 1 (#112) * Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> * bugfix --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> Co-authored-by: Vishal Naik <naik.vishalishwar@gmail.com> * Fixed lint issues (#127) * Add LLM Judges for Anthropic and Together * Remove Bad kwargs for Anthropic Models * Update structured output method for TogetherModel * Add keyword handling for gemini models * Fix PR Comments * Changed version to 0.2.1 (#132) * some improvements, esp for Together.AI models, version bump etc * updated * bugfix in unit tests * minor updates * Add anthropic and together requirements, config example for llama_index * added TRANSFORMER VERBOSITY override --------- Co-authored-by: david-oplatka <david.oplatka@vectara.com> * added CLI options so that users don't have to clone the repo (#134) * added CLI options so that users don't have to clone the repo * version bump * removing unused imports * fixed lint issues (#135) * added METRICS guide (#141) * added METRICS guide * updates based on Vish suggestions * updated * fixed metrics.md * fix(eval): omit empty consistency field in results.json output (#139) * fix(eval): omit empty consistency field in results.json output - Fixed issue where empty `consistency` metrics were still written to results.json - Ensured that `results.json` output only includes non-empty consistency fields - Added unit test `test_results_json_consistency_field.py` to validate the fix - Introduced `requirements-dev.txt` with a full developer toolchain (pytest, linting, pre-commit) * chore(eval): make omit-empty-consistency non-mutating; de-flake test metric name - Return a filtered copy of the report (no in-place mutation) - Make test use a generic non-empty payload instead of a specific metric key * docs(eval): clarify docstring for _omit_empty_consistency to reflect true behavior - Updated docstring to specify that 'consistency' is removed only when present and falsy * update requirements to avoid test failure (#142) * Minimal fix to .dockerignore (#143) * added query generation capability (#144) * added query generation capability * fixed lint issues * added progress bar * updated to work with HF_TOKEN for gated HHEM * updated test action * updated per Tallat suggestions * updated output formatter * version bump * updated for lint * fixed typos --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Vishal Naik <naik.vishalishwar@gmail.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> Co-authored-by: david-oplatka <david.oplatka@vectara.com> Co-authored-by: Sai Vignesh <svnaragoni@gmail.com> Co-authored-by: Tomoki Nakamaru <tomoki.nakamaru@gmail.com>
ofermend
added a commit
that referenced
this pull request
Nov 18, 2025
* Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Added support to process queries in parallel across all connectors. * Version bumped version to 0.1.7 (#110) fixed lint issues and bumped version to 0.1.7 * Merge conflict 1 (#112) * Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> * After rebase1 (#116) * Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * initial * fixed issues from copilot review * Added support to process queries in parallel across all connectors. * Updated CitationMetric * Version bumped version to 0.1.7 (#110) fixed lint issues and bumped version to 0.1.7 * Merge conflict 1 (#112) * Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> * bugfix --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> Co-authored-by: Vishal Naik <naik.vishalishwar@gmail.com> * Fixed lint issues (#127) * Add LLM Judges for Anthropic and Together * Remove Bad kwargs for Anthropic Models * Update structured output method for TogetherModel * Add keyword handling for gemini models * Fix PR Comments * Changed version to 0.2.1 (#132) * some improvements, esp for Together.AI models, version bump etc * updated * bugfix in unit tests * minor updates * Add anthropic and together requirements, config example for llama_index * added TRANSFORMER VERBOSITY override --------- Co-authored-by: david-oplatka <david.oplatka@vectara.com> * added CLI options so that users don't have to clone the repo (#134) * added CLI options so that users don't have to clone the repo * version bump * removing unused imports * fixed lint issues (#135) * added METRICS guide (#141) * added METRICS guide * updates based on Vish suggestions * updated * fixed metrics.md * fix(eval): omit empty consistency field in results.json output (#139) * fix(eval): omit empty consistency field in results.json output - Fixed issue where empty `consistency` metrics were still written to results.json - Ensured that `results.json` output only includes non-empty consistency fields - Added unit test `test_results_json_consistency_field.py` to validate the fix - Introduced `requirements-dev.txt` with a full developer toolchain (pytest, linting, pre-commit) * chore(eval): make omit-empty-consistency non-mutating; de-flake test metric name - Return a filtered copy of the report (no in-place mutation) - Make test use a generic non-empty payload instead of a specific metric key * docs(eval): clarify docstring for _omit_empty_consistency to reflect true behavior - Updated docstring to specify that 'consistency' is removed only when present and falsy * update requirements to avoid test failure (#142) * Minimal fix to .dockerignore (#143) * added query generation capability (#144) * added query generation capability * fixed lint issues * added progress bar * updated to work with HF_TOKEN for gated HHEM * updated test action * updated per Tallat suggestions * updated output formatter * version bump * updated for lint * fixed typos * Add configurable question type distribution for query generation (#146) * now query generation can be configured to control the % of questions per category (roughly) * updated to read env from .env file * Update open_rag_eval/query_generation/llm_generator.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * robust handling when no assignment scores exist * upgraded transformers to remove security vulnerability * updated --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add HHEM API support and fix Anthropic model issues (#147) * initial support for Vectara HHEM via API * minor updates * fixed bug with using Anthropic model * a few bug fixes and esp with Anthropic model usage * added tests * added langchain to requirements * fixed issue with langchain * fixed issue with torch meta-score and BertScore incompatibility * issue with bert score in consistency score around meta-device incompatability to resolve * reverted back to v 4.50.2 of transformers * moved from bert_score to torchmetrics which is more frequently maintained * added max_length to BERT score to avoid going over the model sequence length (truncate if that happens) * fixed unit test --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Vishal Naik <naik.vishalishwar@gmail.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> Co-authored-by: david-oplatka <david.oplatka@vectara.com> Co-authored-by: Sai Vignesh <svnaragoni@gmail.com> Co-authored-by: Tomoki Nakamaru <tomoki.nakamaru@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
ofermend
added a commit
that referenced
this pull request
Dec 15, 2025
* Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Added support to process queries in parallel across all connectors. * Version bumped version to 0.1.7 (#110) fixed lint issues and bumped version to 0.1.7 * Merge conflict 1 (#112) * Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> * After rebase1 (#116) * Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * initial * fixed issues from copilot review * Added support to process queries in parallel across all connectors. * Updated CitationMetric * Version bumped version to 0.1.7 (#110) fixed lint issues and bumped version to 0.1.7 * Merge conflict 1 (#112) * Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> * bugfix --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> Co-authored-by: Vishal Naik <naik.vishalishwar@gmail.com> * Fixed lint issues (#127) * Add LLM Judges for Anthropic and Together * Remove Bad kwargs for Anthropic Models * Update structured output method for TogetherModel * Add keyword handling for gemini models * Fix PR Comments * Changed version to 0.2.1 (#132) * some improvements, esp for Together.AI models, version bump etc * updated * bugfix in unit tests * minor updates * Add anthropic and together requirements, config example for llama_index * added TRANSFORMER VERBOSITY override --------- Co-authored-by: david-oplatka <david.oplatka@vectara.com> * added CLI options so that users don't have to clone the repo (#134) * added CLI options so that users don't have to clone the repo * version bump * removing unused imports * fixed lint issues (#135) * added METRICS guide (#141) * added METRICS guide * updates based on Vish suggestions * updated * fixed metrics.md * fix(eval): omit empty consistency field in results.json output (#139) * fix(eval): omit empty consistency field in results.json output - Fixed issue where empty `consistency` metrics were still written to results.json - Ensured that `results.json` output only includes non-empty consistency fields - Added unit test `test_results_json_consistency_field.py` to validate the fix - Introduced `requirements-dev.txt` with a full developer toolchain (pytest, linting, pre-commit) * chore(eval): make omit-empty-consistency non-mutating; de-flake test metric name - Return a filtered copy of the report (no in-place mutation) - Make test use a generic non-empty payload instead of a specific metric key * docs(eval): clarify docstring for _omit_empty_consistency to reflect true behavior - Updated docstring to specify that 'consistency' is removed only when present and falsy * update requirements to avoid test failure (#142) * Minimal fix to .dockerignore (#143) * added query generation capability (#144) * added query generation capability * fixed lint issues * added progress bar * updated to work with HF_TOKEN for gated HHEM * updated test action * updated per Tallat suggestions * updated output formatter * version bump * updated for lint * fixed typos * Add configurable question type distribution for query generation (#146) * now query generation can be configured to control the % of questions per category (roughly) * updated to read env from .env file * Update open_rag_eval/query_generation/llm_generator.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * robust handling when no assignment scores exist * upgraded transformers to remove security vulnerability * updated --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add HHEM API support and fix Anthropic model issues (#147) * initial support for Vectara HHEM via API * minor updates * fixed bug with using Anthropic model * a few bug fixes and esp with Anthropic model usage * added tests * added langchain to requirements * fixed issue with langchain * fixed issue with torch meta-score and BertScore incompatibility * issue with bert score in consistency score around meta-device incompatability to resolve * reverted back to v 4.50.2 of transformers * moved from bert_score to torchmetrics which is more frequently maintained * added max_length to BERT score to avoid going over the model sequence length (truncate if that happens) * fixed unit test * removed unused import * Adjusted query generation (#150) * updated * updated * updated * Update README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * updated per copilot suggestion --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Add token usage tracking across all LLM metrics (#151) * added token usage * version bump due to minor API update that includes token usage * fixed lint issues and unit tests * fixed copilot suggestions * Support golden answers (#152) * bug fixes * initial code for generating answers and evaluating golden answers * bug fixes; updated docs; removed answer relevancy * further fixes to remove answer relevancy * remove token usage from JSON output to maintain compatibility with openevaluation.ai * fixed copilot suggestions * minor tweaks * one more conflict resolved --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Vishal Naik <naik.vishalishwar@gmail.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> Co-authored-by: david-oplatka <david.oplatka@vectara.com> Co-authored-by: Sai Vignesh <svnaragoni@gmail.com> Co-authored-by: Tomoki Nakamaru <tomoki.nakamaru@gmail.com> Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.