Conversation
Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com>
Enable passing `base_url` to LLMJudge models.
added fixed seed for umbrela
Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section
Update README.md with the new UI
* bump version * bugfix with gemini to catch genai.exceptions
Added support to process queries in parallel across all connectors.
fixed lint issues and bumped version to 0.1.7
* Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com>
* Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * initial * fixed issues from copilot review * Added support to process queries in parallel across all connectors. * Updated CitationMetric * Version bumped version to 0.1.7 (#110) fixed lint issues and bumped version to 0.1.7 * Merge conflict 1 (#112) * Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Update publish_release.yml (#95) Fix issue with ONNX install * merge dev into main for release 0.1.6 (#103) * Clean up merge conflicts from dev -> main (#90) Merge dev to main for v0.1.4 (#82) * upgraded libs * updated to be compliant with PEP 625 * update MANIFESH.in * updated versions to remove security vulnerabilities * Reformat Open-RAG-Eval -> Open RAG Eval. (#76) * Update publish_release.yml (#80) Added OPENAI key (from secrets) for publish script * Update test.yml (#79) * Llama index connector (#78) * initial llama_index_connector * refactored connector to be a true base class with fetch_data CSVConnector (and unit test) removed since it's really just a results loader and not a true connector * fixe lint issues * updated copilot recommendation * updated after fixing tests * added llama_index in requirements * updated * fixed connector tests and moved to use Pandas instead of CSV * moved configs to separate folder * folder re-arranged * fixed unit test * more updated on README * updated per Suleman's comments * added test_rag_results_loader * updated LI connector to include citations * upgraded transformers version * updated * updated llama_index connector * updates to config file comments * Update _version.py (#81) --------- Co-authored-by: Ofer Mendelevitch <ofermend@gmail.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> * Fix issue with release process and ONNX (#96) initial * Enable base_url. * Update .gitignore * Remove print statement. * added fixed seed for umbrela * Update README.md with the new UI Removed "visualize" step for the "run on Vectara vs with a connector" and condensed everything into "Visualization" section * initial * Added evaluation screenshots to ReadMe * fixed issues from copilot review * fixed lint issues * updated per copilot suggestion * added print of no answer in vectara connector * added seed=42 to boost consistency * bump version (#104) * bump version * bugfix with gemini to catch genai.exceptions * bugfix (#105) * fixed lint issue (#106) --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> * bugfix --------- Co-authored-by: Suleman <108358100+sulekz@users.noreply.github.com> Co-authored-by: CJ Cenizal <cj@cenizal.com> Co-authored-by: Suleman Kazi <suleman@vectara.com> Co-authored-by: Renyi Qu <mikustokes@gmail.com> Co-authored-by: Stokes Q <33497497+toastedqu@users.noreply.github.com> Co-authored-by: Donna <yu.donna.dong@gmail.com> Co-authored-by: Vishal Naik <naik.vishalishwar@gmail.com>
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR updates the project’s version to 0.1.7 while standardizing connector class naming and introducing parallel processing capabilities. Key changes include renaming LangchainConnector to LangChainConnector across modules and tests, adding max_workers support for parallel query processing in multiple connectors, and minor updates to constants, logging, and configuration files.
Reviewed Changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_langchain_connector.py | Updated connector name usage to LangChainConnector. |
| open_rag_eval/utils/constants.py | Added new constants for NO_ANSWER and API_ERROR. |
| open_rag_eval/rag_results_loader.py | Added logging and conditions to skip processing when no generated answer. |
| open_rag_eval/metrics/citation_metric.py | Redesigned citation metric computation with improved logging usage. |
| open_rag_eval/connectors/vectara_connector.py | Introduced parallel processing with ThreadPoolExecutor and adjusted retry logging. |
| open_rag_eval/connectors/llama_index_connector.py | Added parallel query processing using ThreadPoolExecutor. |
| open_rag_eval/connectors/langchain_connector.py | Renamed the class to LangChainConnector and implemented parallel processing. |
| open_rag_eval/connectors/connector.py | Minor formatting and query reading improvements. |
| open_rag_eval/connectors/init.py | Updated export to reflect the renamed LangChainConnector. |
| open_rag_eval/_version.py | Bumped package version to 0.1.7. |
| config_examples/*.yaml | Added max_workers parameter for parallel processing in various configs. |
| .pylintrc | Extended disable list with duplicate-code and too-many-positional-arguments. |
Comments suppressed due to low confidence (2)
open_rag_eval/connectors/vectara_connector.py:247
- The function 'islice' is used but not imported. Import it from the 'itertools' module to avoid a NameError.
for idx, result in enumerate(islice(search_results, num_results_used), start=1):
open_rag_eval/metrics/citation_metric.py:136
- Consider combining the message into a single formatted string with placeholders (e.g., using '%s') for improved readability and consistent logging practices.
logging.error("While calculating citation metrics: ", f"Failed to parse response: {response.refusal}", getattr(response, "refusal", "No error details available"))
vish119
approved these changes
Jun 4, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.