`astartes`

[JacksonBurns]

Submitting Author: (@JacksonBurns)
All current maintainers: (@kspieks, @himaghna)
Package Name: astartes
One-Line Description of Package: Better Data Splits for Machine Learning
Repository Link: https://github.com/JacksonBurns/astartes
Version submitted: v1.1.2
Editor: @cmarmo
Reviewer 1: @BerylKanali
Reviewer 2: @du-phan
Archive:
Version accepted: v1.1.3
JOSS DOI:
Date accepted (month/day/year): 10/15/2023

---

Code of Conduct & Commitment to Maintain Package

• I agree to abide by pyOpenSci's Code of Conduct during the review process and in maintaining my package after should it be accepted.
• I have read and will commit to package maintenance after the review as per the pyOpenSci Policies Guidelines.

Description

• Include a brief paragraph describing what your package does:

note: this is a selection from the abstract of the JOSS paper

Machine Learning (ML) has become an increasingly popular tool to accelerate traditional workflows. Critical to the use of ML is the process of splitting datasets into training, validation, and testing subsets that are used to develop and evaluate models. Common practice in the literature is to assign these subsets randomly. Although this approach is fast and efficient, it only measures a model's capacity to interpolate. Testing errors from random splits may be overly optimistic if given new data that is dissimilar to the scope of the training set; thus, there is a growing need to easily measure performance for extrapolation tasks. To address this issue, we report astartes, an open-source Python package that implements many similarity- and distance-based algorithms to partition data into more challenging splits. Separate from astartes, users can then use these splits to better assess out-of-sample performance with any ML model of choice.

Scope

• Please indicate which category or categories.
  Check out our package scope page to learn more about our
  scope. (If you are unsure of which category you fit, we suggest you make a pre-submission inquiry):

  • Data retrieval
  • Data extraction
  • Data processing/munging
  • Data deposition
  • Data validation and testing
  • Data visualization¹
  • Workflow automation
  • Citation management and bibliometrics
  • Scientific software wrappers
  • Database interoperability

Domain Specific & Community Partnerships

- [ ] Geospatial
- [ ] Education
- [ ] Pangeo

Community Partnerships

If your package is associated with an
existing community please check below:

• Pangeo
  • My package adheres to the Pangeo standards listed in the pyOpenSci peer review guidebook

• For all submissions, explain how the and why the package falls under the categories you indicated above. In your explanation, please address the following points (briefly, 1-2 sentences for each):

  • Who is the target audience and what are scientific applications of this package?

The target audience is data scientists, machine learning scientists, and domain scientists using machine learning. The applications of astartes include rigorous ML model validation, automated featurization of chemical data (with flexibility to add others, and instructions for doing so), and reproducibility.

• Are there other Python packages that accomplish the same thing? If so, how does yours differ?

We position astartes as a replacement to scikit-learn's provides train_test_split function, but with greater flexibility for sampling algorithms, and availability of train_val_test_split for more rigorous validation.

• If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted:

N/A

Technical checks

For details about the pyOpenSci packaging requirements, see our packaging guide. Confirm each of the following by checking the box. This package:

• does not violate the Terms of Service of any service it interacts with.
• uses an OSI approved license.
• contains a README with instructions for installing the development version.
• includes documentation with examples for all functions.
• contains a tutorial with examples of its essential functions and uses.
• has a test suite.
• has continuous integration setup, such as GitHub Actions CircleCI, and/or others.

Publication Options

• Do you wish to automatically submit to the Journal of Open Source Software? If so:

JOSS Checks

• The package has an obvious research application according to JOSS's definition in their submission requirements. Be aware that completing the pyOpenSci review process does not guarantee acceptance to JOSS. Be sure to read their submission requirements (linked above) if you are interested in submitting to JOSS.
• The package is not a "minor utility" as defined by JOSS's submission requirements: "Minor ‘utility’ packages, including ‘thin’ API clients, are not acceptable." pyOpenSci welcomes these packages under "Data Retrieval", but JOSS has slightly different criteria.
• The package contains a paper.md matching JOSS's requirements with a high-level description in the package root or in inst/.
  on a separate joss-paper branch
• The package is deposited in a long-term repository with the DOI: 10.5281/zenodo.7884532

Note: JOSS accepts our review as theirs. You will NOT need to go through another full review. JOSS will only review your paper.md file. Be sure to link to this pyOpenSci issue when a JOSS issue is opened for your package. Also be sure to tell the JOSS editor that this is a pyOpenSci reviewed package once you reach this step.

Are you OK with Reviewers Submitting Issues and/or pull requests to your Repo Directly?

This option will allow reviewers to open smaller issues that can then be linked to PR's rather than submitting a more dense text based review. It will also allow you to demonstrate addressing the issue via PR links.

• Yes I am OK with reviewers submitting requested changes as issues to my repo. Reviewers will then link to the issues in their submitted review.

Confirm each of the following by checking the box.

• I have read the author guide.
• I expect to maintain this package for at least 2 years and can help find a replacement for the maintainer (team) if needed.

Please fill out our survey

• Last but not least please fill out our pre-review survey. This helps us track
  submission and improve our peer review process. We will also ask our reviewers
  and editors to fill this out.

P.S. Have feedback/comments about our review process? Leave a comment here

Editor and Review Templates

The editor template can be found here.

The review template can be found here.

Footnotes

1. Please fill out a pre-submission inquiry before submitting a data visualization package. ↩

[cmarmo]

Hi @JacksonBurns,
Thanks for submitting astartes to pyOpenSci and welcome to our submission procedure!
I'm Chiara and I will take care of the process as Editor.

After checking with the editorial board, I'm glad to confirm that your package is considered in scope with a pyOpenSci submission.
I will perform the editor checks by the end of the week, and start looking for reviewers.

In the meanwhile I have a first question for you, if you don't mind.
While the statistical side of the package is clear and answers to a well known need of the scikit-learn user community, I would like to read a bit more about the package connections with chemical engineering. I guess your scientific application was the motivation for this development in the first place and in the documentation the two aspects look quite unrelated. Perhaps a small pedagogical section about chemical applications in the documentation will also be helpful? Thanks!

[JacksonBurns]

@cmarmo thanks for the quick response and intro!

Thank you for sharing the update about scope and reviewers.

We will add a section to the documentation about applications to chemical engineering problems in particular - thank you for this suggestion.

[cmarmo]

Hello @JacksonBurns,

Below are the basic checks that your package needs to pass to begin our review.

Please check our Python packaging guide for more information on the elements below.

• Installation The package can be installed from a community repository such as PyPI (preferred), and/or a community channel on conda (e.g. conda-forge, bioconda).
  • The package imports properly into a standard Python environment import package-name.
• Fit The package meets criteria for fit and overlap.
• Documentation The package has sufficient online documentation to allow us to evaluate package function and scope without installing the package. This includes:
  • User-facing documentation that overviews how to install and start using the package.
  • Short tutorials that help a user understand how to use the package and what it can do for them.
  • API documentation (documentation for your code's functions, classes, methods and attributes): this includes clearly written docstrings with variables defined using a standard docstring format.
• Core GitHub repository Files
  • README The package has a README.md file with clear explanation of what the package does, instructions on how to install it, and a link to development instructions.
  • Contributing File The package has a CONTRIBUTING.md file that details how to install and contribute to the package.
  • Code of Conduct The package has a Code of Conduct file.
  • License The package has an OSI approved license.
    NOTE: We prefer that you have development instructions in your documentation too.
• Issue Submission Documentation All of the information is filled out in the YAML header of the issue (located at the top of the issue template).
• Automated tests Package has a testing suite and is tested via a Continuous Integration service.
• Repository The repository link resolves correctly.
• Package overlap The package doesn't entirely overlap with the functionality of other packages that have already been submitted to pyOpenSci.
• Archive (JOSS only, may be post-review): The repository DOI resolves correctly.
• Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

---

• Initial onboarding survey was filled out
  We appreciate each maintainer of the package filling out this survey individually. 🙌
  Thank you authors in advance for setting aside five to ten minutes to do this. It truly helps our organization. 🙌

---

---

Editor comments

The package is already in excellent shape, congratulations and thanks to all the co-authors for your hard work!
I have some comments, they are not meant to be blockers for the review process, just suggestions for possible improvements.

• It seems to me that the introduction about splitting algorithms is missing some references for those, like me, not really into chemical engineering. In particular I looked for the original work defining the Kennard-Stone algorithm (Kennard, R.W., and Stone, L.A., 1969. Computer aided design of experiments. Technometrics 11, 137-148) and I would like to see it cited in your documentation. The other algorithms also would benefit from some more references, when possible openly accessible, as unfortunately the links to most of the articles are pay-walled.
• Some external links are not correctly rendered ... I'm afraid I couldn't find a solution for backquoted text with external
  link in sphinx... but I think having the link rendered is more readable. (see for example
  ``AIMSim` <https://vlachosgroup.github.io/AIMSim/README.html>`_ can be replaced by `AIMSim <https://vlachosgroup.github.io/AIMSim/README.html>`_)
• Having jupyter notebooks in the documentation helps the user to understand how to use the package: however, I wasn't able to run the binder links (because of the initialization failing with the error "no space left on device").
  I know Binder is downsizing, if you don't have alternatives perhaps it would be safer to just remove the links, in order to not unnecessarily pressure the system.
• The CONTRIBUTING.md file is clear and complete. As users are explicitly asked to provide the package version when opening an issue, I'm suggesting to define an astartes.__version__ attribute and to detaile how to obtain the information when debugging.
• The API section of the documentation would benefit from more examples: the user discovers in it that the package contains a lot more than the train-test-split functions, but it is not always clear how to exploit all this tools.

I think this package may be relevant for the scikit-learn user community, perhaps you might be interested in linking astartes in the related project page among the preprocessor tools, once the review process has come to an end.

---

I'm going to look for the two reviewers now (good news, I already found one! :) ).
I will be back to you in a week (hopefully sooner), to check if we are able to start the process.

[JacksonBurns]

Hi @cmarmo thanks for the speedy work!

• Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

I have just turned on the Zenodo automatic packaging releases (which I also just learned was a thing) so hopefully this doesn't fall behind again! I have also uploaded the current version (1.1.1) to Zenodo manually.

The package is already in excellent shape, congratulations and thanks to all the co-authors for your hard work! I have some comments, they are not meant to be blockers for the review process, just suggestions for possible improvements.

Thank you for the kind words!

• It seems to me that the introduction about splitting algorithms is missing some references for those, like me, not really into chemical engineering. In particular I looked for the original work defining the Kennard-Stone algorithm (Kennard, R.W., and Stone, L.A., 1969. Computer aided design of experiments. Technometrics 11, 137-148) and I would like to see it cited in your documentation. The other algorithms also would benefit from some more references, when possible openly accessible, as unfortunately the links to most of the articles are pay-walled.

The reference for the Kennard-Stone algorithm is in the Implemented Sampling Algorithms table but it is not at all obvious what the links in the References column actually are. We will clarify this, as well as add references where missing.

As far as accessible articles, we can also try and find open-access versions of those which are pay-walled.

• Some external links are not correctly rendered ... I'm afraid I couldn't find a solution for backquoted text with external
  link in sphinx... but I think having the link rendered is more readable. (see for example
  AIMSim <https://vlachosgroup.github.io/AIMSim/README.html>_ can be replaced by AIMSim <https://vlachosgroup.github.io/AIMSim/README.html>_ ``)

👍 will do.

• Having jupyter notebooks in the documentation helps the user to understand how to use the package: however, I wasn't able to run the binder links (because of the initialization failing with the error "no space left on device").
  I know Binder is downsizing, if you don't have alternatives perhaps it would be safer to just remove the links, in order to not unnecessarily pressure the system.

We can definitely do that. We will perhaps try to migrate these to target Google Colab.

• The CONTRIBUTING.md file is clear and complete. As users are explicitly asked to provide the package version when opening an issue, I'm suggesting to define an astartes.__version__ attribute and to detaile how to obtain the information when debugging.

👍 sounds good!

• The API section of the documentation would benefit from more examples: the user discovers in it that the package contains a lot more than the train-test-split functions, but it is not always clear how to exploit all this tools.

👍 will do.

I think this package may be relevant for the scikit-learn user community, perhaps you might be interested in linking astartes in the related project page among the preprocessor tools, once the review process has come to an end.

Didn't know this existed, thanks for sharing! We will do that.

I'm going to look for the two reviewers now (good news, I already found one! :) ). I will be back to you in a week (hopefully sooner), to check if we are able to start the process.

🎉

[JacksonBurns]

@cmarmo quick question - we are getting these suggested changes made now (JacksonBurns/astartes#147). Since we submitted astartes v1.1.1, can we go ahead and merge into main, or will that interfere with the review process?

[cmarmo]

Since we submitted astartes v1.1.1, can we go ahead and merge into main, or will that interfere with the review process?

Would you be ok with making a quick release of the changes before the review starts and update the description? As far as the version does not change during the review process I think this is not an issue. Thanks for addressing the comments already!