Presubmission Inquiry for sciform (float -> string scientific formatting)

[jagerber48]

Submitting Author: Justin Gerber (@jagerber48)
Package Name: sciform
One-Line Description of Package: Provides extended functionality for formatting floats into strings according to scientific standards
Repository Link (if existing): https://github.com/jagerber48/sciform

---

Code of Conduct & Commitment to Maintain Package

• I agree to abide by pyOpenSci's Code of Conduct during the review process and in maintaining my package after should it be accepted.
• I have read and will commit to package maintenance after the review as per the pyOpenSci Policies Guidelines.

Description

• Include a brief paragraph describing what your package does: sciform is used to convert python float objects into strings according to a variety of user-selected scientific formatting options including fixed-pointa and decimal and binary scientific and engineering notations. Where possible, formatting follows documented standards such as those published by BIPM or IEC. sciform provides certain options, such as engineering notation, well-controlled significant figure rounding, and separator customization which are not provided by the python built-in format specification mini-language (FSML). In addition, sciform provides functionality for formatting pairs of floats as value +/- uncertainty pairs according to a variety of scientific standards.

Community Partnerships

We partner with communities to support peer review with an additional layer of
checks that satisfy community requirements. If your package fits into an
existing community please check below:

• Pangeo
  • My package adheres to the Pangeo standards listed in the pyOpenSci peer review guidebook

Scope

• Please indicate which category or categories this package falls under:

Scope

• Please indicate which category or categories.
  Check out our package scope page to learn more about our
  scope. (If you are unsure of which category you fit, we suggest you make a pre-submission inquiry):

  • Data retrieval
  • Data extraction
  • Data processing/munging
  • Data deposition
  • Data validation and testing
  • Data visualization
  • Workflow automation
  • Citation management and bibliometrics
  • Scientific software wrappers
  • Database interoperability

Domain Specific & Community Partnerships

- [ ] Geospatial
- [ ] Education
- [ ] Pangeo
- [ ] Unsure/Other (explain below)

• Explain how and why the package falls under these categories (briefly, 1-2 sentences). Please note any areas you are unsure of:
  Sciform allows for improved formatting of floats into strings according to scientific standards. These strings will be output to terminals, plots, data documents, text documents, and possibly more. Making the displayed strings more readable as per scientific standards improves the visualization of "printed number" data.

There are no existing community partnerships for this project, though there may be opportunities for education around significant figures and uncertainty.

• Who is the target audience and what are the scientific applications of this package?
  Any scientist who uses python is in the potential target audience for this package, but especially those who are concerned with displaying data values in a way that is commensurate with the corresponding uncertainties. Most scientists likely use the python built-in string formatting for this purpose, but there are some shortcomings to python built-in formatting. Scientists who seek more formatting features could consider sciform.

• Are there other Python packages that accomplish similar things? If so, how does yours differ?
  Yes there are similar packages.

1. Python built-in string formatting mini language (https://docs.python.org/3/library/string.html#format-specification-mini-language). sciform includes its own string formatting mini language closely based on the built in one, but with some differences. Notably sciform includes well-controlled significant figure formatting, engineering notation, binary formatting, SI/IEC prefix substitution, digit grouping and decimal symbol options (helpful for a diversity of locales), exponent value coercion, as well as value +/- uncertainty formatting functionality.
2. The uncertainties package (https://pythonhosted.org/uncertainties/). sciform was heavily motivated by this package. This package has sophisticated statistical handling of value +/- uncertainty pairs, handling error propagation and simulation under-the-hood. In addition, it has its own extension of the mini language for formatting value +/- uncertainty pairs. sciform has more formatting functionality than the uncertainties package including, especially, engineering notation, grouping separator controls, and prefix substitution. sciform is also a much lighter weight requirement than the uncertainties package. This may be desirable when a user wants to format strings, but they don't need the rest of the full statistical machinery of the uncertainties package.
3. The prefixed package (https://github.com/Rockhopper-Technologies/prefixed). sciform was also motivated by the prefixed package. This package provides a sort of engineering notation where exponents are rounded to multiples of 3, and then exponents area always replaced with their corresponding SI exponent. prefixed package is a more conservative extension of the built-in formatting language. sciform includes more functionality including engineering notation without prefix substitution and more grouping/decimal symbol control. sciform also includes global configuration options for handling optional SI prefixes such as c, d, da, and h.
4. The sigfig package (https://sigfig.readthedocs.io/en/latest/). The sigfig package has similar functionality to sciform including sig fig rounding, separator control, value +/- uncertainty formatting including some features that are only forthcoming in sciform. sig fig does not currently support binary formatting. sig fig also does not provide a format specification mini language for formatting floats. Rather floats are formatted using an overload of the built-in round function which I find to be slightly awkward compared to a Formatter object or function.

• Any other questions or issues we should be aware of:
  Much of the code is still a work in progress. I'm still working on documenting the existing features, more unit tests are necessary for existing features, and the value +/- uncertainty features are still young and not thoroughly tested. I have important ideas in mind for more value +/- uncertainty formatting features. But I would say the core of the package is in place. One glaring gap for this package is support for Decimal number rather that float numbers. I would like to add that functionality after the functionality for formatting floats is stable.

This package is very new and has 1 user so far. Me. But, I've been kicking around code for this sort of formatting for quite some time now and think many others would find it useful. Having a small authoritative package for this sort of formatting could be useful for the scientific community. There is also some interest in getting some of these features into the python built in string formatting feature set which would be very useful. Having a package like this could be a stepping stone towards that. See https://discuss.python.org/t/new-format-specifiers-for-string-formatting-of-floats-with-si-and-iec-prefixes/26914/46. Though I do note that the format specification mini language is intentionally not 100% backwards compatible with the built in format specification mini language, so it would not be a top candidate for that role.

I'm also not very experienced when it comes to contributing to open source software. This is one of my first forays into that world, so I am learning as I go.

P.S. Have feedback/comments about our review process? Leave a comment here

[NickleDave]

Welcome @jagerber48 and thanks so much for your detailed pre-submission inquiry.

After a first read-through, sciform looks like it could be in scope for a pyOpenSci review, but we'd like to ask you for a little more information.

1. I see above you marked the package as pangeo affiliated -- can you confirm whether that's the case, or was it a mistake? Maybe something about our form wasn't clear there?

2. The goals for sciform seem clear, but can you please say more about who the target audience is, and what the use cases are? We very much welcome newer packages for review, so it's fine if you feel like you currently the only user, but who do you intend to use it more broadly? Along the same lines: the docs have a lot of great detail about how to use sciform, but I'm not finding a lot about who would use it, where and when. Should I use it to format papers or reports? Should I use it to make numerical experiments more replicable? I have a feeling you already have some use cases in mind, but maybe you are so focused on development right now that you just haven't written a lot about those yet. Some vignettes in the documentation with walkthroughs of use cases would really help, something like "here's how you'd use sciform to do x". Please let us know more about who you're developing sciform form and when you see them using it.

[jagerber48]

Hello @NickleDave, thanks for your response!

1. Yes, the Pangeo affiliation was a misunderstanding on my part. I thought pyopensci had a requirement to be conformant with Pangeo in addition to pyopensci, so I was indicating willingness to conform there as well. I've unchecked that box.

2. Good question. I will think about this but here are my off-the-cuff thoughts.

• My main use case for this sort of formatting is when I am printing data analysis results to a terminal (either in some console program) or in a jupyter notebook. I'm trying to view data analysis results part way through a workflow or at the end of the workflow and I would like to see them, e.g. in engineering notation rounded to 3 significant figures. For this case the main "consumers" of sciform outputs are python print or logger functions. Sometimes I make a table using the tabulate package, fill it with formatted strings, and print that to the terminal.
• The next important use case would be integrating these formatted strings into plots in a variety of ways. One very nice example would be to plot tick labels using prefix notation. e.g., instead of having "1e3" out at the edge of the axis, you would have tick labels like 1 k, 2 k, 10 k, 50 k, etc. of course including strings like this on annotations could be useful as well.
• The uncertainties package provides the ability to translate formatted value +/- uncertainty strings into latex code, but I need to understand what the use cases are there. I'm not advanced enough in latex to understand how someone would integrate python data analysis directly into latex source code for a document. I would just copy the number into latex manually and round/format them as needed, but maybe there's a more sophisticated workflow I'm missing. edit: I think the Latex mode might be useful for putting strings with superscripts and pretty +/- symbols into matplotplib graphs.
• There might be use cases for writing formatted data into e.g. csv or other types of files for storing data. But I would suggest that if you are storing numbers you should store them appropriately as ints or floats, not strings.

I can include the tabulate and matplotlib use cases in the documentation. I think those would be illustrative use cases people could look at.

[lwasser]

hey @jagerber48 !! 👋 welcome! I just had a question not related to this specific review. What on that form would make it more clear that pangeo is an option thing? we have an affiliated partner program and that check just allows someone to ALSO become pangeo affiliated. But it's not a requirement. How could we make that more clear as you are not the first person to be confused by that!!

Also i'm wondering then if this tool would really be a support tool for reproducible reports (which is important to our open science goals)? If it's really about printing and output. Does that type of application (reproducible reports/ jupyter notebook output, etc). resonate with your goals for the tool?

[jagerber48]

@lwasser
Thank you for the welcome and your questions/comments!

About the Pangeo option from my perspective: Something like "You may optionally choose to affiliate your package with additional communities by checking the boxes below. These affiliations may come with XYZ benefits/additional requirements" Even just an "(optional)" flag may have cleared me. "If your package fits into an
existing community please check below:" is a challenging sentence be cause I don't know what these communities are and I didn't want to learn it at the time. So yeah, replacing this with something like "If you would like to affiliate your package with an existing community, please check below" would have helped me I think.

Also i'm wondering then if this tool would really be a support tool for reproducible reports (which is important to our open science goals)? If it's really about printing and output. Does that type of application (reproducible reports/ jupyter notebook output, etc). resonate with your goals for the tool?

"would really be a support tool for reproducible reports". What are "reproducible reports"? The tool takes python floats or float pairs and converts them to formatted (hopefully human readable) strings. There are many ways these strings could be used, it sounds like "reproducible reports" is definitely a use case that this tool could support. You mention Jupyter notebook output, that's definitely something I use it for, so I would say this does resonate with my goals for the tool.

[jagerber48]

@NickleDave

I've updated the documentation to include my prototypical use case: https://sciform.readthedocs.io/en/stable/examples.html. Here I am doing two visualization tasks. I have x, y data which I am fitting an extracting best fit parameters for. The first visualization task is plotting the data. The second visualization task is displaying the best fit parameters (and their uncertainties from the fit routine) in a table.

sciform helps with the first plotting task by making it relatively straight forward (though with some admittedly not 100% straightforward helper functions) to convert the tick labels into SI prefix format.

sciform helps with the second table task by making it easy to format value/uncertainty pairs together for easy reading and order of magnitude comparison.

I imagine sciform will typically be used in python scripts or notebooks after some data analysis has been done, and now the user want to print analysis results to the terminal or notebook output. However, the result could also be saved into some sort of human-readable, text-based report which lives in memory or which is saved to the disk.

Instead of using sciform immediately at the conclusion of analysis, users could also use sciform while traversing a non-human readable data file to generate a rounded, human-readable version or summary of that data file. For example if the data file contains numeric or value/uncertainty type data.

I imagine adding an option to format strings into a "pretty" format using unicode characters and also a "latex" format similar to the uncertainties and other float formatting packages I linked above. Especially the "latex" format will open up more use cases for plotting (matplotlib requires latex for some formatting tasks) and report generation.

[jagerber48]

@NickleDave I'm curious what next steps are for this. It seems like the package is likely in scope for pyopensci. Does that mean the next step is to actually submit the package and work towards meeting those requirements?

[NickleDave]

Hi @jagerber48 thank you for your patience--we wanted to get input from other community members about whether this package was in scope.

Thank you also for updating the documentation with a use case. That is exactly the kind of concrete example that really helps users understand what you are trying to do for them.

We have decided that, yes, we will proceed with a review.

Please go ahead and make a full submission. Be sure to mention this issue by number when you do so ("as discussed in #114") and please be sure to complete the pre-review survey when you do make the submission. Appreciate it!

Once you have opened that issue referencing this one, I will close this. We will then put out a call for an editor and reviewers.

[jagerber48]

@NickleDave ok great, thank you for your response! I'll be going on a two week vacation starting this weekend and I haven't yet had time to make the full submission yet. I will work on it, as per all your instructions, when I return.

[NickleDave]

Thanks for letting me know @jagerber48 -- no rush. Have a good vacation!