[CEP] The use of type hints

[kaapstorm]

Abstract

The use of type hints in the CommCare HQ codebase has been discussed in the past. It was decided that we would not adopt them until they can be verified automatically, as part of a build process.

At the time, we evaluated mypy and pytype as tools for doing that verification, and realised that both of them would require an unreasonable amount of effort to configure for use with the commcare-hq repo. Progress stopped there.

Since then, I've been using type hints in my own projects, and not only have I found them to be valuable, but I have also come to believe that verifying them misses a subtle but important aspect of their design, and I no longer think that automatic verification is desirable. More on that under "Specification".

Motivation

Why do we care? Type hints have been controversial. Surely it's easier simply to avoid the controversy and not use them.

We care because the goals of type hints offer tangible value to us. From PEP 484:

Of these goals, static analysis is the most important. This includes ... providing a standard notation that can be used by IDEs for code completion and refactoring.

I am trying to move this initiative forward because I find type hints valuable in my personal projects, and I would like Dimagi to be able to benefit from them too.

Being able to Ctrl + Left-Click to jump straight to the definition of a variable's class can save a lot of time, trying to search for and trace back through function and method calls to find where a variable is first set, in order to figure out its class, and go to that class's definition. Sometimes debugging can make this easier, but not always. Good variable names can also help, but not as often or as much as one might hope.

Type hints are also really valuable when refactoring, to ensure that changes continue to pass parameters in the types that functions and methods expect. If the original author sets those type hints, it saves other developers (or the same developer much later) a lot more time trying to figure what those variable types ought to be. When it comes to the longer-term maintainability of code, the time trade-off that we make for unit tests is applicable to type hints too.

(Type hints do not fulfill the role of unit tests.)

Because they are hints, not declarations, they are a form of documentation. As documentation, they are not necessary or desirable everywhere, and like comments and docstrings, they bring a maintenance burden of their own -- they need to be updated when their variables are refactored. It is the responsibility of the developer to navigate the balance of benefit and burden when determining where to use comments and docstrings, and also type hints.

Specification

Python is designed, and is intended to be used, as a dynamically typed language. Type hints are not meant to encroach on that principle. The authors of PEP 484, Guido van Rossum, Jukka Lehtosalo, and Łukasz Langa, emphasize:

Python will remain a dynamically typed language, and the authors have no desire to ever make type hints mandatory, even by convention.

It seems clear to me that type hints are not meant to be used the way that type declarations are in statically typed languages. In fact, the clue is in the name: they are not declarations or mandates or obligations; they are only hints.

This is the subtle but important aspect of their design that I had initially missed: Developers should be able to set a type-hinted variable with a value of another type.

The obvious use case is when using test fakes in unit tests.

But the practice is based on original Python design philosophies:

1. Python has always treated its developers like adults: "Private" attributes and methods are not private; Python trusts that the developer knows what they are doing, and allows them to access their values. It merely discourages it by making it awkward.

2. Duck-typing is an established Python idiom. It is encouraged, and reinforced by recent language features, like protocols, and structural pattern matching.

These are the reasons why I think that treating type hints as static type declarations is undesirable; maybe even a mistake that we would come to regret.

This CEP does not specify how we should use type hints. (That is covered by existing PEPs.) This CEP specifies the principles to guide when we should use type hints. We have mentioned all of them already. To summarize:

• Type hints are documentation. They add value, and come with a cost. It is the responsibility of the developer to navigate the balance of value and cost.

• Automatic verification of type hints is undesirable and impractical. It is the responsibility of the developer to check that type hints, comments, and docstrings that pertain to their changes are correct. We already do it for comments and docstrings. We can also do it for type hints. (Developers who use PyCharm and VS Code even get a little help.)

• Type hints are not type declarations. Python grants the developer the freedom to assign values of different types, and expects the developer to know when it is and is not best to do so.

• Dimagi has a strong track record of hiring responsible developers, and follows a development process that ensures that poor code choices are addressed before they are released.

I believe that observing these principles will result in practices that increase the value to our codebase, improve its maintainability, and improve our skills as developers, without compromising the quality of the code.

Impact on users

N/A

Impact on hosting

N/A

Backwards compatibility

N/A

Release Timeline

N/A

Open questions and issues

Earlier documents:

• [DRAFT] Guideline for using type annotations in Python
• Revisiting Type Hints

[mjriley]

It seems clear to me that type hints are not meant to be used the way that type declarations are in statically typed languages. In fact, the clue is in the name: they are not declarations or mandates or obligations; they are only hints.

From a cursory glance at the rationale behind type-hints, I come to the opposite conclusion. While it seems that 'static analysis' can be extended to include IDE code-completion and refactoring tools, it has always meant, primarily, offline analysis intended to prevent errors. If we are not planning on retrofitting commcare with type hints, we lose out on the primary value that type hints were intended to provide.

As documentation, they are not necessary or desirable everywhere

Can you link to a conversation suggesting that type hints should be used selectively? My reading of the PEPs is that this is intended for developers who want static analysis throughout their codebase. It's optional because they don't want to disrupt users who enjoy Python as is.

Gradual Typing is referenced in PEP 483, writing: "Large software systems are often developed in multiple languages partly because dynamically typed languages are better for some tasks and statically typed languages are better for others", and goes on to list benefits for dynamic typing, and benefits for static typing. I read these benefits as suggesting that this decision is meant to be decided per system, not per function. For example: "Static type checking catches bugs earlier, thereby removing the greater cost of fixing bugs later in the development cycle or the even greater cost of a bug that occurs in the field." If this was a priority, it would be a priority for all of a system's code, not just one function. How I've seen this actually materialize, is writing a server in C++ or Java, where we care about static analysis, and then writing testing hooks in Ruby, because that code wanted faster iteration and more expressiveness. I feel like this is a similar scenario to Javascript frameworks -- many allow you to port a small segment of your site over, but that is done mainly to aid adoption, with the long-term goal of having all code migrated over to a single framework. Mixing and matching, long-term, is a recipe for confusion and inconsistency.

This is the subtle but important aspect of their design that I had initially missed: Developers should be able to set a type-hinted variable with a value of another type.
The obvious use case is when using test fakes in unit tests....
These are the reasons why I think that treating type hints as static type declarations is undesirable; maybe even a mistake that we would come to regret.

There are exceptions to every rule, but I'm not even sure this is a valid exception, so much as just not wanting to spend the time to create a proper subclass for a test fake (as would be needed in a statically typed language). Either way, this is an exceptional situation, and I'd definitely be concerned seeing types ignored in production code. It seems more beneficial to me to treat type hints as static type declarations, and have to write more complicated test fakes, than the other way around.

Type hints are documentation. They add value, and come with a cost. It is the responsibility of the developer to navigate the balance of value and cost...It is the responsibility of the developer to check that type hints, comments, and docstrings that pertain to their changes are correct. We already do it for comments and docstrings. We can also do it for type hints.

I strongly believe in Uncle Bob's quote: "A comment is an apology for not choosing a more clear name, or a more reasonable set of parameters, or for the failure to use explanatory variables and explanatory functions. Apologies for making the code unmaintainable, apologies for not using well-known algorithms, apologies for writing 'clever' code, apologies for not having a good version control system, apologies for not having finished the job of writing the code, or for leaving vulnerabilities or flaws in the code." Code is for the 'how', comments are for the 'why'. As far as I can tell, type hints can never be the 'why', only the 'how'. While I can see the appeal of being able to click a variable and go to the definition, the fact that the variable is vague in the first place is what I'd prefer to focus on.

This CEP does not specify how we should use type hints. (That is covered by existing PEPs.)

Do you mean that this CEP does not cover how to write type hints? By the wording, I thought the PEPs might have provided guidance on when type hints were, or were not, appropriate, but was unable to do so.

[mjriley]

Personally, I don't want type hints in our codebase. If we were starting new greenfield development, I would be on-board with declaring that project a 'type hints' project. But as it stands, with a codebase as large and old as commcare is, I don't feel the positives outweigh the negatives.

Negatives:

• Ugly (subjective, but it matters). I feel it goes against the Zen of Python's 'readability counts'.
• Selectively applying type hints is confusing. Developers now need to know Python and type hints, rather than just Python...or they only know Python, and thus write code that would meet our criteria for type hints, but doesn't use them.
• It adds an additional 'standard'. I wouldn't feel great about having a codebase with tabs and spaces, inconsistent formatting, etc. I don't like the idea of having code written by 'devs who like type hints' and 'devs who don't like type hints'.
• There is no guarantee type hints will be maintained. With a code base as old as ours, we have plenty of incorrect or out-of-date comments. I'd rather have no type hints, and rely on developers doing their research, than incorrect types hints and wrong assumptions.

[snopoke]

Personally I like using type hints for the benefits I get in terms of documentation and IDE support. In terms of static type checking, I haven't actually found that to be very useful. I have used it on one project and it resulted in me spending a bunch of time just figuring out make the type checker happy or adding exclusions to the rules. Having said that I have once caught a bug because of it (refactor that changed a type but not all places that used it).

In terms of type hints vs variable names, I don't think you can really compare them unless you plan on including type information in the var names:

(types taken from HQ code)

# naming gives you decent info here but doesn't help with IDE features.
# Names can get quite long.
form_datum: SessionDatum
stack_datum: StackDatum
form_datum_meta: FormDatumMeta
workflow_datum_meta: WorkflowDatumMeta

# variable naming doesn't tell you much here unless you give a very long explicit name.
# And still you don't get any of the type hint benefits
datums: List[Union[SessionDatum, StackDatum]]
session_or_stack_datums

In terms of readability I have not found to it be an issue except in rare circumstances which usually means a function is taking in too many args or you should create a type alias.

There is no guarantee type hints will be maintained

That's true, there is no guarantee, but I it is more likely that type hints will be maintained than comments since they do give tangible value to developers and are 'part of the code' in a way that comments aren't.

Overall I'm pro using them (and we already are in parts of HQ).

[millerdev]

TL,DR: I don't want type hints in HQ.

Functions in Python are typing fences. All type checkers that I am aware of do not infer the types of function arguments. If you want autocomplete on arguments inside a function you must declare their types. I am concerned that this incentivizes full type declarations for every function except for the most trivial. There are several reasons why this feels problematic, the most salient being that HQ would likely have a very long if not indefinite period of having type hints scattered inconsistently throughout the code. As long as that is true it will probably be difficult to attain the most important goal listed in PEP 484: static analysis.

As stated previously, I want us to implement a static type checker as part of our build process before we start using type annotations in HQ code. It should be obvious to any contributor when they have broken the code, regardless of what editor they use to make the change. Personally, I don't think the amount work required to do that, including long term maintenance needed to keep a type checker happy, is worth the benefit we will get from it. We have other priorities such as keeping dependencies up to date that seem more important and yet have been hard to resource.

I am concerned that type annotations will make developers feel falsely confident about the types of arguments passed to a function. Without static analysis errors may be hard to detect. Consider this code

def itemize(items: dict) -> Iterable[str]:
    for key, value in items.items():
        yield f"{key}: {value}"

Is it safe to pass in a value that is not a dict, but has an items() method that yields tuples? Should you update the type annotation on items if you do?

Is it safe to refactor this function in a way that would call items.keys()?

On one hand, declaring items to have a type of dict is overly restrictive since the function only uses a small subset of dict methods. Type annotations limit the kinds a ducks that may be passed to a function.

On another hand, someone refactoring that function would feel more confident about using other methods of dict since the variable is declared to have that type. But it is necessary to investigate how the function is called to be confident that a refactor will not break calling code. I think the presence of type annotations, especially on function arguments, will make it less likely for developers to investigate how a function is called.

Finally, I think type hints are ugly. They add boilerplate that clutters the code and sometimes makes it take more lines. Typing verbosity (or lack thereof) is the main reason why I prefer Python to languages like Java and C#. Type hints make Python feel more like Java.

Edit: minor wording improvements and added thoughts about why I don't think it's worth adding a static type checker.

[kaapstorm]

I think there are two main things happening here:

(In reverse order) the second thing is that our perceptions of the value of type hints are different. I can address this one.

The first thing is the aesthetic. I can't do much here. I can only observe that aesthetics are strongly influenced by what we are accustomed to.

It seems as if the people who have used type hints the most are the ones who find the highest value of type hints to be at the level of the function. And the people who have used them less perceive their value to be at the level of the codebase.

We seem to be reading the same few sentences, and understanding them differently. Quoting PEP 484 again, "Rationale and Goals":

This PEP aims to provide a standard syntax for type annotations, opening up Python code to easier static analysis and refactoring, potential runtime type checking, and (perhaps, in some contexts) code generation utilizing type information.

Of these goals, static analysis is the most important. This includes support for off-line type checkers such as mypy, as well as providing a standard notation that can be used by IDEs for code completion and refactoring.

I think the value from offline type checkers is dwarfed by the value of "code completion and refactoring". These are everyday tasks that we do all the time. Shaving off a small amount of effort here compounds into big wins over time and multiple developers.

'static analysis' ... has always meant, primarily, offline analysis intended to prevent errors.

Offline type checking may prevent some errors, but it does not avoid the need for unit tests, and unit tests can and should be responsible for catching those errors in Python. Offline type checking can catch type errors at a lower cost than unit tests, but I think good unit tests diminish that value, because they should exist whether type checking is happening or not.

But I don't think that diminished value is relevant to us. Automated type checking might be a non-starter, because it looks like mypy will take a lot of effort to configure, and pytype will take a long time to run. I'd love someone with more experience with either of those tools to prove me wrong, but I worry this is a distraction from the greater benefit to us:

In the case of commcare-hq, the longer-term benefits of type hints to refactoring, and working with other developers' code, outweigh offline type checking.

Can you link to a conversation suggesting that type hints should be used selectively?

It sounds like you're suggesting that someone else's opinion of what the HQ codebase should look like is more authoritative than our own. pydocstyle (via flake8) thinks that all public modules, classes, methods, and functions should have docstrings. We don't think so, and I think we have every right to disagree. In the same way, it's up to us to decide whether we use type hints universally.

Mixing and matching, long-term, is a recipe for confusion and inconsistency.

In general, that is a truism I agree with. But when it applies to actual examples of type hints, I don't find the following confusing:

from __future__ import annotations
from typing import Any, Protocol

def inc(x):
    return x + 1

def add_one(x: int) -> int:
    return x + 1

def increment(x: Addable) -> Any:
    return x + 1

class Addable(Protocol):
    def __add__(self, other):
        ...

There was a time when I found two-variable list comprehensions confusing. Maybe it just takes time for the pattern recognition to kick in. Over time, I have found that type hints transition from being ugly and confusing, through being unremarkable, to finally being informative -- particularly informative about the author's intention. It is clear to me from the example above that the intention of the author of add_one() was different from the intention of the author of increment(). The intention of the author of inc() could have been the same as add_one(), the same as increment(), something else, or maybe they didn't think about it at all.

And if we are willing to accept that, just like some functions don't need docstrings, some functions don't need type hints, then the example above is also not inconsistent.

I'd definitely be concerned seeing types ignored in production code.

That is a good point. Similarly, I would be concerned seeing production code accessing the private attributes of a class. But I still believe it would be counter the spirit or philosophy of Python for the language to prevent developers from doing it. I happily agree that we wouldn't want to see types ignored in our production code. But, in my mind, dynamic typing is core to Python's nature, and type hints are not intended to change that.

Code is for the 'how', comments are for the 'why'.

I strongly agree.

As far as I can tell, type hints can never be the 'why', only the 'how'.

I'm not certain they fit neatly into either. Maybe they are more accurately described as the "what".

So, again referring to my example above, if I were to read,

incremented = increment(not_a_number)

and my first response might be "wtf?", when I notice the type hint I might realise "oh, that's what the author had in mind all along!"

Developers now need to know Python and type hints, rather than just Python

This sentence jumps out at me. Since September 2015, typing is as much Python as datetime, logging or unittest. Just as developers can choose not to use unittest we could also choose not to use typing, but it's definitely part of the language, and "knowing Python" means "knowing unittest" and "knowing type hints". The almost-eternal, unchanging Python 2.7 lulled us all into a false sense of stasis. But Python has a steady release cycle now, and I'm afraid it's up to us to keep up, or get left behind. Brutal. I know.

I wouldn't feel great about having a codebase with tabs and spaces

I once worked in an old Perl codebase, where some devs had used tabs, some used two-space indents, some three-space indents, and others four-space indents. They also used multiple different ways of passing function parameters ... in the same, 65,000 line file: An undeniable horror show. But I don't think that's really analogous to type hints. Some functions having type hints and others not having type hints is more like some functions having docstrings and others not having docstrings. That feels very different to me than mixing tabs and spaces.

I don't like the idea of having code written by 'devs who like type hints' and 'devs who don't like type hints'.

Yeah. That would be sad. It would be wonderful if others were to track the same journey I took, from "urgh" to "meh" to "oh!" But we already work with code written by "devs who love classes" and "devs who don't love classes". It's not perfect. But it's better than having no freedom, where every minute detail of our work is subject to tyrannical dictate.

I am a big fan of the idea that "There should be one-- and preferably only one --obvious way to do it." But the correct application of "one way to do it" might be realised better at a higher level than the minutiae of "every public function must have a docstring", or "everything should be a class", or "every function must have type hints". I would say that sometimes "one way to do it" refers to a principle that should be applied differently in different circumstances.

If you want autocomplete on arguments inside a function you must declare their types.

No IDE, or human, can tell what a function will be passed in the future. But some IDEs are better than others at figuring out the type of a variable. We should explore to see what tools work best for us.

HQ would likely have a very long if not indefinite period of having type hints scattered inconsistently throughout the code.

I am struggling to understand why it is bad to have some but not all functions with type hints, but it's OK to have some but not all functions with docstrings, or the idea that not everything should be a class. I don't see this as an inconsistency. It's about recognising that not everything is a nail, and we should use the language features that best suit each circumstance.

It should be obvious to any contributor when they have broken the code, regardless of what editor they use to make the change.

An incorrect type declaration in Java breaks the code. In Python it's like an out-of-date comment (although, as we've discussed, it's not a comment). And we already have a process to deal with that.

Personally, I don't think the amount work required to do that, including long term maintenance needed to keep a type checker happy, is worth the benefit we will get from it.

Hard agree. I don't think that is where the value of type hints is for our codebase.

Consider this code

def itemize(items: dict) -> Iterable[str]:
    for key, value in items.items():
        yield f"{key}: {value}"

Is it safe to pass in a value that is not a dict, but has an items() method that yields tuples? Should you update the type annotation on items if you do?

Is it safe to refactor this function in a way that would call items.keys()?

Of course, dropping the type hints would not answer any of those questions. What the type hints give us is an indication that the author considered items to be a dict. So if other developers have respected the author's intentions, then the answer to "Is it safe to refactor this function in a way that would call items.keys()?" is "Yes". (I would still check how the function is currently being called. That feels like due diligence.)

The answer to "Should you update the type annotation on items if you [pass in a value that is not a dict]?" is "Yes". The author was kind enough to tell me what they were thinking, I'd say I ought to be kind enough to the next developer to tell them what I'm thinking. If it were me, I would use a Protocol to show that the items argument needs an items() method that yields tuples.

In the context of commcare-hq, that might also allow the original author to weigh in, and confirm my change, or explain why they chose differently.

If the next developer, keeping with your example, wants to refactor this function in a way that would call items.keys(), then they have a good idea of how the function is currently being used. If that next developer were me, I'd look into the option of adding a keys() method to the class of the last developer's items instance (and to the Protocol). That would take longer to spot if the type hint was missing.

And, finally, back to the "ugly" objection, and particularly, "Type hints make Python feel more like Java." I hear you, especially when it comes to long lists of parameters.

Some mitigating thoughts:

• As @snopoke mentioned, that might be a nudge that the function has too many args, and should be broken up.

• In some cases, type hints are (imho) a noticeable improvement on readability. I think attrs with type hints is a lot more readable than attrs with field assignments. And typing.NamedTuple is so much clearer than calling collections.namedtuple().

• Perhaps not in the commcare-hq codebase, but maybe in your own, call a function with a test fake that is not the hinted type, just to reassure yourself that type hints are not type declarations, that Python still treats you like an adult, and that unittest was the last nauseating chunk of Java code in the Python standard library.

• Oh, lastly, but not seriously, maybe check out the Tomorrow (light) or Tomorrow Night (dark) theme. Instructions for PyCharm | VS Code | Emacs | all of the things. It is a delight, and will make type hints (and stupid self.assertThisOneSpecificThing() calls) a little prettier.

I find that after using type hints in my own projects for a while, they don't grate me any more. The test fakes might have helped.

[mjriley]

Since it seems a strong argument is being made that type hints, at least in our code base, should be used selectively (presumably to increase readability/documentation?). I came across this block while doing other work:

def metrics_counter(name: str, value: float = 1, tags: Dict[str, str] = None, documentation: str = ''):
    provider = _get_metrics_provider()
    provider.counter(name, value, tags=tags, documentation=documentation)

and I felt it was illustrative on how type hints are often unnecessary and hurt readability, and how more readability can be gained with better names than typing. What would be lost if this was instead just:

def metrics_counter(name, value=1.0, tags={}, documentation='')

• name as a string value is redundant
• value already indicates this is going to be a number. If taking float values is important, we can shortcut that here by providing 1.0 rather than 1, but otherwise, that's probably best left to a docstring explaining the limitations on the number. Notably, many systems require positive values, or values underneath a given boundary, which type systems aren't very good at. amount might be more appropriate than value, since value can be confused to be a string
• documentation is easily inferred to be a string. If that's not obvious, documentation_string is explicit

I think the main value gained here was showing that tags is a mapping of strings to strings. And I could see an argument saying that without this typing, that would be confusing. I'd argue that the reason this is confusing isn't the lack of typing, but because the argument was misnamed. tags is used in other Django contexts (Messages) to mean a list of strings (separated by a space, but that's another issue...). After looking at the DataDog documentation, I understand that 'tags' means 'extra properties', but either the person reading this already was familiar with this 'tags' concept and typing adds no benefit, or that person is unfamiliar and typing adds no benefit. propertyMap gets us most of the way there, with a docstring providing further clarification if needed.

I struggle to see how the above type hints would aid in development -- I don't think autocomplete support for string, integers, and dictionaries does much.

As to the specifics of why it hurts readability:

• because it makes it easier to specify the type and stop there. Restrictions breed creativity -- if you know you don't have type annotations, you have to find to a way to clarify tags, which will ultimately be better than what tags can offer here.
• Because the way type hints are read is unnatural. I come from statically typed languages. I find something like:

void metricsCounter(String name, float value=1.0f, HashMap<String, String> tags=null, String documentation='')

relatively readable -- but I made the choice to use Java, and I get all the benefits static typing has to offer. When I'm writing Python, I don't write Python to be Java. Python places a much higher emphasis on human readability than Java does. That's the reason Python has if foo is not None rather than if not foo is None, despite it creating multiple ways to do the same thing. value:float=1 is just a worse float value=1. I want to say "the floating integer 'value' has a default of 1", but instead I'm forced to jump around with "value, floating integer it is, has a default of 1". It's as if Yoda was coding, and it is unpleasant to read.

I'm sure there are other examples I could use that would show a more tangible benefit to typing, but the point I'm trying to make is that there are generally better alternatives. Even if we can agree the typing was unnecessary here, it's still in our code, which tells me that, were we to advocate for type hints going forward, we'd see more unnecessary declarations and less readability. I need more than autocompletion to sell me on that.

[kaapstorm]

I like this example. And I agree with a lot of the conclusions you draw from it.

It could be that this is a useful data point of a function where we don't want to use type hints, because they reduce readability, and they will not help with refactoring in the future. But before we get there ...

I think your point about tags is insightful. This code is a layer that wraps our metrics provider, allowing third-parties who host their own CommCare instances, or us if we need it, the option of using different metrics providers.

A Dimagi developer reading this code is likely (but not certainly) familiar with Datadog. So "tags" might be more familiar to them than "property_map". (I tried to think of a name that means "maps filter names to filter value strings" but "filter_name_to_value_str" seemed very clumsy.) I agree with you that either a docstring, or a dict[str, str] type hint, would make it clear what this thing is.

I see the value of a docstring for a developer who is unfamiliar with this function. But these type hints were added by someone who was refactoring an existing counter() function that has about 200 usages. If I was that author, I would want to use type hints, because I would want my IDE to highlight any calls that are passing the wrong kind of dictionary, so that I can evaluate all those usages quickly.

PEP 484 recommends,

It is recommended but not required that checked functions have annotations for all arguments and the return type.

Let's go with that. I personally like shorter lines (because I split my screen vertically and have tests in the right pane) so I might style this a bit differently:

def metrics_counter(
    name: str,
    value: float = 1.0,
    tags: Optional[dict[str, str]] = None,
    documentation: str = '',
):
    """
    Wraps the delegated metrics provider, and adds ``value`` to the
    counter named ``name``.

    ``tags`` maps tag names to values, which the metrics provider can
    use to allow metrics to be filtered by tag values. Use this to allow
    filtering by domain, or error, for example.

    Use ``documentation`` for additional documentation.
    """
    provider = _get_metrics_provider()
    provider.counter(name, value, tags=tags, documentation=documentation)

I read the parameters slightly differently: "value is a float with a default of 1.0. tags is an optional dict that maps strings to strings with a default value of None."

Alternatively, I like your function signature, but we can't use {} as a default for property_map because it's mutable. (Careful of that. (A good IDE will highlight this.))

def metrics_counter(name, value, property_map=None, documentation=''):
    """
    Wraps the delegated metrics provider, and adds ``value`` to the
    counter named ``name``.

    ``property_map`` maps tag names to values, which the metrics
    provider can use to allow metrics to be filtered by tag values. Use
    this to allow filtering by domain, or error, for example.

    Use ``documentation`` for additional documentation.
    """
    if property_map is None:
        property_map = {}
    provider = _get_metrics_provider()
    provider.counter(name, value, tags=property_map, documentation=documentation)

While it's definitely more readable, I see a problem here that prevents any IDE from providing the author what they really needed: The name "property_map" does suggest that the function expects a dict[str, str], but the default value isn't going to help the IDE figure that out. The author will need to evaluate all ~200 usages carefully to check that the dictionary being passed doesn't have, say, integer, or float tag values.

If this refactor was your task, what would you do?

Maybe use the type hint, complete the refactor, and then delete the type hint afterwards? I've done that before (a few days ago, in fact). Maybe that approach is the best of both worlds; practical and readable.

I'm not sure about it though. It feels bad to me. The next time someone needs to refactor my code, it makes it their problem, all over again. Readability matters. But how many hours is my slightly less verbose function signature going to cost someone else later? Maybe it would be fair if the next developer who needed to refactor my function could call me up, and ask me to put the type hints back in for them, just temporarily, so that they don't have to figure them out themselves.

And in our hypothetical world where the original author adds type hints each time someone wants to refactor, and then deletes them again afterwards, how many times would that happen before the author decided they should rather leave the type hints in?

...

Zooming out a bit, I really appreciate this discussion: Collectively thinking about the trade-offs, and where we, as the developers who have to live with our codebase, feel the balance lies. This is what I was hoping to get out of this CEP.

(As an added bonus, we also get to mention type hint styling along the way.)

[joxl]

Besides some perspective from a "new developer to the codebase", I don't think there is much for me to add that hasn't already been said. @mjriley's comment resonates particularly strongly with me, especially:

With a code base as old as ours, we have plenty of incorrect or out-of-date comments. I'd rather have no type hints ...

I am particularly worried about the inevitable reality of adding type hints to HQ: inconsistency. @millerdev described this well:

... HQ would likely have a very long if not indefinite period of having type hints scattered inconsistently throughout the code

As an experienced developer coming on board and working in the codebase, I can say that dealing with code inconsistencies has been one of the biggest contributors to the steep learning curve that is "understanding how to effectively contribute to commcare-hq". I already spend a sigificant amount of time trying to understand "the '[best] HQ' way to do it", and I'm not enthusiastic about adding more constraints to that already non-trivial task. From the perspective of a developer new to the codebase, it is frustrating to grep around the code looking for the answer to "how this is done in other parts of HQ?" and finding two or more different answers.

HQ is a large, production codebase that is constantly under heavy development by several separate teams. I'd prefer to see existing inconsistent code get refactored to use fewer alternative conventions rather than introducing more inconsistencies. Adding type hints will put further burden on growing HQ dev teams, and I don't think the pros justify adding (more of) them.

[kaapstorm]

Thank you for the input @joxl.

I have been worrying that I'm not giving enough weight to the "it will make things (even) messier" side of the conversation. So I started working on a "style guide" to specify how I think type hints should be written. This contradicts my earlier statement, "This CEP does not specify how we should use type hints." But after some consideration, if we do decide to use type hints, it will improve readability and pattern recognition if they look the same. I hope a style guide can also improve their value to developers wanting to refactor or understand code. e.g. If you are going to the effort of stating argument types, then you should state the return type too.

I am particularly worried about the inevitable reality of adding type hints to HQ: inconsistency.

Could you clarify what you mean by "inconsistency"?

Do you mean that one function definition looks different from another function definition, and all function definitions should look the same? Or do you mean that there exists a principle of when a function definition should have type hints and when it should not, and you are worried that we would apply that principle inconsistently.

There is a principle of when code should have a comment: @mjriley said, "comments are for the 'why'" and I agree with that. We should not delete all comments because they make some code look different. Rather, we should consistently apply the principle: We should add them where they add value, and remove them where they don't. The same is true for docstrings.

Is the same true for type hints?

When do type hints add value? What is that value? What is the principle?

Some of us have been working in codebases that include type hints, and have developed a feel for what they're good for, and when. It seems like these are the people who have expressed support for them to me. Am I right in thinking that those of us who are worried that their value is so low that we should rather never use them at all are the same people who have not worked much with them?

"They're ugly and they smell like Java" is not completely wrong! But are we being like dogs barking at a new postman, and in a few months we could be wagging our tails every time his ugly, Java-smelling face comes to bring us things, because we've learned to appreciate those things?

Maybe we should fire up PyCharm, or VS Code with python.analysis.typeCheckingMode set to "strict", and each start a pet project using, say, Starlette or Flask. Implement Domain Models, and a Repository, and a Service Layer. Use type hints. Refactor some stuff. Get a feel for what type hints are good for, and how we prefer them to look, and where we don't need them. I think we should all have done something like that, in order to make an informed choice, before we decide that HQ should not use them at all.

I am spending my evening writing way-too-long responses (sorry!), because I am worried that we might rule out a feature that could be really beneficial, only because we can't figure out how to use it properly.

[kaapstorm]

I don't want to suggest that your concerns about inconsistency are invalid. The real problem here could be that Dimagi devs don't apply principles consistently. That's outside the scope of this CEP, but something that should get attention. Improving this won't just affect type hints; it will benefit our velocity, our job satisfaction, and all decisions we make about our codebase.

A recent question on Slack is a great example: Older Django apps import signals in models.py or __init__.py; newer Django apps use AppConfig. Why aren't we consistent?

I see two possibilities:

• Maybe we're not all aware of AppConfig, or more generally, the right way to do something. In this case, maybe the solution to not all knowing the same things is to get better at sharing knowledge, or highlighting discoveries, as we do in #gtd-show-and-tell. We should do more of that.

• Maybe we're all onboard with AppConfig, or we all agree on some general "right way", but we just haven't taken the initiative, or made the effort, to bring all the code in line. How much of our time should be spent replacing import signals statements with AppConfig definitions, and how much should be spent doing work that feels like it has tangible impact? Of course, we have to do both. Maybe there is a better balance to be found. I'm up for a conversation about how to do that better.

But for this CEP, I'm just hoping we can figure out when to use type hints.

[millerdev]

I'm not speaking for @joxl, but with the current proposal I am concerned that we will end up with even more inconsistency than we have now.

An example discussed above highlights one of my consistency concerns:

def metrics_counter(name: str, value: float = 1, tags: Dict[str, str] = None, documentation: str = ''):

This is real code written by one of our most experienced developers (and reviewed by others), and the type hints are inconsistent. I mean no disrespect to the developer who wrote that code. It's very easy to make mistakes like that. I expect we would get a fair amount of this type of inconsistency without an additional tool to do automated static analysis as part of each test run.

The other area where I'm worried about (even more widespread) inconsistency (than we have now) is in the application of type hints to new and existing code, were we to agree to adopt them. What would the process of adoption look like? Certainly no one is going to apply type hints to every function in HQ in a single PR, or even as part of a single resourced migration effort. I imagine the most practical approach would be to roll it out in voluntary iterative steps, as developers have time and feel compelled to do so. This sounds like a very long, possibly indefinite, period of inconsistency. This is not appealing to me, especially since I am not a fan of type hints, and therefore would not frequently feel compelled to add them to code I'm working on.

For consistency, I would have preferred that we never allowed type hints in HQ at all, but I also did not want to force anyone to curb their excitement around learning a new language feature by saying "You can't do that here." That is why I agreed to allow developers to use their judgement to apply them consistently within sub-components they maintain and own over an extended period of time. I thought we had agreed that they would not be sprinkled randomly throughout HQ code, and I hope we will stick to that agreement until we can reach some other consensus.

Am I right in thinking that those of us who are worried that their value is so low that we should rather never use them at all are the same people who have not worked much with them?

I don't think so, but I only speak for myself. I have played around with them in a side project. Probably not as extensively as some other developers on the team. I have also read and worked with code written with TypeScript, which has many similarities.

Maybe we should ... each start a pet project [and get more experience with them]

I'm not going to suggest what you do with your spare time, but I have other priorities that feel more important than spending my free time playing with something I am not excited about.

A couple days ago I was listening to an interview with Peter Wang, co-founder of Anaconda, and he said something important:

Types are there to make the compiler writer's jobs easier. ... Types are there because compiler writers are human and they're limited in what they can do.

He wasn't talking specifically about type hints in Python, but he was talking about how Python is easy and beautiful because it does not force you to declare types. The utility of type hints is low without automated static analysis (a kind of compiler). A static analysis tool could possibly catch errors that otherwise would only be caught with slow integration tests. Until we have that, I don't see much value for HQ.

[kaapstorm]

I'm not going to suggest what you do with your spare time

Yeah. That was a little presumptuous of me.

I also did not want to force anyone to curb their excitement around learning a new language feature

I don't think that commcare-hq is the best codebase for learning new language features. In the past I have first tried things out in a toy codebase or a side project. I hope other devs do that too.

Python is easy and beautiful because it does not force you to declare types.

I agree. I don't think type hints are for everywhere. (The principle I use is If other modules call this code, then type hints can help whoever writes those other modules (including myself). I don't use type hints in scripts, and I don't find them useful in top-level modules like views or commands.)

allow developers to use their judgement to apply them consistently within sub-components they maintain and own over an extended period of time. ... The utility of type hints is low without automated static analysis ... Until we have that, I don't see much value for HQ.

OK. What do you all think of the following?

1. We leave the decision to use type hints up to the owner of the module we are working in. It's fine to type our own modules, and help others type theirs, but not OK to commit type hints in someone else's module.

2. We add mypy to test-requirements.in.

3. We draw up a mypy.ini file that sets a level of strictness that feels right for HQ, and create a file that lists the modules "opted in" for mypy to check.

4. We agree that all modules that include type hints must be opted in -- if not immediately then at least planned.

5. We add a style guide to this CEP that provides guidance on how typing should look, and specifies a level of strictness that meets our mypy.ini requirements.

6. (We update the description at the top of the CEP, so that readers don't have to scroll through all this to get to the point.)

[mjriley]

I see three options:

1. Do not use type hints at all.
2. Use type hints everywhere
3. Leave type hint usage up to the developer.

#⁠1 is obviously my preferred option. #⁠2 I'm on board with, provided we agree that the value proposition is high enough to undertake that work. #⁠3 I firmly believe is the worst of all possible worlds, and yet it seems to be the path that is being pushed for. @joxl had an excellent comment pointing out the issues surrounding inconsistency. It seems somewhat consistent to restrict them to modules, but that seems like we're opting in to a bus factor issue in the future, when inevitable turnover happens, and the person picked to maintain the type hints code isn't particularly comfortable using them. I also am not sure if, within the type-hinted module, we are still recommending judgment on what is or is not type hinted.

Researching type hints, I watched the excellent Type-Checked Python in the real world and Putting Type Hints to Work, as well as reading Diminishing returns of static typing. Every presentation or writeup I came across stressed the main value type hints bring is static analysis, i.e. running mypy over your codebase. I ended up being convinced that type hints in Python are essentially like test coverage: 100% coverage is the goal, but it is also unrealistic and often not worth the effort. However, if someone were to submit a PR for entirely new code that lacked tests, I'd want to see good reasons. If we do elect to use typehints in HQ, I'd want a similar standard to apply. If you're writing code, you need to have a very good reason to not have it type-hinted.

Previously, you mentioned:

It sounds like you're suggesting that someone else's opinion of what the HQ codebase should look like is more authoritative than our own.

Yes, I do believe in relying on the collective wisdom of larger companies for best practices. I'm not saying we don't have some specific oddities to our company, but in general, best practices are best practices for a reason. The talks I posted above are from Instagram and Coffee Meets Bagel, and there are many other large, established companies that apply type hints consistently. What about CommCare is so unique that we need to do things our own way here?

However, let's return to option #⁠2. Recently, I had a bit of egg on my face as I added parameters to an existing method, not realizing that it was the base method in an inheritance hierarchy. I ended up having to make a series of staging deploys as I fixed one issue, only for the very same issue to happen with the next child class. This was an area of code that was untested, but to have prevented this issue from happening would have required a fairly specific and unlikely test to be written. Because the failure was in calling an expensive method that ordinarily would have been mocked out, we would have either needed complicated integration tests for all of these child features, or have written tests that verified the specific mocked call parameters. This is a situation that tests are fairly lousy at. On the other hand, had type hints been used consistently throughout our codebase, this is an issue static analysis would have solved trivially. I can see value there.

To @millerdev 's point, the issue of migrating our code base into one that uses type hints appears intimidating, and based on other migrations, would likely take years. That's certainly not something worth doing. However, MonkeyType exists, and that seems like something that could realistically cover most of our code base. My understanding is it uses debugger hooks so that as you run your program as normal, it records the types of parameters in a separate Python Interface (.pyi) file. A separate utility can later be run to apply those interface definitions to your actual code.

That said, I still find them ugly and would prefer to just not use type hints. Foregoing type hints will not leave us in the dust. Typescript has been around for 10 years now, compared to Type Hints' 7. A 2021 language popularity rankings has Typescript as the 8th most popular language, ahead of even Ruby. Yet, the most popular language is still overwhelmingly plain Javascript. I believe Django has been talking about type hints for the last 5 years, but still hasn't put them into the codebase.

Maybe we should fire up PyCharm, or VS Code with python.analysis.typeCheckingMode set to "strict", and each start a pet project using, say, Starlette or Flask. Implement Domain Models, and a Repository, and a Service Layer. Use type hints. Refactor some stuff. Get a feel for what type hints are good for, and how we prefer them to look, and where we don't need them. I think we should all have done something like that, in order to make an informed choice, before we decide that HQ should not use them at all.

Type hints aren't something like a new framework. Statically typed languages, offline static analysis tools to work with those languages, and IDE support for those languages, have existed for a very long time. What else does a C# developer, for example, need to know to have a fair opinion on type hints?

I definitely appreciate the amount of effort everyone has put in to contributing to this (and I learned something about default mutable parameters!), but I do feel like we're liable to continue arguing in circles. I think we should just put this to a vote among anyone on the team who is interested, asking which of the 3 options listed above is preferable, and go from there.