Design Notes on Kotlin Value Classes

[elizarov]

This issue is for discussing and gathering feedback for Design Notes on Kotlin Value Classes: https://github.com/Kotlin/KEEP/blob/master/notes/value-classes.md

[altavir]

Yesterday we had a long discussion in kotlin telegram chat about that. There was a nice proposal there that should be at least discussed. The idea is to be able to isolate all value class mutations into a single mutating lambda function. It could be done in two ways: either allow mutations only in a mutating lambda/function with an appropriate receiver or allow those lambdas alongside simple mutations (one can also allow only lambdas at first and relax the rule later).

There are two practical benefits of such lambdas:

1. If we are doing several mutations, we need to either create several copies of a value object with subsequently changing fields or we need a scope for atomic changes. Subsequent changes could be in theory detected and glued together, but it would require a lot of work from the compiler and in the end, won't be guaranteed to work properly in all cases. The lambda on the other hand could guarantee atomicity of changes.
2. Introducing a lambda provides a small barrier in the usage of such operations. One must remember that copying value-classes is still an expensive operation and one should not use it mindlessly. The performance is maybe not a primary point but it seems a bit unexpected when two different assignment operations give such dramatic performance difference.

Also, I have an additional educational point in favor of isolation - it is a mental model. The mental model explained in the design notes is understandable from the point of view of an experienced programmer, but there are a lot of concepts there. All those mutating modifieds, var declarations, etc add a lot of things to understand and remember. Whereas mutating via lambda keeps things simple - it is just in-place mutation that does not affect previous usages of the variable.

At the end of the design notes document, there is a short discussion about non-constructors for data classes (the idea is really nice, but I think it should be discussed more). Those non-constructors are in fact follow the same idea of mutating lambda, so having them both would probably also simplify the mental model.

[elizarov]

Update on the var keyword reuse

See section on Mutable properties of immutable types

This forum post quotes an important use-case https://discuss.kotlinlang.org/t/inline-classes-var-to-be-deprecated/20762 where inline value class is used as a "handle” to some mutable data storage. It nicely shows how a regular var can be used on immutable value classes for a good reason.

We’d love to rescue and keep this use-case. We don’t have an alternative plan that will make it possible yet, but we are looking for it. There are two potential approaches.

Note: both approaches also open a road to implement regular (non-value) interface with regular var properties by immutable value classes, which is not directly possible if var is hardcoded to have a different meaning in value and non-value contexts.

A new keyword to denote properties with mutating setter (wither)

In addition to two kinds of properties that we already have in Kotlin (val for a read-only property with a getter and var for a writable property with a getter and a setter) we can add a separate keyword for the 3rd kind of a property – a mutating property with a getter and a mutating setter (wither).

How to name it? That is the question!

Introducing the 3rd type of property will make the resulting scheme harder to learn and might require the new hard keyword in the language. However, we can alleviate the former concern by nudging the learners by IDE inspections. For example, value class Foo(var prop: Type) could have an automated suggestion to change var with a backing field (that will not be allowed on a value class) to this new property type.

A new modifier to denote a regularly mutatable property on an immutable value

Alternatively, we can add some new modifier that, being added before a var property on a value class (or interface) turns into a regular mutable property with a setter that is supposed to mutate some mutable data structure and does not affect the value it is being called on.

Basically, this will be a modifier that is opposite to mutating (in the same sense that open is opposite to final). So, that, by default all var properties on a value class/interface will be considered to implicitly have a mutating modifier, while all properties on a regular non-value class/interface will be considered to implicitly have this new "opposite to mutating" modifier.

How to name it? That is the question!

[fvasco]

Please sorry for my trivial question, but I missed something important.

If a value class is an immutable data type, why is it important to mark a property like a var or a val?

All other properties are really similar to regular properties without backing field.

If so, the Grid example can be written as:

value class Grid(ptr: Long) {
    var width: Int
        get() = GridNative.getWidth(ptr)
        set(v) { GridNative.setWidth(ptr, v) }
}

or

value class Grid(ptr: Long)

var Grid.width: Int
        get() = GridNative.getWidth(ptr)
        set(v) { GridNative.setWidth(ptr, v) }

It looks really similar to a regular class

class Grid(val ptr: Long)

var Grid.width: Int
        get() = GridNative.getWidth(ptr)
        set(v) { GridNative.setWidth(ptr, v) }

We really need a new modifier?

[elizarov]

@fvasco

If a value class is an immutable data type, why is it important to mark a property like a var or a val?

Consider, for example, an immutable type for complex numbers with properties re: Double (for its real part), im: Double (for its immaginary part), absoluteValue: Double, etc. For our discussion, it is completely unimportant how these properties are actually stored, whether they have backing fields or not. All of them can be stored in backing fields, or just a few can be stored. Typically re and im will be stored, while others will be computed, but that is not important in larger business data structures, as the "storage approach" for some big enterprise entities might even evolve and change over time.

Regardless of "backed by a field or not", from all these properties, re and im properties are special. You can always reconstruct your complex number with different values of re and im. They are "mutatable properties of an immutable class". They are the ones we'll want to mark as var or in some other way (see the comment above). But, for example, an abosoluteValue property is truly read-only (val property). You can only query it, but you cannot, in general, update an abosluteValue on an arbitrary complex number and expect a meaningful result in all the cases.

For this specific example of a complex number you could define mutation for abosoluteValue if you want, but you'll have to figure out what to do with zero. For larger business entities, there could be lots of readonly (val) properties that cannot be meaningfully mutated at all.

Now, having read this, you can say that Kotlin already has a feature to distinguish these two kinds of properties on immutable classes. In Kotlin, we can define an immutable Complex class like this:

data class Complex(val re: Double, val im: Double) { // constructor properties
    val absoluteValue: Double // other properties
}

That is, we can distinguish different kinds of properties by whether they are constructor properties or body properties. Constructor val properties are actually mutable, since they can be mutated using a copy() function to get a new value. But this constructor/non-constructor is a wrong dichotomy:

• Constructor properties must be backed by a field. While this is a good default, there is no intrinsic relation between mutation ability and being in a stored field (e.g. multiple mutable properties can be packed into a single backing field). We could lift this restriction on the constructor properties (with some effort) but see the other point.
• Constructor properties are a class-only concept. What if you need to define an interface for you class, how do you distinguish those two kinds of properties in an interface? What if you need to define an extension property? How do you distinguish between a truly read-only property and a property that has a mutating setter ("wither") (an example is given in this section of the KEEP).

[Animeshz]

Constructor properties are a class-only concept. What if you need to define an interface for you class, how do you distinguish those two kinds of properties in an interface? What if you need to define an extension property? How do you distinguish between a truly read-only property and a property that has a mutating setter ("wither") (an example is given in this section of the KEEP).

I guess the value classes should just be sort of literals (because they really are not allocating memory on the heap, like another wrapper classes) and hence fully immutable. (Correct me if I'm wrong)

Also written in KEEP:

Values classes are immutable classes that disavow the concept of identity for their instances.

operator fun Complex.add(other: Complex) should return another complex instead of mutating it. Just like operations on primitive doesn't change themselves but returns a new value.

complex.re += 5 doesn't really make much sense but rather complex += Complex(5, 0) do more.

And to extend/shrink modulus of a complex number there can be another function to take the components, form a new complex and return that.

Edit: Also wanna add, that there could be a annotation which can generate a compile-time copy() method (Similar to the kotlinx.serialization) in case there's a need to change only a few fields 👀.

[fvasco]

Hi, @elizarov,
I noticed that I already answer with a really similar example here.

My question is: how many modifier are allowed for re and im properties?

Function's parameters are val by default, and there isn't any other possible modifier.

Sincerely, I didn't see the point to add some kind of wither modifier.

I made a trivial example, just to understand better the issue.

value class IntPointer(val+wither pointer: Long) {

  var value : Int get() ... set() ...
}

How this declaration differs from:

value class IntPointer(val pointer: Long) { ... }
value class IntPointer(pointer: Long) { ... }

If any value class is identified by its state, I don't need to update an unmodifiable type, I have to create a new value with arguments, and it is always possible to create a new value with different values (please correct me if I wrong).

I really see a value class like a primitive type.

var real = 5.0
var imaginary = 2.0 + 3.i

How differs these two variable?
I don't considering primitive/object question (there is no primitive type concept in Kotlin), nor stack/heap allocation (Kotlin does not guarantee allocation type for both type).

[elizarov]

@fvasco The difference is syntactic. The goal is to enable working with immutable classes just like with mutable types. Indeed, every time you "modify" an immutable class you create a new value. We want the syntax for this operation to be as concise as for mutable classes so that people who choose to model their domain with immutable classes don't suffer from the added boilerplate. Now, assume I want to modify a Complex number (that is, to create a new complex value) by adding 5 to its real part (taking example from @Animeshz). I don't want to write this, as this is verbose and does not actually express what I wanted to do:

complex += Complex(5, 0)

I want to write this, just like I do it for mutable classes:

complex.re += 5

The example is not that great for complex numbers, though. The actual design notes have a more striking example where the syntactic difference is quite apparent in the section on deep mutations.

[fvasco]

I agree with you, @elizarov.
But I don't understand how your examples requires a wither.
Maybe we have different view on the same problem and I not exposed my view very well.

I view a strict analogy from base classes (Int, Char, ...) and value classes: both miss of identity and both are immutable.

Considering your example above, in my view it is possible to update a variable if it is declared as a var, regardless it is a base or a value class.

So:

value class Complex(re: Double, im: Double)

var mutableComplex = 1.0 + 2.0.i
mutableComplex += 5.0 // works

val constantComplex = 1.0 + 2.0.i
constantComplex += 5.0 // fail

Regarding the deep mutation, assuming that all types are value classes, the statement order.delivery.status.message = updatedMessage works if order is declared as var order.
Instead it is not possible to update val order (like val c: Char).

[edrd-f]

Some general feedback: value classes for the purpose of optimizations look great. Often it's necessary to model domain objects like Email(value: String) and Username(value: String), and having to box/unbox them all the time is unnecessarily expensive.

On the other hand, the whole "immutable way of working with mutability" (i.e. creating new instances when assigning new values for var/wither properties of value classes) is very implicit. When I see point.x += 10 it's not possible to know if this is an assignment happening to a regular class or a value class without inspecting point. What if the code base is heavily multi-threaded and Point starts as a value class and is later changed to a regular class? With the current design, it seems this would compile with no errors or warnings, however all sorts of bugs would appear.

Using copy makes it explicit, so I know what's going on without having to look at the class definition. Sure, the IDE could give some hints, but we also spend a lot of time reviewing code on platforms such as GitHub, where there's no intelligent code navigation for Kotlin.

I understand copy is much more verbose whenever it's necessary to update deeply nested values, but I'd like to see some real use cases for this. The example given on the Deep mutations section looks ad-hoc and I think it's important to also have real use-cases to see if we're not trying to solve locally what should be solved through better code design.

Another point is how compiler plugins could be used to solve some of the mentioned issues. For example, the following:

order = order.copy(
    delivery = order.delivery.copy(
        status = order.delivery.status.copy(
            message = updatedMessage
        )
    )
)

could be

val updatedOrder = order.deepCopy {
  delivery.status.message.mutate(newValue = updatedMessage) 
}

where deepCopy and mutate could be generated for every value class. This would be more explicit and wouldn't (probably) require language changes.

[philipguin]

@fvasco Continuing from the other thread:

On potentially making val/var superfluous and removing them altogether, there would be a few things to consider:

1. What if, conceptually, we’d like to restrict which combination of things can belong to a value class? In that case, val/var would have some utility, where a library could emit “instances” and the user could “change” only a subset of fields. This means the library author can better protect against misuse.
2. Removing the keyword altogether may introduce parse issues, depending on how it’s implemented. I don’t think there’s a precedent for the syntax anywhere else in Kotlin.
3. What if, on Kotlin/Native for instance, they do want to eventually allow mutating fields directly? In that case, they would definitely want to keep var/val in use.

I’m sure there’s a lot more, but it is still interesting to consider.

More generally, I notice the comparison of value types to primitives occurring frequently, but I have to wonder how useful that analogy really is. In a theoretical sense it may be nice, but the “law of leaky abstractions” probably applies here as well. We are adding a lot of functionality to what was previously just the set of primitives after all, and the list of supported features can only grow.

I also have to wonder if Valhalla won’t be petitioned to add normal field setters at some point in the future - assuming it’s even possible. The desire to focus on immutability rather than just emulate C# structs always struck me as odd, though I’m sure they have good reason for it. Carving out space for this in Kotlin might not be the worst idea, especially for Kotlin/Native.

Finally, on the case of non-mutating custom setters: I’m not sure what the “mutating” keyword would be, but my vote is for option 2 suggested by @elizarov, where the keyword is only needed on the non-mutating case. However, what if we moved the keyword to the setter itself?:

var x: Float
    get() = ...
    const set(v) { ... } // or whichever keyword

By doing so, const is available as it cannot clash with the existing const val. Not sure if there's another consideration.

[fvasco]

@philipguin

What if, conceptually, we’d like to restrict which combination of things can belong to a value class?

This issue should be addressed in the init block.
How some kind of var can forbid something like `Complex(NAN, INFINITY)?

Removing the keyword altogether may introduce parse issues. I don’t think there’s a precedent for the syntax anywhere else in Kotlin.

Nothing new.

class Sum(a: Int, b: Int) {
  val n = a + b
}

Yes, updating the parser is an issue, but we are considering changing the language.

What if, on Kotlin/Native for instance, they do want to eventually allow mutating fields directly?

I will vote no for this proposal.
There are enough mutating types in Kotlin, I think we can define an immutable one, at least.

[fvasco]

@edrd-f

What if the code base is heavily multi-threaded and Point starts as a value class and is later changed to a regular class?

With my proposal, switching from

value class Point(x: Int, y: Int)

to

data class Point(val x: Int, val y: Int)

results in a compilation error.

Instead, switching from immutable to mutable leads to an errors' party.

[elizarov]

@edrd-f

Another point is how compiler plugins could be used to solve some of the mentioned issues.

Everything can be solved by a plugin. The whole idea of having better support for immutability is to have this plugin built-in into the language itself.

For example, the following /skipped/ could be

val updatedOrder = order.deepCopy {
  delivery.status.message.mutate(newValue = updatedMessage) 
}

This is similar to the concerns that @altavir shared above, so let me answer both here. The argument is that these "implicit copies" are spooky, hard to see without IDE, etc, so let's have some more verbose, more explicit, scoped syntax for them. We could invent lots of different approaches to this "more explicit syntax" (the quote above shows just one). These concerns are valid, yet, regardless of how this "more explicit syntax" will look like, they gloss over the key fact:

Mutations (creating new copies) of immutable types are safe (they cannot have side-effects on unrelated parts of the system), while mutations of mutable types are dangerous (they can accidentally affect unrelated code that kept a reference to the same instance).

It is wrong to design a language in such a way that performing a safe and mostly harmless operation (like creating a new immutable value) requires more boilerplate than performing a more dangerous operation (like updating a field in a mutable instance) that requires a lot of attention and forethought from a programmer.

Indeed, if those two kinds of mutations look syntactically the same, then changing a (safe) immutable value class to a (dangerous) mutable class will cause the old code to still compile, yet it will start producing weird bugs. Let's see how this problem can be solved. Both @altavir and @edrd-f propose to distinguish (syntactically or contextually) these two kinds of mutations. However, taking into account the key fact, it means that we have to make (dangerous) mutations of mutable classes more explicit, not vice versa. The safer operation should not be more verbose than a more dangerous one.

There is a simpler solution to this problem. First of all, note that the problem is not novel. It is happening right now all the time. People make mistakes of using mutable classes where they should be using immutable ones. As you write, it compiles "with no errors or warnings, however, all sorts of bugs would appear."

However, as we improve support for immutable classes in the language (make it less burdensome to use immutable values) we can start adding mechanisms to require immutability in various contexts (like asynchronous data pipelines) or at least warn on attempts to use mutable data there. Now if you accidentally use a mutable class where you should have been using an immutable one then you'll get, at least, warned.

[elizarov]

@fvasco But I don't understand how your examples require a wither.

Wither is not needed for mutableComplex += 5.0 (it works in Kotlin now!), wither is needed for order.delivery.status.message = updatedMessage:

• It needs to create a new copy of status with an updated message field.
• Then it needs to create a new copy of delivery with an updated status field.
• Then it needs to create a new copy of order with an updated delivery field.
• Only then it assigns this new copy of order to the order property/local (which must be var or it will not compile).

[philipguin]

Would custom setters be called for each nested “wither” in the example? That would explain the motivation for disallowing them. However, the case I’m interested in is the one lacking a backing field, which I suppose wouldn’t be touched being just functions, essentially.