Skip to content

SIP-69: Existential containers #101

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 13 commits into
base: main
Choose a base branch
from
Open

SIP-69: Existential containers #101

Changes from 1 commit
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
1d78df7
Start SIP on existential containers
kyouko-taiga Nov 25, 2024
1790303
Complete SIP on existential containers
kyouko-taiga Dec 2, 2024
3e6fc1b
Apply Matt's suggestions
kyouko-taiga Dec 2, 2024
4642b49
Submit a SIP on existential containers
kyouko-taiga Dec 20, 2024
03c23a5
Update content/existential-containers.md
kyouko-taiga Jan 24, 2025
8b0807c
Update content/existential-containers.md
kyouko-taiga Jan 24, 2025
5ac6ae4
Update content/existential-containers.md
kyouko-taiga Jan 24, 2025
ef4829d
Update content/existential-containers.md
kyouko-taiga Jan 24, 2025
45f77d9
Avoid mentions of type erasure
kyouko-taiga Jan 24, 2025
dacee46
Apply @lihaoyi's suggestions
kyouko-taiga Jan 24, 2025
85af31b
Merge pull request #1 from kyouko-taiga/more-context
kyouko-taiga Jan 24, 2025
bdbfc35
Use current given syntax
kyouko-taiga Jan 24, 2025
da01f70
Update SIP number
kyouko-taiga May 26, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 116 additions & 34 deletions content/existential-containers.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ title: SIP-NN - Existential Containers
Type classes have become a well-established feature in the Scala ecosystem to escape some of the shortcomings of subtyping with respect to extensibility.
Unfortunately, type classes do not support run-time polymorphism and dynamic dispatch, two features typically taken for granted in Scala.

This SIP proposes a feature called *existential containers* to address this problem.
This SIP proposes a minimal change to the language to support *existential containers*, which address this problem.
An existential container wraps a value together with a witness of its conformance to one or several type classes into an object exposing the API defined by these type classes.

## Motivation
Expand Down Expand Up @@ -65,7 +65,7 @@ In other words, it is impossible to call `largest` with an heterogeneous sequenc
## Proposed solution
Copy link
Contributor

@lihaoyi lihaoyi Jan 24, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section is under-specified. Reading it, I have no idea what the proposed solution is, even though I generally agree on the value of the feature being proposed

We need several more use case examples, which are less contrived than Square Rectangle and Polygon. Preferably from real open-source libraries

Someone should be able to read the proposed solution have zero idea how it is implemented, and still get the general idea of what the proposed language feature is about. Right now, that is not the case, so this section of the proposal is incomplete

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can write other examples but I disagree that the one in the document is contrived. It also generalizes very easily to any situation involving a set of classes conforming to the same type class. Any instance of a situation where the "upper bound" of a set of types is defined retroactively would look like the proposed example.

def lookup(key: List[Containing[Hashable]]) = ???
def zipAll[E](xss: List[Containing[Iterator]{ type Element = E }]) = ???

Instead of justifying existential containers, however, I propose to clarify that the SIP is about sugaring the selection of a method rather than proposing the containers themselves, as those can be defined in library space.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You really need to justify both the language changes and library support. The language change is useless without the library support, but most people would not be familiar with this library-level technique.

List[Containing[Hashable]] is definitely a lot more concrete than Seq[Containing[Polygon]]. We need more examples like that.

  • I've found use for the equivalent of Seq[Containing[upickle.default.Writer]] myself, which contains a list of things that can be converted to JSON

  • Scalatags currently uses methods taking (args: Frag*) with implicit conversions from various types to Frag. it could conceivable instead use (args: Containing[Fragable]*) with a Fragable typeclass to render the value to HTML. Is that better or worse? Why?


The problems raised above can be worked around if, instead of using generic parameters with a context bound, we use pairs bundling each value with its conformance witness.
In broad strokes, our solution generalizes the following possible implementation of `largest`:
In broad strokes, a solution generalizes the following possible implementation of `largest`:

```scala
trait AnyPolygon:
Expand All @@ -78,50 +78,59 @@ def largest(xs: Seq[AnyPolygon]): Option[AnyPolygon] =

The type `AnyPolygon` conceptually represents an arbitrary polygon.
It consists of a pair containing some arbitrary value as well as a witness of that value's type being a polygon.
We call this pair an _existential container_, as a nod to a similar feature in Swift, and the remainder of this SIP explains how to express this idea in a single, type-safe abstraction.
We call this pair an _existential container_, as a nod to a similar feature in Swift.

### Specification
While the above example "hardcodes" the type class `Polygon`, Scala's type system is actually rich enough to safely define a generic abstraction represeting existential containers parameterized by a type class.
A possible implementation is presented in the appendix but its details are actually irrelevant to for the proposed change.
The purpose of this SIP is _only_ to support the selection of an existential container's `value` field implicitly.
That way, one could simply write `xs.maxByOption(_.area)` in the above example, resulting in quite idiomatic scala.

Existential containers are encoded as follows:
To illustrate further, assume the existence of an abstraction named `Containing[TC]` for representing containers pairing an arbitrary value with a witness of its conformance to some type class `TC`.
The proposed change would let the compiler accept the following example:

```scala
import language.experimental.{clauseInterleaving, modularity}
trait Polygon extends TypeClass:
extension (self: Self) def area: Double

/** A type class. */
trait TypeClass:
type Self
def largest(xs: Seq[Containing[Polygon]]): Option[Containing[Polygon]] =
xs.maxByOption(_.area)
```

/** A value together with an evidence of its type conforming to some type class. */
sealed trait Containing[Concept <: TypeClass]:
/** The type of the contained value. */
type Value: Concept as witness
/** The contained value. */
val value: Value
This implementation of `largest` requires existential containers to take and return arbitrary polygons.
Indeed, we wish to operate on a _heterogeneous_ list of polygons (i.e., types conforming to `Polygon`), not a list of a particular type happing to have an instance of the type class.
On the return side, we wish to return any type known to be a polygon paired with the witness of its conformance.
Again, doing so (conveniently) is not possible without existential containers.

object Containing:
/** A `Containing[C]` whose value is known to have type `V`. */
type Precisely[C <: TypeClass, V] =
Containing[C] { type Value >: V <: V }
/** Wraps a value of type `V` into a `Containing[C]` provided a witness that `V is C`. */
def apply[C <: TypeClass](v: Any)[V >: v.type](using V is C) =
new Precisely[C, V] { val value: Value = v }
/** An implicit constructor for `Containing.Precisely[C, V]` from `V`. */
given constructor[C <: TypeClass, V : C]: Conversion[V, Precisely[C, V]] =
apply
The above example generalizes to any occurrence of heterogeneous collection.
For instance:

```scala
trait CustomHashable extends TypeClass:
extension (self: Self) def hashInto(hasher: Hasher)

def customHashValue(xs: List[Containing[CustomHashable]]): Int =
val h = Hasher()
for x <- xs do xs.hashInto(h)
h.finalize()
```

Given a type class `C`, an instance `Containing[C]` is an existential container, similar to `AnyPolygon` shown before.
The context bound on the definition of the `Value` member provides a witness of `Value`'s conformance to `C` during implicit resolution when a method of the `value` field is selected.
The companion object of `Containing` provides basic support to create containers ergonomically.
For instance:
Returning a value paired with its witness generalizes similarly.

```scala
def largest(xs: Seq[Containing[Polygon]]): Option[Containing[Polygon]] =
xs.maxByOption(_.value.area)
trait Sizeable extends TypeClass:
extension (self: Self) def size: Int

def shortest[A: Sizeable, B: Sizeable](a: A, b: B): Containing[Sizeable] =
if b.size < a.size then Containing(a) else Containing(b)
Copy link
Member

@bishabosha bishabosha May 26, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this abstraction seems fundamentally lossy - so if i want to chain some operations over heterogenous data then i guess i need to declare a bag of all the type classes i could want to use ahead of time, but this proposal does not discuss that. (e.g. [T: {Sizeable, Codec}] i would need to capture more than one witness.)

```

To further improve usability, we propose to let the compiler inject the selection of the `value` field implicitly when a method of `Containing[C]` is selected.
That way, one can simply write `xs.maxByOption(_.area)` in the above example, resulting in quite idiomatic scala.
Further motivation for existential containers in Scala have been described in a research paper [2].

### Specification

Assuming the existence of an abstraction named `Containing[TC]` for representing containers pairing an arbitrary value with a witness of its conformance to some type class `TC` in the standard library, the compiler injects the selection of the `value` field implicitly when a method of `Containing[TC]` is selected.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is in the Specificationsection, can I assume that "the compiler injects ..." is in fact part of what this SIP specifies? If so, it should be fleshed out:

  • How does this interact with other adaptations, including implicit conversions and apply insertion?
  • In which condition does this adaptation triggers exactly? Is Containing treated specially by the compiler or can we (and should we) generalize this?

Otherwise, this should be in a separate section (example or appendix)


Illustrating with our running example:

```scala
// Version with subtyping:
Expand All @@ -134,7 +143,7 @@ def largest1(xs: Seq[Polygon1]): Option[Polygon1] =
trait Polygon2 extends TypeClass:
extension (self: Self) def area: Double
def largest2(xs: Seq[Containing[Polygon2]]): Option[Containing[Polygon2]] =
xs.maxByOption(_.area)
xs.maxByOption(_.area) // <- sugared form of `xs.maxByOption(_.value.area)`
```

### Compatibility
Expand Down Expand Up @@ -189,9 +198,49 @@ In contrast, to avoid possible undesirable complications, this proposal does not
Rust also supports existential containers in a similar way, writing `dyn P` to denote a container bundling some value of a type conforming to `P`.
Similar to Swift, existential containers in Rust are considered supertypes of the types conforming to their bound.

Existential contains are also featured in Haskell, under the [`ExistentialQuantification`](https://wiki.haskell.org/Heterogenous_collections) extension.
Unlike in Swift and Rust, packing and unpacking in and out of existential containers requires more boilerplate:

```haskell
{-# LANGUAGE ExistentialQuantification #-}

class Polygon a where
area :: a -> Double

data AnyPolygon = forall a . Polygon a => MakePolygon a
pack :: Polygon a => a -> AnyPolygon
pack = MakePolygon

instance Polygon Square where
area s = 1.0
instance Polygon Hexagon where
area h = 1.0

largest :: [AnyPolygon] -> Maybe AnyPolygon
largest (x : xs) = case (largest xs) of
Nothing -> Just x
Just(y) -> Just (if (f x) < (f y) then y else x)
where f (MakePolygon a) = area a
largest [] = Nothing
```

A more formal exploration of the state of the art as been documented in a research paper presented prior to this SIP [2].

Copy link
Contributor

@lihaoyi lihaoyi Jan 24, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This proposal is missing a section on Alternatives. I can think of a few myself:

  • What if we just add a Containing[T] type to the standard library; people could then use it without needing a language/compiler change at all? Could it even be a third-party library?

  • What about implicit conversions and implicit constructors? Those are widely used today, often referred to as the "magnet pattern"

Why is the proposed solution better than the alternatives I listed here? What other alternatives should we be aware of, if any?

## Alternatives considered

### No change to the language

As already mentioned, Scala's type system is currently strong enough to support the definition of existential containers.
Hence, no change to the language is strictly necessary to support them.
The gain offered by the proposal is that method selection on an existential container will look more familiar.

### Converting to and from wrappers

As mentioned in our motivations, the problem of creating a heterogeneous collections can be addressed by defining custom wrappers.
One can further define implicit conversions to alleviate the syntactic burden.
This approach is nonetheless strictly more verbose since it requires the definition of a specific container for each type class used in conjunction with heterogeneous collections.
In that sense, existential containers can be understood as a generalized wrapper.

## FAQ

#### Is there any significant performance overhead in using existential containers?
Expand All @@ -204,3 +253,36 @@ This overhead drops below 10% on larger, more realistic benchmarks [2].
1. Stefan Wehr and Peter Thiemann. 2011. JavaGI: The Interaction of Type Classes with Interfaces and Inheritance. ACM Transactions on Programming Languages and Systems 33, 4 (2011), 12:1–12:83. https://doi.org/10.1145/1985342.1985343
2. Dimi Racordon and Eugene Flesselle and Matt Bovel. 2024. Existential Containers in Scala. ACM SIGPLAN International Conference on Managed Programming Languages and Runtimes, pp. 55-64. https://doi.org/10.1145/3679007.3685056


## Appendix

The following is a possible implementation of existential containers.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here's a version that doesn't rely on the experimental modularity language import (and doesn't require the typeclass to extend TypeClass):

/** A value together with an evidence of its type conforming to some type class. */
sealed trait Containing[TC[_]]:
  /** The type of the contained value. */
  type Value: TC as witness

  /** The contained value. */
  val value: Value

object Containing:

  /** Wraps a value of type `V` and a corresponding typeclass instance `TC[V]` into a `Containing[TC]`. */
  def apply[TC[_]](v: Any)[V >: v.type](using TC[V]): Containing[TC] =
    new Containing[TC]:
      type Value >: V <: V
      val value: Value = v


```scala
import language.experimental.{clauseInterleaving, modularity}

/** A type class. */
trait TypeClass:
type Self

/** A value together with an evidence of its type conforming to some type class. */
sealed trait Containing[Concept <: TypeClass]:
Copy link
Member

@bishabosha bishabosha Jan 24, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does it only work with this upper bound or can it be { type Self <: AnyKind } - so you are not forced to extend this TypeClass trait?


/** The type of the contained value. */
type Value: Concept as witness

/** The contained value. */
val value: Value

object Containing:

/** Wraps a value of type `V` into a `Containing[TC]` provided a witness that `V is TC`. */
def apply[TC <: TypeClass](v: Any)[V >: v.type](using V is TC) =
new Containing[TC]:
type Value >: V <: V
val value: Value = v
```

Given a type class `C`, an instance `Containing[TC]` is an existential container.
The context bound on the definition of the `Value` member provides a witness of `Value`'s conformance to `TC` during implicit resolution when a method of the `value` field is selected.
The companion object of `Containing` provides basic support to create containers ergonomically.