`!` should partipicate in null shorting

[stereotype441]

This issue captures discusison that has been happening in dart-lang/sdk#43093.
Quoting @lrhn:

@eernstg We have formally specified null shortening now, right?
I would expect ! to be included in the suffix operators that are shortened, but please check.

I definitely do not want

 x?.foo?.bar;

and

 x?.foo!.bar;

to behave differently when x is null or x.foo is non-null. Users should be able to think that ?. can be replaced by !. when they know that the value will not be null.

Quoting @eernstg:

Given that ! is a selector when it is used to check for non-null values, I believe the consistent approach is to treat ! accordingly, by adding a rule along the following lines:

If `e` translates to `F` then `e!` translates to: `PASSTHRU[F, fn[x] => x!]`

I believe this would ensure that x?.foo!.bar evaluates to null when x is null, and throws when x is non-null but its foo is null.

Also, it seems less useful to let x?.foo!.bar mean (x?.foo)!.bar, because this would imply that we didn't mean ?. in the first place.

Proposed spec update here.

[munificent]

I worry that users will find the resulting type of the expression to be very unintuitive. A couple of examples:

expectInt(int i) {}

test(List<int?>? list) {
  expectInt(list?.first!); // Error.

  list?.first! + 2; // Error.

  var x = list?.first!;
  x.isEven; // Error.
}

So the user is explicitly writing ! after an expression, but the thing they get in response is nullable!

One option would be to only null short ! when immediately followed by another selector:

list?.first! // Equivalent to: `(list?.first)!`
list?.first!.isEven // Equivalent to: `(list != null) ? (list.first!.isEven) : null`

In other words, we short ! when it appears like !. or ![...], but not when a subexpression of other kinds of expressions. I don't know if I like this idea or not. :-/

[leafpetersen]

What about making it an error to apply ! to a null shortened expression? I think it's always the case that anything you write of the form e! where e is a null shorting expression can be rewritten by replacing the null shorting operators in e with !, right? That is, foo?.bar()?[3]! is equivalent to foo!.bar()![3].

Alternatively, say that it does participate in null shorting, but make it a warning to avoid the confusion that @munificent points out (which I agree is really bogus - it's really surprising that an expression of the form e! can evaluate to null).

[stereotype441]

What about making it an error to apply ! to a null shortened expression? I think it's always the case that anything you write of the form e! where e is a null shorting expression can be rewritten by replacing the null shorting operators in e with !, right? That is, foo?.bar()?[3]! is equivalent to foo!.bar()![3].

Personally I'm not a huge fan of this. I buy Lasse's argument (from our meeting this morning) that allowing ! to participate in null shortening is valuable because it lets you write code like a?.b!.c (meaning a == null ? null : a.b!.c).

Alternatively, say that it does participate in null shorting, but make it a warning to avoid the confusion that @munificent points out (which I agree is really bogus - it's really surprising that an expression of the form e! can evaluate to null).

I like this, assuming that when you say "make it a warning", you mean "make it a warning when a null-shorted e! is not immediately followed by another selector". The warning could link to some documentation where we explain why it doesn't do what the user probably wants, and the analysis server could provide a quick fix that changes a?.b! into (a?.b)! (or a semantically equivalent expression like a!.b or a!.b!, depending on the type of b).

@bwilkerson @pq for visibility on the warning/quick fix idea.

[jakemac53]

expectInt(list?.first!); // Error.

For situations like this - if we don't do null shortening then it really converts it to list!.first!, which doesn't seem right to me. If users want to assert the entire chain is non-null, they should use !. and not ?. throughout.

[leafpetersen]

Ok, let me try to pull together some pieces here. Suppose we have:

class A {
  int get zero => 0;
  int? get zeroOrNull => null;
}

Suppose we have a local variable A? a = A()

Consider a?.zero!:

• If ! participates, this has type int? and evaluates the same as a?.zero
  • This seems bad to me, the user never wants this (should be error or warning if we take this interpretation)
• If ! does not participate this has type int and evaluates the same as (a?.zero)!
  • This seems likely to be what the user wanted when they wrote this.

Consider a?.zeroOrNull!

• If ! participates, this has type int? and evaluates the same as (a == null) ? null : a.zeroOrNull!
  • The user may have wanted this interpretation: they want to assert that when a is non-null, a.zeroOrNull is non-null
• If ! does not participate this has type int and evaluates the same as a!.zeroOrNull!
  • The user may have wanted this interpretation. In fact, if they wrote int x = a?.zeroOrNull, the obvious fix for the error is to add a ! to the end, and then be puzzled that the error doesn't go away.

Consider a?.zero!.isEven:

• If ! participates, then this has type bool?, and the ! is a no-op and should be an error or warning.
• If ! does not participate, then this has type bool, and the meaning is `(a?.zero)!.isEven
• This seems likely to be what the user intended when they wrote this.

Consider a?.zeroOrNull!.isEven

• If ! participates, then this has type bool?, and the meaning is (a == null) ? null : a.zeroOrNull!.isEven.
  • A user might reasonably write this and expect it to work.
• If ! does not participate, then this has type bool, and the meaning is `a!.zeroOrNull!.isEven
  • It's possible that this is what the user intended when they wrote this, but seems less likely.

Summarizing, if we agree with my evaluations above... let's say that the user falls into the pit of success if they get the interpretation they expected, the pit of non-failure if they get an error, and the pit of failure if they sometimes get the wrong behavior, and the pit of despair if the always get the wrong behavior.

If ! participates, we get the following:

Expression	Intention	Behavior	Which pit
a?.zero!	(a?.zero)!	Error	Non-failure
a?.zeroOrNull!	Unknown	(a == null) ? null : a.zeroOrNull!	Failure
a?.zero!.isEven	`(a?.zero)!.isEven	Error	Non-failure
a?.zeroOrNull!.isEven	More likely (a == null) ? null : a.zeroOrNull!.isEven	(a == null) ? null : a.zeroOrNull!.isEven	Likely success

If ! does not participate, we get the following:

Expression	Intention	Behavior	Which pit
a?.zero!	(a?.zero)!	(a?.zero)!	Success
a?.zeroOrNull!	Unknown	a!.zeroOrNull!	Failure
a?.zero!.isEven	`(a?.zero)!.isEven	`(a?.zero)!.isEven	Success
a?.zeroOrNull!.isEven	More likely (a == null) ? null : a.zeroOrNull!.isEven	a!.zeroOrNull!.isEven	Likely failure

Does that summary seem fair? The likely success/failure example could be argued to be a failure. Other things that I got wrong? Are there other forms we should be considering?

[lrhn]

@munificent If a user writes list?.first! and expects a non-nullable result, they get a nullable result and will likely recognize this because nullable results are highly visible. They hopefully then writes (list?.first)!, which is not the same as list!.first when first can be nullable. The (list?.first)! will throw if either list or list.first is null. However, that's not a usage I'd recommend, you should not be using ! to report null errors, you should use it when you know that it won't throw, and even if you do use it just to report null errors, I'd recommend list!.first! instead as more readable.

@leafpetersen The summary seems fine. It's worth noticing that the two Failure entries are both where the intention is unknown. That is, we don't know whether it's a failure or success.

I'll still argue that consistency and simplicity are sufficient to choose ! always participating in the null-shortening. rather than trying to guess at the user intent.

• All other suffix operations participate.
• The ! participates in cascades: a?..zero!, a?..zeroOrNull!, a?..zero!.isEven and z?.zeroOrNull!.isEven all include the !, and x?..foo should always be equivalent to x?.foo when there is only one part of the casecade and the value isn't used.
• It's easy to explain: "! is a suffix operator", "?. can always be replaced by !.".

Also, if ! does not participate, then there is no way to make it. a?.foo! cannot be converted to! applied to foo in any way, it means the same as (a?.foo)!, but if it does participate, you can use parentheses to make it non-participating: a?.foo! and (a?.foo)! means different things, both of which might be the intent of someone.

[eernstg]

@leafpetersen wrote:

If ! participates, we get the following: ...
If ! does not participate, we get the following: ...

It looks like there's a rather strong argument in favor of ! not participating, because we have two outcomes of 'Success' over 'Non-failure' for the latter and a single, weaker 'Likely success' over a 'Likely failure' for the former.

However, I agree with @lrhn that it is more consistent to let ! participate than not. I also think it's important that "the other choice" cannot directly be expressed if ! does not participate.

But even more, we can't say that we are avoiding the parentheses: Being a selector, ! won't apply to any other expression than t <selector>* where t is <primary> or <constructorInvocation>, plus a similar situation in a <cascadeSectionTail>. For instance, a + b! won't mean (a + b)!, simply because a selector can't be applied to a + b. So the intuition that we can "just add ! at the end of any expression" won't work, it generally parses as applied to some subexpression near the end of the expression as a whole. This means that the parentheses will be needed very frequently in any case, and it won't help developers much to allow them to omit the parentheses in very special cases like a?.b!.

I believe we can't change ! to be a postfix operator, because that's deeply breaking, and then we couldn't even have a.b!.c().

Next, I'm not convinced that a developer who writes a?.zero! actually intends it to mean (a?.zero)!. That would be natural to assume if it would work to say "just add ! at the end", but we know that this won't work in general. Also, in the case where the intention is actually (a?.zero)!, the natural way to write it would be a!.zero. Do we really want to bend the language design in a direction where "a?.zero! works", if it should have been written differently in the first place?

The same consideration applies to a?.zero!.isEven: If the intention is (a?.zero)!.isEven then the proper way to write it would presumably be a!.zero.isEven.

What could the intention be for a?.zeroOrNull!? I find it natural to assume that ! participates in null-shorting, in which case it is a null check on the zeroOrNull that we will apply when we have already detected that a is non-null. This works, so why would that be a failure? The argument would presumably be that this is useless: The whole expression is still nullable. That is true, but even that could make sense: We may have business logic reasons for insisting that zeroOrNull returns a non-null value when we call it, and we might still be happy about the whole expression evaluating to null in the case where a is null.

With a?.zeroOrNull!.isEven the intention matches what I described, and I don't see why that shouldn't be a plain success. ;-)

So I'd still prefer that ! participates in null-shorting.

However, should we then emit a hint in the case where ! occurs at the end of an expression (that is: as the last selector) and the expression type is potentially nullable? As I mentioned above, I think there are legitimate usages. But it is possible that this is so rare that we'd prefer to hint against it.

[munificent]

@leafpetersen, thanks for writing up all of the interesting cases. When I look at them, the question I find myself asking is, "If a user wrote that, would it be useful?" By that I mean:

1. Does it do something meaningful?
2. Is there no better way to express the same behavior?

The only two cases that I see being useful—where I would be happy to see a user write them—are:

a?.zeroOrNull!
a?.zeroOrNull!.isEven

And they are only useful if ! null-shorts. The second case here is particularly compelling. I think it would be actively frustrating if I could not write that, akin to the pain we feel today when you want to stuff an await in the middle of a method chain and can't.

If we short, the other two cases become pointless but harmless:

a?.zero!
a?.zero!.isEven

Fortunately, they give you a static warning letting you know that what you wrote is pointless.

On the other hand, if we don't short, every one of these cases becomes what I'd consider bad code. There's always a better way to express the same thing:

a?.zero!              // Should instead be: a!.zero
a?.zeroOrNull!        // Should instead be: a!.zeroOrNull!
a?.zero!.isEven       // Should instead be: a!.zero.isEven
a?.zeroOrNull!.isEven // Should instead be: a!.zeroOrNull!.isEven

I see a compelling argument that shorting is the right behavior. I was worried that it would be hard to explain why an expression that textually ends with ! might have a nullable type, but now I could explain how that behavior is better than the alternative.

[leafpetersen]

So, oddly, I find myself becoming a hold out here.

@munificent

"If a user wrote that, would it be useful?" By that I mean:

I actually think this is the wrong question, or at least, only one of the questions that needs to be asked. The question I find myself asking is "what did the user most likely intend?". And I cannot convince myself that a user who writes a?.b! actually intended to only apply the ! to the read of b and not to the entire expression. We may think of ! as a selector, but users, I claim will not. They will think of !. as a selector, and ! as a postfix operator. I see a very clear pit of failure here where:

• a user writes: takesInt(a?.b)
• the analyzer yells at them for passing a nullable thing where a non-nullable thing was expected
• the user does the completely obvious fix and writes takesInt(a?.b!)
• the analyzer yells at them for passing a nullable thing where a non-nullable thing was expected
• the user is sad and concludes that Dart is one weird language

@lrhn

• It's easy to explain: "! is a suffix operator", "?. can always be replaced by !.".

I strongly doubt this statement.

I'm fairly strongly of the opinion right now that if we do this, we should make it an error to terminate a null-shorting expression with a ! selector, because it almost never is what the user wants, and when it is, there are better ways to write it.

I think that if we allow it to be the case that an expression of the form e! can have nullable type, this will be very, very confusing to users.

[eernstg]

Here's a similar story:

• a user writes: takesInt(a?.b)
• the analyzer yells at them for passing a nullable thing where a non-nullable thing was expected
• the user does the completely obvious fix and writes takesInt(a!.b)

Writing a?.<anything> is an explicit request for a nullable result, and it seems reasonable to say that the toolbox for removing nullability should include both "change ?. to !." and "add ! at the end".

We can still hint in the cases like a?.zeroOrNull! where e! has a nullable type, at least in situations where the context accepts null (such that the developer doesn't get any other diagnostics).

[lrhn]

Summary: I firmly believe that a?.b!.c must have the ! participating, anything else is strictly less useful and potentially dangerous, and making a?.b! by itself not participate is also useless (and dangerous).
The only thing I think could be potentially useful, is making a?.b! invalid.
I just don't think the extra exception is worth the complexity. I'd rather encourage the analyzer to provide hints whenever a ! is used unnecessarily, where the non-nullability of the expression isn't actually need, and that's not just in that one case.

(Basically: What Bob said!)

---

• the user does the completely obvious fix and writes takesInt(a?.b!)
• the analyzer yells at them for passing a nullable thing where a non-nullable thing was expected

The analyzer also yells at them for using a ! on a non-nullable thing. If it's a good error message, it tells them that b has a type of int.

the user is sad and concludes that Dart is one weird language

I definitely do not want a?.b!.c to not include !.c in the null-shortening. If we do that, then they can truly claim that Dart is a weird language. When you grok that ?. is now shortening, it looks exactly like something which should shorten.
One thing I have repeatedly mentioned as a design goal is that the user to expect that changing e?.select to e!.selector is always allowed if you know e evaluates to a non-null value. Since we shorten e?.foo?.bar, we should shorten e?.foo!.bar too.

Then we can still make not make a trailing ! not participate in the shortening, but that's not actually useful. Doing x?.y! where ! does not participate is almost invariably an error, and if not, it should be written as x!.y! (or even x!.y if y happens to be non-nullable). Allowing it, especially silently is directly harmful. The expression looks like it's non-nullable, but it's internally inconsistent because x?. only makes sense if x can actually be null, otherwise you should write x!..
Then it's better to completely disallow a trailing ! because it's never going to be the right thing.

Spreading !s blindly around your code is not something we want to encourage. Allowing it silently is doing the users a disservice.

If we let a trailing ! participate in the null-shortening, and you write var x = a?.b! you are effectively asserting that b is not null. It's slightly weird because it's technically unnecessary, but it might be considered useful as a run-time sanity check.

If we make a trailing ! on a null-shortening into an error, then a?.b! is invalid. There is no practical use for it in the type system, so it's no big loss. It's an extra special case, though, and it's not very consistent.
Should we disallow a..b! as well, the ! is equally invalid there. (I'd actually say "yes", because I still want cascades and null-shortening to behave similarly).
How about a.b!; where the value is not used (expression statement, any void context)? The ! is also unnecessary there.
Or int? x = a.b!; - the ! has no type effect, it's just asserting that b is not null, even though it's not important. That's exactly the same as a?.b! where ! participates in the null shortening.

If we start disallowing useless things, I think we should be consistent about it, otherwise it's just a weird wart on the language, not a useful feature.

If the user gets confused because adding ! doesn't change the type, they need to understand precedence. We should help them with that, perhaps with better warning or error messages, not by making the code do something which isn't useful anyway.
There are many other expressions where adding a ! does not make the expression non-nullable:

• a + b!
• a ? b : c!
• a?..foo!

We expect users to recognize that because they know how ! binds, and how to parse the structure of an expression.
Making ! act differently between x?.y! and x?.y!.foo is not helping that.

I want users to think of ?. the same way they think of ?.., and they should have the same extent.
You need parentheses in the same situations where you want to limit the scope of the cascade/null-shortening.

• (a?..foo.bar).toString() // limits cascade to bar
• (a?.foo.bar).toString() // limits the null-shortening to bar.

[leafpetersen]

Here's a similar story:

• a user writes: takesInt(a?.b)
• the analyzer yells at them for passing a nullable thing where a non-nullable thing was expected
• the user does the completely obvious fix and writes takesInt(a!.b)

I tested this out on a couple of people last night, neither of them saw this as an option, and neither of them did this.