Some new co19_2 tests of experimental features failing on the analyzer

[whesse]

Some co19_2 tests of experimental features are now behaving differently, since the

// SharedOptions=--enable-experiment=constant-update-2018
and
// SharedOptions=--enable-experiment=set-literals

annotations in co19_2 test source are being obeyed after landing
https://dart-review.googlesource.com/c/sdk/+/88704
which makes test.py treat co19_2 tests the same as all other tests.

These have been preapproved, to land the CL, but these failures should be investigated and fixed.
If they are mistakes in the test, issues should be filed against the dart-lang/co19 repository to fix them.

BOT TEST RESULT EXPECTED
analyzer-linux-release/ co19_2/LanguageFeatures/Constant_update2018/TypeTestOperator_A04_t01 Pass Fail
analyzer-linux-release/ co19_2/LanguageFeatures/Constant_update2018/TypeTestOperator_A04_t02 Pass Fail
analyzer-linux-release/ co19_2/LanguageFeatures/Set-literals/semantics_A01_t01/01 Pass Fail
analyzer-linux-release/ co19_2/LanguageFeatures/Set-literals/semantics_A01_t01/02 Pass Fail
analyzer-linux-release/ co19_2/LanguageFeatures/Set-literals/semantics_A01_t01/09 Pass Fail
analyzer-linux-release/ co19_2/LanguageFeatures/Set-literals/semantics_A01_t01/10 Pass Fail
analyzer-linux-release/ co19_2/LanguageFeatures/Set-literals/set_literals_A03_t01/03 Pass Fail
analyzer-linux-release/ co19_2/LanguageFeatures/Set-literals/set_literals_A03_t01/04 Pass Fail
analyzer-linux-release/ co19_2/LanguageFeatures/Set-literals/set_literals_A03_t02/01 Pass Fail
analyzer-linux-release/ co19_2/LanguageFeatures/Set-literals/set_literals_A03_t02/02 Pass Fail

[whesse]

Only the first 4 Set-literals test failures seem to be true failures of the analyzer: the following two expressions should be static errors (because an empty { } is a Map, not a Set)
// var v1 = {}; //# 01: compile-time error
// var v2 = const {}; //# 02: compile-time error

The last 4 errors in set_literals_A03_t01/03 and 04 and _t02/01 and 02 seem to be errors in the test - I have filed issue dart-lang/co19#220 about them.

[bwilkerson]

Thanks for looking at the failures!

Actually, many of those tests are broken. I triaged them yesterday and opened issues, most of which have now been fixed. Here are the issues I opened (one per file rather than one per test):

dart-lang/co19#214
dart-lang/co19#215
dart-lang/co19#216
dart-lang/co19#217
dart-lang/co19#218
dart-lang/co19#219

[bwilkerson]

(But as far as I know the fixes have not yet been rolled into our repo.)

[stereotype441]

As of Friday (581af6e) co19 was rolled to dart-lang/co19@31f7dc1, so it should contain the fixes for all the issues Brian filed. But it looks like we are still failing 7 tests:

                                  name                                   expected   unexpected_result  
 ---------------------------------------------------------------------- ---------- ------------------- 
  co19_2/LanguageFeatures/Set-literals/exact_types_of_literals_A01_t03   Pass       CompileTimeError   
  co19_2/LanguageFeatures/Set-literals/semantics_A05_t01                 Pass       CompileTimeError   
  co19_2/LanguageFeatures/Set-literals/set_literals_A03_t01/03           Fail       MissingError       
  co19_2/LanguageFeatures/Set-literals/set_literals_A03_t01/04           Fail       MissingError       
  co19_2/LanguageFeatures/Set-literals/set_literals_A03_t02/01           Fail       MissingError       
  co19_2/LanguageFeatures/Set-literals/set_literals_A03_t02/02           Fail       MissingError       
  co19_2/LanguageFeatures/Set-literals/set_literals_A04_t01              Pass       CompileTimeError   

[stereotype441]

I believe these failures are due to incorrect tests:

• co19_2/LanguageFeatures/Set-literals/exact_types_of_literals_A01_t03
• co19_2/LanguageFeatures/Set-literals/semantics_A05_t01
• co19_2/LanguageFeatures/Set-literals/set_literals_A04_t01

I will follow up with the language team to confirm, and file co19 bugs as needed.

[stereotype441]

I believe these failures are due to analyzer bug #35742:

• co19_2/LanguageFeatures/Set-literals/set_literals_A03_t01/03
• co19_2/LanguageFeatures/Set-literals/set_literals_A03_t01/04
• co19_2/LanguageFeatures/Set-literals/set_literals_A03_t02/01
• co19_2/LanguageFeatures/Set-literals/set_literals_A03_t02/02

@bwilkerson would you mind having a look at how difficult it would be to fix this?

[stereotype441]

The analyzer is now passing all co19 tests of set literals (other than the buggy ones - see dart-lang/co19#230 and dart-lang/co19#231)