Skip to content

Commit 506fc08

Browse files
authored
Merge pull request #12767 from dotnet/merges/main-to-release/dev17.2
Merge main to release/dev17.2
2 parents e7d8762 + 45518d2 commit 506fc08

File tree

90 files changed

+9474
-5674
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

90 files changed

+9474
-5674
lines changed

docs/debug-emit.md

Lines changed: 0 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -504,7 +504,3 @@ Some design-time services are un-implemented by F#:
504504
* Unimplemented: [Proximity expressions](https://github.com/dotnet/fsharp/issues/4271) (for Autos window)
505505

506506
These are major holes in the F# experience and should be implemented.
507-
508-
### Missing debug emit for F# Interactive
509-
510-
For F# Interactive [we do not currently emit debug information for script code](https://github.com/dotnet/fsharp/issues/5457). This is because of a missing piece of functionality in the Reflection.Emit APIs, and means we have to change our approach to emitting code fragments in F# Interactive to no longer use dynamic assemblies.

docs/fsi-emit.md

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
---
2+
title: F# Interactive Emit
3+
category: Compiler Internals
4+
categoryindex: 200
5+
index: 375
6+
---
7+
# F# Interactive Code Generation
8+
9+
F# Interactive (`dotnet fsi`) accepts incremental code fragments. This capability is also used by [hosted execution capability of the FSharp.Compiler.Service API](fcs/interactive.fsx) which is used to build the F# kernel for .NET Interactive notebooks.
10+
11+
Historically F# Interactive code was emitted into a single dynamic assembly using Reflection.Emit and ilreflect.fs (meaning one assembly that is continually growing). However, .NET Core Reflection.Emit does not support the emit of debug symbols for dynamic assemblies, so in Feb 2022 we switched to emitting multiple non-dynamic assemblies (meaning assemblies dynamically created in-memory using ilwrite.fs, and loaded, but not growing).
12+
13+
The assemblies are named:
14+
15+
`FSI-ASSEMBLY1`
16+
`FSI-ASSEMBLY2`
17+
18+
etc.
19+
20+
## Compat switch
21+
22+
There is a switch `fsi --multiemit` that turns on the use of multi-assembly generation (when it is off, we use Reflection Emit for single-dynamic-assembly generation). This is on by default for .NET Core, and off by default for .NET Framework for compat reasons.
23+
24+
## Are multiple assemblies too costly?
25+
26+
There is general assumption in this that on modern dev machines (where users execute multiple interactions) then generating 50, 100 or 1,000 or 10,000 dynamic assemblies by repeated manual execution of code is not a problem: the extra overheads of multiple assemblies compared to one dynamic assembly is of no real significance in developer REPL scenarios given the vast amount of memory available on modern 64-bit machines.
27+
28+
Quick check: adding 10,000 `let x = 1;;` interactions to .NET Core `dotnet fsi` adds about 300MB to the FSI.EXE process, meaning 30K/interaction. A budget of 1GB for interactive fragments (reasonable on a 64-bit machine), and an expected maximum of 10000 fragments before restart (that's a lot!), then each fragment can take up to 100K. This is well below the cost of a new assembly.
29+
30+
Additionally, these costs are not substantially reduced if `--multiemit` is disabled, so they've always been the approximate costs of F# Interactive fragment generation.
31+
32+
## Internals and accessibility across fragments
33+
34+
Generating into multiple assemblies raises issues for some things that are assembly bound such as "internals" accessibility. In a first iteration of this we had a failing case here:
35+
36+
```fsharp
37+
> artifacts\bin\fsi\Debug\net50\fsi.exe --optimize-
38+
...
39+
// Fragment 1
40+
> let internal f() = 1;;
41+
val internal f: unit -> int
42+
43+
// Fragment 2 - according to existing rules it is allowed to access internal things of the first
44+
f();;
45+
System.MethodAccessException: Attempt by method '<StartupCode$FSI_0003>.$FSI_0003.main@()' to access method 'FSI_0002.f()' failed.
46+
at <StartupCode$FSI_0003>.$FSI_0003.main@()
47+
```
48+
49+
This is because we are now generating into multiple assemblies. Another bug was this:
50+
51+
```fsharp
52+
> artifacts\bin\fsi\Debug\net50\fsi.exe --optimize+
53+
...
54+
// Fragment 1 - not `x` becomes an internal field of the class
55+
> type C() =
56+
> let mutable x = 1
57+
> member _.M() = x
58+
> ;;
59+
...
60+
// Fragment 2 - inlining 'M()' gave an access to the internal field `x`
61+
> C().M();;
62+
...<bang>...
63+
```
64+
65+
According to the current F# scripting programming model (the one checked in the editor), the "internal" thing should be accessible in subsequent fragments. Should this be changed? No:
66+
67+
* It's very hard to adjust the implementation of the editor scripting model to consider fragments delimited by `;;` to be different assemblies, whether in the editor or in F# Interactive.
68+
* And would we even want to? It's common enough for people to debug code scattered with "internal" declarations.
69+
* In scripts, the `;;` aren't actually accurate markers for what will or won't be sent to F# Interactive, which get added implicitly.
70+
71+
For example, consider the script
72+
73+
```fsharp
74+
let internal f() = 1;;
75+
f();;
76+
```
77+
78+
In the editor should this be given an error or not? That is, should the `;;` be seen as accurate indicators of separate script fragments? (Answer: yes if we know the script will be piped-to-input, no if the script is used as a single file entry - when the `;;` are ignored)
79+
80+
* Further, this would be a breaking change, e.g. it could arise in an automated compat situation if people are piping into standard input and the input contains `;;` markers.
81+
82+
Because of this we emit IVTs for the next 30 `FSI-ASSEMBLYnnn` assemblies on each assembly fragment, giving a warning when an internal thing is accessed across assembly boundaries within that 30 (reporting it as a deprecated feature), and give an error if internal access happens after that.
83+
84+
From a compat perspective this seems reasonable, and the compat flag is available to return the whole system to generate-one-assembly behavior.

src/fsharp/AttributeChecking.fs

Lines changed: 3 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -125,7 +125,7 @@ type AttribInfo =
125125
/// Check custom attributes. This is particularly messy because custom attributes come in in three different
126126
/// formats.
127127
let AttribInfosOfIL g amap scoref m (attribs: ILAttributes) =
128-
attribs.AsList |> List.map (fun a -> ILAttribInfo (g, amap, scoref, a, m))
128+
attribs.AsList() |> List.map (fun a -> ILAttribInfo (g, amap, scoref, a, m))
129129

130130
let AttribInfosOfFS g attribs =
131131
attribs |> List.map (fun a -> FSAttribInfo (g, a))
@@ -443,7 +443,7 @@ let MethInfoIsUnseen g (m: range) (ty: TType) minfo =
443443
// We are only interested in filtering out the method on System.Object, so it is sufficient
444444
// just to look at the attributes on IL methods.
445445
if tcref.IsILTycon then
446-
tcref.ILTyconRawMetadata.CustomAttrs.AsArray
446+
tcref.ILTyconRawMetadata.CustomAttrs.AsArray()
447447
|> Array.exists (fun attr -> attr.Method.DeclaringType.TypeSpec.Name = typeof<TypeProviderEditorHideMethodsAttribute>.FullName)
448448
else
449449
false
@@ -452,9 +452,7 @@ let MethInfoIsUnseen g (m: range) (ty: TType) minfo =
452452
false
453453
#endif
454454

455-
//let isUnseenByBeingTupleMethod () = isAnyTupleTy g ty
456-
457-
isUnseenByObsoleteAttrib () || isUnseenByHidingAttribute () //|| isUnseenByBeingTupleMethod ()
455+
isUnseenByObsoleteAttrib () || isUnseenByHidingAttribute ()
458456

459457
/// Indicate if a property has 'Obsolete' or 'CompilerMessageAttribute'.
460458
/// Used to suppress the item in intellisense.

src/fsharp/CompilerConfig.fs

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -447,27 +447,27 @@ type TcConfigBuilder =
447447
mutable showTimes: bool
448448
mutable showLoadedAssemblies: bool
449449
mutable continueAfterParseFailure: bool
450+
450451
#if !NO_EXTENSIONTYPING
451452
/// show messages about extension type resolution?
452453
mutable showExtensionTypeMessages: bool
453454
#endif
454455

455-
/// pause between passes?
456+
/// Pause between passes?
456457
mutable pause: bool
457-
/// whenever possible, emit callvirt instead of call
458+
459+
/// Whenever possible, emit callvirt instead of call
458460
mutable alwaysCallVirt: bool
459461

460-
/// if true, strip away data that would not be of use to end users, but is useful to us for debugging
461-
// REVIEW: "stripDebugData"?
462+
/// If true, strip away data that would not be of use to end users, but is useful to us for debugging
462463
mutable noDebugAttributes: bool
463464

464-
/// if true, indicates all type checking and code generation is in the context of fsi.exe
465+
/// If true, indicates all type checking and code generation is in the context of fsi.exe
465466
isInteractive: bool
466-
isInvalidationSupported: bool
467467

468-
/// used to log sqm data
468+
isInvalidationSupported: bool
469469

470-
/// if true - every expression in quotations will be augmented with full debug info (filename, location in file)
470+
/// If true - every expression in quotations will be augmented with full debug info (filename, location in file)
471471
mutable emitDebugInfoInQuotations: bool
472472

473473
mutable exename: string option
@@ -477,9 +477,14 @@ type TcConfigBuilder =
477477

478478
/// When false FSI will lock referenced assemblies requiring process restart, false = disable Shadow Copy false (*default*)
479479
mutable shadowCopyReferences: bool
480+
480481
mutable useSdkRefs: bool
482+
481483
mutable fxResolver: FxResolver option
482484

485+
// Is F# Interactive using multi-assembly emit?
486+
mutable fsiMultiAssemblyEmit: bool
487+
483488
/// specify the error range for FxResolver
484489
rangeForErrors: range
485490

@@ -660,6 +665,7 @@ type TcConfigBuilder =
660665
shadowCopyReferences = false
661666
useSdkRefs = true
662667
fxResolver = None
668+
fsiMultiAssemblyEmit = true
663669
internalTestSpanStackReferring = false
664670
noConditionalErasure = false
665671
pathMap = PathMap.empty
@@ -921,6 +927,7 @@ type TcConfig private (data: TcConfigBuilder, validate: bool) =
921927
#endif
922928
None, data.legacyReferenceResolver.Impl.HighestInstalledNetFrameworkVersion()
923929

930+
member _.fsiMultiAssemblyEmit = data.fsiMultiAssemblyEmit
924931
member x.FxResolver = data.FxResolver
925932
member x.primaryAssembly = data.primaryAssembly
926933
member x.noFeedback = data.noFeedback

src/fsharp/CompilerConfig.fsi

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -272,6 +272,7 @@ type TcConfigBuilder =
272272
mutable shadowCopyReferences: bool
273273
mutable useSdkRefs: bool
274274
mutable fxResolver: FxResolver option
275+
mutable fsiMultiAssemblyEmit: bool
275276
rangeForErrors: range
276277
sdkDirOverride: string option
277278

@@ -458,6 +459,9 @@ type TcConfig =
458459
member isInteractive: bool
459460
member isInvalidationSupported: bool
460461

462+
/// Indicates if F# Interactive is using single-assembly emit via Reflection.Emit, where internals are available.
463+
member fsiMultiAssemblyEmit: bool
464+
461465
member xmlDocInfoLoader: IXmlDocumentationInfoLoader option
462466

463467
member FxResolver: FxResolver

src/fsharp/CompilerImports.fs

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -650,7 +650,7 @@ let MakeScopeRefForILModule (ilModule: ILModuleDef) =
650650
| None -> ILScopeRef.Module (mkRefToILModule ilModule)
651651

652652
let GetCustomAttributesOfILModule (ilModule: ILModuleDef) =
653-
(match ilModule.Manifest with Some m -> m.CustomAttrs | None -> ilModule.CustomAttrs).AsList
653+
(match ilModule.Manifest with Some m -> m.CustomAttrs | None -> ilModule.CustomAttrs).AsList()
654654

655655
let GetAutoOpenAttributes ilModule =
656656
ilModule |> GetCustomAttributesOfILModule |> List.choose TryFindAutoOpenAttr
@@ -669,7 +669,7 @@ type RawFSharpAssemblyDataBackedByFileOnDisk (ilModule: ILModuleDef, ilAssemblyR
669669
member _.TryGetILModuleDef() = Some ilModule
670670

671671
member _.GetRawFSharpSignatureData(m, ilShortAssemName, filename) =
672-
let resources = ilModule.Resources.AsList
672+
let resources = ilModule.Resources.AsList()
673673
let sigDataReaders =
674674
[ for iresource in resources do
675675
if IsSignatureDataResource iresource then
@@ -688,7 +688,7 @@ type RawFSharpAssemblyDataBackedByFileOnDisk (ilModule: ILModuleDef, ilAssemblyR
688688

689689
member _.GetRawFSharpOptimizationData(m, ilShortAssemName, filename) =
690690
let optDataReaders =
691-
ilModule.Resources.AsList
691+
ilModule.Resources.AsList()
692692
|> List.choose (fun r -> if IsOptimizationDataResource r then Some(GetOptimizationDataResourceName r, (fun () -> r.GetBytes())) else None)
693693

694694
// Look for optimization data in a file
@@ -733,14 +733,14 @@ type RawFSharpAssemblyData (ilModule: ILModuleDef, ilAssemblyRefs) =
733733
member _.TryGetILModuleDef() = Some ilModule
734734

735735
member _.GetRawFSharpSignatureData(_, _, _) =
736-
let resources = ilModule.Resources.AsList
736+
let resources = ilModule.Resources.AsList()
737737
[ for iresource in resources do
738738
if IsSignatureDataResource iresource then
739739
let ccuName = GetSignatureDataResourceName iresource
740740
yield (ccuName, fun () -> iresource.GetBytes()) ]
741741

742742
member _.GetRawFSharpOptimizationData(_, _, _) =
743-
ilModule.Resources.AsList
743+
ilModule.Resources.AsList()
744744
|> List.choose (fun r -> if IsOptimizationDataResource r then Some(GetOptimizationDataResourceName r, (fun () -> r.GetBytes())) else None)
745745

746746
member _.GetRawTypeForwarders() =
@@ -1499,7 +1499,7 @@ and [<Sealed>] TcImports(tcConfigP: TcConfigProvider, initialResolutions: TcAsse
14991499

15001500
let phase2 () =
15011501
#if !NO_EXTENSIONTYPING
1502-
ccuinfo.TypeProviders <- tcImports.ImportTypeProviderExtensions (ctok, tcConfig, filename, ilScopeRef, ilModule.ManifestOfAssembly.CustomAttrs.AsList, ccu.Contents, invalidateCcu, m)
1502+
ccuinfo.TypeProviders <- tcImports.ImportTypeProviderExtensions (ctok, tcConfig, filename, ilScopeRef, ilModule.ManifestOfAssembly.CustomAttrs.AsList(), ccu.Contents, invalidateCcu, m)
15031503
#endif
15041504
[ResolvedImportedAssembly ccuinfo]
15051505
phase2
@@ -1593,7 +1593,7 @@ and [<Sealed>] TcImports(tcConfigP: TcConfigProvider, initialResolutions: TcAsse
15931593
match ilModule.TryGetILModuleDef() with
15941594
| None -> () // no type providers can be used without a real IL Module present
15951595
| Some ilModule ->
1596-
let tps = tcImports.ImportTypeProviderExtensions (ctok, tcConfig, filename, ilScopeRef, ilModule.ManifestOfAssembly.CustomAttrs.AsList, ccu.Contents, invalidateCcu, m)
1596+
let tps = tcImports.ImportTypeProviderExtensions (ctok, tcConfig, filename, ilScopeRef, ilModule.ManifestOfAssembly.CustomAttrs.AsList(), ccu.Contents, invalidateCcu, m)
15971597
ccuinfo.TypeProviders <- tps
15981598
#else
15991599
()

src/fsharp/IlxGen.fs

Lines changed: 17 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -1565,11 +1565,11 @@ type TypeDefBuilder(tdef: ILTypeDef, tdefDiscards) =
15651565
let gnested = TypeDefsBuilder()
15661566

15671567
member b.Close() =
1568-
tdef.With(methods = mkILMethods (tdef.Methods.AsList @ ResizeArray.toList gmethods),
1569-
fields = mkILFields (tdef.Fields.AsList @ ResizeArray.toList gfields),
1570-
properties = mkILProperties (tdef.Properties.AsList @ HashRangeSorted gproperties ),
1571-
events = mkILEvents (tdef.Events.AsList @ ResizeArray.toList gevents),
1572-
nestedTypes = mkILTypeDefs (tdef.NestedTypes.AsList @ gnested.Close()))
1568+
tdef.With(methods = mkILMethods (tdef.Methods.AsList() @ ResizeArray.toList gmethods),
1569+
fields = mkILFields (tdef.Fields.AsList() @ ResizeArray.toList gfields),
1570+
properties = mkILProperties (tdef.Properties.AsList() @ HashRangeSorted gproperties ),
1571+
events = mkILEvents (tdef.Events.AsList() @ ResizeArray.toList gevents),
1572+
nestedTypes = mkILTypeDefs (tdef.NestedTypes.AsList() @ gnested.Close()))
15731573

15741574
member b.AddEventDef edef = gevents.Add edef
15751575

@@ -1618,11 +1618,11 @@ and TypeDefsBuilder() =
16181618
let tdef = b.Close()
16191619
// Skip the <PrivateImplementationDetails$> type if it is empty
16201620
if not eliminateIfEmpty
1621-
|| not tdef.NestedTypes.AsList.IsEmpty
1622-
|| not tdef.Fields.AsList.IsEmpty
1623-
|| not tdef.Events.AsList.IsEmpty
1624-
|| not tdef.Properties.AsList.IsEmpty
1625-
|| not (Array.isEmpty tdef.Methods.AsArray) then
1621+
|| not (tdef.NestedTypes.AsList()).IsEmpty
1622+
|| not (tdef.Fields.AsList()).IsEmpty
1623+
|| not (tdef.Events.AsList()).IsEmpty
1624+
|| not (tdef.Properties.AsList()).IsEmpty
1625+
|| not (Array.isEmpty (tdef.Methods.AsArray())) then
16261626
yield tdef ]
16271627

16281628
member b.FindTypeDefBuilder nm =
@@ -1704,7 +1704,9 @@ type AssemblyBuilder(cenv: cenv, anonTypeTable: AnonTypeGenerationTable) as mgbu
17041704
let ilFieldDefs =
17051705
mkILFields
17061706
[ for _, fldName, fldTy in flds ->
1707-
let fdef = mkILInstanceField (fldName, fldTy, None, ILMemberAccess.Private)
1707+
// The F# Interactive backend may split to multiple assemblies.
1708+
let access = (if cenv.opts.isInteractive then ILMemberAccess.Public else ILMemberAccess.Private)
1709+
let fdef = mkILInstanceField (fldName, fldTy, None, access)
17081710
fdef.With(customAttrs = mkILCustomAttrs [ g.DebuggerBrowsableNeverAttribute ]) ]
17091711

17101712
// Generate property definitions for the fields compiled as properties
@@ -4588,7 +4590,7 @@ and GenFormalReturnType m cenv eenvFormal returnTy : ILReturn =
45884590
| None -> ilRet
45894591
| Some ty ->
45904592
match GenReadOnlyAttributeIfNecessary cenv.g ty with
4591-
| Some attr -> ilRet.WithCustomAttrs (mkILCustomAttrs (ilRet.CustomAttrs.AsList @ [attr]))
4593+
| Some attr -> ilRet.WithCustomAttrs (mkILCustomAttrs (ilRet.CustomAttrs.AsList() @ [attr]))
45924594
| None -> ilRet
45934595

45944596
and instSlotParam inst (TSlotParam(nm, ty, inFlag, fl2, fl3, attrs)) =
@@ -5094,7 +5096,7 @@ and GenStaticDelegateClosureTypeDefs cenv (tref: ILTypeRef, ilGenParams, attrs,
50945096
// Apply the abstract attribute, turning the sealed class into abstract sealed (i.e. static class).
50955097
// Remove the redundant constructor.
50965098
tdefs |> List.map (fun td -> td.WithAbstract(true)
5097-
.With(methods= mkILMethodsFromArray (td.Methods.AsArray |> Array.filter (fun m -> not m.IsConstructor))))
5099+
.With(methods= mkILMethodsFromArray (td.Methods.AsArray() |> Array.filter (fun m -> not m.IsConstructor))))
50985100

50995101
and GenGenericParams cenv eenv tps =
51005102
tps |> DropErasedTypars |> List.map (GenGenericParam cenv eenv)
@@ -8648,9 +8650,7 @@ open System
86488650

86498651
/// The lookup* functions are the conversions available from ilreflect.
86508652
type ExecutionContext =
8651-
{ LookupFieldRef: ILFieldRef -> FieldInfo
8652-
LookupMethodRef: ILMethodRef -> MethodInfo
8653-
LookupTypeRef: ILTypeRef -> Type
8653+
{ LookupTypeRef: ILTypeRef -> Type
86548654
LookupType: ILType -> Type }
86558655

86568656
// A helper to generate a default value for any System.Type. I couldn't find a System.Reflection
@@ -8672,7 +8672,7 @@ let LookupGeneratedValue (amap: ImportMap) (ctxt: ExecutionContext) eenv (v: Val
86728672
try
86738673
// Convert the v.Type into a System.Type according to ilxgen and ilreflect.
86748674
let objTyp() =
8675-
let ilTy = GenType amap v.Range TypeReprEnv.Empty v.Type (* TypeReprEnv.Empty ok, not expecting typars *)
8675+
let ilTy = GenType amap v.Range TypeReprEnv.Empty v.Type
86768676
ctxt.LookupType ilTy
86778677
// Lookup the compiled v value (as an object).
86788678
match StorageForVal amap.g v.Range v eenv with

src/fsharp/IlxGen.fsi

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -79,8 +79,6 @@ type public IlxGenResults =
7979
/// Used to support the compilation-inversion operations "ClearGeneratedValue" and "LookupGeneratedValue"
8080
type ExecutionContext =
8181
{
82-
LookupFieldRef: ILFieldRef -> FieldInfo
83-
LookupMethodRef: ILMethodRef -> MethodInfo
8482
LookupTypeRef: ILTypeRef -> Type
8583
LookupType: ILType -> Type
8684
}

0 commit comments

Comments
 (0)