[generator] Use an intermediate C# model (#674)

`generator` is an amazingly powerful tool that has proven to be very
versatile over the past decade.  However, many of the remaining bugs
fall into a category that is hard to fix given `generator`'s current
design, which is:

 1. Read in Java model
 2. Make tweaks to Java model
 3. Open `<TYPE>.cs`
 4. Loop through Java model determining what C# to write

The issue with this is that we are deciding **what** to generate
on-the-fly to a write-only stream.  This means that once we've written
to the stream we cannot change it if future information suggests that
we should.

A good example is issue #461.

The Java model is: #461

Given:

	// Java:
	interface AnimatorListener {
	  onAnimationEnd (int p0);
	  onAnimationEnd (int p0, p1);
	}

Looping through this, we see we need to generate an `EventArgs` class for
`AnimatorListener.onAnimationEnd(int p0)`, so we do:

	public partial class OnAnimationEndEventArgs : global::System.EventArgs {
	    int p0;

	    public OnAnimationEndEventArgs (int p0) { this.p0= p0; }

	    public int P0 { get { return p0; } }
	}

Then we get to the second method
`AnimatorListener.onAnimationEnd(int p0, int p1)` and see that we need
to add additional parameters to `OnAnimationEndEventArgs`.  However we
cannot modify the already written class, so we generate a second
`OnAnimationEndEventArgs` with different parameters, which causes
CS0101 compilation errors in C#.

Another example is we can generate an empty `InterfaceConsts` class.
This is because we write the class opening because we think we have
members to place in it (`public static class InterfaceConsts {`), but
as we loop through the members we thought we were going to add they
all get eliminated.  At that point all we can do is close the class
with no members.

The solution to both examples above is "simple": we need to first loop
through the rest of the Java model and determine what we **actually**
need to generate before we start writing anything.  We can then merge
the two `OnAnimationEndEventArgs` classes, or forgo writing an empty
`InterfaceConsts` type.

However, rather than adding this additional logic in each instance
that needs it, there is enough benefit to convert the entire
`generator` architecture to work by first building a C# model of what
needs to be generated that can be tweaked before writing to a file,
resulting in a design of:

 1. Read in Java model
 2. Make tweaks to Java model
 3. ***Build C# model from Java model***
 4. ***Make tweaks to C# model***
 5. Open `<TYPE>.cs`
 6. Loop through C# model, writing code

Having a C# model to tweak should also help in a few other tricky
cases we fail at today, like determining `override`s (#367, #586) and
implementing `abstract` methods for `abstract` classes inheriting an
`interface` (#470).

Additionally, having distinct logic and source writing steps will make
unit testing better.  Currently, the only way to test the *logic* of if
we are going to generate something is to write out the source code and
compare it to "expected" source code.  This means tiny fixes can have
many "expected" files that need to change (#625).

With separate steps, we can have a set of unit tests that test what
code we are writing, and a set that tests the logic of determining what
to generate.  To test the above "2 `EventArgs` classes" example, we can
test a fix by looking at the created C# model to ensure that only a
single combined `OnAnimationEndEventArgs` class is created.  We would
not need to write out the generated source code and compare it to
expected source code.

To assist in this new design, add a new `Xamarin.SourceWriter.dll`
assembly which is responsible for generating the C# source code.
This eliminates a lot of hard to read and error prone code such as:

	writer.WriteLine ("{0}{1}{2}{3}{4} unsafe {5} {6}{7} ({8})",
	        indent,
	        visibility,
	        static_arg,
	        virt_ov,
	        seal,
	        ret,
	        is_explicit ? GetDeclaringTypeOfExplicitInterfaceMethod (method.OverriddenInterfaceMethod) + '.' : string.Empty,
	        method.AdjustedName,
	        method.GetSignature (opt));

and replaces it with:

	var m = new MethodWriter {
	    IsPublic = true,
	    IsUnsafe = true,
	    IsOverride = true,
	    Name = method.AdjustedName
	};

	m.Write (…);

which will emit:

	public unsafe override void DoSomething ()
	{
	}

`Xamarin.SourceWriter.CodeWriter` is a wrapper around a
`System.IO.Stream` that tracks the current indent level, rather than
needing to pass `indent` around to every method.

	cw.WriteLine ("{");
	cw.Indent ();
	// ... write block body ...
	cw.Unindent ();
	cw.WriteLine ("}");

~~ Testing ~~

Many existing unit tests call directly into methods like
`CodeGenerator.WriteProperties()`.  These methods are no longer
available when using `JavaInterop`/`XAJavaInterop`.  These tests were
either changed to use `CodeGenerator.WriteType()` or were moved to only
run for `XamarinAndroid` codegen target.

Tests were updated to ignore whitespace and line endings differences,
as the refactor did not preserve 100% identical whitespace.

Author: jpobst

Java.Interop.sln

@@ -93,6 +93,10 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Java.Interop.Tools.Generato
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "java-source-utils", "tools\java-source-utils\java-source-utils.csproj", "{F46EDFA5-C52A-4F0C-B5A2-5BB67E0D8C74}"
 EndProject
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Xamarin.SourceWriter", "src\Xamarin.SourceWriter\Xamarin.SourceWriter.csproj", "{C5B732C8-7AF3-41D3-B903-AEDFC392E5BA}"
+EndProject
+Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Xamarin.SourceWriter-Tests", "tests\Xamarin.SourceWriter-Tests\Xamarin.SourceWriter-Tests.csproj", "{6CF94627-BA74-4336-88CD-7EDA20C8F292}"
+EndProject
 Global
 	GlobalSection(SharedMSBuildProjectFiles) = preSolution
 		src\Java.Interop.NamingCustomAttributes\Java.Interop.NamingCustomAttributes.projitems*{58b564a1-570d-4da2-b02d-25bddb1a9f4f}*SharedItemsImports = 5
@@ -257,6 +261,14 @@ Global
 		{F46EDFA5-C52A-4F0C-B5A2-5BB67E0D8C74}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{F46EDFA5-C52A-4F0C-B5A2-5BB67E0D8C74}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{F46EDFA5-C52A-4F0C-B5A2-5BB67E0D8C74}.Release|Any CPU.Build.0 = Release|Any CPU
+		{C5B732C8-7AF3-41D3-B903-AEDFC392E5BA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{C5B732C8-7AF3-41D3-B903-AEDFC392E5BA}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{C5B732C8-7AF3-41D3-B903-AEDFC392E5BA}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{C5B732C8-7AF3-41D3-B903-AEDFC392E5BA}.Release|Any CPU.Build.0 = Release|Any CPU
+		{6CF94627-BA74-4336-88CD-7EDA20C8F292}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{6CF94627-BA74-4336-88CD-7EDA20C8F292}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{6CF94627-BA74-4336-88CD-7EDA20C8F292}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{6CF94627-BA74-4336-88CD-7EDA20C8F292}.Release|Any CPU.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE
@@ -301,6 +313,8 @@ Global
 		{C2FD2F12-DE3B-4FB9-A0D3-FA3EF597DD04} = {0998E45F-8BCE-4791-A944-962CD54E2D80}
 		{7F4828AB-3908-458C-B09F-33C74A1368F9} = {271C9F30-F679-4793-942B-0D9527CB3E2F}
 		{F46EDFA5-C52A-4F0C-B5A2-5BB67E0D8C74} = {C8F58966-94BF-407F-914A-8654F8B8AE3B}
+		{C5B732C8-7AF3-41D3-B903-AEDFC392E5BA} = {0998E45F-8BCE-4791-A944-962CD54E2D80}
+		{6CF94627-BA74-4336-88CD-7EDA20C8F292} = {271C9F30-F679-4793-942B-0D9527CB3E2F}
 	EndGlobalSection
 	GlobalSection(ExtensibilityGlobals) = postSolution
 		SolutionGuid = {29204E0C-382A-49A0-A814-AD7FBF9774A5}

Makefile

@@ -28,7 +28,8 @@ TESTS = \
 	bin/Test$(CONFIGURATION)/generator-Tests.dll \
 	bin/Test$(CONFIGURATION)/Xamarin.Android.Tools.ApiXmlAdjuster-Tests.dll \
 	bin/Test$(CONFIGURATION)/Xamarin.Android.Tools.Bytecode-Tests.dll \
-	bin/Test$(CONFIGURATION)/Java.Interop.Tools.Generator-Tests.dll
+	bin/Test$(CONFIGURATION)/Java.Interop.Tools.Generator-Tests.dll \
+	bin/Test$(CONFIGURATION)/Xamarin.SourceWriter-Tests.dll
 
 PTESTS = \
 	bin/Test$(CONFIGURATION)/Java.Interop-PerformanceTests.dll

build-tools/automation/azure-pipelines.yaml

@@ -55,7 +55,7 @@ jobs:
     inputs:
       solution: build-tools/scripts/RunNUnitTests.targets
       configuration: $(Build.Configuration)
-      msbuildArguments: /p:TestAssembly="bin\Test$(Build.Configuration)\generator-Tests.dll;bin\Test$(Build.Configuration)\Java.Interop.Tools.JavaCallableWrappers-Tests.dll;bin\Test$(Build.Configuration)\logcat-parse-Tests.dll;bin\Test$(Build.Configuration)\Xamarin.Android.Tools.ApiXmlAdjuster-Tests.dll;bin\Test$(Build.Configuration)\Xamarin.Android.Tools.Bytecode-Tests.dll;bin\Test$(Build.Configuration)\Java.Interop.Tools.Generator-Tests.dll"
+      msbuildArguments: /p:TestAssembly="bin\Test$(Build.Configuration)\generator-Tests.dll;bin\Test$(Build.Configuration)\Java.Interop.Tools.JavaCallableWrappers-Tests.dll;bin\Test$(Build.Configuration)\logcat-parse-Tests.dll;bin\Test$(Build.Configuration)\Xamarin.Android.Tools.ApiXmlAdjuster-Tests.dll;bin\Test$(Build.Configuration)\Xamarin.Android.Tools.Bytecode-Tests.dll;bin\Test$(Build.Configuration)\Java.Interop.Tools.Generator-Tests.dll;bin\Test$(Build.Configuration)\Xamarin.SourceWriter-Tests.dll"
     condition: succeededOrFailed()
 
   - task: PublishTestResults@2

build-tools/automation/templates/core-tests.yaml

@@ -51,6 +51,13 @@ steps:
     arguments: bin/Test$(Build.Configuration)/Java.Interop.Tools.JavaSource-Tests.dll
   continueOnError: true
 
+- task: DotNetCoreCLI@2
+  displayName: 'Tests: Xamarin.SourceWriter'
+  inputs:
+    command: test
+    arguments: bin/Test$(Build.Configuration)/Xamarin.SourceWriter-Tests.dll
+  continueOnError: true
+
 # Running native Java.Interop tests are not yet supported on .NET Core
 #- task: DotNetCoreCLI@2
 #  displayName: 'Tests: Java.Interop'

src/Xamarin.SourceWriter/CodeWriter.cs

@@ -0,0 +1,74 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Text;
+
+namespace Xamarin.SourceWriter
+{
+	public class CodeWriter : IDisposable
+	{
+		TextWriter stream;
+		bool owns_stream;
+		int indent;
+		bool need_indent = true;
+		string base_indent;
+
+		public CodeWriter (string filename)
+		{
+			stream = File.CreateText (filename);
+			owns_stream = true;
+		}
+
+		public CodeWriter (TextWriter streamWriter, string baseIndent = "")
+		{
+			stream = streamWriter;
+			base_indent = baseIndent;
+		}
+
+		public void Write (string value)
+		{
+			WriteIndent ();
+			stream.Write (value);
+		}
+
+		public void WriteLine ()
+		{
+			WriteIndent ();
+			stream.WriteLine ();
+			need_indent = true;
+		}
+
+		public void WriteLine (string value)
+		{
+			WriteIndent ();
+			stream.WriteLine (value);
+			need_indent = true;
+		}
+
+		public void WriteLine (string format, params object[] args)
+		{
+			WriteIndent ();
+			stream.WriteLine (format, args);
+			need_indent = true;
+		}
+
+		public void Indent (int count = 1) => indent += count;
+		public void Unindent (int count = 1) => indent -= count;
+
+		private void WriteIndent ()
+		{
+			if (!need_indent)
+				return;
+
+			stream.Write (base_indent + new string ('\t', indent));
+
+			need_indent = false;
+		}
+
+		public void Dispose ()
+		{
+			if (owns_stream)
+				stream?.Dispose ();
+		}
+	}
+}

src/Xamarin.SourceWriter/Enumerations/Visibility.cs

@@ -0,0 +1,15 @@
+using System;
+using System.Collections.Generic;
+using System.Text;
+
+namespace Xamarin.SourceWriter
+{
+	public enum Visibility
+	{
+		Default = 0,
+		Private = 1,
+		Public = 2,
+		Protected = 4,
+		Internal = 8
+	}
+}

src/Xamarin.SourceWriter/Extensions/StringExtensions.cs

@@ -0,0 +1,11 @@
+using System;
+using System.Collections.Generic;
+using System.Text;
+
+namespace Xamarin.SourceWriter
+{
+	public static class StringExtensions
+	{
+		public static bool HasValue (this string str) => !string.IsNullOrWhiteSpace (str);
+	}
+}

src/Xamarin.SourceWriter/Models/AttributeWriter.cs

@@ -0,0 +1,11 @@
+using System;
+using System.Collections.Generic;
+using System.Text;
+
+namespace Xamarin.SourceWriter
+{
+	public abstract class AttributeWriter
+	{
+		public virtual void WriteAttribute (CodeWriter writer) { }
+	}
+}

src/Xamarin.SourceWriter/Models/ClassWriter.cs

@@ -0,0 +1,25 @@
+using System;
+using System.Collections.Generic;
+using System.Collections.ObjectModel;
+using System.Text;
+
+namespace Xamarin.SourceWriter
+{
+	public class ClassWriter : TypeWriter
+	{
+		public ObservableCollection<ConstructorWriter> Constructors { get; } = new ObservableCollection<ConstructorWriter> ();
+
+		public ClassWriter ()
+		{
+			Constructors.CollectionChanged += MemberAdded;
+		}
+
+		public override void WriteConstructors (CodeWriter writer)
+		{
+			foreach (var ctor in Constructors) {
+				ctor.Write (writer);
+				writer.WriteLine ();
+			}
+		}
+	}
+}

src/Xamarin.SourceWriter/Models/CommentWriter.cs

@@ -0,0 +1,22 @@
+using System;
+using System.Collections.Generic;
+using System.Text;
+
+namespace Xamarin.SourceWriter
+{
+	public class CommentWriter : ISourceWriter
+	{
+		public string Value { get; set; }
+		public int Priority { get; set; }
+
+		public CommentWriter (string value)
+		{
+			Value = value;
+		}
+
+		public virtual void Write (CodeWriter writer)
+		{
+			writer.WriteLine (Value);
+		}
+	}
+}

src/Xamarin.SourceWriter/Models/ConstructorWriter.cs

@@ -0,0 +1,21 @@
+using System;
+using System.Collections.Generic;
+using System.Text;
+
+namespace Xamarin.SourceWriter
+{
+	public class ConstructorWriter : MethodWriter
+	{
+		public string BaseCall { get; set; }
+
+		protected override void WriteReturnType (CodeWriter writer)
+		{
+		}
+
+		protected override void WriteConstructorBaseCall (CodeWriter writer)
+		{
+			if (BaseCall.HasValue ())
+				writer.Write ($" : {BaseCall}");
+		}
+	}
+}

src/Xamarin.SourceWriter/Models/DelegateWriter.cs

@@ -0,0 +1,114 @@
+using System;
+using System.Collections.Generic;
+using System.Text;
+
+namespace Xamarin.SourceWriter
+{
+	public class DelegateWriter : ISourceWriter, ITakeParameters
+	{
+		Visibility visibility;
+
+		public string Name { get; set; }
+		public List<MethodParameterWriter> Parameters { get; } = new List<MethodParameterWriter> ();
+		public TypeReferenceWriter Type { get; set; }
+		public List<string> Comments { get; } = new List<string> ();
+		public List<AttributeWriter> Attributes { get; } = new List<AttributeWriter> ();
+		public bool IsPublic { get => visibility.HasFlag (Visibility.Public); set => visibility = value ? Visibility.Public : Visibility.Default; }
+		public bool UseExplicitPrivateKeyword { get; set; }
+		public bool IsInternal { get => visibility.HasFlag (Visibility.Internal); set => visibility = value ? Visibility.Internal : Visibility.Default; }
+		public bool IsConst { get; set; }
+		public string Value { get; set; }
+		public bool IsStatic { get; set; }
+		public bool IsReadonly { get; set; }
+		public bool IsPrivate { get => visibility.HasFlag (Visibility.Private); set => visibility = value ? Visibility.Private : Visibility.Default; }
+		public bool IsProtected { get => visibility.HasFlag (Visibility.Protected); set => visibility = value ? Visibility.Protected : Visibility.Default; }
+		public int Priority { get; set; }
+		public bool IsShadow { get; set; }
+
+		public void SetVisibility (string visibility)
+		{
+			switch (visibility?.ToLowerInvariant ()) {
+				case "public":
+					IsPublic = true;
+					break;
+				case "internal":
+					IsInternal = true;
+					break;
+				case "protected":
+					IsProtected = true;
+					break;
+				case "private":
+					IsPrivate = true;
+					break;
+			}
+		}
+
+		public virtual void Write (CodeWriter writer)
+		{
+			WriteComments (writer);
+			WriteAttributes (writer);
+			WriteSignature (writer);
+		}
+
+		public virtual void WriteComments (CodeWriter writer)
+		{
+			foreach (var c in Comments)
+				writer.WriteLine (c);
+		}
+
+		public virtual void WriteAttributes (CodeWriter writer)
+		{
+			foreach (var att in Attributes)
+				att.WriteAttribute (writer);
+		}
+
+		public virtual void WriteSignature (CodeWriter writer)
+		{
+			if (IsPublic)
+				writer.Write ("public ");
+			else if (IsInternal)
+				writer.Write ("internal ");
+			else if (IsPrivate)
+				writer.Write ("private ");
+
+			if (IsStatic)
+				writer.Write ("static ");
+			if (IsReadonly)
+				writer.Write ("readonly ");
+			if (IsConst)
+				writer.Write ("const ");
+
+			if (IsShadow)
+				writer.Write ("new ");
+
+			writer.Write ("delegate ");
+
+			WriteType (writer);
+
+			writer.Write (Name + " ");
+			writer.Write ("(");
+
+			WriteParameters (writer);
+
+			writer.Write (")");
+
+			writer.Write (";");
+		}
+
+		protected virtual void WriteParameters (CodeWriter writer)
+		{
+			for (var i = 0; i < Parameters.Count; i++) {
+				var p = Parameters [i];
+				p.WriteParameter (writer);
+
+				if (i < Parameters.Count - 1)
+					writer.Write (", ");
+			}
+		}
+
+		protected virtual void WriteType (CodeWriter writer)
+		{
+			Type.WriteTypeReference (writer);
+		}
+	}
+}