Further memory and performance optimizations

[sungam3r]

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.16299.726 (1709/FallCreatorsUpdate/Redstone3)
Intel Core i7 CPU 920 2.67GHz (Nehalem), 1 CPU, 8 logical and 4 physical cores
Frequency=2607420 Hz, Resolution=383.5209 ns, Timer=TSC
.NET Core SDK=5.0.101
  [Host]     : .NET Core 3.1.10 (CoreCLR 4.700.20.51601, CoreFX 4.700.20.51901), X64 RyuJIT
  DefaultJob : .NET Core 3.1.10 (CoreCLR 4.700.20.51601, CoreFX 4.700.20.51901), X64 RyuJIT

Before all ROM stuff (f951f8a):

Method	name	options	Mean [μs]	Error [μs]	StdDev [μs]	Gen 0	Gen 1	Gen 2	Allocated [B]
Parse	escapes	None	99.236	0.1343	0.1191	4.0283	0.2441	-	17,320
Parse	escapes	IgnoreComments	99.095	0.0316	0.0296	4.0283	0.2441	-	17,320
Parse	github	None	10,055.023	195.8361	304.8932	390.6250	187.5000	-	2,492,664
Parse	github	IgnoreComments	10,280.116	134.9776	126.2582	390.6250	187.5000	-	2,492,664
Parse	hero	None	1.341	0.0010	0.0009	0.2327	-	-	976
Parse	hero	IgnoreComments	1.343	0.0008	0.0007	0.2327	-	-	976
Parse	introspection	None	34.671	0.0276	0.0245	3.9063	0.0610	-	16,440
Parse	introspection	IgnoreComments	34.521	0.0321	0.0300	3.9063	0.0610	-	16,440
Parse	kitchen	None	29.264	0.0319	0.0267	3.0823	0.0305	-	12,968
Parse	kitchen	IgnoreComments	29.349	0.0314	0.0294	3.0823	0.0305	-	12,968
Parse	params	None	18.273	0.0174	0.0162	2.3499	-	-	9,952
Parse	params	IgnoreComments	18.611	0.0189	0.0158	2.3499	-	-	9,952
Parse	variables	None	61.891	0.0900	0.0842	6.3477	0.1221	-	26,736
Parse	variables	IgnoreComments	61.971	0.1221	0.1082	6.3477	0.1221	-	26,738

Current:

Method	name	options	Mean [μs]	Error [μs]	StdDev [μs]	Median [μs]	Gen 0	Gen 1	Gen 2	Allocated [B]
Parse	escapes	None	83.959	0.0468	0.0438	83.975	1.9531	-	-	8,240
Parse	escapes	IgnoreComments	84.116	0.0966	0.0904	84.123	1.9531	-	-	8,240
Parse	escapes	IgnoreCommentsAndLocations	83.887	0.0314	0.0293	83.891	1.8311	-	-	8,120
Parse	github	None	6,901.345	8.3651	7.8247	6,903.978	273.4375	132.8125	-	1,631,232
Parse	github	IgnoreComments	6,864.271	3.5723	3.3415	6,864.424	273.4375	132.8125	-	1,631,232
Parse	github	IgnoreCommentsAndLocations	6,437.079	10.0350	9.3867	6,436.322	226.5625	109.3750	-	1,315,592
Parse	hero	None	1.640	0.0036	0.0033	1.641	0.1869	-	-	784
Parse	hero	IgnoreComments	1.614	0.0035	0.0033	1.613	0.1869	-	-	784
Parse	hero	IgnoreCommentsAndLocations	1.544	0.0039	0.0037	1.543	0.1602	-	-	672
Parse	introspection	None	37.331	0.0329	0.0308	37.327	2.9907	0.0610	-	12,608
Parse	introspection	IgnoreComments	37.325	0.0551	0.0515	37.313	2.9907	0.1221	-	12,608
Parse	introspection	IgnoreCommentsAndLocations	35.704	0.0192	0.0179	35.703	2.4414	-	-	10,464
Parse	kitchen	None	32.810	0.0220	0.0206	32.814	2.3193	-	-	9,880
Parse	kitchen	IgnoreComments	33.042	0.0276	0.0258	33.031	2.3193	-	-	9,880
Parse	kitchen	IgnoreCommentsAndLocations	31.714	0.0425	0.0397	31.698	1.9531	-	-	8,320
Parse	params	None	26.691	0.4859	0.7122	26.735	1.4343	-	-	6,040
Parse	params	IgnoreComments	25.947	0.5004	0.6329	25.550	1.4343	-	-	6,040
Parse	params	IgnoreCommentsAndLocations	24.337	0.0113	0.0100	24.337	1.2207	-	-	5,176
Parse	variables	None	65.777	0.0539	0.0504	65.779	4.6387	-	-	19,552
Parse	variables	IgnoreComments	66.626	0.0732	0.0685	66.625	4.6387	-	-	19,552
Parse	variables	IgnoreCommentsAndLocations	63.839	0.0316	0.0295	63.841	3.6621	0.1221	-	15,800

[Shane32]

Note: these performance benchmarks include the lexer

before rom changes

Method	name	options	Mean [ns]	Error [ns]	StdDev [ns]	Gen 0	Gen 1	Gen 2	Allocated [B]
Parse	escapes	None	51,314.6	37.97	35.52	2.7466	0.1221	-	17,320
Parse	escapes	IgnoreComments	54,416.9	247.40	231.42	2.7466	0.1221	-	17,320
Parse	github	None	3,935,955.5	18,907.23	16,760.76	394.5313	195.3125	-	2,492,667
Parse	github	IgnoreComments	3,999,250.4	29,322.76	27,428.52	390.6250	195.3125	-	2,492,675
Parse	hero	None	675.3	0.93	0.87	0.1554	-	-	976
Parse	hero	IgnoreComments	679.9	0.72	0.68	0.1554	-	-	976
Parse	introspection	None	18,297.2	47.84	42.41	2.5940	0.1221	-	16,440
Parse	introspection	IgnoreComments	18,299.6	33.06	29.31	2.5940	0.1221	-	16,440
Parse	kitchen	None	14,211.6	119.60	111.88	2.0599	0.0916	-	12,968
Parse	kitchen	IgnoreComments	14,148.0	82.20	72.86	2.0599	0.0916	-	12,968
Parse	params	None	9,168.1	26.71	23.68	1.5717	0.0305	-	9,952
Parse	params	IgnoreComments	9,418.6	36.18	33.84	1.5717	0.0305	-	9,952
Parse	variables	None	29,636.0	39.43	34.96	4.2419	0.3357	-	26,736
Parse	variables	IgnoreComments	30,390.0	112.48	105.22	4.2419	0.3357	-	26,736

before this

Method	name	options	Mean [ns]	Error [ns]	StdDev [ns]	Gen 0	Gen 1	Gen 2	Allocated [B]
Parse	escapes	None	40,100.7	25.28	21.11	1.2817	-	-	8,296
Parse	escapes	IgnoreComments	41,036.4	31.11	27.58	1.2817	-	-	8,296
Parse	github	None	3,469,282.1	3,159.47	2,800.79	289.0625	144.5313	-	1,825,006
Parse	github	IgnoreComments	3,460,697.3	8,399.87	7,014.27	289.0625	144.5313	-	1,825,005
Parse	hero	None	820.7	1.00	0.94	0.1326	-	-	832
Parse	hero	IgnoreComments	823.7	0.85	0.80	0.1326	-	-	832
Parse	introspection	None	18,472.9	19.28	18.03	2.1362	0.0916	-	13,520
Parse	introspection	IgnoreComments	18,562.2	20.58	19.25	2.1362	0.0916	-	13,520
Parse	kitchen	None	15,437.8	17.90	15.87	1.6785	0.0610	-	10,704
Parse	kitchen	IgnoreComments	15,507.4	15.97	14.16	1.6785	0.0610	-	10,704
Parse	params	None	12,411.2	12.89	12.06	1.0376	0.0153	-	6,600
Parse	params	IgnoreComments	12,728.2	8.05	7.53	1.0376	0.0153	-	6,600
Parse	variables	None	32,655.4	33.97	26.52	3.4180	0.2441	-	21,736
Parse	variables	IgnoreComments	32,672.7	32.24	30.16	3.4180	0.2441	-	21,737

after

Method	name	options	Mean [ns]	Error [ns]	StdDev [ns]	Gen 0	Gen 1	Gen 2	Allocated [B]
Parse	escapes	None	42,001.3	42.46	37.64	1.2817	-	-	8,241
Parse	escapes	IgnoreComments	39,629.8	40.25	37.65	1.2817	-	-	8,240
Parse	escapes	IgnoreCommentsAndLocations	40,113.1	23.85	22.31	1.2817	-	-	8,120
Parse	github	None	3,424,604.2	4,065.52	3,802.89	257.8125	128.9063	-	1,631,237
Parse	github	IgnoreComments	3,436,195.8	3,416.93	2,667.71	257.8125	128.9063	-	1,631,255
Parse	github	IgnoreCommentsAndLocations	3,218,633.4	3,368.03	2,985.67	207.0313	101.5625	-	1,315,615
Parse	hero	None	835.3	0.74	0.69	0.1249	-	-	784
Parse	hero	IgnoreComments	841.3	0.53	0.45	0.1249	-	-	784
Parse	hero	IgnoreCommentsAndLocations	786.2	1.05	0.93	0.1068	-	-	672
Parse	introspection	None	18,697.8	26.83	25.10	1.9836	0.0610	-	12,608
Parse	introspection	IgnoreComments	18,627.9	14.33	12.70	1.9836	0.0610	-	12,608
Parse	introspection	IgnoreCommentsAndLocations	17,756.8	23.87	22.32	1.6479	0.0610	-	10,464
Parse	kitchen	None	15,553.0	31.81	29.76	1.5564	0.0305	-	9,880
Parse	kitchen	IgnoreComments	15,708.0	15.21	14.23	1.5564	0.0305	-	9,880
Parse	kitchen	IgnoreCommentsAndLocations	14,867.5	12.53	11.72	1.3123	0.0305	-	8,320
Parse	params	None	12,803.6	11.50	10.76	0.9613	0.0153	-	6,040
Parse	params	IgnoreComments	12,636.8	13.51	11.97	0.9613	0.0153	-	6,040
Parse	params	IgnoreCommentsAndLocations	12,408.8	9.74	9.11	0.8240	-	-	5,176
Parse	variables	None	33,289.7	28.95	27.08	3.1128	0.1831	-	19,552
Parse	variables	IgnoreComments	33,569.3	42.05	39.33	3.1128	0.1831	-	19,552
Parse	variables	IgnoreCommentsAndLocations	31,911.7	34.42	32.20	2.5024	0.1221	-	15,800

[sungam3r]

Although the memory improvements are obvious, some cases run a little slower. I've tried several times to think of something to speed up processing, but I feel like it's time to quit and stop.

[Shane32]

Fine with me.

[sungam3r]

I'm done here. I updated results in initial comment.