CI failure: Unhandled timeout exception in System.Text.RegularExpressions.Tests.RegexMatchTests.Match_InstanceMethods_DefaultTimeout_Throws

[carlossanlop]

• Found in a release/7.0 backport PR: [PERF][Release/7.0] Update release/7.0 Perf infrastructure to use Perf repo release/7.0 branch #77699
• Queue: Libraries Test Run release coreclr windows x64 Debug
• Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=70816&view=logs&j=55b384d0-d6b0-5894-fd8a-d754d713246e&t=8dcacd81-cb5e-5ac9-93e5-c3f19b19e19b&l=105
• Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-77699-merge-97f7d283ec0d40fa98/System.Text.RegularExpressions.Tests/1/console.dd432b67.log?helixlogtype=result

----- start Wed 11/02/2022 18:59:47.06 ===============  To repro directly: ===================================================== 
pushd C:\h\w\AAD408C2\w\B4860A36\e\
"C:\h\w\AAD408C2\p\dotnet.exe" exec --runtimeconfig System.Text.RegularExpressions.Tests.runtimeconfig.json --depsfile System.Text.RegularExpressions.Tests.deps.json xunit.console.dll System.Text.RegularExpressions.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

C:\h\w\AAD408C2\w\B4860A36\e>"C:\h\w\AAD408C2\p\dotnet.exe" exec --runtimeconfig System.Text.RegularExpressions.Tests.runtimeconfig.json --depsfile System.Text.RegularExpressions.Tests.deps.json xunit.console.dll System.Text.RegularExpressions.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Text.RegularExpressions.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.RegularExpressions.Tests (found 319 of 345 test cases)
  Starting:    System.Text.RegularExpressions.Tests (parallel test collections = on, max threads = 2)
Unhandled exception. System.Text.RegularExpressions.RegexMatchTimeoutException: The Regex engine has timed out while trying to match a pattern to an input string. This can occur for many reasons, including very large inputs or excessive backtracking caused by nested quantifiers, back-references and other factors.
   at System.Text.RegularExpressions.RegexRunner.<CheckTimeout>g__ThrowRegexTimeout|25_0() in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexRunner.cs:line 323
   at Regex1_TryMatchAtCurrentPosition(RegexRunner, ReadOnlySpan`1)
   at Regex1_Scan(RegexRunner, ReadOnlySpan`1)
   at System.Text.RegularExpressions.CompiledRegexRunner.Scan(ReadOnlySpan`1 text)
   at System.Text.RegularExpressions.Regex.ScanInternal(RegexRunnerMode mode, Boolean reuseMatchObject, String input, Int32 beginning, RegexRunner runner, ReadOnlySpan`1 span, Boolean returnNullIfReuseMatchObject)
   at System.Text.RegularExpressions.Regex.RunSingleMatch(RegexRunnerMode mode, Int32 prevlen, String input, Int32 beginning, Int32 length, Int32 startat)
   at System.Text.RegularExpressions.Regex.Match(String input)
   at Microsoft.CodeAnalysis.ProjectInfo.ProjectAttributes..ctor(ProjectId id, VersionStamp version, String name, String assemblyName, String language, String filePath, String outputFilePath, String outputRefFilePath, CompilationOutputInfo compilationOutputFilePaths, String defaultNamespace, Boolean isSubmission, Boolean hasAllInformation, Boolean runAnalyzers, Guid telemetryId)
   at Microsoft.CodeAnalysis.ProjectInfo.Create(ProjectId id, VersionStamp version, String name, String assemblyName, String language, String filePath, String outputFilePath, CompilationOptions compilationOptions, ParseOptions parseOptions, IEnumerable`1 documents, IEnumerable`1 projectReferences, IEnumerable`1 metadataReferences, IEnumerable`1 analyzerReferences, IEnumerable`1 additionalDocuments, Boolean isSubmission, Type hostObjectType, String outputRefFilePath)
   at Microsoft.CodeAnalysis.Solution.AddProject(ProjectId projectId, String name, String assemblyName, String language)
   at Microsoft.CodeAnalysis.Solution.AddProject(String name, String assemblyName, String language)
   at System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(ValueTuple`4[] regexes, CancellationToken cancellationToken)
   at System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(String pattern, CultureInfo culture, Nullable`1 options, Nullable`1 matchTimeout, CancellationToken cancellationToken)
   at System.Text.RegularExpressions.Tests.RegexHelpers.GetRegexAsync(RegexEngine engine, String pattern, Nullable`1 options, Nullable`1 matchTimeout)
   at System.Text.RegularExpressions.Tests.RegexMatchTests.<>c.<<Match_InstanceMethods_DefaultTimeout_Throws>b__14_0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args)
--- End of stack trace from previous location ---
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args)
    System.Text.RegularExpressions.Tests.RegexMatchTests.Match_InstanceMethods_DefaultTimeout_Throws(engine: SourceGenerated) [FAIL]
      Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Remote process failed with an unhandled exception.
      Stack Trace:
        
        Child exception:
          System.Text.RegularExpressions.RegexMatchTimeoutException: The Regex engine has timed out while trying to match a pattern to an input string. This can occur for many reasons, including very large inputs or excessive backtracking caused by nested quantifiers, back-references and other factors.
        /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexRunner.cs(323,0): at System.Text.RegularExpressions.RegexRunner.<CheckTimeout>g__ThrowRegexTimeout|25_0()
           at Regex1_TryMatchAtCurrentPosition(RegexRunner, ReadOnlySpan`1)
           at Regex1_Scan(RegexRunner, ReadOnlySpan`1)
           at System.Text.RegularExpressions.CompiledRegexRunner.Scan(ReadOnlySpan`1 text)
           at System.Text.RegularExpressions.Regex.ScanInternal(RegexRunnerMode mode, Boolean reuseMatchObject, String input, Int32 beginning, RegexRunner runner, ReadOnlySpan`1 span, Boolean returnNullIfReuseMatchObject)
           at System.Text.RegularExpressions.Regex.RunSingleMatch(RegexRunnerMode mode, Int32 prevlen, String input, Int32 beginning, Int32 length, Int32 startat)
           at System.Text.RegularExpressions.Regex.Match(String input)
           at Microsoft.CodeAnalysis.ProjectInfo.ProjectAttributes..ctor(ProjectId id, VersionStamp version, String name, String assemblyName, String language, String filePath, String outputFilePath, String outputRefFilePath, CompilationOutputInfo compilationOutputFilePaths, String defaultNamespace, Boolean isSubmission, Boolean hasAllInformation, Boolean runAnalyzers, Guid telemetryId)
           at Microsoft.CodeAnalysis.ProjectInfo.Create(ProjectId id, VersionStamp version, String name, String assemblyName, String language, String filePath, String outputFilePath, CompilationOptions compilationOptions, ParseOptions parseOptions, IEnumerable`1 documents, IEnumerable`1 projectReferences, IEnumerable`1 metadataReferences, IEnumerable`1 analyzerReferences, IEnumerable`1 additionalDocuments, Boolean isSubmission, Type hostObjectType, String outputRefFilePath)
           at Microsoft.CodeAnalysis.Solution.AddProject(ProjectId projectId, String name, String assemblyName, String language)
           at Microsoft.CodeAnalysis.Solution.AddProject(String name, String assemblyName, String language)
           at System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(ValueTuple`4[] regexes, CancellationToken cancellationToken)
           at System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(String pattern, CultureInfo culture, Nullable`1 options, Nullable`1 matchTimeout, CancellationToken cancellationToken)
           at System.Text.RegularExpressions.Tests.RegexHelpers.GetRegexAsync(RegexEngine engine, String pattern, Nullable`1 options, Nullable`1 matchTimeout)
           at System.Text.RegularExpressions.Tests.RegexMatchTests.<>c.<<Match_InstanceMethods_DefaultTimeout_Throws>b__14_0>d.MoveNext()
        --- End of stack trace from previous location ---
           at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args)
        
        Child process:
          System.Text.RegularExpressions.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 System.Text.RegularExpressions.Tests.RegexMatchTests+<>c System.Threading.Tasks.Task <Match_InstanceMethods_DefaultTimeout_Throws>b__14_0(System.String)
        
        Child arguments:
          3
        
  Finished:    System.Text.RegularExpressions.Tests
=== TEST EXECUTION SUMMARY ===
   System.Text.RegularExpressions.Tests  Total: 28393, Errors: 0, Failed: 1, Skipped: 0, Time: 114.838s
----- end Wed 11/02/2022 19:01:44.71 ----- exit code 1 ----------------------------------------------------------

{
  "ErrorMessage": "The Regex engine has timed out while trying to match a pattern to an input string. This can occur for many reasons, including very large inputs or excessive backtracking caused by nested quantifiers, back-references and other factors.",
  "BuildRetry": false
}

Tagging subscribers to this area: @dotnet/area-system-text-regularexpressions
See info in area-owners.md if you want to be subscribed.

Issue Details

---

• Found in a release/7.0 backport PR: [PERF][Release/7.0] Update release/7.0 Perf infrastructure to use Perf repo release/7.0 branch #77699
• Queue: Libraries Test Run release coreclr windows x64 Debug
• Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=70816&view=logs&j=55b384d0-d6b0-5894-fd8a-d754d713246e&t=8dcacd81-cb5e-5ac9-93e5-c3f19b19e19b&l=105
• Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-77699-merge-97f7d283ec0d40fa98/System.Text.RegularExpressions.Tests/1/console.dd432b67.log?helixlogtype=result

----- start Wed 11/02/2022 18:59:47.06 ===============  To repro directly: ===================================================== 
pushd C:\h\w\AAD408C2\w\B4860A36\e\
"C:\h\w\AAD408C2\p\dotnet.exe" exec --runtimeconfig System.Text.RegularExpressions.Tests.runtimeconfig.json --depsfile System.Text.RegularExpressions.Tests.deps.json xunit.console.dll System.Text.RegularExpressions.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing 
popd
===========================================================================================================

C:\h\w\AAD408C2\w\B4860A36\e>"C:\h\w\AAD408C2\p\dotnet.exe" exec --runtimeconfig System.Text.RegularExpressions.Tests.runtimeconfig.json --depsfile System.Text.RegularExpressions.Tests.deps.json xunit.console.dll System.Text.RegularExpressions.Tests.dll -xml testResults.xml -nologo -nocolor -notrait category=IgnoreForCI -notrait category=OuterLoop -notrait category=failing  
  Discovering: System.Text.RegularExpressions.Tests (method display = ClassAndMethod, method display options = None)
  Discovered:  System.Text.RegularExpressions.Tests (found 319 of 345 test cases)
  Starting:    System.Text.RegularExpressions.Tests (parallel test collections = on, max threads = 2)
Unhandled exception. System.Text.RegularExpressions.RegexMatchTimeoutException: The Regex engine has timed out while trying to match a pattern to an input string. This can occur for many reasons, including very large inputs or excessive backtracking caused by nested quantifiers, back-references and other factors.
   at System.Text.RegularExpressions.RegexRunner.<CheckTimeout>g__ThrowRegexTimeout|25_0() in /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexRunner.cs:line 323
   at Regex1_TryMatchAtCurrentPosition(RegexRunner, ReadOnlySpan`1)
   at Regex1_Scan(RegexRunner, ReadOnlySpan`1)
   at System.Text.RegularExpressions.CompiledRegexRunner.Scan(ReadOnlySpan`1 text)
   at System.Text.RegularExpressions.Regex.ScanInternal(RegexRunnerMode mode, Boolean reuseMatchObject, String input, Int32 beginning, RegexRunner runner, ReadOnlySpan`1 span, Boolean returnNullIfReuseMatchObject)
   at System.Text.RegularExpressions.Regex.RunSingleMatch(RegexRunnerMode mode, Int32 prevlen, String input, Int32 beginning, Int32 length, Int32 startat)
   at System.Text.RegularExpressions.Regex.Match(String input)
   at Microsoft.CodeAnalysis.ProjectInfo.ProjectAttributes..ctor(ProjectId id, VersionStamp version, String name, String assemblyName, String language, String filePath, String outputFilePath, String outputRefFilePath, CompilationOutputInfo compilationOutputFilePaths, String defaultNamespace, Boolean isSubmission, Boolean hasAllInformation, Boolean runAnalyzers, Guid telemetryId)
   at Microsoft.CodeAnalysis.ProjectInfo.Create(ProjectId id, VersionStamp version, String name, String assemblyName, String language, String filePath, String outputFilePath, CompilationOptions compilationOptions, ParseOptions parseOptions, IEnumerable`1 documents, IEnumerable`1 projectReferences, IEnumerable`1 metadataReferences, IEnumerable`1 analyzerReferences, IEnumerable`1 additionalDocuments, Boolean isSubmission, Type hostObjectType, String outputRefFilePath)
   at Microsoft.CodeAnalysis.Solution.AddProject(ProjectId projectId, String name, String assemblyName, String language)
   at Microsoft.CodeAnalysis.Solution.AddProject(String name, String assemblyName, String language)
   at System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(ValueTuple`4[] regexes, CancellationToken cancellationToken)
   at System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(String pattern, CultureInfo culture, Nullable`1 options, Nullable`1 matchTimeout, CancellationToken cancellationToken)
   at System.Text.RegularExpressions.Tests.RegexHelpers.GetRegexAsync(RegexEngine engine, String pattern, Nullable`1 options, Nullable`1 matchTimeout)
   at System.Text.RegularExpressions.Tests.RegexMatchTests.<>c.<<Match_InstanceMethods_DefaultTimeout_Throws>b__14_0>d.MoveNext()
--- End of stack trace from previous location ---
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args)
--- End of stack trace from previous location ---
   at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args)
    System.Text.RegularExpressions.Tests.RegexMatchTests.Match_InstanceMethods_DefaultTimeout_Throws(engine: SourceGenerated) [FAIL]
      Microsoft.DotNet.RemoteExecutor.RemoteExecutionException : Remote process failed with an unhandled exception.
      Stack Trace:
        
        Child exception:
          System.Text.RegularExpressions.RegexMatchTimeoutException: The Regex engine has timed out while trying to match a pattern to an input string. This can occur for many reasons, including very large inputs or excessive backtracking caused by nested quantifiers, back-references and other factors.
        /_/src/libraries/System.Text.RegularExpressions/src/System/Text/RegularExpressions/RegexRunner.cs(323,0): at System.Text.RegularExpressions.RegexRunner.<CheckTimeout>g__ThrowRegexTimeout|25_0()
           at Regex1_TryMatchAtCurrentPosition(RegexRunner, ReadOnlySpan`1)
           at Regex1_Scan(RegexRunner, ReadOnlySpan`1)
           at System.Text.RegularExpressions.CompiledRegexRunner.Scan(ReadOnlySpan`1 text)
           at System.Text.RegularExpressions.Regex.ScanInternal(RegexRunnerMode mode, Boolean reuseMatchObject, String input, Int32 beginning, RegexRunner runner, ReadOnlySpan`1 span, Boolean returnNullIfReuseMatchObject)
           at System.Text.RegularExpressions.Regex.RunSingleMatch(RegexRunnerMode mode, Int32 prevlen, String input, Int32 beginning, Int32 length, Int32 startat)
           at System.Text.RegularExpressions.Regex.Match(String input)
           at Microsoft.CodeAnalysis.ProjectInfo.ProjectAttributes..ctor(ProjectId id, VersionStamp version, String name, String assemblyName, String language, String filePath, String outputFilePath, String outputRefFilePath, CompilationOutputInfo compilationOutputFilePaths, String defaultNamespace, Boolean isSubmission, Boolean hasAllInformation, Boolean runAnalyzers, Guid telemetryId)
           at Microsoft.CodeAnalysis.ProjectInfo.Create(ProjectId id, VersionStamp version, String name, String assemblyName, String language, String filePath, String outputFilePath, CompilationOptions compilationOptions, ParseOptions parseOptions, IEnumerable`1 documents, IEnumerable`1 projectReferences, IEnumerable`1 metadataReferences, IEnumerable`1 analyzerReferences, IEnumerable`1 additionalDocuments, Boolean isSubmission, Type hostObjectType, String outputRefFilePath)
           at Microsoft.CodeAnalysis.Solution.AddProject(ProjectId projectId, String name, String assemblyName, String language)
           at Microsoft.CodeAnalysis.Solution.AddProject(String name, String assemblyName, String language)
           at System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(ValueTuple`4[] regexes, CancellationToken cancellationToken)
           at System.Text.RegularExpressions.Tests.RegexGeneratorHelper.SourceGenRegexAsync(String pattern, CultureInfo culture, Nullable`1 options, Nullable`1 matchTimeout, CancellationToken cancellationToken)
           at System.Text.RegularExpressions.Tests.RegexHelpers.GetRegexAsync(RegexEngine engine, String pattern, Nullable`1 options, Nullable`1 matchTimeout)
           at System.Text.RegularExpressions.Tests.RegexMatchTests.<>c.<<Match_InstanceMethods_DefaultTimeout_Throws>b__14_0>d.MoveNext()
        --- End of stack trace from previous location ---
           at Microsoft.DotNet.RemoteExecutor.Program.Main(String[] args)
        
        Child process:
          System.Text.RegularExpressions.Tests, Version=7.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51 System.Text.RegularExpressions.Tests.RegexMatchTests+<>c System.Threading.Tasks.Task <Match_InstanceMethods_DefaultTimeout_Throws>b__14_0(System.String)
        
        Child arguments:
          3
        
  Finished:    System.Text.RegularExpressions.Tests
=== TEST EXECUTION SUMMARY ===
   System.Text.RegularExpressions.Tests  Total: 28393, Errors: 0, Failed: 1, Skipped: 0, Time: 114.838s
----- end Wed 11/02/2022 19:01:44.71 ----- exit code 1 ----------------------------------------------------------

Author:	carlossanlop
Assignees:	-
Labels:	area-System.Text.RegularExpressions, test-failure
Milestone:	-

[danmoseley]

what happened to most of the line numbers..

[danmoseley]

Anyway, coincidentally, this test which is designed to verify that RMTE is thrown is failing because RMTE is thrown on an unexpected line. It's being thrown where it's source-generating the regex code, inside Roslyn where it's running this regex
(@"^(?<name>.*?)\s*\((?<flavor>.*?)\)$", RegexOptions.Compiled);
on the string Test

the system is so busy this took more than negligible time, but even then it would not be a problem except that the line above we set the timeout to 100ms.

runtime/src/libraries/System.Text.RegularExpressions/tests/FunctionalTests/Regex.Match.Tests.cs

Line 1256 in 5d0943f

AppDomain.CurrentDomain.SetData(RegexHelpers.DefaultMatchTimeout_ConfigKeyName, TimeSpan.FromMilliseconds(100));

We need to move that to just below the throwing line in this test and the one next to it.

[danmoseley]

It's a bit trickier, see the PR I closed. I'll let area owners fix this.