Skip to content

Investigate and fix instability in tests #264

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
HowardWolosky opened this issue Jul 16, 2020 · 0 comments · Fixed by #265 or #268
Closed

Investigate and fix instability in tests #264

HowardWolosky opened this issue Jul 16, 2020 · 0 comments · Fixed by #265 or #268
Labels
bug This relates to a bug in the existing module. help wanted Anyone in the community is welcome to do this work tests A change related to the Pester tests for the module. up for grabs Anyone in the community is welcome to do this work

Comments

@HowardWolosky
Copy link
Contributor

Issue Details

Lately, the CI build has been failing inconsistently against a couple different tests. The goal here is to identify the problem and get these tests passing consistently so that we can rely on the results coming out of the CI builds again.

Steps to reproduce the issue

Tests that seem to be regularly, but inconsistently, failing (to be updated further as past test results are further analyzed):

Context 'Remove Repo project via pipeline' {
$project = $repo | New-GitHubProject -ProjectName $defaultRepoProject -Description $defaultRepoProjectDesc
$project | Remove-GitHubProject -Force
It 'Project should be removed' {
{$project | Get-GitHubProject} | Should -Throw
}

"Expected an exception, to be thrown, but no exception was thrown."
at <ScriptBlock>, D:\a\1\s\Tests\GitHubProjects.tests.ps1: line 622
622:                 {$project | Get-GitHubProject} | Should -Throw

Requested Assignment

I'm just reporting this problem, but don't want to fix it.

Operating System

OsName               : Microsoft Windows 10 Pro
OsOperatingSystemSKU : 48
OsArchitecture       : 64-bit
WindowsVersion       : 1909
WindowsBuildLabEx    : 18362.1.amd64fre.19h1_release.190318-1202
OsLanguage           : en-US
OsMuiLanguages       : {en-US}

PowerShell Version

Name                           Value
----                           -----
PSVersion                      5.1.18362.752
PSEdition                      Desktop
PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}
BuildVersion                   10.0.18362.752
CLRVersion                     4.0.30319.42000
WSManStackVersion              3.0
PSRemotingProtocolVersion      2.3
SerializationVersion           1.1.0.1

Module Version

Running: 0.14.0
Installed:

@HowardWolosky HowardWolosky added bug This relates to a bug in the existing module. up for grabs Anyone in the community is welcome to do this work tests A change related to the Pester tests for the module. help wanted Anyone in the community is welcome to do this work labels Jul 16, 2020
HowardWolosky added a commit that referenced this issue Jul 18, 2020
We're seeing some inconsistent failures in some of the Pester tests.

The hypothesis is that GitHub may need a little bit more time after the creation of objects before performing certain operations on them (like renaming repos), or may need more time after deleting them before it will successfully return a 404 on a successive Get request.

I have added a number of `Start-Sleep`'s throughout the test codebase wherever we've encountered inconsistent failures, and that appears to have resolved the problem.  We may need to continue to do more of these if additional ones pop up.

The duration of the sleep itself is controlled by `$script:defaultSleepSecondsForReliability` which is defined in `Tests/Common.ps1`.

Long term, I have opened #267 which poses the idea of switching over to mocking out `Invoke-WebRequest` for the majority of the tests, and instead focus on validating the data that it's sending matches the expected values per the API documentation, and having just a limited number of tests that do actual end-to-end testing.

Fixes #264
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug This relates to a bug in the existing module. help wanted Anyone in the community is welcome to do this work tests A change related to the Pester tests for the module. up for grabs Anyone in the community is welcome to do this work
Projects
None yet
1 participant