fix: fix concurrency bugs causing panics and race conditions in internal/syncutil

[Wwwsylvia]

1. Remove the use of atomic.Value
2. Cancel all go-routines before releasing semaphore permits
3. Wait for schedule go-routines to release the semaphore permits before returning
4. Add unit tests

Fixes #916, fixes #908

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.98%. Comparing base (fdf2c51) to head (26ee98a).
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #921      +/-   ##
==========================================
+ Coverage   80.24%   80.98%   +0.74%     
==========================================
  Files          63       63              
  Lines        6053     6063      +10     
==========================================
+ Hits         4857     4910      +53     
+ Misses        858      811      -47     
- Partials      338      342       +4     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Wwwsylvia]

Mmmm the missing lines are tricky to cover

[shizhMSFT]

For the code coverage, let's discuss offline.

[shizhMSFT]

LGTM