Flaky testsuit track list

[soulomoon]

Cateogries

Fakiness types:

1. busy file handler, should use atomic write instead.
2. progress notification missed, should emit at least start and end for it in lsp during test.
3. vanished pipe, can be fixed by stopping the sender in lsp

Trackers

Some tests are really falky, it is important for us to take a look and solve them.It is crucial important to the HLS stability.
It should be part of #3736

• Flaky simple-multi-def-test on windows #4270

• Flaky test failure result in error of GetLinkable #4093 fixed by Stabilize the build system by correctly house keeping the dirtykeys and rule values [flaky test #4185 #4093] #4190

• Cabal Plugin Test Case is flaky #3333 fixed by Stabilize the build system by correctly house keeping the dirtykeys and rule values [flaky test #4185 #4093] #4190

• ghcide-tests' addDependentFile test #4194 fixed by Stabilize the build system by correctly house keeping the dirtykeys and rule values [flaky test #4185 #4093] #4190

• error-order-test‘s InternalError over InvalidParams get stuck https://github.com/haskell/haskell-language-server/actions/runs/8868613651/job/24372650102?pr=4195

• ghcide test session-deps-are-picked-up https://github.com/haskell/haskell-language-server/actions/runs/8900448886/job/24442056968?pr=4199

• ghcide test IfaceTests, we should remove interface cache dir before we run the test, or the diganostic simply gone #4200

• UnitTests's notification handlers run in priority order
  ghcide/test/exe/UnitTests.hs:91: expected: [20,19,18,17,16,15,14,13,12,11,10,9,8,7,6,5,4,3,2,1] but got: [20,19]

• hls-splice:
  TDeclKindError (golden): FAIL (66.44s) Timed out waiting to receive a message from the server. Last message received: { "jsonrpc": "2.0", "method": "$/progress", "params": { "token": "35", "value": { "kind": "end", "message": "Finished indexing 3 files" } } }

• ghcide-tests'

   th-linking-test:                                                                                                         FAIL
     Exception: Timed out waiting to receive a message from the server.
     Last message received:
     {
         "jsonrpc": "2.0",
         "method": "$/progress",
         "params": {
             "token": "7",
             "value": {
                 "kind": "report",
                 "message": "5/5"
             }
         }
     }
  

• 'sends indefinite progress notifications' test seems to be nondeterministic #4627

PS. I am doing some migration in #4173 collecting these

Set up long run CI to test the flaky test for 500 times

We can set up a standard to verify a flaky test is gone by running it consectively 500 times.
Since our CI is likely to miss this, we can set up a long running CI elsewhere to track our main branch and run the flaky test we pick, so we can see status of it .

@jhrcek already develop a script to run these test. we can build the long running CI based on it

# recommended to build test binary separately and then run it in a loop (to avoid running cabal test in a loop)
# Run tests in a loop
for i in {1..500}; do
    echo "Iteration $i" &&
    LSP_TEST_LOG_MESSAGES=0 LSP_TEST_LOG_STDERR=0 TASTY_PATTERN="Notification Handlers" $(find dist-newstyle -name ghcide-tests -type f | head -n 1) \
    || {
        echo "Warning: error at iteration $i"
        break
        }; done

[soulomoon]

HLS Graph: Build System and State Inconsistencies

This document explains the cause of issues like #4194 and #4093, which arise from malformed status handling in the HLS graph. It also demonstrates how the problem occurs and outlines solutions.

---

Build System

The build system in HLS is based on hls-graph, a lightweight Shake-like system.
hls-graph operates across sessions. Whenever something is marked as dirty in a session, the session is restarted, and all dirty keys (together with their reverse dependencies) are re-evaluated.

---

Registration of Keys

Keys are registered with their compute function inside TheRules.

---

State of Keys

Internally, hls-graph is essentially a key-value store.
However, the value is not just a plain result — it encodes the status of the key:

data Status
    = Clean !Result
    | Dirty (Maybe Result)
    | Running {
        runningStep   :: !Step,
        runningWait   :: !(IO ()),
        runningResult :: Result,     -- LAZY
        runningPrev   :: !(Maybe Result)
        }

---

Sessions

When a session starts:

• All dirty keys and their reverse dependencies are marked dirty in the key-value store.
• They are then scheduled for recomputation in parallel. Scheduling is handled automatically by the RTS, based on the Status.
• If additional keys are marked dirty, the session is restarted.

---

Dependencies and Shortcuts

Before a key is recomputed, its dependencies are fired first. This determines the run mode used during computation:

data RunMode
    = RunDependenciesSame     -- ^ Dependencies unchanged since last time.
    | RunDependenciesChanged  -- ^ At least one dependency has changed, or no dependencies recorded.
      deriving (Eq,Show)

Based on the run mode and possible cached results, the compute function determines the new value.
Afterward:

• The result is recorded with build/changed times.
• Dependencies and reverse dependencies are updated.
• Finally, the key is marked as Clean.

---

The Problem

There are two sources of truth for a key’s state:

1. Inside the hls-graph database.
2. Outside, in the actual compute functions (e.g. defineEarlyCutoff'), which maintain their own copy of dirtiness and cached values.

Problem 1: Inconsistent State During Session Restart

When a session restarts, the current session is stopped.
If a key is interrupted mid-computation, the two sources of state (graph DB vs external cache) can diverge — leaving the system in an inconsistent state. Nothing sensible can be done in this scenario.

Problem 2: Premature Removal of Dirty Keys

Once a key is marked dirty, it should remain dirty until the next session starts.
Currently, dirty keys may be removed before a session restart, leading to missed recomputations.

---

Solutions

Solution to Problem 1: Centralized Housekeeping

To keep state consistent, dirty key and cache housekeeping should happen inside the hls-graph database itself.

A new field runHook has been added to RunResult.
This hook is invoked whenever a key is marked Clean, and performs housekeeping (e.g. syncing values in shakeExtra) atomically.

Solution to Problem 2: Session-Level Dirty Key Management

Dirty keys are now sent directly to the restarted session.
They are only added to the session between restarts, and removed only when the session actually restarts.

---