Bug: incorrect and confusing Link Checker reports

[hsnabszhdn]

Consider this Link Checker comment in one of the PRs: #564

Errors in content/terraform/v1.3.x/docs/language/style.mdx

• [404] https://unified-docs-frontend-preview-rc25odzu1-hashicorp.vercel.app/terraform/language/settings/backends/configuration | Error (cached)

It says the "/terraform/language/settings/backends/configuration" link is broken in the "content/terraform/v1.3.x/docs/language/style.mdx" file, but it's not how the links on the website work.

On the actual website, if you are viewing the "Style Guide" page in its v1.3.x version, all the links on that page are prefixed with v1.3.x while the Link Checker ignores the version and checks the link against the latest version (i.e. no version in the URL at all).

For example, when viewing "/terraform/language/v1.3.x/style", the "backend configuration" is linked to "/terraform/language/v1.3.x/settings/backends/configuration" which is valid (i.e. 200 OK):

...but Link Checker was checking it against /terraform/language/settings/backends/configuration (no version in the URL) which is 404 Not Found because the link has changed in newer versions.

Naturally, in order to make Link Checker happy, I'd go and do a find-replace all links in the repo with /terraform/language/backend which is the updated link in the newer Terraform version. While Link Checker becomes happy and reports all ✅, I've broken the link for v1.3.x ❌

Expected behaviour

Link Checker should prefix all the links in content/terraform/<VERSION>/docs/... with <VERSION> (if <VERSION> is not the latest version) and NOT check them against the latest version.

If <VERSION> is the latest version in the docs, it should check against both <VERSION> and latest version (not <VERSION> in the URL).

I hope I was able to report it in an easy-to-understand way, but let me know if the issue is still not clear.

[hsnabszhdn]

Noticed this project is using lychee for link checking. So, I jury-rigged a script using the same tool (it's got a --remap argument) and realised the situation is more dire than it looks 😅

Caution

Not only is the current implementation of Link Checker reporting false positives, but it's also giving false negative results. That's double bad! ❌ ❌

Save this script in the root of the repo as link-checker.sh. It's just checking v1.3.x of the "Style Guide" page to make things simple/fast. (Note to future readers: tested on Ubuntu 22.04, GNU Bash v5.2.21, against d42f924 SHA of this repo):

#!/bin/bash

base_url="https://developer.hashicorp.com"
version="v1.3.x"
directory="content/terraform/${version}/docs/language/style.mdx"

docker run \
    --init \
    -it \
    --rm \
    -w /input \
    -v $(pwd):/input:ro \
    lycheeverse/lychee:sha-696a7ca \
    "${directory}" \
    --base-url "${base_url}" \
    --exclude-all-private \
    --exclude '\\.(svg|gif|jpg|png)' \
    --accept 200,429 \
    --timeout=60 \
    --max-concurrency 24 \
    ${USE_REMAP:+ --remap "${base_url}/terraform/language/(.*) ${base_url}/terraform/language/${version}/\$1"}

Let's try running it with/without USE_REMAP. First without it, current implementation (I've trimmed the output):

$ ./link-checker.sh

[content/terraform/v1.3.x/docs/language/style.mdx]:
     [404] https://developer.hashicorp.com/terraform/language/settings/backends/configuration

It says v1.3.x of the "Style Guide" page has one broken link to /terraform/language/settings/backends/configuration. It's this link (screenshot 👇):

---

---

But, it's NOT broken. Guess what? There is a broken link on that page (but it's not "backend configuration") that the tool didn't reveal. Let's try the --remap argument this time:

$ USE_REMAP=1 ./link-checker.sh

[content/terraform/v1.3.x/docs/language/style.mdx]:
     [404] https://developer.hashicorp.com/terraform/language/v1.3.x/tests

This time, it did NOT report /terraform/language/settings/backends/configuration as broken (which is correct ✅), but it's now reporting the /terraform/language/v1.3.x/tests link being broken (which again is correct ✅). It's this link on the page (screenshot 👇):

---

---

Go to the page and try both links in your browser...

[hsnabszhdn]

Just adding some numbers (why not? I'm an engineer! 🤓):

There are 13 versions in the content/terraform/v* directory and the tool only works reliably in one of them (latest, v1.13.x). Not that the result on other 12 versions are wrong, but they are not reliable as they include both false positives and false negatives.

Applying the same logic to content/terraform-enterprise/v*, the reliability/confidence rate of Link Checker is calculated as follows 🧮:

• For terraform ⇒ 1/13≈8%
• For terraform-enterprise ⇒ 1/49≈2%