Inline sourcepos fixes.

[kivikakk]

• Fixes Inconsistent sourcepos for two link types #478.
• Fixes sourcepos is Unicode-naïve. #495 — not by doing anything, but by resolving that we can only really use byte-oriented sourcepos when referring to the source document, and being careful when we transform the source in any way while interpreting it (such as with smart punctuation). Anything else is making stuff up.
• Fixes HtmlInline (and possibly other inlines) incorrect end line calculation #501.
• Fixes sourcepos not correct for inline code #503.
• Aim is to fully address Fix inline sourcepos listed in src/tests/sourcepos.rs. #541.
  • All the inlines. The block levels ones will require some deeper thinking; the base cases for these (List/Item) are broken even in the current version of cmark-gfm we match. Does upstream cmark also report them that way?
    • See also sourcepos is incorrect for <script> tags #448, sourcepos for description lists not correct #502, Incorrect sourcepos for a html comment of HtmlBlock #507. commonmark/cmark@3460cd8 doesn't get <script></script> right and I assume not others either.

Still to do:

• Decide how to handle smart punctuation and sourcepos, particularly in light of the lovely hack in the autolink extension.

To be merged after 0.36.0 release; no voy a publicar esto un viernes ..!

[github-actions]

Command	Mean [ms]	Min [ms]	Max [ms]	Relative
./bench.sh ./comrak-f8b232b	319.9 ± 3.4	315.9	327.0	2.79 ± 0.18
./bench.sh ./comrak-main	321.2 ± 3.6	316.6	330.2	2.80 ± 0.18
./bench.sh ./pulldown-cmark	114.7 ± 7.2	108.3	140.3	1.00
./bench.sh ./cmark-gfm	117.2 ± 3.4	113.7	128.9	1.02 ± 0.07
./bench.sh ./markdown-it	486.8 ± 8.7	476.5	517.4	4.24 ± 0.28

Run on Fri Feb 28 05:30:17 UTC 2025

[kivikakk]

/run-bench

[kivikakk]

#3682081876: cov: 64468 ft: 57371 corp: 19044 exec/s 5500 oom/timeout/crash: 0/0/0 time: 92355s job: 2476 dft_time: 0
^C
…
==4214== libFuzzer: run interrupted; exiting
stat::number_of_executed_units: 17349
stat::average_exec_per_sec:     1156
stat::new_units_added:          2372
stat::slowest_unit_time_sec:    0
stat::peak_rss_mb:              544
INFO: exiting: 18432 time: 92375s

[kivikakk]

I think the better fix for 8964f32 is to remove the paragraph, as one doesn't actually exist.

[digitalmoksha]

Dang, some serious work into this. Nice job @kivikakk! 🚀

[kivikakk]

Thank you! Just got the last block issues to fix (which afaict are all the one issue, also present upstream), and then we might have full, complete sourcepos!

[gjtorikian]

Dang. Super nice!

[digitalmoksha]

@kivikakk I don't think 0.36.0 included this. Would it be possible to create a new release with it when you have some spare time? 🙏

[kivikakk]

Indeed, it intentionally did not! I'll do a release. Keep in note there are still outstanding block-level cases.

[kivikakk]

Done! https://github.com/kivikakk/comrak/releases/tag/v0.37.0

[digitalmoksha]

Thanks! 🚀 I have a specific bug open in our code on inline code, so want to finish that off.

Keep in note there are still outstanding block-level cases.

No problem - I don't think any of the block-level ones that were working are now broken, so it shouldn't be an issue.