Revert Bash rewrite

[MarcoPolo]

I think it's a mistake to rewrite the testing framework in Bash, and I think we should revert the recent rewrite.

First the changes were made without any peer review. Second it introduced backwards incompatible changes (see #789). Third the rewrite used Bash, which is notoriously difficult at scale and lacks many niceties of a modern programming language. I'm fairly convinced that the majority of the bash code is LLM slop (take for example https://github.com/libp2p/test-plans/blob/master/lib/lib-filter-engine.sh). Fourth the result output is not very well structured and hard to parse (compare this https://github.com/libp2p/test-plans/blob/f3d57d4c7d74646d79fc845c685cd186fa78a377/transport/README.md#test-pass-transport-interop-030430-25-12-2025 with https://gist.github.com/MarcoPolo/a167b94a427d7372462d8eb46d3863e9 (this is a small go only test I rendered just now because the prior ones expired from the GitHub actions log)).

I could go on about why the bash rewrite is not the correct approach, but I hope the rest of it can be self evident.

I don't think the prior version was perfect by any means. Plenty of opportunity to improve. But he improvements should be done with care and engineering detail, not with a sledgehammer of LLM assisted bash scripts.

[dhuseby]

I understand your concerns, but this repo was stagnant, not meeting our needs, and prior attempts at improvement went ignored when flagged for review. The libp2p project as a whole was suffering from the lack of maintenance of this repo so I took it upon myself to fix it. You may not agree with my requirements for the final solution but the result is complete. This works. It is demonstrably better than what we had before.

1. I agree with you that Bash is not and ideal solution, however it is universal. The previous npm based solution added extra steps for developers to test locally and due to the persistent lack of maintenance, we were constantly getting flagged with Dependabot security warnings. Maintaining npm dependencies is always a chore and one we could do without. The current Bash solution is more of a write it once and forget it kind of solution.

2. I did use AI to help audit, review and optimize this. Some of the early prototypes were AI slop for sure. But I hand wrote most of the critical pieces, including the filtering engine you think is slop. The filtering engine is critical to the ease of use of this new testing framework. If you have suggestions on how to improve it, please file an issue and tag me on it.

3. The results output is now being rendered to a YAML file and no longer injected into the README.md files. You may not have noticed but we have a new project website as well. That is where the results will be rendered. Here's the latest version of the site as of today...it's not ready to push to production yet, but it will be soon: https://libp2p.myfilebase.com/ipfs/bafybeiddbgadyxyhhj67gssb4zoa3pf72unkf3okg3op6vo56yud2hffl4/

4. I would love for you to go on and on about why the Bash rewrite is not the correct approach. Please let us know what requirements you have that aren't met or exceeded. If there are any, I'm happy to address them. Based on the capabilities of this rewrite, compared to what we used to have, I think the only thing that is self-evident is that reverting these changes would deal a hugely negative blow to the project and set us back tremendously. The new testing regime works. It is demonstrably better in its capabilities, optimization, developer ergonomics, and maintainability.

I would like both you @MarcoPolo and @jxs to explain to the rest of the libp2p community why this new testing framework—that works better, and meets our needs—is actually a bad thing and that we all need to go back to the old testing framework that was worse, unmaintained, and not serving the project's goals well.

If you want to review the code and give suggestions for how to improve it. I am more than happy to engage in code review and improvement. There are some research and development needs that demanded action in this repository that are now met. Reverting would grind those efforts to a halt.

If you really really want the old testing framework, I'm happy to create a fork from before this rewrite and park it in a test-plans-old repo for you.

[dhuseby]

I think the burden is upon you @MarcoPolo to provide an alternate solution that accomplishes the same capabilities as this new testing framework (e.g. reproducibility, portability, optimized caching, snapshots, portability, etc etc etc). Reverting back to an old system that doesn't do those things would grind the 2026 R&D efforts to a halt. It make me wonder what your motives are.

[sukunrt]

I'd vote for this framework to be written in any one of js, ts, python, or go. Bash is very difficult to read and maintain. It being "universal" isn't enough to justify the burden of reading and maintaining it.

@dhuseby is there any text as to why we needed this rewrite? Or can you elaborate on how the previous one wasn't offering reproducibility, portability, optimized caching, snapshots, portability, etc etc etc.