Use the native bun s3 client when possible?

[m1212e]

Hi, did you consider using or even already use the native Bun S3 client where applicable? Might be a cool thing since it is very performant. One could abstract away the differences in the api layout and dynamically import the client and use it depending on if the bun runtime is detected.

https://bun.sh/docs/api/s3
https://bun.sh/guides/util/detect-bun

[good-lly]

I like the idea - but as I am avoiding (unintentionally) to use bun by default I always see some other more important tasks - like for example use "native" webcrypto for hashing if available on global scope (for example in CF workers) ...

what you think? does it make sense to make wrapper abstractions?

[m1212e]

I'd have to look into that but my thought behind the abstraction thing was, to make s3mini a kind of "runtime independent" choice for s3. So a dev who wants to use s3 but wants to stay runtime independent can use it to automatically achieve the best performance no matter what runtime they are in. The benefit would be that no one has to worry about changes in library calls and behavior depending on which runtime is used. Abstracting that away would have a real benefit imo.

But I totally see your point and you should prioritize as you please. If you are open to contributions, maybe you can direct me at relevant parts of the code and project so I can attempt this?

[good-lly]

My goal was to build an S3 client alternative that is smaller, faster, and runtime-independent while developer-friendly (I hate dealing with cursors and pagination when building stuff!). To achieve this, we could split builds into multiple tree-shakeable exports (s3mini/cloudflare, s3mini/bun, etc.), but wouldn't all those layered abstractions eventually grow and build up complexity, causing us to shoot ourselves in the foot?

Another question: Do you know how Bun achieves such performance with S3? It would be great to use it, but even better if we could learn from it and deliver similar performance improvements to other runtimes as well. If you're up for some quick contributions, I think there are a few tasks we could start working on or at least plan for the future:

0. Multi-export + multi-environment e2e testing (would be nice)
1. Use of native crypto (WebCrypto) instead of node:crypto (use only as fallback?)
2. Learn Bun S3 "hacks" and come up with improvements for other runtimes as well (this might be a bit of a challenge)
3. Add native Bun S3 support (shouldn't be too complicated)
4. Add smart errors and tests for failure scenarios
5. Add proper unit testing

Is there anything here that resonates with you? If so, I can open an issue and add better explanatory descriptions.

[m1212e]

Hi, sounds reasonable. I'd like to do some proof of concept for use of the bun native client, which I started over here, if you'd like, let me hear what you think. I don't think that any of the perfomance gains could be replecated by a pure js library since this is mostly about memory management. They explain it in more detail in the docs I linked above.

Methods like text(), json(), bytes(), or arrayBuffer() avoid duplicating the string or bytes in memory when possible.

If the text happens to be ASCII, Bun directly transfers the string to JavaScriptCore (the engine) without transcoding and without duplicating the string in memory. When you use .bytes() or .arrayBuffer(), it will also avoid duplicating the bytes in memory.

These helper methods not only simplify the API, they also make it faster.

[m1212e]

I think multiple runtime specific exports would not be that good of an idea. I'd try to hide any runtime specific code behind the already existing library and try to not require the user to need to choose where to import from since that would kill all the benefit of beeing runtime agnostic.

So ideally you import s3mini and it decides on its own what kind of internal structure, class and client is used depending on what runtime it is in.

[good-lly]

ok, I am only skeptic about this approach as there might a few more bytes that will be "unused" - which might slow down the bootup for workers/edge runtimes - it will need some performance tests/overview about the impacts. Plus - I dont find specific imports as "runing the DX" point. But lets keep it all open and decide when we see some benchmarks instead