Sendable Config

[piotrkowalczuk]

This PR is a follow-up for #172 , it introduces a Sendable Config, built around BinaryDistinctString and BinaryDistinctCharacter.

Motivation

• Ensure Config conforms to Sendable.
• Provide an explicit API instead of relying on @dynamicMemberLookup.
• Minimize changes to business logic unless strictly necessary.

TODO

• Ensure all existing tests pass.
• Implement JSON serialization for Config, particularly in chat template contexts.
• Add explicit tests for UTF-16 handling and protocol conformance.
• Review and address TODO comments in the code.
• Remove any unused code.
• Bring back dot notation while accessing Config dict values.

Open Questions

• Should BinaryDistinctString and BinaryDistinctCharacter be adopted more widely across the package, replacing String and NSString where applicable?
  • In practice, String already contains all the necessary information for binary-distinct comparisons.
  • The key difference lies in how the == operator affects serialization, deserialization, and hashing for uniqueness.
• Making tokenizers Sendable would require deep changes to the Tokenizers package. What other improvements should be incorporated next to the Experimental: tokenizers with and without templates #168?

Follow-ups

• Improve the Hub package, particularly by making Downloader conform to Sendable so the entire target can be marked as Swift6 compatible.
• Redesign the Tokenizers package.

[FL33TW00D]

This is looking good so far @piotrkowalczuk!

[ZachNagengast]

You might be planning this already but @dynamicMemberLookup seems to be missing from your new config struct. Would be nice to keep some of the dot syntax for properties that were previously defined like vocab, also general syntactic sugar vs string based access pattern - in your original proposal it was included #172 (comment), has your thinking changed to where that is no longer the case?

[piotrkowalczuk]

I can certainly revert the change. My initial experience with the codebase was confusing because it was difficult to quickly determine whether a property was statically defined or dynamically accessed. Using explicit subscript notation clarified this distinction and increased my confidence in relying on the type system for accuracy.

[ZachNagengast]

I'll let @FL33TW00D @pcuenca et al. weigh in, my main concern besides syntax is backwards compatibility if any public properties (that may be accessed in projects that import this one) become inaccessible from this change.

For example, this was possible via public methods before this PR:
config.tokenizerData.model.vocab
and if merged these errors will propagate and require updating
Value of type 'Config' has no member 'model'

Perhaps a path forward could be making those objects with defined properties serializable (and probably defining more common properties that were previously dynamic) and leaving the remaining dynamic properties to your subscript pattern, just one idea.

[piotrkowalczuk]

If there is no further input on this topic, I will revert to the original behavior.

[piotrkowalczuk]

@pcuenca @FL33TW00D @ZachNagengast you might want to have a look :)

[FL33TW00D]

@piotrkowalczuk Could you resolve the conflicts first? <3

[piotrkowalczuk]

@FL33TW00D done

[FL33TW00D]

Sorry @piotrkowalczuk could you format it with the following instructions: https://github.com/huggingface/swift-transformers/actions/runs/14204745034/attempts/2#summary-39804901716

Will make this process smoother shortly.

[piotrkowalczuk]

@FL33TW00D done.

Apparently Testing package is not available in CI. On it.

/Users/runner/work/swift-transformers/swift-transformers/Tests/HubTests/ConfigTests.swift:10:8: error: no such module 'Testing'
import Testing
       ^

[piotrkowalczuk]

I changed the package manifest. Here is where my ability to debug the CI build ends. @FL33TW00D could you help me?

[FL33TW00D]

@piotrkowalczuk so Testing is Swift 6 and above.
EDIT: we will need to make a choice on dropping support for Swift <6. Discussing with @pcuenca.

[ZachNagengast]

@piotrkowalczuk The CI runners are using this workflow on this image: https://github.com/actions/runner-images/blob/macos-14-arm64/20250324.1158/images/macos/macos-14-arm64-Readme.md which uses Xcode 15.4 by default. To get the tests you may need to make some edits such that it can build on that version of xcode. There are some apps that make it easier to switch between versions in your local development environment like https://github.com/XcodesOrg/xcodes or it can also be downloaded directly from the Apple's website.

[piotrkowalczuk]

I experimented with several options locally using https://github.com/XcodesOrg/xcodes and found that all versions of the Testing package available within the Xcode toolchain support Swift 6.

I also attempted to explicitly import swift-testing:0.5.0, but it is relatively immature and lacks some features, resulting in tests that do not compile.

Available options:

• Rewrite the tests to use XCTest. This is feasible, but due to the parametrized nature of the ConfigTests, it would require either custom glue code or a significant amount of redundancy.
• Update the toolchain. Due to limited access to CI, I believe the most effective approach would be for a member of the maintainers team to handle this outside of this PR.

@FL33TW00D I'll wait for the verdict regarding the toolchain.

[FL33TW00D]

I experimented with several options locally using https://github.com/XcodesOrg/xcodes and found that all versions of the Testing package available within the Xcode toolchain support Swift 6.

I also attempted to explicitly import swift-testing:0.5.0, but it is relatively immature and lacks some features, resulting in tests that do not compile.

Available options:

• Rewrite the tests to use XCTest. This is feasible, but due to the parametrized nature of the ConfigTests, it would require either custom glue code or a significant amount of redundancy.
• Update the toolchain. Due to limited access to CI, I believe the most effective approach would be for a member of the maintainers team to handle this outside of this PR.

@FL33TW00D I'll wait for the verdict regarding the toolchain.

I'm really not keen on dropping support for Swift <6, as Swift 6 only came out 6 months ago it seems premature.
However, I do appreciate the difficulties of parameterised tests using XCTest.