security add-trusted-cert fails with errSecInternalComponent on macOS 11

[djc]

Description

I ran into this in the rustls-native-certs repo:

https://github.com/rustls/rustls-native-certs/runs/4511747465?check_suite_focus=true

Because this library is about inspecting the system certificate store, we'd like to be able to mutate the store in order to test that we're correctly reading out of it. However, adding a root certificate fails now:

  sudo bash integration-tests/macos.sh
  shell: /bin/bash -e {0}
+ ANY_CA_PEM=integration-tests/one-existing-ca.pem
+ ANY_CA_SUBJECT='OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign'
+ reset
+ security remove-trusted-cert integration-tests/one-existing-ca.pem
SecTrustSettingsRemoveTrustSettings: No Trust Settings were found.
+ true
+ list
+ cargo test util_list_certs -- --nocapture
+ grep 'OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign'
cert[113] = OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign
+ test_distrust_existing_root
+ assert_exists 'OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign'
+ list
+ cargo test util_list_certs -- --nocapture
+ grep 'OU=GlobalSign Root CA - R3, O=GlobalSign, CN=GlobalSign'
+ security add-trusted-cert -r deny integration-tests/one-existing-ca.pem
SecTrustSettingsSetTrustSettings: errSecInternalComponent

Previously we used add-trusted-cert -d, but it seems to fail whether -d is present or not. I found https://docs.github.com/en/actions/deployment/deploying-xcode-applications/installing-an-apple-certificate-on-macos-runners-for-xcode-development which solves this problem for signing Xcode applications, but since that solution relies on adding a separate/new keychain I don't think it can be used to solve our problem (where we rely on being able to change the default keychain).

Virtual environments affected

• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11
• Windows Server 2016
• Windows Server 2019
• Windows Server 2022

Image version and build link

Operating System
macOS
11.6.1
20G224
Virtual Environment
Environment: macos-11
Version: 20211207.1
Included Software: https://github.com/actions/virtual-environments/blob/macOS-11/20211207.1/images/macos/macos-11-Readme.md
Image Release: https://github.com/actions/virtual-environments/releases/tag/macOS-11%2F20211207.1

Is it regression?

Works with 10.15: https://github.com/rustls/rustls-native-certs/runs/4511924574?check_suite_focus=true

Expected behavior

Would like to be able to mutate the default keychain's trust store.

Actual behavior

Get an error when trying to add a trust root.

Repro steps

1. Execute security add-trusted-cert -d -r deny $ANY_CA_PEM within a sudoed bash script

[nikolai-frolov]

Hello @djc. Thanks for reporting, we'll take care about the issue investigation.

[miketimofeev]

Hi @djc!
Could you try running this before adding the cert?
sudo security authorizationdb write com.apple.trust-settings.admin allow
More details can be found here #4519

[djc]

That worked great, thanks!

Feel free to close this issue as you see fit.

[miketimofeev]

@djc thanks for the confirmation! Just for reference — we investigated the possible impact and decided not to modify this setting on the image due to security concerns.