pair.encodePkcs8() not interoperable

[rvalle]

• I'm submitting a ...

• Bug report
• Feature request
• Support request
• Other

• What is the current behavior and expected behavior?

Polkadot jS api provides a function encodePkcs8() to encode an private key of type ed25519 to the PKCS8 standard.

We cannot achieve interoperability with other software using the same standard, in particular NodeJS crypto package or OpenSSL.

• What is the motivation for changing the behavior?

The data type returned by encodePkcs8() should conform with the standard and ideally be easy to use with other software using the same standard.

• Please tell us about your environment:

• Version: Node 18 / Openssl

• Environment: Ubutnu

  • Node.js
  • Browser
  • Other (limited support for other environments)

• Language:

  • JavaScript
  • [X ] TypeScript (include tsc --version)
  • [X ] Other

• How to reproduce

We first generate a keypair and then encode it as PKCS8:

        const mnemonic = mnemonicGenerate();
        const k8 = pair.encodePkcs8();

        const node_key = crypto.createPrivateKey({
            key: Buffer.from(k8),
            format: 'der',
            type: 'pkcs8',
        });

NodeJS returns:

Error: error:0680009B:asn1 encoding routines::too long

We can also export the PKCS #8 for processing with OpenSSL as:

        fs.writeFileSync('key.pkcs8', k8);

And then try to parse it with OpenSSL by Doing:

(venv) rafael@tuxraf:~/Development/valletech/dinfra/packages/keytool$ openssl pkey -inform DER -outform PEM -in key.pkcs8 -out key.pem
unable to load key
140232103302464:error:0D07209B:asn1 encoding routines:ASN1_get_object:too long:../crypto/asn1/asn1_lib.c:91:
140232103302464:error:0D068066:asn1 encoding routines:asn1_check_tlen:bad object header:../crypto/asn1/tasn_dec.c:1137:
140232103302464:error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:698:
140232103302464:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:614:
140232103302464:error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149:
140232103302464:error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713:
140232103302464:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=n, Type=RSAPrivateKey
140232103302464:error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:133:
140232103302464:error:0D07209B:asn1 encoding routines:ASN1_get_object:too long:../crypto/asn1/asn1_lib.c:91:
140232103302464:error:0D068066:asn1 encoding routines:asn1_check_tlen:bad object header:../crypto/asn1/tasn_dec.c:1137:
140232103302464:error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:572:Field=attributes, Type=PKCS8_PRIV_KEY_INFO

Accoring to the RFC5208, which defines PKCS8 the main data structure that holds the key includes information about the PrivateKeyAlgorithmIdentifier while the output from the OpenSSL command seems to identify the key as RSAPrivateKey which makes me believe that there is something fundamentally wrong.

[rvalle]

On the other hand, the I cannot locate the implementation of the function, as this has been imported from WASM, and I cannot review what it actually does. I cannot find a function with that name neither in Substrate or Polkadot repositories.

Anybody knows where is encodePkcs8 implemented?

[jacogr]

It is not a Polkadot/Substrate standard. It is purely a Polkadot Js function used to encode the keyring for internal usage and storage. It is always assumed that anything encoded with this will be decoded with the same (reverse) function.

As such there is no appetite to change it since -

1. This would require a Polkadot JS ecosystem-wide upgrade to ensure that keypairs can be shared (an upgrade is planned for version 3 -> 4, but it will drop the current encoding format completely - obviously catering for old decoding as well, just new default encoding would be 4, not pkcs8-like, but rather different)
2. It is an internal keyring function (obviously some people may have use for it elsewhere, in that case the limitiations needs to be kept in mind)

The Polkadot JS-specific encoding is handled here - https://github.com/polkadot-js/common/tree/master/packages/util-crypto/src/json

[rvalle]

@jacogr Thanks, I can see where the issue may be coming from. It could be that the header used is actually for an RSA key, I will dig deeper.

As such there is no appetite to change it since ...

Well, at least we should add documentation if it turns out that PKCS8 is actually not PKCS8. Also I believe that adding an `encodeStandardPkcs8' would definitely not hurt. Even better and less confusing would be to add encodePEM() as implemented as Standard PKCS8 in PEM format, and ensuring interoperability at least with NodeJS crypto packages and OpenSSL.

ensure that keypairs can be shared (an upgrade is planned for version 3 -> 4

We are specifically looking into key interoperability as part of DINFRA project, which is in the Web3 Grants program. We are going to be attempting key interoperability with a number of other opensource projects, so we are happy contribute to this.

We are happy to participate in selecting which format/s would be nice to have for best interoperability.