Base64 is generating invalid length base64 strings - cross language interop

[curtislinden]

Hi.

I'm experiencing a cross language interoperability problem that comes from removing the '=' padding on the end of the string. This breaks strict implementations of base64 ( ruby lets one get away with a missing length). I'm seeing this issue in the Elixir implementation and the native erlang implementation.

Is it possible to stop removing the additional = ?

[grempe]

I think this is not a bug. The '=' is not a url safe character and the RFC's seem to indicate that decoding a base64url string without the '=' padding chars should just work. If you left it in, you would then have to take the additional steps of % encoding the '=' on encode/decode.

There is a very good discussion about the Ruby implementation here:

https://bugs.ruby-lang.org/issues/10740

The Ruby implementation of base64url in the stdlib (which this library does not use) adds the padding by default, but provides and option to remove the padding. When decoding (after a patch related to the bug above) it will decode properly with or without padding.

The implementation in this library outputs exactly the same output as the stdlib Base64.urlsafe_encode64 does and the Ruby implementation also strips the '=' when you pass in the padding: false option.

See : http://ruby-doc.org/stdlib-2.3.0/libdoc/base64/rdoc/Base64.html#method-i-urlsafe_encode64

Here is some sample output. I used the test samples from the padding section here:

https://en.wikipedia.org/wiki/Base64#Padding

irb(main):004:0> list = ['any carnal pleasure.', 'any carnal pleasure', 'any carnal pleasur', 'any carnal pleasu', 'any carnal pleas']
=> ["any carnal pleasure.", "any carnal pleasure", "any carnal pleasur", "any carnal pleasu", "any carnal pleas"]

# ruby stdlib version with padding
irb(main):006:0> list.each {|l| puts Base64.urlsafe_encode64(l)}
YW55IGNhcm5hbCBwbGVhc3VyZS4=
YW55IGNhcm5hbCBwbGVhc3VyZQ==
YW55IGNhcm5hbCBwbGVhc3Vy
YW55IGNhcm5hbCBwbGVhc3U=
YW55IGNhcm5hbCBwbGVhcw==

# ruby stdlib with padding turned off
irb(main):008:0> list.each {|l| puts Base64.urlsafe_encode64(l, padding: false)}
YW55IGNhcm5hbCBwbGVhc3VyZS4
YW55IGNhcm5hbCBwbGVhc3VyZQ
YW55IGNhcm5hbCBwbGVhc3Vy
YW55IGNhcm5hbCBwbGVhc3U
YW55IGNhcm5hbCBwbGVhcw

# as implemented in this library
irb(main):009:0> list.each {|l| puts Base64.encode64(l).tr('+/', '-_').gsub(/[\n=]/, '')}
YW55IGNhcm5hbCBwbGVhc3VyZS4
YW55IGNhcm5hbCBwbGVhc3VyZQ
YW55IGNhcm5hbCBwbGVhc3Vy
YW55IGNhcm5hbCBwbGVhc3U
YW55IGNhcm5hbCBwbGVhcw

# round trip encode/decode of any of the unpadded output works fine:
irb(main):014:0> out = []; list.each {|l| out << Base64.encode64(l).tr('+/', '-_').gsub(/[\n=]/, ''); true}; out.each {|o| p Base64.decode64(o)}
"any carnal pleasure."
"any carnal pleasure"
"any carnal pleasur"
"any carnal pleasu"
"any carnal pleas"
=> ["YW55IGNhcm5hbCBwbGVhc3VyZS4", "YW55IGNhcm5hbCBwbGVhc3VyZQ", "YW55IGNhcm5hbCBwbGVhc3Vy", "YW55IGNhcm5hbCBwbGVhc3U", "YW55IGNhcm5hbCBwbGVhcw"]

Elixir implementation does indeed seem to throw an error on decode if the padding is removed:

http://elixir-lang.org/docs/stable/elixir/Base.html#url_encode64/1
http://elixir-lang.org/docs/stable/elixir/Base.html#url_decode64/1

~$ iex
Erlang/OTP 18 [erts-7.2.1] [source] [64-bit] [smp:8:8] [async-threads:10] [hipe] [kernel-poll:false] [dtrace]

Interactive Elixir (1.2.2) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> Base.url_encode64(<<255,127,254,252>>)
"_3_-_A=="
iex(2)> Base.url_decode64("_3_-_A==")
{:ok, <<255, 127, 254, 252>>}
iex(3)> Base.url_decode64("_3_-_A")
:error
iex(4)>

When I took a quick look at the Elixir source code it seems that Base.url_decode internally calls Base.url_decode! which will raise an error if the padding is not present. So I think you might want to talk to @elixir-lang or @josevalim about that to see why. I'm not really sure if there is a difference between url_decode64(string) and url_decode64!(string) in reality right now.

See source code : https://github.com/elixir-lang/elixir/blob/v1.2.2/lib/elixir/lib/base.ex#L303

Sorry I was so long winded. I took a look and couldn't stop myself. :-)

Cheers.

[josevalim]

The spec mentions the padding could be skipped in base64url: https://tools.ietf.org/html/rfc4648#section-5

That said, Elixir's implementation is incomplete. I have opened na issue here and I will try to include a fix in the upcoming v1.2.3 release: elixir-lang/elixir#4316

[grempe]

Great news @josevalim Thanks. I think that @curtislinden can close this bug now since I think its not a bug in ruby-jwt.

Cheers.

[aj-michael]

Thanks for the in depth discussion guys. We will leave this for @curtislinden to close.

[curtislinden]

@josevalim Thank you for your attention.
@grempe Thank you for your thoroughness.

[grempe]

Follow up. Elixir was patched to support creating and decoding Base64 with or without padding (triggered by this discussion).

See elixir-lang/elixir#4316

It was included in v1.2.3 of Elixir (the latest release).

http://elixir-lang.org/docs/stable/elixir/Base.html#encode64/2
http://elixir-lang.org/docs/stable/elixir/Base.html#decode64/2

👍