Skip to content

FrameCryptor: a failed ratchet attempt puts back the old key over one set while it ran #338

Open

Description

@mdepinet

When a receiver can't decrypt a frame and ratchet_window_size is above 0, it ratchets the key to try to catch up. If none of the ratchets work, it puts back the key it started with. It does that unconditionally, so a key set with SetSharedKey() or SetKey() while the ratchets were running is overwritten by the old one. The receiver then keeps failing with the old key until a key is set again.

Where
In FrameCryptorTransformer::decryptFrame(), on m150_release:

auto initialKeyMaterial = key_set->material;
...
  // after up to ratchet_window_size failed ratchets:
  if (!decryption_success ||
      ratchet_count >= key_provider_->options().ratchet_window_size) {
    key_handler->SetKeyFromMaterial(initialKeyMaterial, key_index);
  }

Each ratchet attempt runs a key derivation, so the window between saving initialKeyMaterial and restoring it isn't small. If a receiver is getting video it can't decrypt, a frame is nearly always in that window. DataPacketCryptor::Decrypt() has the same pattern.

What we saw
We're building a Godot extension on the LiveKit C++ SDK (client-sdk-cpp 1.13.1, which builds client-sdk-rust with webrtc a545de4). This happened on Linux x86_64 and on an Android x86_64 emulator, against livekit-server --dev, with the default key provider options (ratchet_window_size 16, failure_tolerance -1):

  1. Room 1 publishes VP8 video with shared key A. Room 2 subscribes with shared key B, so it can't decrypt anything.
  2. After 8 seconds, room 2 calls SetSharedKey(A, 0) while the video keeps arriving.
  3. Room 1 sends a reliable data message, waits for it, and then keeps publishing video for another 8 seconds.

In 25 of 25 runs on Linux, room 2 decoded no video after the key change. It got the data message in 22 runs, within 70 ms of the key change, and not in the other 3. With ratchet_window_size 0 on room 2, it decoded video after the key change and got the data message in 25 of 25 runs.

Possible fix
Before restoring initialKeyMaterial, check that the key at key_index is still the one the ratchets started from, for example by comparing it under the key handler's lock, or with a generation counter that SetKey() increments. Then restore only if nothing replaced it.


Written with Claude Opus 5.5 (an AI model by Anthropic), working with @mdepinet.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions