Pull audio track from avatar worker in useVoiceAssistant #1130
Conversation
🦋 Changeset detectedLatest commit: 10c2380 The changes in this PR will be included in the next version bump. This PR includes changesets to release 5 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
size-limit report 📦
|
| @@ -66,6 +78,7 @@ export function useVoiceAssistant(): VoiceAssistant { | |||
| agent, | |||
| state, | |||
| audioTrack, | |||
| videoTrack, | |||
There was a problem hiding this comment.
wouldn't we simply want to return all tracks that the agent publishes?
not a fan of audioTrack in the first place, but adding videoTrack doesn't make it any better.
Generally we have microphoneTrack and cameraTrack in our glossary of constructs, but not sure how well that works for agents
There was a problem hiding this comment.
yeah i don't have a strong opinion really. this seemed least disruptive and i think if you really want all the tracks you can get them from agent anyways
| const agent = useRemoteParticipants().find((p) => p.kind === ParticipantKind.AGENT); | ||
| const audioTrack = useParticipantTracks([Track.Source.Microphone], agent?.identity)[0]; | ||
| const agent = useRemoteParticipants().find( | ||
| (p) => p.kind === ParticipantKind.AGENT && !('lk.publish_on_behalf' in p.attributes), |
There was a problem hiding this comment.
where are we documenting these attributes?
There was a problem hiding this comment.
we probably need a full reference of special attributes but for now this is going into the avatar integration docs https://github.com/livekit/web/pull/1165/files#diff-42c10a5d6ec7f1e0d9ade709732ba1d4c3697b6d411523ecdc5a2ac75a9457efR46
There's a remaining bug which is that useTrackTranscriptions doesn't work, as the agent won't publish legacy transcriptions if it has no audio track id. so we either need to fix something in the agents sdk so it can publish them anyways or just port this hook over to text streams
but for now this fixes the agent playground for video + audio at least