getDisplayMedia in headless

[ldenoue]

I would like to record the current page using getDisplayMedia().

It works now in headful, not headless.
I launch the browser using these args:

browser = await playwright.launch({
		headless:**false**,
		ignoreHTTPSErrors:true,
		args: [
      '--enable-experimental-web-platform-features',
      '--disable-infobars',
      '--enable-usermedia-screen-capturing',
      '--allow-http-screen-capture',
      '--auto-select-desktop-capture-source=**MYPAGETITLE**',
      '--ignore-certificate-errors',
      '--no-sandbox',
      '--autoplay-policy=no-user-gesture-required',
      '--disable-gpu',
      '--hide-scrollbars'
    ],
	});

and inside my web page loaded by the browser, I can now use

canvasStream = await navigator.mediaDevices.getDisplayMedia();

and I indeed get the stream of the tab.

However, it fails when headless = true;

Do you think others might benefit from this as well?
I believe that running headless is much easier since it doesn't require us running a virtual display (e.g. xvfb).

[aslushnikov]

@ldenoue There's no screen capture in headless chrome, so it won't work there. And I'm not aware of any plans to support it.
However, playwright has a screen recording API that works across broowsers: https://playwright.dev/docs/videos

Would it work for you?

[ldenoue]

@aslushnikov not quite since I need audio as well. and I've noticed that with Chromium, Page.startScreenCast wasn't generating smooth videos.
However, it looks like there's a custom grabber for Firefox

playwright/browser_patches/firefox-stable/juggler/screencast/nsScreencastService.cpp

Line 211 in b4261ec

"NotifyScreencastFrame", [client, base64, deviceWidth, deviceHeight]() -> void {

Instead of dispatching NotifyScreencastFrame it would be amazing to feed this directly to the video encoder, or ffmpeg!!!
We wouldn't have to wait until the end of the recording to get a video, allowing more use-cases (e.g. ffmpeg output to RTMP for example).

[aslushnikov]

Instead of dispatching NotifyScreencastFrame it would be amazing to feed this directly to the video encoder

@ldenoue Actually, we do this already:
https://ffsearch.azurewebsites.net/#path=%2Fhome%2Fjoe%2Fplaywright%2Fbrowser_patches%2Ffirefox%2Fcheckout%2Fjuggler%2Fscreencast%2FScreencastEncoder.cpp&line=228

Or do I fail to understand something?

not quite since I need audio as well.

Interesting! Would you mind sharing your usecase? We didn't know audio was of much importance for anybody.

[ldenoue]

@aslushnikov My use-case to also record audio is that I want to record a webrtc-based video session on the server. Currently I'm rendering all incoming videos into a CANVAS element and use a MediaRecorder to generate a WEBM or MP4 that I pass back to the NodeJS process to write to a file. Using a more native solution using Playwright's record video function looks like much better: I could render anything I want on the web page without doing the compositing by hand inside a CANVAS element nor use a MediaRecorder.

@aslushnikov thanks for sending me this link, I now understand that on Firefox (and webkit too I think), the video is generated on the fly using a VPX encoder.
I noticed that the resulting videos are much more grainy than when using Chromium (which uses FFMPEG I believe).
Is there a reason why the webm generated by Firefox and Webkit are of inferior quality?
I've attached 3 screenshots to illustrate: one for chromium, firefox and webkit.

I'm using MacOS, perhaps people have no issue on other platforms?

I also noticed that the fps seems hardcoded to 25. Could we change that when requesting to capture a video?
https://ffsearch.azurewebsites.net/#path=%2Fhome%2Fjoe%2Fplaywright%2Fbrowser_patches%2Ffirefox%2Fcheckout%2Fjuggler%2Fscreencast%2FScreencastEncoder.h&line=24

For audio capture, do you know where we could we tap into the browser's audio output, like e.g. Firefox does in HeadlessCapture.cpp ? There must be a way to get audio similar to what extensions do to obtain a stream that contains audio and video using tapCapture:

chrome.tabCapture.capture(
		{
			audio: true,
			video: true,
		},
		(stream) => {
			if (!stream) return;
})

Chrome

Firefox

Webkit

[aslushnikov]

thanks for sending me this link, I now understand that on Firefox (and webkit too I think), the video is generated on the fly using a VPX encoder.

@ldenoue yes, this is the case. Just in case, here's a WebKit reference.

I also noticed that the fps seems hardcoded to 25. Could we change that when requesting to capture a video?

Sure! Do you want to submit a PR?

For audio capture, do you know where we could we tap into the browser's audio output, like e.g. Firefox does in HeadlessCapture.cpp ?

Unfortunately, I didn't look in the audio capture. Do you want to investigate?

Is there a reason why the webm generated by Firefox and Webkit are of inferior quality?

Actually, I'd recommend talking to @pavelfeldman on slack: https://aka.ms/playwright-slack He's currently refactoring screencast implementation across browsers, so he has a lot of relevant context. And it looks like we all are in the same time zone :)

[aslushnikov]

The discussion here seemed to stale - closing this for now. Please re-submit to continue discussion (we're much better at looking in open issues than through closed ones)

[ldenoue]

@aslushnikov I was on vacation last week. I found a project https://github.com/centricular/gstcefsrc that uses Chromium Embedded Framework to provide video and audio recording.
Perhaps the source code would be useful to know how they tap into the audio part?
Notice that at the moment gstcefsrc requires running a frame buffer like xvfb-run, so it's not completely headless.

I won't submit a PR now for the FPS because the quality issues don't come from the frame rate, but most likely from the parameters of the VP8 encoder (notice the pixelation for both Firefox and Webkit, whereas Chrome's frames look much higher quality because "we" use ffmpeg instead of the VP8 encoder I guess).