CAMERA.RECORD from STREAM - Invalid Video Length property

[GaryOkie]

The problem

MP4 recordings from the camera.record service are created with actual lengths that are fairly close to the requested duration+lookback (within a few seconds). However, the video length property is way off. The length property that is saved in the file is improperly listed as just 1-4 seconds long, regardless of the actual length.

The biggest issue here is that (some*) web browsers ignore the full length of the actual video and play only the much shorter listed length, nothing more. Note that other players (VLC, MS MediaPlayer) also show the incorrect short listed length on the playback timeline, but continue on and play the entire video.

*Note: I normally use FireFox where this issue was discovered. Just tried Chrome, and it will playback the entire video. Other players may also have a truncated timeline, but play on. Still - the video length property should match the actual recorded length.

Environment

• Home Assistant Core release with the issue: 108.8
• Last working Home Assistant Core release (if known): before STREAM? Similar to other reported issues with camera.record duration that are still open.
• Operating environment: Home Assistant, RasPi4 2G 32bit
• Integration causing this issue: STREAM & CAMERA
• Link to integration documentation on our website

Problem-relevant configuration.yaml

configuration.yaml

stream:
camera:
  - platform: ffmpeg 
    input: !secret rtsp
    name: "Amcrest Doorbell"

automation.yaml

    - service: script.video_record
      data_template:
        entity_id: camera.amcrest_doorbell
        filename: '{{ states("input_text.filename") }}'
        duration: 5
        lookback: 3

scripts.yaml

video_record:   
  sequence:
    - condition: state    # continue only if no previous recording already in progress  
      entity_id: input_boolean.video_recording
      state: 'off' 

    - service: input_boolean.turn_on   # turn on recording in progress flag
      entity_id: input_boolean.video_recording

    - service: camera.record
      data_template:
        entity_id: '{{ entity_id }}'
        filename: '{{ filename }}.mp4'
        duration: '{{ duration }}' 
        lookback: '{{ lookback }}' 

Traceback/Error logs

NO errors

Additional information

I ran a series of recording tests with durations from 1 to 10 seconds long, all with a lookback of 3 seconds. As you can see, the expected length was reasonably close to the actual recording length, but the file length recorded in the mp4 file is nothing close to reasonable for the longer recordings.

Shown here is the file properties of the 10+3 second duration MP4 recording. Expected length 13 seconds, actual length 10 seconds, calculated Video Length property - 2 seconds.

[dshokouhi]

So the docs do mention that you won't get exact values, have you tried adjusting the settings to be higher to see if the clip is longer?

Both duration and lookback options are suggestions, but should be consistent per camera. The actual length of the recording may vary. It is suggested that you tweak these settings to fit your needs.

[probot-home-assistant]

Hey there @hunterjm, mind taking a look at this issue as its been labeled with a integration (stream) you are listed as a codeowner for? Thanks!

[GaryOkie]

Thanks for the quick reply and suggestions. Yes, I am aware that I cannot expect exact durations, and have mentioned that the expected duration vs actual are "reasonable" and generally within a few seconds. I have no problem with that.

There is a video length property stored in the actual MP4 file that is part of the header for playback. This length is just flat wrong, and there is nothing that I have found that can make it anywhere close to reality except for the shortest recordings.

Some kind of calculation in the code seems to be stuck between 1-4 seconds for recordings regardless of actual length. I have not tried longer than 10+3 seconds since my intention is to transmit activity MP4 via Telegram. I've even looked at the code, but it's a major undertaking to unwind coming in cold.

[hunterjm]

This is a known quirk and is related to how I am manually concatenating the mpeg-ts segments (which have no header info due to being used in live streaming situations) into a single file and remuxing it to an MP4 container.

The trade off here was valid mp4 headers vs resource consumption preparing the recording and allowing for “lookback” to account for delays in automations being triggered due to prerequisites like image processing components for object detection.

Since most media players are able to play the file that is created with no issue anyway, I decided the trade off was worth it, even if it’s not technically “correct”.

[hunterjm]

For reference, here is the relevant code: https://github.com/home-assistant/core/blob/dev/homeassistant/components/stream/recorder.py#L18-L39

[GaryOkie]

Thanks for the explanation Jason. I understand the tradeoff given that "most" players overlook it except for the playback timeline.

I'll look at the code to see if I can understand where the "1-4" seconds video length comes from. I was wondering if the video length in the header could simply be the duration+lookback time and call it a day? Yeah, it wouldn't be completely accurate of course, but it should be a lot closer than what it is now and not require any resource consumption.

[hunterjm]

The actual encoding is all low level libav (the library that powers ffmpeg) work. I’ve no idea how to manually manipulate the output headers.

[GaryOkie]

oh, so dead-end and nothing we can do then to improve the video len?

EDIT: not ideal, but since my recordings are short, I'll experiment with ffmpeg remuxing the mp4 file immediately after it has been recorded to see if I can get a reasonably accurate, if not perfect, video length that way. Since camera.record runs asynchronously, coming up with automations that do this sort of thing in sequence is tricky, but I'll manage.

[hunterjm]

There is always something that can be done, but at this point it’s an effort/reward trade off. I found something suggesting adding a flushing packet to the end might help. We can also validate the frame pts/dts are monotonically incrementing as we remux the segments which could cause weirdness if there is issue there.

I don’t think it’s worth refactoring the entire thing to solve though.

[GaryOkie]

I understand and agree. I should have added "within reason" to my comment. That flushing packet idea does sound worth trying though.

During those 10 back to back (non-simultaneous) sequential recordings for my video length test, one of them did spit out this error:
ERROR (recorder_save_worker) [libav.mp4] Application provided invalid, non monotonically increasing dts to muxer in stream 0: 44405280 >= 43708500
But I can't say I've seen that show up during normal day to day recordings.

And FYI - I'm making good progress with a workaround.
ffmpeg -i input.mp4 -c copy output.mp4
As I expected, this fixes the video length in the header and is really fast. Now to just work out the async sequencing in the automation...

EDIT: .... and it's working. FireFox and me are happy now.

@hunterjm - So shall I close this, or leave it open for a little while in case you plan on giving that packet flush a whirl down the loo?

[hunterjm]

@GaryOkie - I ran a few tests on my dev instance, and VLC is showing the correct duration... so I pulled up recordings from my production instance and ran ffprobe on them. They too are showing the correct duration:

Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'driveway_20200423_193422.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1iso6mp41
    encoder         : Lavf58.29.100
  Duration: 00:00:29.98, start: 0.000000, bitrate: 4840 kb/s
    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuvj420p(pc, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 4839 kb/s, 12.01 fps, 30 tbr, 12288 tbn, 24 tbc (default)
    Metadata:
      handler_name    : VideoHandler

I have my lookback set to 10 and duration at 20, matching a multiplier of the I-Frame interval of my camera almost exactly so that I get consistent recording lengths.

I then ran it through MP4Box and that gave me a little more info about the file being produced:

* Movie Info *
	Timescale 1000 - Duration 00:00:02.500
	1 track(s)
	Fragmented File: yes - duration 00:00:00.000
11 fragments - 0 SegmentIndexes
	File suitable for progressive download (moov before mdat)
	File Brand isom - version 512
	Created: UNKNOWN DATE	Modified: UNKNOWN DATE
File has no MPEG4 IOD/OD

iTunes Info:
	Encoder Software: Lavf58.29.100
1 UDTA types: meta (1) 

Track # 1 Info - TrackID 1 - TimeScale 12288 - Media Duration 00:00:02.500
Media Info: Language "und (und)" - Type "vide:avc1" - 30 samples
Fragmented track: 330 samples - Media Duration 00:00:27.483
Visual Track layout: x=0 y=0 width=1920 height=1080
MPEG-4 Config: Visual Stream - ObjectTypeIndication 0x21
AVC/H264 Video - Visual Size 1920 x 1080
	AVC Info: 1 SPS - 1 PPS - Profile Main @ Level 4.1
	NAL Unit length bits: 32
	Pixel Aspect Ratio 1:1 - Indicated track size 1920 x 1080
	SPS#1 hash: 93F040E857E027E6EFB591393FB4380B0309B107
	PPS#1 hash: E2C4336287B33A2D561825FC8AC3C9878F254511
Self-synchronized
	RFC6381 Codec Parameters: avc1.4d0029
	Average GOP length: 30 samples

What that says is it's a fragmented track with 11 segments (should be 12). Which makes sense, because we are combining multiple mpeg-ts fragments into a single file.

The packet flush did nothing, and the timestamps were fine, so I'm out of ideas unless there is a way to ask libav to create a non-fragmented file from multiple fragments, but a quick google search turned up nothing useful.

[hunterjm]

My recommendation would be to close this as a quirk in how Firefox's video decoder works. A quick google of "Firefox MP4 won't play" comes up with tons of results from other sources claiming compatibility in every browser except FF. When I open in Chrome, I'm getting the full correct timeline, so was unable to reproduce that aspect.

[McGiverGim]

If it helps, I have the same problem. I store the videos and I try to reproduce them with Windows player, not any browser. The length in the player appears as 10-12 seconds but the videos are 30 seconds length.
Is not a fatal problem, the video plays complete but I can't move the slider to select the playback point if the point is bigger than 10 seconds ;)