New content pack causes issues w/ displayed figures in Video Manager

[benjaoming]

Summary

It seems like the duplication of video nodes is somehow causing a glitch. I can hardly believe that 11,406 videos have been removed and only 394 added in the video set.

System information

• Operating system: (any)
• Version: 0.17.4 + test content pack from Fix videos only showing in one content node ka-lite-content-packs#78
• Browser: (any)

Traceback or relevant snippet from server.log

n/a

[benjaoming]

A quick verification...

I have 9855 videos on my drive from the previous content pack.

Videos that are marked as available by videoscan:

SELECT COUNT(*) FROM (SELECT COUNT(youtube_id) from item WHERE available=1 GROUP by youtube_id)
> 8879

Videos that are marked as unavailable by videoscan:

SELECT COUNT(*) FROM (SELECT COUNT(youtube_id) from item WHERE available=1 GROUP by youtube_id)
> 450

This suggests that neither the function that summarizes missing videos, nor the one that summarizes how many have been downloaded take uniqueness of the youtube_id into account.

[benjaoming]

Hi @mrpau-eugene -- would you be able to look at this? @rtibbles will help out with any questions you might have :)

[mrpau-eugene]

@benjaoming sure! will take a look into this one and would find a fix for it 👍

[mrpau-eugene]

@benjaoming seems like it doesn't account the uniqueness of each youtube_id
https://github.com/learningequality/ka-lite/blob/master/kalite/updates/static/js/updates/bundle_modules/update_videos.js#L160

Should this logic be changed given that we now have duplicated youtube_ids?

There is also a problem with staging.learningequality.org coz I can't download videos. I'm always getting a 404.

[benjaoming]

There is also a problem with staging.learningequality.org coz I can't download videos. I'm always getting a 404.

This server is used because of using kalite.project.settings.dev, you can reconfigure it to use the real server by changing the values locally, put this in ~/.kalite/settings.py:

from kalite.project.settings.dev import *

SECURESYNC_PROTOCOL = "http"
CENTRAL_SERVER_HOST = "kalite.%s" % CENTRAL_SERVER_DOMAIN
CENTRAL_SERVER_URL = "%s://%s" % (SECURESYNC_PROTOCOL, CENTRAL_SERVER_HOST)

...and then invoke kalite without setting --settings.

[mrpau-eugene]

@benjaoming I just found out that this doesn't seem to be the issue with the new content packs since I also experience a glitch using the old content packs.

What I did was:

1. Download Class 5 (India)
2. Check the Math checkbox to select all videos inside Math

But if you uncheck at least one of the videos, it will have a different value

It seems to be a problem on how the remote_size of the Math topic is computed in the database. I have a guess that it has something to do when building the content packs.

In the screenshot above, the remote size is just around 2.5GB which is equivalent to the size when we just select the Math topic checkbox

[benjaoming]

@mrpau-eugene yes, the sizes are wrong, too :) And they have been so for a while, I'm afraid that it has gone under the radar -- I can't find an issue for it.

What worries me here is that the file count is wrong, too -- which AFAIK is a new issue. And very problematic, too, because a user can select to delete videos in Topic A and then they're deleted in Topic B as well.

[mrpau-eugene]

@benjaoming I think I seem to have found a fix regarding the sizes of the video. The only problem so far is the file count.

If we have this kind of scenario:

Math (12 total_files)
  Early Math
    ... 5 Videos (3 unique videos)
  Algebra I
    ... 3 Videos (1 unique video)
  Algebra II
    ... 4 Videos (2 unique videos)

Should the 12 total_files stay as is, or should it be 6 total files because we only have 6 unique videos?

Regarding video deletion, I have a suggestion..

1. Check the videos by its path instead of its youtube_id
2. If the video is deleted, mark that video with that path as available = False
3. Check that youtube_id if other topics are still using it. If no topic uses it, we should completely delete it, else do nothing.

This might work, however when a user performs another videoscan I think videos using those youtube_ids will be marked as available = True again so I dont think this fixes the problem at all (?)

Do you have other suggestions?