Replace raw SQL queries in videoDeletionModule & autoRemovalQueries by MaienM · Pull Request #878 · DialmasterOrg/Youtarr

MaienM · 2026-09-25T00:19:10Z

This replaces all the raw SQL in these modules with sequelize functions, and fixes some bugs in the process.

I'll list some notes + a formatted comparison of the raw SQL queries before and after this change for each of the functions below.

`autoRemovalQueries.getChannelKeepRecentIds`

This one contained two queries, one to get a list of channels with applicable removal options and one to get the recent videos for a single channel. Both of these queries are functionally equivalent to the old ones.

Channels:

SELECT
    channel_id,
    auto_removal_keep_recent_count AS "keepCount"
FROM
    channels
WHERE
    auto_removal_keep_recent_count > 0
    AND auto_removal_protected = 0
    AND enabled = 1
    AND channel_id IS NOT NULL

SELECT
    `channel_id`,
    `auto_removal_keep_recent_count`
FROM
    `channels` AS `Channel`
WHERE
    `Channel`.`auto_removal_keep_recent_count` > 0
    AND `Channel`.`auto_removal_protected` = false
    AND `Channel`.`enabled` = TRUE
    AND `Channel`.`channel_id` IS NOT NULL;

Videos:

SELECT
    videos.id,
    MAX(
        COALESCE(
            videos.last_downloaded_at,
            jobs.time_created,
            STR_TO_DATE(videos.original_date, '%Y%m%d')
        )
    ) AS timeCreated
FROM
    videos
    LEFT JOIN jobvideos ON videos.id = jobvideos.video_id
    LEFT JOIN jobs ON jobs.id = jobvideos.job_id
WHERE
    videos.removed = 0
    AND videos.protected = 0
    AND videos.channel_id = '...'
GROUP BY
    videos.id
HAVING
    timeCreated IS NOT NULL
ORDER BY
    timeCreated DESC
LIMIT
    3

SELECT
    `Video`.`id`,
    MAX(
        COALESCE(
            `Video`.`last_downloaded_at`,
            `jobVideos->job`.`time_created`,
            STR_TO_DATE(`Video`.`original_date`, '%Y%m%d')
        )
    ) AS `timeCreated`
FROM
    `videos` AS `Video`
    LEFT OUTER JOIN `jobvideos` AS `jobVideos` ON `Video`.`id` = `jobVideos`.`video_id`
    LEFT OUTER JOIN `jobs` AS `jobVideos->job` ON `jobVideos`.`job_id` = `jobVideos->job`.`id`
WHERE
    `Video`.`removed` = false
    AND `Video`.`protected` = false
    AND `Video`.`channel_id` = '...'
GROUP BY
    `Video`.`id`
HAVING
    `timeCreated` IS NOT NULL
ORDER BY
    `timeCreated` DESC
LIMIT
    3;

`autoRemovalQueries.getRecentVideoIds`

This query is functionally equivalent to the old one.

SELECT
    videos.id,
    MAX(
        COALESCE(
            videos.last_downloaded_at,
            jobs.time_created,
            STR_TO_DATE(videos.original_date, '%Y%m%d')
        )
    ) AS timeCreated
FROM
    videos
    LEFT JOIN jobvideos ON videos.id = jobvideos.video_id
    LEFT JOIN jobs ON jobs.id = jobvideos.job_id
    LEFT JOIN channels AS protchannel ON protchannel.channel_id = videos.channel_id
    AND protchannel.enabled = 1
WHERE
    videos.removed = 0
    AND videos.protected = 0
    AND COALESCE(protchannel.auto_removal_protected, 0) = 0
GROUP BY
    videos.id
HAVING
    timeCreated IS NOT NULL
ORDER BY
    timeCreated DESC
LIMIT
    3

SELECT
    `Video`.`id`,
    MAX(
        COALESCE(
            `Video`.`last_downloaded_at`,
            `jobVideos->job`.`time_created`,
            STR_TO_DATE(`Video`.`original_date`, '%Y%m%d')
        )
    ) AS `timeCreated`
FROM
    `videos` AS `Video`
    LEFT OUTER JOIN `jobvideos` AS `jobVideos` ON `Video`.`id` = `jobVideos`.`video_id`
    LEFT OUTER JOIN `jobs` AS `jobVideos->job` ON `jobVideos`.`job_id` = `jobVideos->job`.`id`
    LEFT OUTER JOIN `channels` AS `channel` ON `Video`.`channel_id`
    AND `channel`.`enabled` = TRUE
WHERE
    (
        COALESCE(`channel`.`auto_removal_protected`, false) = false
    )
    AND `Video`.`removed` = false
    AND `Video`.`protected` = false
GROUP BY
    `id`
HAVING
    `timeCreated` IS NOT NULL
ORDER BY
    `timeCreated` DESC
LIMIT
    3;

`autoRemovalQueries.getWatchedRemovalCandidates`

This query is functionally equivalent to the old one.

SELECT
    videos.id,
    videos.youtube_id AS "youtubeId",
    videos.youtube_video_name AS "youTubeVideoName",
    videos.youtube_channel_name AS "youTubeChannelName",
	(COALESCE(videos.file_size, 0) + COALESCE(videos.audio_file_size, 0)) AS "fileSize",
    MAX(
        COALESCE(
            videos.last_downloaded_at,
            jobs.time_created,
            STR_TO_DATE(videos.original_date, '%Y%m%d')
        )
    ) AS timeCreated
FROM
    videos
    LEFT JOIN jobvideos ON videos.id = jobvideos.video_id
    LEFT JOIN jobs ON jobs.id = jobvideos.job_id
    LEFT JOIN channels AS protchannel ON protchannel.channel_id = videos.channel_id
    AND protchannel.enabled = 1
WHERE
    videos.removed = 0
    AND videos.protected = 0
    AND COALESCE(protchannel.auto_removal_protected, 0) = 0
    AND (
        EXISTS (
            SELECT
                1
            FROM
                video_watch_status vws
            WHERE
                vws.video_id = videos.id
                AND vws.played = 1
        )
        AND NOT EXISTS (
            SELECT
                1
            FROM
                video_watch_status vws
            WHERE
                vws.video_id = videos.id
                AND vws.played = 1
                AND (
                    vws.last_watched_at IS NULL
                    OR vws.last_watched_at > DATE_SUB(NOW(), INTERVAL 7 DAY)
                )
        )
    )
    AND videos.id NOT IN (1, 2, 3)
GROUP BY
    videos.id
HAVING
    timeCreated IS NOT NULL
    AND timeCreated < DATE_SUB(NOW(), INTERVAL 60 DAY)
ORDER BY
    timeCreated ASC

SELECT
    `Video`.`id`,
    MAX(
        COALESCE(
            `Video`.`last_downloaded_at`,
            `jobVideos->job`.`time_created`,
            STR_TO_DATE(`Video`.`original_date`, '%Y%m%d')
        )
    ) AS `timeCreated`,
    `Video`.`youtube_id` AS `youtubeId`,
    `Video`.`youtube_video_name` AS `youTubeVideoName`,
    `Video`.`youtube_channel_name` AS `youTubeChannelName`,
    (COALESCE(Video.file_size, 0) + COALESCE(Video.audio_file_size, 0)) AS `fileSize`
FROM
    `videos` AS `Video`
    LEFT OUTER JOIN `jobvideos` AS `jobVideos` ON `Video`.`id` = `jobVideos`.`video_id`
    LEFT OUTER JOIN `jobs` AS `jobVideos->job` ON `jobVideos`.`job_id` = `jobVideos->job`.`id`
    LEFT OUTER JOIN `channels` AS `channel` ON `Video`.`channel_id`
    AND `channel`.`enabled` = TRUE
WHERE
    (
        COALESCE(`channel`.`auto_removal_protected`, false) = false
        AND (
            EXISTS (
                SELECT
                    1
                FROM
                    video_watch_status vws
                WHERE
                    vws.video_id = Video.id
                    AND vws.played = 1
            )
            AND NOT EXISTS (
                SELECT
                    1
                FROM
                    video_watch_status vws
                WHERE
                    vws.video_id = Video.id
                    AND vws.played = 1
                    AND (
                        vws.last_watched_at IS NULL
                        OR vws.last_watched_at > DATE_SUB(NOW(), INTERVAL 7 DAY)
                    )
            )
        )
    )
    AND `Video`.`removed` = false
    AND `Video`.`protected` = false
    AND `Video`.`id` NOT IN (1, 2, 3)
GROUP BY
    `Video`.`id`
HAVING
    (
        `timeCreated` IS NOT NULL
        AND `timeCreated` < DATE_SUB(NOW(), INTERVAL 60 DAY)
    )
ORDER BY
    `timeCreated` ASC;

`videoDeletionModule.getVideosOlderThanThreshold`

The replacement query is functionally different from the old query as it (like the queries in autoRemovalQueries) will only consider the last download for videos that have multiple associated jobs. The old query always considered all jobs which means that if a video had been downloaded multiple times it would match this filter if the oldest exceeded the threshold rather than the newest, which was a bug.

SELECT
    DISTINCT videos.id,
    videos.youtube_id AS "youtubeId",
    videos.youtube_video_name AS "youTubeVideoName",
    videos.youtube_channel_name AS "youTubeChannelName",
	(COALESCE(videos.file_size, 0) + COALESCE(videos.audio_file_size, 0)) AS "fileSize",
    COALESCE(
        videos.last_downloaded_at,
        jobs.time_created,
        STR_TO_DATE(videos.original_date, '%Y%m%d')
    ) AS timeCreated
FROM
    videos
    LEFT JOIN jobvideos ON videos.id = jobvideos.video_id
    LEFT JOIN jobs ON jobs.id = jobvideos.job_id
    LEFT JOIN channels AS protchannel ON protchannel.channel_id = videos.channel_id
    AND protchannel.enabled = 1
WHERE
    videos.removed = 0
    AND videos.protected = 0
    AND COALESCE(protchannel.auto_removal_protected, 0) = 0
    AND COALESCE(
        videos.last_downloaded_at,
        jobs.time_created,
        STR_TO_DATE(videos.original_date, '%Y%m%d')
    ) IS NOT NULL
    AND COALESCE(
        videos.last_downloaded_at,
        jobs.time_created,
        STR_TO_DATE(videos.original_date, '%Y%m%d')
    ) < DATE_SUB(NOW(), INTERVAL 7 DAY)
    AND videos.id NOT IN (1, 2, 3)
ORDER BY
    timeCreated ASC

SELECT
    `Video`.`id`,
    MAX(
        COALESCE(
            `Video`.`last_downloaded_at`,
            `jobVideos->job`.`time_created`,
            STR_TO_DATE(`Video`.`original_date`, '%Y%m%d')
        )
    ) AS `timeCreated`,
    `Video`.`youtube_id` AS `youtubeId`,
    `Video`.`youtube_video_name` AS `youTubeVideoName`,
    `Video`.`youtube_channel_name` AS `youTubeChannelName`,
    (COALESCE(Video.file_size, 0) + COALESCE(Video.audio_file_size, 0)) AS `fileSize`
FROM
    `videos` AS `Video`
    LEFT OUTER JOIN `jobvideos` AS `jobVideos` ON `Video`.`id` = `jobVideos`.`video_id`
    LEFT OUTER JOIN `jobs` AS `jobVideos->job` ON `jobVideos`.`job_id` = `jobVideos->job`.`id`
    LEFT OUTER JOIN `channels` AS `channel` ON `Video`.`channel_id`
    AND `channel`.`enabled` = TRUE
WHERE
    (
        COALESCE(`channel`.`auto_removal_protected`, false) = false
    )
    AND `Video`.`removed` = false
    AND `Video`.`protected` = false
    AND `Video`.`id` NOT IN (1, 2, 3)
GROUP BY
    `Video`.`id`
HAVING
    (
        `timeCreated` IS NOT NULL
        AND `timeCreated` < DATE_SUB(NOW(), INTERVAL 7 DAY)
    )
ORDER BY
    `timeCreated` ASC;

`videoDeletionModule.getOldestVideos`

The replacement query is functionally different from the old query as it (like the queries in autoRemovalQueries) will only consider the last download for videos that have multiple associated jobs. The old query always considered all jobs which means that if a video had been downloaded multiple times it would use the oldest of those jobs to determine whether this video should be in this list rather than the newest, which was a bug.

SELECT
    DISTINCT videos.id,
    videos.youtube_id AS "youtubeId",
    videos.youtube_video_name AS "youTubeVideoName",
    videos.youtube_channel_name AS "youTubeChannelName",
	(COALESCE(videos.file_size, 0) + COALESCE(videos.audio_file_size, 0)) AS "fileSize",
    COALESCE(
        videos.last_downloaded_at,
        jobs.time_created,
        STR_TO_DATE(videos.original_date, '%Y%m%d')
    ) AS timeCreated
FROM
    videos
    LEFT JOIN jobvideos ON videos.id = jobvideos.video_id
    LEFT JOIN jobs ON jobs.id = jobvideos.job_id
    LEFT JOIN channels AS protchannel ON protchannel.channel_id = videos.channel_id
    AND protchannel.enabled = 1
WHERE
    videos.removed = 0
    AND videos.protected = 0
    AND COALESCE(protchannel.auto_removal_protected, 0) = 0
    AND COALESCE(
        videos.last_downloaded_at,
        jobs.time_created,
        STR_TO_DATE(videos.original_date, '%Y%m%d')
    ) IS NOT NULL
    AND videos.id NOT IN (1, 2, 3)
ORDER BY
    timeCreated ASC
LIMIT
    50

SELECT
    `Video`.`id`,
    MAX(
        COALESCE(
            `Video`.`last_downloaded_at`,
            `jobVideos->job`.`time_created`,
            STR_TO_DATE(`Video`.`original_date`, '%Y%m%d')
        )
    ) AS `timeCreated`,
    `Video`.`youtube_id` AS `youtubeId`,
    `Video`.`youtube_video_name` AS `youTubeVideoName`,
    `Video`.`youtube_channel_name` AS `youTubeChannelName`,
    (COALESCE(Video.file_size, 0) + COALESCE(Video.audio_file_size, 0)) AS `fileSize`
FROM
    `videos` AS `Video`
    LEFT OUTER JOIN `jobvideos` AS `jobVideos` ON `Video`.`id` = `jobVideos`.`video_id`
    LEFT OUTER JOIN `jobs` AS `jobVideos->job` ON `jobVideos`.`job_id` = `jobVideos->job`.`id`
    LEFT OUTER JOIN `channels` AS `channel` ON `Video`.`channel_id`
    AND `channel`.`enabled` = TRUE
WHERE
    (
        COALESCE(`channel`.`auto_removal_protected`, false) = false
    )
    AND `Video`.`removed` = false
    AND `Video`.`protected` = false
    AND `Video`.`id` NOT IN (1, 2, 3)
GROUP BY
    `Video`.`id`
HAVING
    `timeCreated` IS NOT NULL
ORDER BY
    `timeCreated` ASC
LIMIT
    50;

`storageUsage.getDownloadedBytes`

This query is functionally equivalent to the old one.

SELECT
    COALESCE(
        SUM(
            (
                COALESCE(videos.file_size, 0) + COALESCE(videos.audio_file_size, 0)
            )
        ),
        0
    ) AS "totalBytes"
FROM
    videos
WHERE
    videos.removed = 0

SELECT
    COALESCE(
        SUM(
            (
                COALESCE(Video.file_size, 0) + COALESCE(Video.audio_file_size, 0)
            )
        ),
        0
    ) AS `totalBytes`
FROM
    `videos` AS `Video`
WHERE
    `Video`.`removed` = false
LIMIT
    1;

MaienM · 2026-09-25T16:06:59Z

Needs rebasing due to recent changes, and I'll also include the new query in storageUsage since that's pretty intertwined with these modules.

Old: ```sql SELECT videos.id, MAX( COALESCE( videos.last_downloaded_at, jobs.time_created, STR_TO_DATE(videos.original_date, '%Y%m%d') ) ) AS timeCreated FROM videos LEFT JOIN jobvideos ON videos.id = jobvideos.video_id LEFT JOIN jobs ON jobs.id = jobvideos.job_id LEFT JOIN channels AS protchannel ON protchannel.channel_id = videos.channel_id AND protchannel.enabled = 1 WHERE videos.removed = 0 AND videos.protected = 0 AND COALESCE(protchannel.auto_removal_protected, 0) = 0 GROUP BY videos.id HAVING timeCreated IS NOT NULL ORDER BY timeCreated DESC LIMIT 3 ``` New: ```sql SELECT `Video`.`id`, MAX( COALESCE( `Video`.`last_downloaded_at`, `jobVideos->job`.`time_created`, STR_TO_DATE(`Video`.`original_date`, '%Y%m%d') ) ) AS `timeCreated` FROM `videos` AS `Video` LEFT OUTER JOIN `jobvideos` AS `jobVideos` ON `Video`.`id` = `jobVideos`.`video_id` LEFT OUTER JOIN `jobs` AS `jobVideos->job` ON `jobVideos`.`job_id` = `jobVideos->job`.`id` LEFT OUTER JOIN `channels` AS `channel` ON `Video`.`channel_id` AND `channel`.`enabled` = TRUE WHERE ( COALESCE(`channel`.`auto_removal_protected`, false) = false ) AND `Video`.`removed` = false AND `Video`.`protected` = false GROUP BY `id` HAVING `timeCreated` IS NOT NULL ORDER BY `timeCreated` DESC LIMIT 3; ```

Old: ```sql SELECT videos.id, MAX( COALESCE( videos.last_downloaded_at, jobs.time_created, STR_TO_DATE(videos.original_date, '%Y%m%d') ) ) AS timeCreated FROM videos LEFT JOIN jobvideos ON videos.id = jobvideos.video_id LEFT JOIN jobs ON jobs.id = jobvideos.job_id WHERE videos.removed = 0 AND videos.protected = 0 AND videos.channel_id = '...' GROUP BY videos.id HAVING timeCreated IS NOT NULL ORDER BY timeCreated DESC LIMIT 3 ``` New: ```sql SELECT `Video`.`id`, MAX( COALESCE( `Video`.`last_downloaded_at`, `jobVideos->job`.`time_created`, STR_TO_DATE(`Video`.`original_date`, '%Y%m%d') ) ) AS `timeCreated` FROM `videos` AS `Video` LEFT OUTER JOIN `jobvideos` AS `jobVideos` ON `Video`.`id` = `jobVideos`.`video_id` LEFT OUTER JOIN `jobs` AS `jobVideos->job` ON `jobVideos`.`job_id` = `jobVideos->job`.`id` WHERE `Video`.`removed` = false AND `Video`.`protected` = false AND `Video`.`channel_id` = '...' GROUP BY `Video`.`id` HAVING `timeCreated` IS NOT NULL ORDER BY `timeCreated` DESC LIMIT 3; ```

Old: ```sql SELECT videos.id, videos.youtube_id AS "youtubeId", videos.youtube_video_name AS "youTubeVideoName", videos.youtube_channel_name AS "youTubeChannelName", videos.file_size AS "fileSize", MAX( COALESCE( videos.last_downloaded_at, jobs.time_created, STR_TO_DATE(videos.original_date, '%Y%m%d') ) ) AS timeCreated FROM videos LEFT JOIN jobvideos ON videos.id = jobvideos.video_id LEFT JOIN jobs ON jobs.id = jobvideos.job_id LEFT JOIN channels AS protchannel ON protchannel.channel_id = videos.channel_id AND protchannel.enabled = 1 WHERE videos.removed = 0 AND videos.protected = 0 AND COALESCE(protchannel.auto_removal_protected, 0) = 0 AND ( EXISTS ( SELECT 1 FROM video_watch_status vws WHERE vws.video_id = videos.id AND vws.played = 1 ) AND NOT EXISTS ( SELECT 1 FROM video_watch_status vws WHERE vws.video_id = videos.id AND vws.played = 1 AND ( vws.last_watched_at IS NULL OR vws.last_watched_at > DATE_SUB(NOW(), INTERVAL 7 DAY) ) ) ) AND videos.id NOT IN (1, 2, 3) GROUP BY videos.id HAVING timeCreated IS NOT NULL AND timeCreated < DATE_SUB(NOW(), INTERVAL 60 DAY) ORDER BY timeCreated ASC ``` New: ```sql SELECT `Video`.`id`, MAX( COALESCE( `Video`.`last_downloaded_at`, `jobVideos->job`.`time_created`, STR_TO_DATE(`Video`.`original_date`, '%Y%m%d') ) ) AS `timeCreated`, `Video`.`youtube_id` AS `youtubeId`, `Video`.`youtube_video_name` AS `youTubeVideoName`, `Video`.`youtube_channel_name` AS `youTubeChannelName`, `Video`.`file_size` AS `fileSize` FROM `videos` AS `Video` LEFT OUTER JOIN `jobvideos` AS `jobVideos` ON `Video`.`id` = `jobVideos`.`video_id` LEFT OUTER JOIN `jobs` AS `jobVideos->job` ON `jobVideos`.`job_id` = `jobVideos->job`.`id` LEFT OUTER JOIN `channels` AS `channel` ON `Video`.`channel_id` AND `channel`.`enabled` = TRUE WHERE ( COALESCE(`channel`.`auto_removal_protected`, false) = false AND ( EXISTS ( SELECT 1 FROM video_watch_status vws WHERE vws.video_id = Video.id AND vws.played = 1 ) AND NOT EXISTS ( SELECT 1 FROM video_watch_status vws WHERE vws.video_id = Video.id AND vws.played = 1 AND ( vws.last_watched_at IS NULL OR vws.last_watched_at > DATE_SUB(NOW(), INTERVAL 7 DAY) ) ) ) ) AND `Video`.`removed` = false AND `Video`.`protected` = false AND `Video`.`id` NOT IN (1, 2, 3) GROUP BY `Video`.`id` HAVING ( `timeCreated` IS NOT NULL AND `timeCreated` < DATE_SUB(NOW(), INTERVAL 60 DAY) ) ORDER BY `timeCreated` ASC; ```

The replacement query is functionally different from the old query as it (like the queries in autoRemovalQueries) will only consider the last download for videos that have multiple associated jobs. The old query always considered all jobs which means that if a video had been downloaded multiple times it would match this filter if the _oldest_ exceeded the threshold rather than the _newest_, which was a bug. Old: ```sql SELECT DISTINCT videos.id, videos.youtube_id AS "youtubeId", videos.youtube_video_name AS "youTubeVideoName", videos.youtube_channel_name AS "youTubeChannelName", videos.file_size AS "fileSize", COALESCE( videos.last_downloaded_at, jobs.time_created, STR_TO_DATE(videos.original_date, '%Y%m%d') ) AS timeCreated FROM videos LEFT JOIN jobvideos ON videos.id = jobvideos.video_id LEFT JOIN jobs ON jobs.id = jobvideos.job_id LEFT JOIN channels AS protchannel ON protchannel.channel_id = videos.channel_id AND protchannel.enabled = 1 WHERE videos.removed = 0 AND videos.protected = 0 AND COALESCE(protchannel.auto_removal_protected, 0) = 0 AND COALESCE( videos.last_downloaded_at, jobs.time_created, STR_TO_DATE(videos.original_date, '%Y%m%d') ) IS NOT NULL AND COALESCE( videos.last_downloaded_at, jobs.time_created, STR_TO_DATE(videos.original_date, '%Y%m%d') ) < DATE_SUB(NOW(), INTERVAL 7 DAY) AND videos.id NOT IN (49, 50, 51, 40, 41, 42) ORDER BY timeCreated ASC ``` New: ```sql SELECT `Video`.`id`, MAX( COALESCE( `Video`.`last_downloaded_at`, `jobVideos->job`.`time_created`, STR_TO_DATE(`Video`.`original_date`, '%Y%m%d') ) ) AS `timeCreated`, `Video`.`youtube_id` AS `youtubeId`, `Video`.`youtube_video_name` AS `youTubeVideoName`, `Video`.`youtube_channel_name` AS `youTubeChannelName`, `Video`.`file_size` AS `fileSize` FROM `videos` AS `Video` LEFT OUTER JOIN `jobvideos` AS `jobVideos` ON `Video`.`id` = `jobVideos`.`video_id` LEFT OUTER JOIN `jobs` AS `jobVideos->job` ON `jobVideos`.`job_id` = `jobVideos->job`.`id` LEFT OUTER JOIN `channels` AS `channel` ON `Video`.`channel_id` AND `channel`.`enabled` = TRUE WHERE ( COALESCE(`channel`.`auto_removal_protected`, false) = false ) AND `Video`.`removed` = false AND `Video`.`protected` = false AND `Video`.`id` NOT IN (47, 48, 49, 40, 41, 42) GROUP BY `Video`.`id` HAVING ( `timeCreated` IS NOT NULL AND `timeCreated` < DATE_SUB(NOW(), INTERVAL 7 DAY) ) ORDER BY `timeCreated` ASC; ```

The replacement query is functionally different from the old query as it (like the queries in autoRemovalQueries) will only consider the last download for videos that have multiple associated jobs. The old query always considered all jobs which means that if a video had been downloaded multiple times it would use the _oldest_ of those jobs to determine whether this video should be in this list rather than the _newest_, which was a bug. Old: ```sql SELECT DISTINCT videos.id, videos.youtube_id AS "youtubeId", videos.youtube_video_name AS "youTubeVideoName", videos.youtube_channel_name AS "youTubeChannelName", videos.file_size AS "fileSize", COALESCE( videos.last_downloaded_at, jobs.time_created, STR_TO_DATE(videos.original_date, '%Y%m%d') ) AS timeCreated FROM videos LEFT JOIN jobvideos ON videos.id = jobvideos.video_id LEFT JOIN jobs ON jobs.id = jobvideos.job_id LEFT JOIN channels AS protchannel ON protchannel.channel_id = videos.channel_id AND protchannel.enabled = 1 WHERE videos.removed = 0 AND videos.protected = 0 AND COALESCE(protchannel.auto_removal_protected, 0) = 0 AND COALESCE( videos.last_downloaded_at, jobs.time_created, STR_TO_DATE(videos.original_date, '%Y%m%d') ) IS NOT NULL AND videos.id NOT IN (1, 2, 3) ORDER BY timeCreated ASC LIMIT 50 ``` New: ```sql SELECT `Video`.`id`, MAX( COALESCE( `Video`.`last_downloaded_at`, `jobVideos->job`.`time_created`, STR_TO_DATE(`Video`.`original_date`, '%Y%m%d') ) ) AS `timeCreated`, `Video`.`youtube_id` AS `youtubeId`, `Video`.`youtube_video_name` AS `youTubeVideoName`, `Video`.`youtube_channel_name` AS `youTubeChannelName`, `Video`.`file_size` AS `fileSize` FROM `videos` AS `Video` LEFT OUTER JOIN `jobvideos` AS `jobVideos` ON `Video`.`id` = `jobVideos`.`video_id` LEFT OUTER JOIN `jobs` AS `jobVideos->job` ON `jobVideos`.`job_id` = `jobVideos->job`.`id` LEFT OUTER JOIN `channels` AS `channel` ON `Video`.`channel_id` AND `channel`.`enabled` = TRUE WHERE ( COALESCE(`channel`.`auto_removal_protected`, false) = false ) AND `Video`.`removed` = false AND `Video`.`protected` = false AND `Video`.`id` NOT IN (1, 2, 3) GROUP BY `Video`.`id` HAVING `timeCreated` IS NOT NULL ORDER BY `timeCreated` ASC LIMIT 50; ```

Old: ```sql SELECT channel_id, auto_removal_keep_recent_count AS "keepCount" FROM channels WHERE auto_removal_keep_recent_count > 0 AND auto_removal_protected = 0 AND enabled = 1 AND channel_id IS NOT NULL ``` New: ```sql SELECT `channel_id`, `auto_removal_keep_recent_count` FROM `channels` AS `Channel` WHERE `Channel`.`auto_removal_keep_recent_count` > 0 AND `Channel`.`auto_removal_protected` = false AND `Channel`.`enabled` = TRUE AND `Channel`.`channel_id` IS NOT NULL; ```

Old: ```sql SELECT COALESCE( SUM( ( COALESCE(videos.file_size, 0) + COALESCE(videos.audio_file_size, 0) ) ), 0 ) AS "totalBytes" FROM videos WHERE videos.removed = 0 ``` New: ```sql SELECT COALESCE( SUM( ( COALESCE(Video.file_size, 0) + COALESCE(Video.audio_file_size, 0) ) ), 0 ) AS `totalBytes` FROM `videos` AS `Video` WHERE `Video`.`removed` = false LIMIT 1; ```

MaienM · 2026-09-25T23:34:51Z

Rebased, added the query in storageUsage and updated the queries in the PR description.

dialmaster

I ran the old and new version of each query side by side against my dev database. getDownloadedBytes matched exactly, but two things broke (details inline):

The channel join never matches, so videos from auto_removal_protected channels can get deleted.
Per-channel keep-recent lost its LIMIT, so those channels keep everything.

With the fixes I suggested inline, every function returned the same videos as the old SQL on my data. The order only differed between videos with the exact same timestamp.

This also needs a rebase onto dev. It merges without conflicts, but 3 tests that were added to videoDeletionModule.test.js since you branched (the "free-space cleanup after earlier strategies" block) fail after the merge, because they still feed data through mockSequelize.query. The other 2 in that block still pass, but only because they get no data now, so they need the same update.

dialmaster · 2026-10-02T23:18:22Z

 Video.hasMany(VideoWatchStatus, { foreignKey: 'video_id', as: 'watchStatuses' });
 VideoWatchStatus.belongsTo(Video, { foreignKey: 'video_id', as: 'video' });

+Video.belongsTo(Channel, { foreignKey: 'channel_id', as: 'channel' });


Without a targetKey, Sequelize assumes this points at channels.id (the integer primary key). But videos.channel_id holds the YouTube channel ID ("UC..."), and that lives in channels.channel_id. Both lines need to point at that column:

Suggestion:

Video.belongsTo(Channel, { foreignKey: 'channel_id', targetKey: 'channel_id', as: 'channel', constraints: false }); Channel.hasMany(Video, { foreignKey: 'channel_id', sourceKey: 'channel_id', as: 'videos', constraints: false });

(constraints: false because there's no real foreign key between these columns.)

dialmaster · 2026-10-02T23:19:46Z

+        model: Channel,
+        as: 'channel',
+        attributes: [],
+        on: {


This is the big one. When a value in on is a bare sequelize.col(), Sequelize drops the key, so this comes out as:

ON `Video`.`channel_id` AND `channel`.`enabled` = true

(It's in the "after" SQL in the PR description too.) MariaDB reads 'UC...' as 0, so the join never matches anything and every protected channel looks unprotected. I ran the old and new queries side by side on my dev DB, and all 15 videos from my one protected channel showed up as deletion candidates.

With the targetKey fix in models/index.js, the association can build the join itself, eg:

required: false, where: { enabled: true },

That gives ON Video.channel_id = channel.channel_id AND channel.enabled = true, and with both changes every query matched the old SQL on my data.

dialmaster · 2026-10-02T23:20:25Z

+          joinChannel: false,
        });
+        options.where.channel_id = channel.channel_id;
+        options.limit = channel.keepCount;


keepCount was an alias from the old raw query. The new Channel.findAll doesn't create it, so this is undefined, the LIMIT gets dropped, and a channel set to keep 10 videos keeps all of them.

Suggestion:

options.limit = channel.auto_removal_keep_recent_count;

dialmaster · 2026-10-02T23:21:49Z

-      mockSequelize.query
-        .mockResolvedValueOnce([
+      mockChannel.findAll
+        .mockResolvedValue([


The real Channel model never has a keepCount field, so this mock is why the test still passes. Can you return auto_removal_keep_recent_count here instead?

dialmaster · 2026-10-02T23:22:27Z

+            as: 'channel',
+            attributes: [],
+            on: {
+              id: mockSequelize.col('Video.channel_id'),


Since these tests check the options object, they're asserting the broken on: { id: ... } as correct. I think at least one test should check the real SQL: use the real models, spy on sequelize.query to grab the generated string (no DB needed), then check for ON `Video`.`channel_id` = `channel`.`channel_id and the LIMIT. That would have caught both bugs.

dialmaster · 2026-10-02T23:23:28Z

+
+    MockSequelize = {
+      QueryTypes: { SELECT: 'SELECT' },
+      Op: {


gt is missing here, so [Op.gt]: 0 turns into { undefined: 0 } in both the code and the test, and that assertion doesn't really check anything.

MaienM marked this pull request as draft September 25, 2026 16:07

MaienM added 8 commits September 26, 2026 01:26

refactor: Add relationship between Video and Channel models

557afc5

MaienM force-pushed the feature/reduce-raw-sql-video-auto-removal branch from 21f519b to 2e2ee97 Compare September 25, 2026 23:34

MaienM marked this pull request as ready for review September 25, 2026 23:35

dialmaster requested changes Oct 2, 2026

View reviewed changes

Navigation Menu

Repository navigation

Replace raw SQL queries in videoDeletionModule & autoRemovalQueries - #878

Replace raw SQL queries in videoDeletionModule & autoRemovalQueries#878
MaienM wants to merge 8 commits into
DialmasterOrg:devfrom
MaienM:feature/reduce-raw-sql-video-auto-removal

MaienM commented Sep 25, 2026 •

edited

Loading

Uh oh!

MaienM commented Sep 25, 2026

Uh oh!

MaienM commented Sep 25, 2026

Uh oh!

dialmaster left a comment

Uh oh!

dialmaster Oct 2, 2026

Uh oh!

dialmaster Oct 2, 2026

Uh oh!

dialmaster Oct 2, 2026

Uh oh!

dialmaster Oct 2, 2026

Uh oh!

dialmaster Oct 2, 2026

Uh oh!

dialmaster Oct 2, 2026

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

MaienM commented Sep 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

autoRemovalQueries.getChannelKeepRecentIds

autoRemovalQueries.getRecentVideoIds

autoRemovalQueries.getWatchedRemovalCandidates

videoDeletionModule.getVideosOlderThanThreshold

videoDeletionModule.getOldestVideos

storageUsage.getDownloadedBytes

Uh oh!

MaienM commented Sep 25, 2026

Uh oh!

MaienM commented Sep 25, 2026

Uh oh!

dialmaster left a comment

Choose a reason for hiding this comment

Uh oh!

dialmaster Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

dialmaster Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

dialmaster Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

dialmaster Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

dialmaster Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

dialmaster Oct 2, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

MaienM commented Sep 25, 2026 •

edited

Loading

`autoRemovalQueries.getChannelKeepRecentIds`

`autoRemovalQueries.getRecentVideoIds`

`autoRemovalQueries.getWatchedRemovalCandidates`

`videoDeletionModule.getVideosOlderThanThreshold`

`videoDeletionModule.getOldestVideos`

`storageUsage.getDownloadedBytes`