This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

As for teded, it was zh-Hant-TW.

And you mentioned khan-videos_es_playlists but this one has noting to do with this. it's a recipe issue.

@kelson42, youtube sometimes uses non-standard codes for subtitles. That's something we have to react to when we encounter it but it's not a regression.

I've added mapping for the khan-videos-bn culprit (zh-Hans-CN) and I've tried to find the list of all possible codes from the YT subtitle uploads UI. I found a couple of codes that would have failed and mapped those as well.

Unfortunately, that UI had no reference of that zh-Hans-CN code for instance so it means the UI doesn't expose all the codes they are using, meaning we might encounter a similar issue in the future.

I've added a debug print of such code so that in the event of a similar issue, we'll save a lot of time and be able to fix it immediately.

So, next time, open a new ticket with the lang code.

https://farm.openzim.org/pipeline/5f5f3154cf078db7bac5284e/debug

See https://farm.openzim.org/pipeline/b2e47e70159a9be6d08f6ff5/debug

[youtube2zim::2021-01-10 13:20:01,636] INFO:download all author's profile pictures [youtube2zim::2021-01-10 13:20:01,636] DEBUG:query youtube-api for Channel #UC6nSFpj9HTCZ5t-N3Rm3-HA [youtube2zim::2021-01-10 13:20:01,832] INFO:update general metadata [youtube2zim::2021-01-10 13:20:01,832] DEBUG:query youtube-api for Channel #CSANnvayMtMLizmxt8koEHH5nj1ZR7xq6Pnu-VMYg [youtube2zim::2021-01-10 13:20:01,959] ERROR:FAILED. An error occurred: 'items' [youtube2zim::2021-01-10 13:20:01,959] ERROR:'items' Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/entrypoint.py", line 202, in main scraper.run() File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 310, in run self.update_metadata() File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 727, in update_metadata main_channel_json = get_channel_json(self.main_channel_id) File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/youtube.py", line 88, in get_channel_json channel_json = req.json()["items"][0] KeyError: 'items'

This is happening because of inconsistent data returned by the Youtube API:

When querying playlist PLZRRxQcaEjA4qyEuYfAMCazlL0vQDkIj2, the API tells us the creator_id is Channel CSANnvayMtMLizmxt8koEHH5nj1ZR7xq6Pnu-VMYg.

As you can see, this channel doesn't raise a 404 as incorrect IDs does but the page is blank. If you click on the playlist link above, it says the channel is Channel vsauce1 (which apparently is same as User Vsauce.

So we have no way, from the API, to know about Vsauce and we can only query for that channel ID returned which failes because there is no item returned by the API for it.

I don't want to fix this because:

• this shouldn't happen. this is clearly a bug on YT API.
• making it work will have unwanted side effects.
• this is happening only on this recipe and is about past playlists that won't evolve. Actually, it was supposed to be publicly available only for a month and that's why the recipe was created in the first place.

I guess this is due to the fact that this was a Youtube Originals content (reserved to paying members) that got opened later.

I suggest we close this and disable the recipe.

This pull request has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

Same problem with :

• https://farm.openzim.org/recipes/khan-videos_es_playlists
• https://farm.openzim.org/recipes/khan-videos_bn_playlists

we'll probably go straight to opds 2.0 (json-based) sometime this year

@eshellman From our perspective having a proper OPDS would be preferable, because this is a standard.

I'm again unsure after the helpful comment of @dattaz. If I understand right now, the goal would be to add the following buttons?

I'm not really sure "Active" makes a lot of sense, but would the two other ones be doable in pure javascript (not the buttons, the sorting) based on the DOM infos?

@hksingh4545 Thx

Now JS of DataTables-1.10.13 and Responsive-2.1.1 will be added during setup

Let's close it.

[download] Destination: /output/tmpobb8pd_4/videos/Ouy2ocDbFYs/video.f251.webm

[download] 0.0% of 19.67MiB at 405.05KiB/s ETA 00:49 [download] 0.0% of 19.67MiB at 1.16MiB/s ETA 00:17 [download] 0.0% of 19.67MiB at 2.67MiB/s ETA 00:07 [download] 0.1% of 19.67MiB at 5.64MiB/s ETA 00:03 [download] 0.2% of 19.67MiB at 2.10MiB/s ETA 00:09 [download] 0.3% of 19.67MiB at 2.71MiB/s ETA 00:07 [download] 0.6% of 19.67MiB at 3.60MiB/s ETA 00:05 [download] 1.3% of 19.67MiB at 5.44MiB/s ETA 00:03 [download] 2.5% of 19.67MiB at 8.77MiB/s ETA 00:02 [download] 5.1% of 19.67MiB at 14.54MiB/s ETA 00:01 [download] 10.2% of 19.67MiB at 24.86MiB/s ETA 00:00 [download] 20.3% of 19.67MiB at 25.95MiB/s ETA 00:00 [download] 40.7% of 19.67MiB at 30.64MiB/s ETA 00:00 [download] 49.6% of 19.67MiB at 28.05MiB/s ETA 00:00 [download] 49.6% of 19.67MiB at 416.76KiB/s ETA 00:24 [download] 49.6% of 19.67MiB at 1.19MiB/s ETA 00:08 [download] 49.6% of 19.67MiB at 2.73MiB/s ETA 00:03 [download] 49.7% of 19.67MiB at 5.78MiB/s ETA 00:01 [download] 49.8% of 19.67MiB at 2.10MiB/s ETA 00:04 [download] 49.9% of 19.67MiB at 2.59MiB/s ETA 00:03 [download] 50.2% of 19.67MiB at 3.58MiB/s ETA 00:02 [download] 50.9% of 19.67MiB at 5.24MiB/s ETA 00:01 [download] 52.1% of 19.67MiB at 8.46MiB/s ETA 00:01 [download] 54.7% of 19.67MiB at 14.15MiB/s ETA 00:00 [download] 59.8% of 19.67MiB at 24.26MiB/s ETA 00:00 [download] 69.9% of 19.67MiB at 30.94MiB/s ETA 00:00 [download] 90.3% of 19.67MiB at 35.36MiB/s ETA 00:00 [download] 98.8% of 19.67MiB at 35.40MiB/s ETA 00:00 [download] 98.8% of 19.67MiB at 438.87KiB/s ETA 00:00 [download] 98.8% of 19.67MiB at 1.25MiB/s ETA 00:00 [download] 98.9% of 19.67MiB at 2.87MiB/s ETA 00:00 [download] 98.9% of 19.67MiB at 6.03MiB/s ETA 00:00 [download] 99.0% of 19.67MiB at 2.09MiB/s ETA 00:00 [download] 99.1% of 19.67MiB at 2.67MiB/s ETA 00:00 [download] 99.5% of 19.67MiB at 3.56MiB/s ETA 00:00 [download] 100.0% of 19.67MiB at 5.10MiB/s ETA 00:00 [download] 100% of 19.67MiB in 00:00 [ffmpeg] Merging formats into "/output/tmpobb8pd_4/videos/Ouy2ocDbFYs/video.webm" Deleting original file /output/tmpobb8pd_4/videos/Ouy2ocDbFYs/video.f244.webm (pass -k to keep) Deleting original file /output/tmpobb8pd_4/videos/Ouy2ocDbFYs/video.f251.webm (pass -k to keep) [zimscraperlib::2021-01-11 02:18:34,602] INFO:Encode /output/tmpobb8pd_4/videos/Ouy2ocDbFYs/video.webm -> /output/tmpobb8pd_4/videos/Ouy2ocDbFYs/video.webm video format = .webm [youtube2zim::2021-01-11 02:31:44,534] DEBUG:Uploading video file for Ouy2ocDbFYs to cache ... [youtube2zim::2021-01-11 02:31:49,370] ERROR:webm/low/Ouy2ocDbFYs failed to upload to cache: Failed to upload /output/tmpobb8pd_4/videos/Ouy2ocDbFYs/video.webm to org-kiwix-youtube/webm/low/Ouy2ocDbFYs: An error occurred (TemporarilyUnavailable) when calling the CreateMultipartUpload operation (reached max retries: 4): Maximum number of server active requests exceeded [youtube2zim::2021-01-11 02:31:49,370] DEBUG:Attempting to download thumbnail for Ouy2ocDbFYs from cache... [youtube2zim::2021-01-11 02:31:58,227] ERROR:FAILED. An error occurred: local variable 'meta' referenced before assignment [youtube2zim::2021-01-11 02:31:58,227] ERROR:local variable 'meta' referenced before assignment Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/entrypoint.py", line 202, in main scraper.run() File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 294, in run succeeded, failed = self.download_video_files( File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 506, in download_video_files return self.download_video_files_batch(options, self.videos_ids) File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 698, in download_video_files_batch if self.download_video(video_id, options) and self.download_thumbnail( File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 651, in download_thumbnail if self.download_from_cache(s3_key, thumbnail_path, preset.VERSION): File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 564, in download_from_cache if not self.s3_storage.has_object_matching_meta( File "/usr/local/lib/python3.8/site-packages/kiwixstorage-0.4-py3.8.egg/kiwixstorage/init.py", line 430, in has_object_matching_meta return meta.get(tag) == value UnboundLocalError: local variable 'meta' referenced before assignment

In c’est pas sorcier recipe https://farm.openzim.org/pipeline/0cb0223bedd943b9b7537ff5/debug

upstream bug https://github.com/kiwix/python_storagelib/issues/5

https://farm.openzim.org/pipeline/5e291209e3a713c9cbdebff5/debug

[youtube2zim::2021-01-11 06:20:15,354] ERROR:FAILED. An error occurred: 404 Client Error: Not Found for url: https://www.googleapis.com/youtube/v3/playlistItems?playlistId=+PLrei6UENqoSpmwTS-ALRpGR3icX6HlRmB&part=snippet%2CcontentDetails&key=AIzaSyCjggfpmMSXFLoMhZQgyMlPBHoUXgQb5uc&maxResults=50
[youtube2zim::2021-01-11 06:20:15,354] ERROR:404 Client Error: Not Found for url: https://www.googleapis.com/youtube/v3/playlistItems?playlistId=+PLrei6UENqoSpmwTS-ALRpGR3icX6HlRmB&part=snippet%2CcontentDetails&key=AIzaSyCjggfpmMSXFLoMhZQgyMlPBHoUXgQb5uc&maxResults=50
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/entrypoint.py", line 202, in main
    scraper.run()
  File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 274, in run
    self.extract_videos_list()
  File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 458, in extract_videos_list
    videos_json = get_videos_json(playlist.playlist_id)
  File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/youtube.py", line 185, in get_videos_json
    req.raise_for_status()
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.googleapis.com/youtube/v3/playlistItems?playlistId=+PLrei6UENqoSpmwTS-ALRpGR3icX6HlRmB&part=snippet%2CcontentDetails&key=AIzaSyCjggfpmMSXFLoMhZQgyMlPBHoUXgQb5uc&maxResults=50

Spaces in playlist IDs. Recipe mistake.

https://farm.openzim.org/pipeline/5e291209e3a713c9cbdebff5/debug

[youtube2zim::2021-01-11 06:20:15,354] ERROR:FAILED. An error occurred: 404 Client Error: Not Found for url: https://www.googleapis.com/youtube/v3/playlistItems?playlistId=+PLrei6UENqoSpmwTS-ALRpGR3icX6HlRmB&part=snippet%2CcontentDetails&key=AIzaSyCjggfpmMSXFLoMhZQgyMlPBHoUXgQb5uc&maxResults=50
[youtube2zim::2021-01-11 06:20:15,354] ERROR:404 Client Error: Not Found for url: https://www.googleapis.com/youtube/v3/playlistItems?playlistId=+PLrei6UENqoSpmwTS-ALRpGR3icX6HlRmB&part=snippet%2CcontentDetails&key=AIzaSyCjggfpmMSXFLoMhZQgyMlPBHoUXgQb5uc&maxResults=50
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/entrypoint.py", line 202, in main
    scraper.run()
  File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 274, in run
    self.extract_videos_list()
  File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 458, in extract_videos_list
    videos_json = get_videos_json(playlist.playlist_id)
  File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/youtube.py", line 185, in get_videos_json
    req.raise_for_status()
  File "/usr/local/lib/python3.8/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://www.googleapis.com/youtube/v3/playlistItems?playlistId=+PLrei6UENqoSpmwTS-ALRpGR3icX6HlRmB&part=snippet%2CcontentDetails&key=AIzaSyCjggfpmMSXFLoMhZQgyMlPBHoUXgQb5uc&maxResults=50

[download] Destination: /output/tmpobb8pd_4/videos/Ouy2ocDbFYs/video.f251.webm

[download] 0.0% of 19.67MiB at 405.05KiB/s ETA 00:49 [download] 0.0% of 19.67MiB at 1.16MiB/s ETA 00:17 [download] 0.0% of 19.67MiB at 2.67MiB/s ETA 00:07 [download] 0.1% of 19.67MiB at 5.64MiB/s ETA 00:03 [download] 0.2% of 19.67MiB at 2.10MiB/s ETA 00:09 [download] 0.3% of 19.67MiB at 2.71MiB/s ETA 00:07 [download] 0.6% of 19.67MiB at 3.60MiB/s ETA 00:05 [download] 1.3% of 19.67MiB at 5.44MiB/s ETA 00:03 [download] 2.5% of 19.67MiB at 8.77MiB/s ETA 00:02 [download] 5.1% of 19.67MiB at 14.54MiB/s ETA 00:01 [download] 10.2% of 19.67MiB at 24.86MiB/s ETA 00:00 [download] 20.3% of 19.67MiB at 25.95MiB/s ETA 00:00 [download] 40.7% of 19.67MiB at 30.64MiB/s ETA 00:00 [download] 49.6% of 19.67MiB at 28.05MiB/s ETA 00:00 [download] 49.6% of 19.67MiB at 416.76KiB/s ETA 00:24 [download] 49.6% of 19.67MiB at 1.19MiB/s ETA 00:08 [download] 49.6% of 19.67MiB at 2.73MiB/s ETA 00:03 [download] 49.7% of 19.67MiB at 5.78MiB/s ETA 00:01 [download] 49.8% of 19.67MiB at 2.10MiB/s ETA 00:04 [download] 49.9% of 19.67MiB at 2.59MiB/s ETA 00:03 [download] 50.2% of 19.67MiB at 3.58MiB/s ETA 00:02 [download] 50.9% of 19.67MiB at 5.24MiB/s ETA 00:01 [download] 52.1% of 19.67MiB at 8.46MiB/s ETA 00:01 [download] 54.7% of 19.67MiB at 14.15MiB/s ETA 00:00 [download] 59.8% of 19.67MiB at 24.26MiB/s ETA 00:00 [download] 69.9% of 19.67MiB at 30.94MiB/s ETA 00:00 [download] 90.3% of 19.67MiB at 35.36MiB/s ETA 00:00 [download] 98.8% of 19.67MiB at 35.40MiB/s ETA 00:00 [download] 98.8% of 19.67MiB at 438.87KiB/s ETA 00:00 [download] 98.8% of 19.67MiB at 1.25MiB/s ETA 00:00 [download] 98.9% of 19.67MiB at 2.87MiB/s ETA 00:00 [download] 98.9% of 19.67MiB at 6.03MiB/s ETA 00:00 [download] 99.0% of 19.67MiB at 2.09MiB/s ETA 00:00 [download] 99.1% of 19.67MiB at 2.67MiB/s ETA 00:00 [download] 99.5% of 19.67MiB at 3.56MiB/s ETA 00:00 [download] 100.0% of 19.67MiB at 5.10MiB/s ETA 00:00 [download] 100% of 19.67MiB in 00:00 [ffmpeg] Merging formats into "/output/tmpobb8pd_4/videos/Ouy2ocDbFYs/video.webm" Deleting original file /output/tmpobb8pd_4/videos/Ouy2ocDbFYs/video.f244.webm (pass -k to keep) Deleting original file /output/tmpobb8pd_4/videos/Ouy2ocDbFYs/video.f251.webm (pass -k to keep) [zimscraperlib::2021-01-11 02:18:34,602] INFO:Encode /output/tmpobb8pd_4/videos/Ouy2ocDbFYs/video.webm -> /output/tmpobb8pd_4/videos/Ouy2ocDbFYs/video.webm video format = .webm [youtube2zim::2021-01-11 02:31:44,534] DEBUG:Uploading video file for Ouy2ocDbFYs to cache ... [youtube2zim::2021-01-11 02:31:49,370] ERROR:webm/low/Ouy2ocDbFYs failed to upload to cache: Failed to upload /output/tmpobb8pd_4/videos/Ouy2ocDbFYs/video.webm to org-kiwix-youtube/webm/low/Ouy2ocDbFYs: An error occurred (TemporarilyUnavailable) when calling the CreateMultipartUpload operation (reached max retries: 4): Maximum number of server active requests exceeded [youtube2zim::2021-01-11 02:31:49,370] DEBUG:Attempting to download thumbnail for Ouy2ocDbFYs from cache... [youtube2zim::2021-01-11 02:31:58,227] ERROR:FAILED. An error occurred: local variable 'meta' referenced before assignment [youtube2zim::2021-01-11 02:31:58,227] ERROR:local variable 'meta' referenced before assignment Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/entrypoint.py", line 202, in main scraper.run() File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 294, in run succeeded, failed = self.download_video_files( File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 506, in download_video_files return self.download_video_files_batch(options, self.videos_ids) File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 698, in download_video_files_batch if self.download_video(video_id, options) and self.download_thumbnail( File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 651, in download_thumbnail if self.download_from_cache(s3_key, thumbnail_path, preset.VERSION): File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 564, in download_from_cache if not self.s3_storage.has_object_matching_meta( File "/usr/local/lib/python3.8/site-packages/kiwixstorage-0.4-py3.8.egg/kiwixstorage/init.py", line 430, in has_object_matching_meta return meta.get(tag) == value UnboundLocalError: local variable 'meta' referenced before assignment

In c’est pas sorcier recipe https://farm.openzim.org/pipeline/0cb0223bedd943b9b7537ff5/debug

See https://farm.openzim.org/pipeline/b2e47e70159a9be6d08f6ff5/debug

[youtube2zim::2021-01-10 13:20:01,636] INFO:download all author's profile pictures [youtube2zim::2021-01-10 13:20:01,636] DEBUG:query youtube-api for Channel #UC6nSFpj9HTCZ5t-N3Rm3-HA [youtube2zim::2021-01-10 13:20:01,832] INFO:update general metadata [youtube2zim::2021-01-10 13:20:01,832] DEBUG:query youtube-api for Channel #CSANnvayMtMLizmxt8koEHH5nj1ZR7xq6Pnu-VMYg [youtube2zim::2021-01-10 13:20:01,959] ERROR:FAILED. An error occurred: 'items' [youtube2zim::2021-01-10 13:20:01,959] ERROR:'items' Traceback (most recent call last): File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/entrypoint.py", line 202, in main scraper.run() File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 310, in run self.update_metadata() File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/scraper.py", line 727, in update_metadata main_channel_json = get_channel_json(self.main_channel_id) File "/usr/local/lib/python3.8/site-packages/youtube2zim-2.1.12-py3.8.egg/youtube2zim/youtube.py", line 88, in get_channel_json channel_json = req.json()["items"][0] KeyError: 'items'