Author Topic: Record On-demand files at 320kbps with YouTube-dl.exe  (Read 142 times)

0 Members and 1 Guest are viewing this topic.

Offline Ed

  • Full Member
  • ***
  • Posts: 119
    • View Profile
Record On-demand files at 320kbps with YouTube-dl.exe
« on: August 05, 2017, 06:06:52 PM »
This is a brief note intended to help listeners to the BBC iPlayer service, following recent changes to it that have hidden many of the audio streams that run at more than 128 kbps.



BBC iPlayer DASH audio streams

I have, with regret, abandoned the FFMPEG program entirely.

The BBC now use two types of audio stream, HLS and DASH. FFMPEG can only download HLS, but I cannot find any HLS on-demand streams that are genuinely running at more than 128 kbps (i.e. the entire 'Listen-again' service is only running at 128 kbps, even streams that are advertised as having a higher bitrate).

Solutions I've suggested in the past on Beebotron for HLS streams will still work. And the live streams can still be accessed at the 320 kbps bitrate. Even the On-demand streams can be accessed, but only at 128 kbps, not at 320 kbps.

But I would now recommend Livestreamer.exe as the best program for accessing all of those streams, not ffmpeg.

Livestreamer is here:

https://github.com/chrippa/livestreamer/releases/download/v1.12.2/livestreamer-v1.12.2-win32-setup.exe

And for anyone who wants the 320 kbps bitrate, only the DASH streams now provide that, for the On-demand streams (and the BBC seem to be moving gradually towards using only DASH). So welcome to the world of YouTube-dl.



YouTube-dl

This is a downloader program which downloads DASH streams. It runs on Windows (and other platforms), and was originally developed to download videos from YouTube.com (hence the fact that it's called YouTube Downloader), but it also works with other sites that use DASH.

Get YouTube-dl here:

  https://rg3.github.io/youtube-dl/

The Firefox script by Andrea has been rewritten from scratch for the new version (version 4), to cope with the latest changes to iPlayer. The version 4 script doesn't use FFMPEG, instead it takes advantage of YouTube-dl : a program which has DASH recording capability.

Get Andrea's script (currently v4.0.5), for Firefox, here:

  https://greasyfork.org/en/scripts/8161-bbc-iplayer-video-download


The audio streams are downloaded in their original formats. Usually this results in an .MP4 file for tv shows, and .M4A for radio shows. The file plays in GOM media player (and, in my tests, in most media players that run on Windows - except, sadly, Winamp).

The script enables you to download (i.e. record) video or audio in the DASH format, by providing the necessary command line. You also need to use youtube-dl to actually record the streams (https://rg3.github.io/youtube-dl/).


So all you need is the YouTube-dl.exe program, in order to download a DASH audio file using the new type of command line, which Andrea's script will give you (currently the script is version 4.0.5).

I use Firefox 39, with the Greasemonkey add-on. Andrea's script installs in Firefox as a Greasemonkey script, as described in my other posts on Beebotron about the iPlayer 320 kbps streams.

Here's an example of the command line that appears on the iPlayer page when you open it in Firefox. This is an episode of 'To the Manor Born' (from the radio series) -

Code: [Select]
youtube-dl -f bestaudio "http://aod-dash-uk-live.akamaized.net/usp/auth/vod/piff_abr_full_audio/648c91-b0059prl/vf_b0059prl_3e1c3c39-8dd1-4b21-9b3b-a3a7656fc519.ism/pc_hd_abr_v2_uk_dash_master.mpd?__gda__=1501970164_f704cf749be1d3bcabe59ff62e003d1a" -o To_the_Manor_Born_The_Rhythms_of_the_Earth
Copy-and-paste the command line from Firefox into a batch file (a .bat or .cmd file in Windows), put the batch file in the same folder as YouTube-dl.exe, then run the batch file. The file which downloads must then be given the extension .mp4 or .m4a by you.



MPEG-DASH

Dynamic Adaptive Streaming over HTTP (DASH), also known as MPEG-DASH, works by breaking the content into a sequence of small HTTP-based file segments, each segment containing a short interval of playback time of content.

MPEG-DASH should not be confused with a transport protocol — the transport protocol that MPEG-DASH uses is TCP.

It can use content encoded with any codec. The BBC currently use both HLS and MPEG-DASH.



« Last Edit: August 05, 2017, 06:42:01 PM by Ed »

ilainftw

  • Guest
Re: Record On-demand files at 320kbps with YouTube-dl.exe
« Reply #1 on: August 08, 2017, 02:14:01 PM »
But I would now recommend Livestreamer.exe as the best program for accessing all of those streams, not ffmpeg.
Livestreamer is long-abandoned. It would probably be worth looking into its successor Streamlink (https://streamlink.github.io/). A plugin with some iPlayer TV support has been added to Streamlink. Hopefully, someone will add an iPlayer radio plugin at some point.
The BBC now use two types of audio stream, HLS and DASH. FFMPEG can only download HLS, but I cannot find any HLS on-demand streams that are genuinely running at more than 128 kbps (i.e. the entire 'Listen-again' service is only running at 128 kbps, even streams that are advertised as having a higher bitrate).
320k HLS streams for most on-demand radio programmes are most definitely available, but it's not worth the bother extracting them yourself for use with ffmpeg. Both get_iplayer (https://github.com/get-iplayer/get_iplayer/wiki) and youtube-dl can access them. But with 320k DASH streams available, they aren't altogether necessary.
And for anyone who wants the 320 kbps bitrate, only the DASH streams now provide that, for the On-demand streams (and the BBC seem to be moving gradually towards using only DASH). So welcome to the world of YouTube-dl.
There are some wrinkles to be aware of with youtube-dl. The files it produces from DASH streams are fragmented M4A, which is why some media players (e.g., iTunes) don't like them. You can work around that by using --add-metadata in your youtube-dl command (requires ffmpeg). The metadata added is negligible, but the file is losslessly re-muxed (and thus defragmented) in the process. You can also do it yourself after download, of course. If you download an HLS stream with youtube-dl, it should automatically re-mux the resulting MPEG-TS file to M4A if ffmpeg is available. If not it will leave a .m4a file that does not really have an MP4 container - it is MPEG-TS. The other small gotcha is that you have to sign in to the BBC (where required) for the Greasemonkey script to work.

For those who don't want to use Firefox or Greasemonkey (or Chrome equivalents), you can accomplish much the same thing with youtube-dl alone. Assuming you want to restrict yourself to DASH streams as the Greasemonkey script does (usually not necessary, but may be faster downloads), you can use commands in the form:
Code: [Select]
youtube-dl -f "best[protocol*=dash]" --restrict-filename --add-metadata http://www.bbc.co.uk/programmes/b007jpzwThe [protocol*=dash] filter restricts youtube-dl to DASH streams and --restrict-filename replaces white space and unfriendly characters in the output filename.

Use [protocol*=m3u8] filter to restrict youtube-dl to HLS streams:
Code: [Select]
youtube-dl -f "best[protocol*=m3u8]" --restrict-filename --add-metadata http://www.bbc.co.uk/programmes/b007jpzw
One place the Greasmonkey script may be useful is in constructing an output filename (the -o option). Because of the way youtube-dl extracts BBC radio programme titles, the series and episode names are usually reversed in the default filenames it generates. Also, the command generated by the Greasemonkey script may take slightly less time to run because it doesn't have to filter the available streams.

It's a matter of taste, but for me get_iplayer is a simpler solution. You just need to type "get_iplayer " at a command prompt, then copy/paste the programme's URL (from browser) after it, then hit Return:
Code: [Select]
get_iplayer http://www.bbc.co.uk/programmes/b007jpzwBy default, it uses DASH streams (or HLS if DASH not available) with the highest available bit rate (usually 320k) to produce a defragmented M4A file with full metadata and a reasonable filename.

Use --radiomode=daf option to restrict get_iplayer to DASH streams:
Code: [Select]
get_iplayer http://www.bbc.co.uk/programmes/b007jpzw --radiomode=daf
Use --radiomode=haf option to restrict get_iplayer to HLS streams:
Code: [Select]
get_iplayer http://www.bbc.co.uk/programmes/b007jpzw --radiomode=haf
« Last Edit: August 09, 2017, 01:12:06 PM by ilainftw »

Offline Ed

  • Full Member
  • ***
  • Posts: 119
    • View Profile
Re: Record On-demand files at 320kbps with YouTube-dl.exe
« Reply #2 on: August 31, 2017, 02:44:19 AM »
It's nice to hear from someone who understands these programs. I use them, but my familiarity with them is purely empirical.

In your view, is the use of FFMPEG to extract part of the downloaded file (trimming off the announcer at the start and the end of the file) a sufficient re-muxing to achieve the result you describe? It results in an iso2 file, in m4a format.

I use this command:


Code: [Select]
::  **  INPUT File : iso6 (iso6/dash) **
    SET file=C:\Path\to\file\filename.m4a

    ::  Specify STARTING POINT in seconds
    SET start=0

    ::  Specify NUMBER of frames to extract (duration of part extracted) (46.875 frames per second)
    SET frames=99999

::  ** Extract PART of .M4A (AAC)  [Output = isom/iso2] **
    SET extract=-ss %start% -aframes %frames%
    SET options=-map_metadata 0 -movflags +faststart -flags global_header
    ffmpeg.exe  -i "%file%"  -vn -acodec copy %extract%  %options%  "%file%.m4a"


Is it necessary to include  --add-metadata  as well?  (I can't in fact find that option, in the ffmpeg help files. And the program throws an error if I use that option.)



It is NOT necessary to sign-in to a BBC account before you can use Andrea's Greasemonkey script:

  How to listen to iPlayer radio without signing-in -
  https://forum.beebotron.org/index.php?topic=66093.0

I use the script all the time, and have never signed in. Due to those solutions.



My only objection to get_iplayer was the complexity (for the average user) of having to install Perl in order to use it.

But it also offers the user no way out in the event of any problem. Every time the BBC make yet another change to the iplayer site, it typically breaks the current version of get_iplayer. All one can do is wait in the hope that a new version of get_iplayer will eventually be released.

Any more 'hands-on' approach gives the user options to fix a problem himself, immediately, perhaps by switching to a new url. A problem is user-repairable if all it needs is for the user to substitute a new BBC url (for a non-working one) in a batch script.




« Last Edit: September 05, 2017, 10:34:58 PM by Ed »