Author Topic: Download TV or Radio using only the 8-digit PID  (Read 943 times)

0 Members and 1 Guest are viewing this topic.

Offline Ed

  • Full Member
  • ***
  • Posts: 132
    • View Profile
Download TV or Radio using only the 8-digit PID
« on: January 02, 2018, 07:12:02 AM »
Here is a method for downloading from iPlayer using only the url address of the iPlayer page. This method works for Radio shows and for TV shows.

This means that -

  • You can download without having to log-in to a BBC account
  • You can download without having to install Adobe's flash player
  • You can download without having to find the broadcast's hidden vpid number



One preliminary step, first. Download the YouTube-dl program (originally developed for downloading from YouTube, but its latest version can download from many websites which use DASH streams including the BBC's iPlayer site) -

  https://rg3.github.io/youtube-dl/

The program does NOT require installing. Just put it in any convenient folder/directory.



And a second preliminary step. Download the ffmpeg program -

  http://ffmpeg.zeranoe.com/builds/

Put it in the same folder/directory with the YouTube-dl program. That is now your working directory.



For a Radio show, use the following command line (actually two lines in this case!) -

Code: [Select]
    SET url=https://www.bbc.co.uk/programmes/00000000

    YouTube-dl.exe  -f "bestaudio[protocol*=dash]"  --add-metadata  "%url%"


1. Copy-and-paste the command (above) into a batch file (in Windows, a batch file is a plain text file with the extension .bat or .cmd instead of .txt), and put the batch file in the same folder as YouTube-dl.exe and ffmpeg.exe

2. Substitute the actual URL address of the iplayer page which plays the radio show for the one shown in my example. In effect you are just replacing the final 8 zero's in my example with the actual 8-digit PID (Programme ID).

3. Then run the batch file by double-clicking it.

The file which downloads should already have the extension .mp4 or .m4a (but if by chance it has not, you simply add either of those extensions). M4A is the iTunes extension.



For a TV show, use the following command line (actually two lines!) -

Code: [Select]
    SET url=https://www.bbc.co.uk/iplayer/episode/00000000

    YouTube-dl.exe  "%url%"  -f best  --add-metadata


1. Copy-and-paste the command (above) into a batch file (in Windows, a batch file is a plain text file with the extension .bat or .cmd instead of .txt), and put the batch file in the same folder as YouTube-dl.exe and ffmpeg.exe

2. Substitute the actual URL address of the iplayer page which plays the TV show for the one shown in my example. In effect you are just replacing the final 8 zero's in my example with the actual 8-digit PID (Programme ID).

3. Then run the batch file by double-clicking it.

The file which downloads should already have the extension .mp4 (but if by chance it has not, you simply add that extension).

Note that the command does not mention ffmpeg. This is NOT a mistake. The YouTube-dl program calls ffmpeg when it requires to use certain functions of ffmpeg, and calls it automatically so that you don't need to. Just bear in mind that things will go wrong if ffmpeg is not in the same folder as YouTube-dl.

Note that a tv show uses a different url address to that for a radio show, so be sure to use 2 separate batch files, as there is no benefit in trying to use only one. The 2nd line is also different in the case of a tv show.



That's all there is to it. But there are some additional options for a tv show, if you want to be in better control of the file you end up with.

Unlike radio (which usually has only 2 streams, a 128 kbps one and a 320 kbps one), TV has a large number of available streams. You can use the above command to get the default stream.

But if the default stream gets you a file that won't play (many of the available streams use the new avc3 codec, which only Windows 10 can play, so if you use Windows 7 you will instead prefer to have a file containing a stream which has the avc1 codec), then you need to get a list of the available streams ("formats").

Put the following command in a batch file (name it, say, TV_streams.bat) and run it (substitute the TV show's actual url for the one in my example, just as you did above) -

Code: [Select]
    SET url=https://www.bbc.co.uk/iplayer/episode/00000000

    YouTube-dl.exe "%url%" --list-formats > Formats.txt

Wait, then open the text file Formats.txt using Notepad (it will take perhaps a minute to download all the data into it). Then pick a stream: they will be listed by video size, e.g. 640x360, and by codec, e.g. avc1.

The following is a substitute batch file to download the TV stream you choose. This is a simple example (there are many other possibilities); but this is designed to avoid complicating matters -


Code: [Select]
    SET url=https://www.bbc.co.uk/iplayer/episode/00000000

    SET format=stream-uk-iptv_streaming_concrete_combined_sd_mf_akamai_uk_hls_https-1836

    YouTube-dl.exe  "%url%"  -f %format%  --add-metadata


Ideally, you will choose a stream which (as in this example) is a "combined" stream - as stated in the name of the stream. A combined stream is one which contains both audio and video.

I recommend the 1836 stream (shown in this example) because it contains AVC video (a.k.a. avc1 video), UK-style 25 fps (frames per second), an image size of 960x540 pixels, and good quality audio at 128 kbps.

Do NOT choose any video stream containing the avc3 codec, unless you have satisfied yourself that your computer can play that! If you are running Windows XP, Windows Vista, or Windows 7 it's a good bet you can't play a file containing the avc3 codec stream.


1. Copy-and-paste the command into a batch file (in Windows, a batch file is a plain text file with the extension .bat or .cmd instead of .txt), name the file (for example) TV_1836stream.BAT, and put the batch file in the same folder as YouTube-dl.exe and ffmpeg.exe

2. Substitute the actual URL address of the iplayer page which plays the TV show for the one shown in my example. In effect you're just replacing the final 8 zero's in my example with the actual 8-digit PID (Programme ID).

3. Substitute the actual name of your chosen stream, if you are not opting for my choice of the 1836 stream. (If you choose separate audio and video files, instead of a combined file, you'll need to make the necessary changes to line 3 of the batch file - so you'll need to do a Google search on the term ffmpeg to find an appropriate command.)

4. Then run the batch file by double-clicking it.













« Last Edit: January 02, 2018, 08:31:57 AM by Ed »

Offline Janaru

  • Administrator
  • Hero Member
  • *****
  • Posts: 18,401
    • View Profile
Re: Download TV or Radio using only the 8-digit PID
« Reply #1 on: January 05, 2018, 10:04:20 AM »
I'd love to try this, but my old brain got lost. :(  Maybe I''ll give it another try later on. Thanks so much, Ed. :D
Note to self:  Use your powers for good, not for evil.....

Offline TallforaDuck

  • Full Member
  • ***
  • Posts: 113
    • View Profile
Re: Download TV or Radio using only the 8-digit PID
« Reply #2 on: January 05, 2018, 02:49:54 PM »
I'd love to try this, but my old brain got lost. :(  Maybe I''ll give it another try later on. Thanks so much, Ed. :D

My exact thoughts
'Anima Mea' -My Soul

Offline Ed

  • Full Member
  • ***
  • Posts: 132
    • View Profile
The simplified version...
« Reply #3 on: January 06, 2018, 12:15:48 AM »
In brief, for a Radio show all you need is a .bat file with the following contents  -

Code: [Select]
    SET url=https://www.bbc.co.uk/programmes/00000000

    YouTube-dl.exe  -f "bestaudio[protocol*=dash]"  --add-metadata  "%url%"


1. Copy-and-paste the command (above) into an otherwise empty .bat file (on Windows, this is a file called filename.bat). Put that file in the same folder as both YouTube-dl.exe and ffmpeg.exe:

   YouTube-dl  -
   https://rg3.github.io/youtube-dl/

   ffmpeg -
   http://ffmpeg.zeranoe.com/builds/

2. Substitute the actual URL address of the iplayer page which plays the radio show for the one shown in my example, i.e. replace the final 8 zero's in my example with the actual 8-digit PID (Programme ID) from the url address of the BBC page.

3. Then run the .bat file by double-clicking it.

Offline Janaru

  • Administrator
  • Hero Member
  • *****
  • Posts: 18,401
    • View Profile
Re: Download TV or Radio using only the 8-digit PID
« Reply #4 on: January 10, 2018, 10:06:29 AM »
Thank you for dumbing it down, Ed. Unfortunately, I'm dumber even that that! :P  I'll have my husband look at that and see if  he can walk me through it. I really appreciate you taking the time. :)
Note to self:  Use your powers for good, not for evil.....

Offline TallforaDuck

  • Full Member
  • ***
  • Posts: 113
    • View Profile
Re: Download TV or Radio using only the 8-digit PID
« Reply #5 on: January 14, 2018, 09:40:46 PM »
Very much like Jan here, I will try this again midweek but at the minute I download single PID
recordings at a time by cutting and pasting commands.
What I''d like to know is what command do I need to record at 128k instead of 320k as that is all I require?
'Anima Mea' -My Soul

Offline Ironeyes

  • Hero Member
  • *****
  • Posts: 1,335
    • View Profile
Re: Download TV or Radio using only the 8-digit PID
« Reply #6 on: January 18, 2018, 12:38:17 PM »
I've been using get iplayer for some weeks now and have used it to download programmes for me to enjoy during our stay in the antipodes. For those that I wish to keep and listen to again, I always use Audacity to clean up the start and finish (mostly the latter, can't stand an untidy finish), and also then convert and save the download to a 128k MP3 file. The whole process can, with a good download speed, take a matter of minutes.

Offline TallforaDuck

  • Full Member
  • ***
  • Posts: 113
    • View Profile
Re: Download TV or Radio using only the 8-digit PID
« Reply #7 on: January 18, 2018, 04:07:11 PM »
Thanks Ironeyes, that's a reasonable idea, I have uninstalled Audacity as it seemed to take an eternity to load and save programmes,  my laptop is getting on now and though I purchased a newer on eight months ago, I still feel more comfortable with this old thing.

Currently, I do something similar but using the Freemake Video Converter, which has a good Audio option with it and if I want to tidy up start and finish, as like you it's a bugbear, I use the excellent MP3Cut which doesn't have to rerecord, but instantly cuts unwanted portions.

It is just slightly irritating that in the instructions for the old version it was quite a simple option to put 128k as the defaut, this new version doesn't want to play ball.

Thanks for the idea though.

Oh! forgot to say, I have a nice simple Audio restoration program Magix Audio Cleaner, which is simpler than the old version of Audacity I had.

I'll see if I can find a newer version of Audacity online.
'Anima Mea' -My Soul

Offline Ed

  • Full Member
  • ***
  • Posts: 132
    • View Profile
Re: Download TV or Radio using only the 8-digit PID
« Reply #8 on: February 05, 2018, 07:47:01 AM »
Okay, in response to the latest changes on iPlayer I have written a batch script for downloading a radio show from the 'listen again' pages, that needs only the URL address of the page. And all you need is the last 8 digits from that URL, the 'PID' (Programme ID), which you type into the .bat file.

If you are running Windows this script will work for you, because the script uses some built-in functions of Windows. Including some functions of Internet Explorer. (The script accesses these things for itself, you don't need to do anything.)

I have bundled with it the only program you will need, which is the YouTube-dl.exe program.

The beauty of this script is that it runs without you being signed-in on the BBC website.

Also, I have designed the script to use the https secure stream, which gives a much better result: a fast download, without the stream continually pausing, and without the frequent failures that the other streams suffer from. The https stream is far more reliable, and far faster, taking only 60 seconds to download a 30 minute radio show.


1. Get the script here, in a .zip file:
http://www.mediafire.com/file/bxrskhdatv54z71/Download_from_iPlayer_-_Build_2018-02-05.zip

2. Open the .zip file, and take the .bat file from the directory named 'iPlayer - Download Automatically'. Also take out the YouTube-dl.exe program, which put in the folder/directory C:\Program Files\YouTube-dl

3. Open the .bat file with any text editor, type in the 8 digits from the URL address of the iplayer page (where it says "SET PID=" in the .bat file, you'll be changing that to something like: SET PID=12345678). Then double-click the .bat file to run it.


Optionally, I have included at the top of the .bat file a space for you to include a file name if you want to. If you leave it blank, some details from the iplayer page will be used instead, i.e. as the file name of the downloaded radio programme.

Remember that you can still listen to any radio show without signing in, simply by opening the iplayer page in Internet Explorer 8.


For anyone keen to record at 128 kbps, use your text editor to search the .bat file, and replace each instance of the value 320 with the value 128 instead. In fact, in this version of the .bat file there is only one command that specifies that value (plus it's mentioned in a couple of places in the notes/hints/annotations, but those places don't control the script).


.
« Last Edit: February 05, 2018, 09:11:01 AM by Ed »

Offline Ed

  • Full Member
  • ***
  • Posts: 132
    • View Profile
Re: Download TV or Radio using only the 8-digit PID
« Reply #9 on: February 05, 2018, 08:36:06 AM »
@TallForADuck -

Here is a method for quickly editing an audio file, using a .bat file (batch file).

This is fast and lossless, because it uses a special form of the ffmpeg command, one that does NOT involve re-encoding the input file. This is an ffmpeg "stream copy" operation, that copies the selected section of audio without re-encoding it. A raw copy of this sort will take less than ten seconds (with a 30 minute radio show).

Requires ffmpeg.exe from:
http://ffmpeg.zeranoe.com/builds/


Code: [Select]
::  *** Extract PART of M4A (AAC) File ***

::  **  INPUT File (WITHOUT .m4a Extension) **
    SET file=The Daughter of Time - Episode 1

::  ** Specify STARTING POINT in seconds **
    SET start=1.0

::  ** Specify NUMBER of frames to extract (46.875 frames per second) **
    SET duration=79176

::  ==================================================================  **

::  Location of FFMPEG
    SET ffmpeg="C:\Program Files\FFmpeg\ffmpeg.exe" -hide_banner

::  Safety Check
    ATTRIB -R -A -S -H  "%file%.m4a"

::  Extract PART of .M4A (AAC) File
    SET extract=-ss %start% -aframes %duration%
    SET options=-map_metadata 0  -flags:a global_header  -movflags faststart
    "%file%.m4a"  -vn  -acodec copy  %extract%  %options%  "%file%[%duration%].m4a"

    :: Command will fail unless -flags options are the final options in the chain,
    :: i.e. final options before the output file.


To convert the file from its normal .m4a iTunes format to mp3, simply change the final line so that the output file has the mp3 extension instead (and if it is desired to convert the entire file to mp3, rather than extracting just a part of it, simply omit the option %extract% from this final line) -

Code: [Select]
    "%file%.m4a"  -vn  -acodec copy  %extract%  %options%  "%file%[%duration%].mp3"


If you don't like using command programs, then for editing an mp3 file I recommend mp3DirectCut (which cuts instantly, as it doesn't have to re-encode). But I would point out that it - like all mp3 editors - can only edit an mp3 or an mp2 file. It can't edit, for example, an m4a iTunes format file (the iPlayer type): you must convert that to mp3 before you can use an mp3 editor on it.

Get mp3DirectCut here:  http://mpesch3.de1.cc/#mp3dc


.


« Last Edit: February 05, 2018, 09:06:54 AM by Ed »

Offline Ed

  • Full Member
  • ***
  • Posts: 132
    • View Profile
Download TV or Radio using only the 8-digit PID
« Reply #10 on: May 02, 2018, 08:25:56 AM »
While I'm on here today, I'll just mention a couple of new points that have come up in the past month.


First, and most important, the BBC website has abandoned support for the http format. Most software programs, however, have been built on the assumption that http:// is a valid url format.

Those programs are now broken. So are scripts based on them.

From 1st May forward, the BBC website will only recognise a request for a url if the request is made using the https:// format. Whereas, up until last month, the website would serve a stream if the request was made in either http or https format, from now on it will only serve a stream if the secure https stream is requested.

If you access their site, and are refused, this is the first thing to check. Make certain that the url address of the audio stream begins with https:// (and not http://).


Secondly, but probably equally important, from 1st May the BBC's servers will only serve a stream if the request is made using the correct security certificate. Previously, their servers did not ask for this.

Again, many software programs, and scripts based on them, will be broken as a result of this change.

In the case of YouTube-dl.exe the solution is to add to the command line the option --no-check-certificate because without that the server will simply send a refusal instead of the requested stream. My above-mentioned batch file must be modified by you, to add that option to the command line which mentions YouTube-dl.

Again, if you access their site, and are refused, this is the second thing to check.

So far my researches have not extended beyond avoiding the check. I can't yet comment on the specifics of the particular security certificate, or how to comply rather than simply side-step.


Thirdly, I regret to announce that my favourite trick for avoiding signing-in on the BBC website will no longer work: they have now fixed their earlier mistake, and radio shows will no longer play in Internet Explorer 8.

I currently use a different trick to play a radio show without signing in.

The method is to slightly modify my above-mentioned batch file, by substituting for the command line in it either of the following instead -


Code: [Select]
    "%YouTube-dl%"  -o -  %url%   --no-check-certificate | "C:\Program Files\VLC\vlc.exe" -
Code: [Select]
    "%YouTube-dl%"  -o -  %url%   --no-check-certificate | "C:\Program Files\MPlayer\mplayer.exe" -

My above-mentioned batch file contains the script needed to extract the correct url from the BBC's web pages. You will need to install either VLC player or MPlayer on your computer, to actually play the .m4a (itunes audio format) audio stream.

Get VLC media player :  http://www.videolan.org/vlc/

Get MPlayer for Windows : http://mplayerwin.sourceforge.net/downloads.html






Offline monkeyglands

  • Newbie
  • *
  • Posts: 1
    • View Profile
Re: Download TV or Radio using only the 8-digit PID
« Reply #11 on: May 04, 2018, 04:22:00 PM »
First, and most important, the BBC website has abandoned support for the http format. Most software programs, however, have been built on the assumption that http:// is a valid url format.

Those programs are now broken. So are scripts based on them.

From 1st May forward, the BBC website will only recognise a request for a url if the request is made using the https:// format. Whereas, up until last month, the website would serve a stream if the request was made in either http or https format, from now on it will only serve a stream if the secure https stream is requested.

If you access their site, and are refused, this is the first thing to check. Make certain that the url address of the audio stream begins with https:// (and not http://).
Completely wrong. http:// URL requests are automatically redirected to https:// equivalent. The BBC has been doing that for a long time.
Secondly, but probably equally important, from 1st May the BBC's servers will only serve a stream if the request is made using the correct security certificate. Previously, their servers did not ask for this.

Again, many software programs, and scripts based on them, will be broken as a result of this change.

In the case of YouTube-dl.exe the solution is to add to the command line the option --no-check-certificate because without that the server will simply send a refusal instead of the requested stream. My above-mentioned batch file must be modified by you, to add that option to the command line which mentions YouTube-dl.

Again, if you access their site, and are refused, this is the second thing to check.

So far my researches have not extended beyond avoiding the check. I can't yet comment on the specifics of the particular security certificate, or how to comply rather than simply side-step.
Also completely wrong. The BBC servers do not check individuals' client certificates (that would make no sense), and that is not what --no-check-certificate does. It means the client (youtube-dl in this case) doesn't attempt to verify the certificate presented by the server.
« Last Edit: May 04, 2018, 04:23:42 PM by monkeyglands »

Offline Ed

  • Full Member
  • ***
  • Posts: 132
    • View Profile
Re: Download TV or Radio using only the 8-digit PID
« Reply #12 on: May 12, 2018, 06:40:58 AM »
This isn't a theoretical forum, we only deal with practical solutions to real problems here.


1.  The plain fact is that from the 1st of this month the BBC servers are rejecting any attempt to connect to the iplayer radio streams using http:// and are currently allowing only requests using https:// to connect. (The BBC servers might behave inconsistently in the first stages of this changeover, as has sometimes happened in the past, so some requests using http:// might still be honoured, but inconsistently.)

This change might not be obvious to anyone using a web browser to listen-in, since browsers often substitute a secure connection automatically in this situation, without the user having to intervene, so the user may be unaware of that happening.

Accordingly, if you are using a program such as YouTube-dl or get_iplayer to connect, in the event of a failure/refusal the first thing to do is make sure you are using https:// (the secure protocol) in the url address of the stream.


2.  Also from the 1st of this month, the BBC servers are refusing an ordinary attempt to connect to them using YouTube-dl.exe but will only allow the connection if the documented option --no-check-certificate is added to the command line.

Accordingly, if you are using the program YouTube-dl to connect, in the event of a failure/refusal the second thing to do is make sure you add --no-check-certificate to the command line (somewhere prior to the output file name).




.

Offline ricklous

  • Administrator
  • Sr. Member
  • *****
  • Posts: 321
    • View Profile
Re: Download TV or Radio using only the 8-digit PID
« Reply #13 on: May 13, 2018, 11:10:29 AM »
Ed,
Is there a link or anything from the BBC that I can take a look at to confirm the switch to HTTPS? I may need to rewrite a few things.

I can only find 2 blog posts, neither of which refers to a 1st May cutoff:
http://www.bbc.co.uk/blogs/internet/entries/f6f50d1f-a879-4999-bc6d-6634a71e2e60
and:
http://www.bbc.co.uk/blogs/internet/entries/eb4fdb3a-fa91-49ad-bb71-bbe82dab2bd3

Also, monkeyglands is right. On the youtube-dl documentation, that options is explained as:

Code: [Select]
--no-check-certificate           Suppress HTTPS certificate validation
...which means that youtube-dl is not checking the certificate that the BBC feeds back to it against its own set of trusted CAs. If you're having to use the option to connect to the bbc https url, then either
1. the BBC has a problem with it's certificates,
2. youtube-dl doesn't have an up-to-date library of CAs, or some other issue
3. the certificate arriving at your PC isn't a certificate sent by the BBC.

I think we can discount the first.
Also, I think it's unlikely that the BBC would flat-out reject url requests via http. The Beebotron has been redirecting http to https for about two years, and it's trivial. There is no reason to reject a url instead of rewriting, and it would cause massively more problems than it solves. I wonder if there is something else tripping up your access to the BBC. For example, your ISP could be making a clumsy intervention in your traffic which is tripping things up when youtube-dl tries to switch to the BBCs https url. Mobile ISPs are often found to be doing something they shouldn't when people have issues browsing. Are there any proxies/vpns etc that you know of, on your connection? How do you connect?

Also can you give us an example of a url that trips up, so that we can test it from different locations/connections?