Latest News
The Power of the URL Service…
Time does fly! I know that I promised this blog a while ago now and it seems like Christmas, New Year and then life have just eaten the last couple of months, leaving nothing! Thankfully, i’m starting to find myself with time again and keen to get back to blogging about Channel development!
In the last blog, we covered how to implement a very basic video Channel, which relied on URL Service support for URLs hosted by blip.tv. In this post, we’re going to discover what a URL Service actually is and their power! The actual blip.tv service is a little complicated for this short post, so instead we will focus on a nice simple one for the Euronewswebsite.
A URL Service is essentially a mechanism for Plex to translate a given URL into the associated metadata and available media items. The specific framework documentation can be found here but basically as a developer, you are required to implement two specific functions:
MetadataObjectForURL(url)
- Returns a metadata object for the given URL (VideoClipObject, MovieObject, EpisodeObject, TrackObject, PhotoObject)
MediaObjectsForURL(url)
- Returns a list of MediaObject’s which represent the video streams available for the specific video/photo/music
- This function is expected to execute and return very quickly. It should avoid making any HTTP request which could cause a delay.
There is also an optional function that should be implemented when a single site provides multiple URLs for the same video. This is as follows:
NormalizeURL(url)
- Returns the standard normalised URL
- This function is expected to execute and return very quickly. It should avoid making any HTTP request which could cause a delay
Let’s start by first looking into how these are defined within a plug-in. Plex Media Server needs to know some basic information about the implemented URL Service. All Service related code is placed within a separate subfolder within the bundle called “Services”. It then has its own plist file. The following picture gives a good illustration of the file hierarchy:
It should be pretty obvious the service’s code is contained within the ServiceCode.pys file. However, we should take a quick look over the information contained within the plist file…
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>URL</key> <dict> <key>Euronews</key> <dict> <key>URLPatterns</key> <array> <string>http://([^.]+.)?euronews.net/.+</string> </array> <key>TestURLs</key> <array> <string>http://www.euronews.net/nocomment/2012/01/04/high-winds-and-heavy-rain-lash-uk/</string> </array> </dict> </dict> </dict> </plist>
The initial key defines the name and therefore sub-directory where the actual code resides, under the URL folder. This can be essentially anything, but is always best to keep it short but descriptive. The next point of interest is the URLPatterns. This array of string define regular expressions which represents the URLs supported by this specific service. Think of it as an easy way for Plex to workout which URL Service should be execute against a requested URL. Last, but not least, the TestURLs element defines a collection of URLs which can be used to test the service. This is extremely important in order to quickly determine when a URL Service brakes due to a site change, or other dependent changes, etc.
Okay, now that the service is correctly defined within the ServiceInfo.plist, we must create the necessary folder structure for the code to reside. As seen in the above picture, there are a number of sub-folders required that contain all associated Services, along with one specifically for URL Service. Once these have been created, the actual code file needs to be created named “ServiceCode.pys”. This file should contain all the code associated with your service, implementing all mandatory functions defined earlier. Lets start by looking at an implementation to obtain the associated metadata…
def MetadataObjectForURL(url):
# Request the URL
page = HTML.ElementFromURL(url)
# Extract the details available directly form the page.
title = page.xpath("//head//meta[@property='og:title']")[0].get('content')
description = page.xpath("//head//meta[@name='description']")[0].get('content')
thumb = page.xpath("//head//meta[@property='og:image']")[0].get('content')
return VideoClipObject(
title = title,
summary = description,
thumb = thumb)
As you can see, this is fairly basic. The Euronews website provides a number of video clips of topical news reports. You’ll often find that some URL specific information can be obtained from the HEAD of the HTML document, rather than more complicated XPath into the more embedded information of the page. The functions available to you as a developer are identical to those you know (and love) in the framework. The common scenario is:
- Make a request for the page
- Extract information via XPath
- Return the suitable metadata object
The amount of information available depends really on what’s provided by the site. It’s normal to have the title, description and thumb for a given item. However, sometimes obtaining further metadata can be a little bit more tricky but definitely worth it in the long run!
Once we’ve got all of our nice juicy metadata, we can start by looking at how video(s) are available. The Euronews site actually provides a single FLV file which the embedded player utilities. This was simply found by viewing the source from a page, and searching for common video file extensions. There is only a single quality available so we can only return a single MediaObject. Here’s the code:
def MediaObjectsForURL(url):
return [
MediaObject(
video_codec = VideoCodec.VP6,
audio_codec = AudioCodec.MP3,
container = 'flv',
parts = [PartObject(key=Callback(PlayVideo, url = url))]
)
]
You might wonder how to determine the correct video/audio codecs to report. However, the best application to use is a free program called MediaInfo. It will give you lots of information about the specific file. You’ll often find that for one particular site, the video/audio codecs are always the same. You’ll also notice from the above code that the parts defines a PartObject for which a Callback is required. As you might remember, this function needs to return quickly and cannot do any HTTP requests. Therefore, we know that an FLV file will be available, we just don’t know where it is. The callback will only be executed if the user actively selects that video item to play. When they do, the PlayVideo function is called. Here’s the code for that:
BASE_URL = 'http://video.euronews.net/'
RE_VIDEO_URL = Regex('videofile:"(?P<video_url>[^"]+)"')
def PlayVideo(url):
# Request the URL
page = HTTP.Request(url).content
# The source of the page actually contains a link to the associated flv file. We can simply find
# this by using a regular expression to find it. Then, we just redirect.
video_url = RE_VIDEO_URL.search(page).group('video_url') + ".flv"
return Redirect(BASE_URL + video_url)
This is doing the basic HTML request and using a regular expression to locate the known FLV file. The use of the Regex class provides a pre-compiled version of the expression, similar to re.compile. Once the actual video URL is found, returning a simple Redirect will result in the client being redirected to the actual video file.
Once your happy with our implementation and you want to try it out, it’s very easy! You need to manually construct a URL Service lookup URL and well, hit it. The Channel documentation covers this here. You basically take the following PMS URL and add an encoded version of the URL that you want to test. The easiest way to encode your test URL is to use some quick and easy online tool, like this one:
So, that’s basically it! We’ve managed to provide support for Plex to translate a URL from a specific site and convert it into the associated metadata and media content, allowing for a number of cool features. However, there is still more magic to explain… 
Some of you might be thinking that the test URLs defined within the plug-in might expire. This happens frequently when content is only available for a limited amount of time. Therefore, tests can begin failing but are actually because the Test URL is wrong, rather than the service. Luckily, the framework also provides a mechanism to programmatically define the Test URLs via a function, rather than staticly defined. Here’s a little example of how to do this…
def TestURLs():
test_urls = []
page = HTML.ElementFromURL('http://www.euronews.net/')
for link in page.xpath("//span[@class='vid']/.."):
if len(test_urls) < 3:
url = link.get('href')
url = "http://www.euronews.net" + url
if url not in test_urls:
test_urls.append(url)
else:
break
return test_urls
All this is really doing is obtaining the main page for the site, and using XPath to find a view recent videos to be used. If you want to double check that this function has worked correctly, the easiest thing to do is to hit the Test URLs PMS URL to see what you plug-in has returned. You can either obtain all test URLs associated with all channels, or a specific one using the following PMS URLs:
- http://localhost:32400/:/plugins/com.plexapp.system/serviceTestURLs
- http://localhost:32400/:/plugins/com.plexapp.system/serviceTestURLs/com.plexapp.plugins.blip
Ok, so creating test URLs might not be the sexiest thing to do, but there’s one more thing! A while back Plex introduced myPlex along with a Queue of media, which is added to via the bookmarklet. Well, the bookmarklet basically loads some javascript which captures the current URL from the browser and sends it to the myPlex servers. Once the servers have that URL, how do you think it converts it to the metadata and media you access in the clients… yup! URL Services! Now there’s only one small change to support myPlex. Instead of Plex needing to look through all Channels available from the store, the URL Services are moved into a centralized Services.bundle. This code is available via GitHub here. It’s as simple as copying the files that you’ve already written and moving them into the appropriate sub-directories. Fork the repo and give it a go!
A Beginners Guide to v2.1
For fun, I thought it might be quite useful to run through a very simple example of a new v2.1 plug-in. For those of you who are familiar with plug-ins, this will probably be fairly basic but for those keen to learn and hopefully start writing your own plug-ins, it should be a nice little walkthough and explanation…
The simplest type of plug-in is one which is constructed based upon an RSS feed. RSS is a family of web feed formats used to publish frequently updated works – such as blog entries, news headlines, audio, and video – in a standardized format. An RSS document (aka feed) includes full or summarized text, plus metadata such as publishing dates and authorship. Plug-ins which are created using these RSS feeds generally present a list of the feed entries to the user, allowing them to select the one of interest and therefore play the associated media. There are a variety of examples available on GitHub but for this example, we will consider Ask A Ninja.
Ask A Ninja is a somewhat “strange” site but provides access to a number of short clips where a Ninja attempts to answer some of the world’s questions. The videos are actually hosted by a popular video site called blip.tv. This makes the process of creating a plug-in even simpler for us, since a URL Service for blip.tv already exists. If you are not familiar with a URL Service, it is the code responsible for translating a webpage’s url into the actual video url and associated metadata. We’re going to come back to this in a future blog.
Right, lets make a start, starting with the very basics. A plug-in is essentially a folder which contains a number of source files and resources. The folder name is commonly the name of the desired plug-in with an extension of “.bundle”. If you’re using OSX, and you want to take a look in a bundle, simply right click on the file and select “Show Package Contents”. An example folder contents is as follows:
We’re going to skip over most of the plug-ins configuration and resource files, but great documentation can be found here. For now, lets consider the actual source code, which is contained within the “__init__.py” file.
To begin with, a Plug-in must provide an entry point which gives the Plex client information about the plug-in. Lets jump in and start looking at some code…
TITLE = 'Ask A Ninja'
RSS_FEED = 'http://askaninja.blip.tv/rss'
NS = {'blip':'http://blip.tv/dtd/blip/1.0',
'media':'http://search.yahoo.com/mrss/'}
ART = 'art-default.jpg'
ICON = 'icon-default.png'
ICON_SEARCH = 'icon-search.png'
#####################################################################
# This (optional) function is initially called by the PMS framework to
# initialize the plug-in. This includes setting up the Plug-in static
# instance along with the displayed artwork.
def Start():
# Initialize the plug-in
Plugin.AddViewGroup("Details", viewMode="InfoList", mediaType="items")
Plugin.AddViewGroup("List", viewMode="List", mediaType="items")
# Setup the default attributes for the ObjectContainer
ObjectContainer.title1 = TITLE
ObjectContainer.view_group = 'List'
ObjectContainer.art = R(ART)
# Setup the default attributes for the other objects
DirectoryObject.thumb = R(ICON)
DirectoryObject.art = R(ART)
VideoClipObject.thumb = R(ICON)
VideoClipObject.art = R(ART)
#####################################################################
@handler('/video/askaninja', TITLE)
def MainMenu():
oc = ObjectContainer()
return oc
As the above code suggests, Plug-ins can optionally define a “Start” method. This can be used for initializing the plug-in and telling the framework information about itself, for example it’s default icon and artwork for constructed ObjectContainers, etc.
Let’s break this down a little further:
# Initialize the plug-in
Plugin.AddViewGroup("Details", viewMode="InfoList", mediaType="items")
Plugin.AddViewGroup("List", viewMode="List", mediaType="items")
These lines are configuring the type of ViewTypes that the plug-in might want to use. I’m sure you’ve all noticed that within the OSX client of Plex it’s possible to change the type of view being displayed at any time. However, for the iOS client (and probably the Andriod one too), this is not configurable. Therefore, the developer needs to decide which view types make sense for each Container object returned by the Plug-in. This plug-in defines a single group, called “List” which actually maps to the Plex View Type of “List”. You should note that you can add mulitple ones of these for different types.
The next section is configuring the default values for both the ObjectContainer, DirectoryObject and VideoClipObjects. You’ll note that when assigning the attributes associated with the icon and art work, the string is wrapped with an “R(X)”. This tells Plex that it’s actually a name of a file contained within the Plug-in’s Resource directory. It would also work if this was a URL to an online resource.
# Setup the default attributes for the ObjectContainer ObjectContainer.title1 = TITLE ObjectContainer.view_group = 'List' ObjectContainer.art = R(ART) # Setup the default attributes for the other objects DirectoryObject.thumb = R(ICON) DirectoryObject.art = R(ART) VideoClipObject.thumb = R(ICON) VideoClipObject.art = R(ART)
Some of you familiar with older versions of the framework might not have come across the following syntax yet, but this is something brand new in v2.1!
@handler('/video/askaninja', TITLE)
def MainMenu():
oc = ObjectContainer()
return oc
This declaration is actually performing a number of tasks. It is firstly defining a “Prefix”. The “Prefix” is basically a unique identifier for the plug-in which not only defines its identifier, but also it’s type. It’s important to note that this is of the format “/video/*” meaning that it is a Video plug-in. For Music plug-ins, this would be “/music/*” and for Photo plug-ins, this would be “/photos/*”. The general convention would be to just use the name of the actual plug-in, i.e. skygo or spotify, etc. This attribute accepts a number of optional parameters (art=…, thumb=…) for specifying the resources to be used in the Channel selection menu. By default, these will be “art-default.png” and “icon-default.png” but anything can be specified manually. The last important aspect of this code is that it is marking a method to be called when the user selects this particular plugin (which is displayed using the defined TITLE). If the user starts this plug-in, this MainMenu function will be called and is therefore responsible for returning an ObjectContainer which contains a number of Objects. As I mentioned earlier, this plug-in is going to iterate over the items contained within the associated RSS Feed and display these as potential videos to play. Let’s just jump straight in again…
@handler('/video/askaninja', TITLE)
def MainMenu():
oc = ObjectContainer()
for video in XML.ElementFromURL(RSS_FEED).xpath('//item'):
url = video.xpath('./link')[0].text
title = video.xpath('./title')[0].text
date = video.xpath('./pubDate')[0].text
date = Datetime.ParseDate(date)
summary = video.xpath('./blip:puredescription', namespaces=NS)[0].text
thumb = video.xpath('./media:thumbnail', namespaces=NS)[0].get('url')
if thumb[0:4] != 'http': thumb = 'http://a.images.blip.tv' + thumb
duration_text = video.xpath('./blip:runtime', namespaces=NS)[0].text
duration = int(duration_text) * 1000
oc.add(VideoClipObject(
url = url,
title = title,
summary = summary,
thumb = Callback(Thumb, url=thumb),
duration = duration,
originally_available_at = date))
return oc
#####################################################################
def Thumb(url):
try:
data = HTTP.Request(url, cacheTime = CACHE_1MONTH).content
return DataObject(data, 'image/jpeg')
except:
return Redirect(R(ICON))
Okay, so what’s happening here then? We’re going to have to process each item located in the RSS feed, which is being performed by the following for loop:
for video in XML.ElementFromURL(RSS_FEED).xpath('//item'):
# Do Something
This is making use of the XML helper functions provided by Plex. This will make a HTTP request to the given URL and construct an object representation of the XML document in memory. It is then using XPATH to query the document to obtain all instances where we find a node with the name “item”. If you’re a little unclear with this, it’s best to navigate to the RSS URL and have a look at it’s source. You should be able to see the different items that we are iterating over…
def Thumb(url):
try:
data = HTTP.Request(url, cacheTime = CACHE_1MONTH).content
return DataObject(data, 'image/jpeg')
except:
return Redirect(R(ICON))
The second bit of magic that some of you might be wondering is how is Plex able to extract the actual video from just the actual web pages URL. This is done by utilizing another important addition to the v2.1 framework, called a URL Service. The formal documentation for this can be found here, but i’m going to re-visit shortly in another blog… If you note, the url attribute of the VideoClipObject has been assigned with the URL of the webpage:
oc.add(VideoClipObject( url = url, title = title, summary = summary, thumb = Function(Thumb, url=thumb), duration = duration, originally_available_at = date))
By doing this, Plex will automatically locate and call the URL Service’s MediaObjectForURL function. This includes the logic for extracting this video and redirecting the client to the appropriate video file. Now, isn’t that Magic! For those of you who are observant and have looked at the latest code in GitHub, you might be wondering why I also left out references to the Search Service. Well, we’ll be back to go into this in a lot more detail…
A GitHub Guide To Fixing A Plug-in…
There’s that plug-in that you’d love if only it added that certain show, or maybe your favourite plug-in has just died because of recent website changes and the original developer has gone AWOL, what do you do? You might be happy having a play with the code, but how do you actually maintain your changes and submit them to the store… Well, in this post, we’ll look at some really basic setups and workflow on how to push your fixes/improvements to Plex.
In the development of software, it’s incredibly important to track and maintain all versions of your code. This can be used for simply reverting back to a previous version of the application or even attempting to determine when a specific bug got introduced. This type of system is known as a Version Control System. There are many alternatives out there, each have their own advantages and disadvantages. Git is an example of a distributed Version Control System which is currently used by Plex. Since it’s classed as a “distributed” system, the underlying data can be hosted in various locations, i.e. locally on your computer or even online by a 3rd party server. A popular (free) public repository is available from GitHub and if you browse around, you’ll be able to find all currently available Plex Plug-ins. I would worry too much how this system works under the hood, for now we’ll simply walk through a number of scenarios on how to set it up and how to commit changes.
There are numerous different 3rd party GUI applications that provide a front-end for Git. However, since we’re going to be following some fairly simple workflows, we’ll just focus on using the command line. After you’ve gained confidence in this, you can then checkout the applications available and find one that best suites your own needs.
GitHub Setup
Okay, so the first thing that any budding plug-in developer needs is a GitHub account. It’s quick and simple to sign up just by hitting this URL and selecting the “Free Account”. After you’ve run through the process of creating an account, you’ll need to setup your main development machine. There are walkthroughs available for all operating systems:
Forking a Repository
Now that your machine is all ready, it’s time to create your own copy of the plug-in. The terminology used is to “fork” the plug-in’s repository. This is essentially copying the entire plug-in for your own work. You can then be free to make any changes you desire, before pushing these back to the original repository. Luckily GitHub has also provided us a nice walkthrough on this, including setting up your local copy, which can be found here. Remember, all official plug-ins that you might want to fork are kept here.
Work in Progress
What you might not know is that Plex Media Server will continually replace the plug-in bundles stored on your machine. This is to ensure that users always have the most up to date version as available in the store. While being incredibly helpful to most users, it can be a bit of a pain for plug-in developers since if you simply modify the working copy, it will be periodically reverted. There are basically a couple of workarounds for this problem. The first is to simply keep the bundle separate (i.e. on your desktop) and copy into the Plug-ins folder when ready to test. Another is to set the PlexPluginDevMode parameter to “1″ in the plug-ins plist so that Plex ignores this particular instance. The final option (if your running OSX or Linux) is to create a symbolic link from the Plug-ins folder to your local git repository. A nice simple tool for doing this can be found here. Once you’ve finished with your symbolic link, simply delete it and Plex Media Server will eventually replace the now missing plug-in with that currently available from the store (hopefully your new and improved one!).
Please remember, always change your local repository of the plug-in, and NOT the one downloaded from the store!
Adding/Removing/Committing Changes
So, you’ve modified a few lines, maybe added a new file and even removed another. After testing your changes to make sure that they are now ready to be committed to the repository we must first double check that the changed files are as expected. The following command line is a quick and simple way to get details of the current state of your repository:
git status
If you’re happy with the changed files, here are a few command lines to help you mark them as such:
git add path/to/file git rm path/to/file
You’ll notice that the add command is actually appropriate for either new files or simply modified files. Once you’ve marked all your files appropriately, it’s now time to commit them to your local repository. This can be done with the following:
git commit -m "This is a commit message"
Now that you’ve done this once, you can repeat this cycle as many times as you need. It’s best to keep commits relatively small and focussed on a particular issue/area. This makes the repositories commit history much more readable as how the plug-in progressed development. After each commit, it’s perfectly fine to update GitHub (see next section) or not. It’s entirely up to you as to what works best for you but think of it this way. If it’s been pushed to GitHub, it’s safe from any type of failure, e.g. machine spontaneously blowing up
Updating GitHub
Your local repository now has all the commits required and you’d like to update the repository stored on GitHub. This is particularly simple:
git push origin master
Bingo, all done. Now check on the GitHub website and you should see that your forked repository now shows all the commits that you’ve made locally. If this is the development complete, then there is only one thing left to do, send a Pull Request so that the Plex guys can pick up your changes…
Creating a Pull Request
A Pull Request is exactly what it sounds like. It collects all the commits that you’ve made since you forked the original repository and asks the owners of the original (i.e. Plex) if they would like to pull your changes too. The guys over at GitHub have been incredibly helpful and even produced a nice step by step guide on how to do this. It’s worth spending a little bit of time on the request’s title and details. It’s possible that following a quick review, the Plex guys might ask for a few changes which will be done via the Pull Request’s comments field. If you make any further changes to your local repository, and update the repository in GitHub, the Pull Request will be automatically updated to now contain your new commits.
[Advanced] Rebasing
Ok, so this is a slightly more advanced topic that I don’t want to go into too much detail, but what happens if someone else modifies the same Plex repository as you. E.g. you forked a repo a week ago, then yesterday another user had a Pull Request accepted which has now led to modifications of the original. They might be in a slightly different area to yours, but you still want to pull these new changes back over. If you’ve followed the instructions linked to above for when Forking a Repository, the “original” repository will be known as “upstream”. However, if you omitted this, you can easily add a remote repo using the following command:
git remote add upstream git://github.com/...git
Now you need to pull the recent changes from the original repository and rebase your local copy. This is done as follows:
git fetch upstream git rebase upstream/master
This will essentially replay the new commits sequentially to your local instance. It’s possible that you may have made some conflicting changes and therefore Git is unable to continue the process. Thankfully, it will notify you of the file it is struggling with and allow you to have a look at it, decide what the correct combination of commits should look like, and then continue.
Right, it’s that simple! Please don’t get put off by this. Once you’ve done it a couple of times, you’ll start nailing those command lines without even thinking about it. Also, if you’re unsure at all about any of this, just post a query here or jump into the Plex Campfire Chat Room and shout for a Plug-in Developer.
Enjoy!
1 comment