Easier openelec testing - boot from a USB key while leaving internal version intact

Ok so something very useful about openelec is that it will happily run off a small cheap USB key - 1GB is enough.

This means that you can have a nicely set up main box with an SSD/HD for general 'wife friendly' use, and periodically just whack a new testing version on to your USB stick and boot that up to see how it's all going.

However, if you actually try this, you will find the openelec won't boot from the USB stick - it will in fact start to boot from the usb stick but then partially boot from the internal HD and fall in a big heap.  If you unplug your internal hard drive it will boot fine off the USB stick.  But that's a pain.

Some quick instructions to fix this:

(I did this in my linux VM running on my windows server - it's proving very handy!)

Plug your USB stick into your linux box and run:
df -h
to determine what device your USB stick is - we're looking for two partitions, /System and /Storage.  Usually these will be /dev/sdb1 /dev/sdb2 but it will depend on your system.

Then, you need to change those partition labels as root:

sudo e2label /dev/sdb1 System2

sudo e2label /dev/sdb2 Storage2

Now as root, edit the boot file in the system partition

sudo nano /media/System/extLinux.conf

In there change the reference to System (LABEL=System) to System2 and Label=Storage to refer to Storage2.

Now, on your openelec box set it to boot from the USB stick first, and your internal HD second.  This means if you have the stick plugged in it will boot your testing version, otherwise it will just boot your proper set up as normal.

Wa la, easy testing!

OzWeather - my first XBMC Add On!

I've just coded my first XBMC add on!

Was quite tough as I tried to simultaneously learn git, xbmc add ons, and python at the same time ;) - and of course none of these things have anything resembling good documentation (well, other than git).

I'd wanted better weather data for a while, and every Aussie knows the only decent weather data here is from the BOM.  Unfortunately, their API is shite (basically they don't really have one and what they do have is very poorly documented).  So in the end I wrote is as a scraper of the site weatherzone.com.au which uses BOM data anyway.

Anyway, with the recent change to add on based XBMC weather, it was a good moment to de-construct ronie's official wunderground add on and  write my own based on that.  I used this awesome tool from the collective to do the scraping - was super easy. 

Unfortunately I still can't get my XBMC repo to work so it's install from .zip for the moment and no auto updates till I nut that out or some kind soul takes pity on me and shows my my stupid mistake.

You can find the release thread with installation instructions here - http://forum.xbmc.org/showthread.php?t=116905

 My eventual plan is to spend some time adding some Squeezebox like features to XBMC music playing which is pretty crude so far.

Bringing Live TV into XBMC (using a Sony Play TV and TVHeadEnd in a virtual machine!)

Introduction
 
Ok, so this was a little tricky.

But totally worth it - the results are really very good indeed,and  even with all sorts of beta/development version involved, it's all looking good and fairly stable.

For a long time, I have wanted to get rid of our cable TV package (Foxtel).  We barely ever use it and it costs us $60+ a month I think - it's kind of like a gym membership, this vampire like thign that sucks you dry while you're not really watching.

I finally convinced my wife that the only thing we really watch on Cable these days is kids stuff - notably, ABC4Kids which is a free to air digital channel featuring excellent pre-school programming.  Cable also delivers to us the over-the-air free channels, so I've never actually just used the tuner side of my TV until now.

(As an aside - the ABC here in Australia is a totally awesome, progressive, and tech friendly organisation with some fantastic stuff, like iView, and their iApps etc. - this is how a TV statuon should be, if you ask me.  They actually treat their viewers with some respect, offer interesting programming, and don't make it hard to consume or have 17+ minutes of add per hour like the commercial networks!  There are XBMC plugins for the ABC streams, so it's really easy to use inside of XBMC too).

So here was the goal - an digital TV server running on our main home server machine (which runs Windows 7 Ultimate and is regularly used for other work as well, so it needs to be Windows).  Feeding the TV signal over Ethernet to gorgeous XBMC clients - so one media centre to rule them all!

*** Hardware Required

We still do want at least some access to Live TV, so the first thing I did was buy a TV tuner for the PC as my plan was to have this on the PC - and there was no aerial port in the TV room as it happens, so I though I'd start there and then get an aerial port put in the TV room.

My research led me to the Sony PlayTV - this is inexpensive, USB based, and a neat little box that requires only one aerial input for two tuners, so it's tidier.  You can get there cheaper if you only want one tuner, but I'd stick to USB based tuners and check carefully about Windows and Linux support for your chosen device before you embark on all this, as good drivers are essential.  PCI devices don't work well with virtual machines, or at least as well as USB ones, so USB is the way to go even though it means an external box.

The Sony Play TV was $85 from JB Hi-Fi, you can get them even cheaper off eBay.  Not bad for a two tuner unit, really.  It's USB powered, so it just needs one USB connection to the PC, and a single standard aerial jack plugged in works for both tuners.  Very simple, and uses bugger all power.  It doesn't get hot, and this unit is very well shielded against RF interference unlike some of the little stick ones.

So, I bought this, and though I'd try it at work.  No real luck here, couldn't get a signal really.  I wasn't too bothered, as I have never really used TV at my workplace so I don't know about the state of the aerial.  I brought it home and tried our aerial jacks - no go their either.  Oh dear.  I decided it might be the device, so I tried with a  rabbit ears and was able to get at least some response.  So I realised both work and home have crappy aerials.

I had an aerial guy come in and put in a new fancy pants digital aerial (turns out the old one was  a bizarre home made frankenaerial - all sorts of weird plastic and just a heap of crap basically - no wonder there was no signal, and God knows what the previous owners were watching?!).  The cost was $350 all up for a new top of the line aerial and install of two aerial points, which in our old house was not simple, so I think these guys (New Image Antennas) did a great job (thanks Eric!).  And on the very same day I rang them, no less.

At this point, I finally had a working aerial port both at my server and next to my TV, and the TV itself was getting excellent digital reception - very occasional minor glitches but far better than most digital installs I have seen.

*** Software
Moving beyond just Digital TV to the TV - Hello XBMC based PVR

Next test was to try Windows MCE to confirm the PlayTV was fully working at least under Windows. 

Ok - about drivers - this was a bit hard.

Under Windows7 - to get the PlayTV working reliably - you need to install these drivers and this technique - http://www.avforums.com/forums/11785322-post4.html  - do not use the hacked pinnacle drivers, these don't change channels properly on Win7!  Note be careful when extracting this .exe (use 7 zip) - I think there are two .inf files in there and one doesn't work (not I am not talking about the HID inf, ignore that, but two .inf files for the actual elgato device itself).  If you get an error saying 'these drivers are not for your device' then re-extract the archive manually and try the other .inf) - it was a bit wierd, I could not get this to work on my laptop but iot went straight on the server, and the onyl differnce was how I extracted the archive, so be warned!).

Once the Elgato drivers were in place, Windows MCE instantly tuned all 20+ channels and all tests were perfect in terms of rapid channel changing etc.  Maybe 2-3 seconds to change channels, could record one and watch another etc.  All good.

However, MCE doesn't help me much - I want to be able to serve the TV up to my XBMC clients.  My reading suggest the easiest approach, and the one the current XBMC PVR main developer is using, is to use the linux only TvHeadEnd package to serve up the TV feed over ethernet to my XBMC clients.  So now I wanted to get this working under linux - either running direct on one of my rather feeble XBMC clients (all Atom/ION type machines), or ideally on my main server - which is Windows and needs to be for many reasons - I don't have any Linux server.

Alternatives are other Windows based TV servers, like MediaPortal but everything suggested TvHeadEnd was very much the way to go.  And I have never like MediaPortal for some reason....the one time I tried it I found it very clunky and fiddly, but that was years ago.


Enter Linux - Ubuntu Oncelot Virtual Machine in WMWare Workstation

So it seemed my best option was a Linux virtual machine running on my Windows server.  Not only would this potentially give me a proper TV feed, but I could potentially start using linux for other servive tasks, like Calibre book serving, etc - a lot of these things run easier and better on Linux than Windows.  But first things first, let's try TV....

My first tests were done under VirtualBox which I have used before and is very easy.  Slightly slicker to use than VMWare Player in my experience.  However, I installed the device and TvHeadEnd on this ang got it running and the USB device was recognised and working - but no matter what, I could not get TvHeadEnd to tune.  I saw other eports of TvHeadEnd failing under VirtualBox.

(TvHeadEnd install instructions are here - http://forum.xbmc.org/showthread.php?t=92315 - but I found that script doesn't do TvHeadEnd, just the depends really, so I manually installed it, see below).

I almost gave up at this point (the one paragraph above sounds like not much hassle but in reality it was a LOT of mucking about and trying for zero result, so it was a bit dis-motivating - after several days of mucking about with the aerial ports and then VBox, I'd got basically nowhere).

I decided to throw jsut a bit more time at it and couldn't find any reference to VMWare Player and TVHeadEnd but decided to try that as well - thankfully, this was a lot easier really!

In the end, I am actually using VMWare Workstation as it has more options, but you can test you install under the basic, free Player first, and if it works for you, consider the $195 cost of Workstation if you want the extra features...

So, I set up a pretty standard Ubuntu install under VMWare Player.  30GM hard drive, 1GB Ram, USB passthrough of the SonyPlayTV etc.  Running in Bridge networking mode so that the VM gets its own IP, I couldn't get the server visible externally using the basic NAT mode for some reason.  I actually like it having it's own IP as it means I can set up testing servers of things I already have and easily switch between them by changing one IP digit, rather than port numbers.

I installed TvHeadEnd using the pulse eight .deb Packages available here - note I used the lucid packages as there's no specific Oncelot version (so tvheadend_2.12.99~pulse-2_i386.deb in the lucid folder)....

....just downloaded that and double clicked it I think, and ubuntu went ahead and installed it.  (I tried other ways and versions of tvheadend but this seems to be an easy and reliable way as opdenkamp the XBMC dev (aka dushmaniac on the forums) is clearly working with these guys and so it appears this is a relatively stable source of the latest PVR code both at the server and client ends). 

Initially this seemed to work ok, but damn it still no actual service results in TvHeadEnd.  Both my tuners came up as available, and I could at last see Muxes (I used the 'Add Mux by location' button or whatever it is called to add the Melbourne muxes).  This got me Muxes with good signal quality, but grrr still no services.  Grrr.

I checked dmesg and saw there was some errors associated with the PlayTV device...much googling let me to this problem and solution:

Working PlayTV Under Ubuntu Oncelot (11.10) - the latest linux kernel has broken some usb-dvb support.  So I had to rebuild the drivers per these bug notes -
(https://bugs.launchpad.net/ubuntu/+s...ux/+bug/838130)
..so it appears it will be properly patched in later kernels but an easy fix is to follow this method to manually re-build the drivers
https://bugs.launchpad.net/ubuntu/+s...30/comments/17

Compiled and installed per those instructions, rebooted the VM...and wa la - instant awesomeness in TvHeadEnd.  Both tuners found, and all the expected services.  I then used the auto 'map services to channels' button and I had 20+ channels ready to go.  Note, make sure you do both tuners the same, in terms of mapping channels etc. or you might get wierdness at the client end if two clients are tuned to different channels.

(Incidentally, if have lots of clients, there's no reason you couldn't add another PlayTV for another two tuners, or indeed multiples for many many tuners - TVHeadEnd is happy to use many tuners).

One other thing I did was give all my channels a logo URL for the TV station logos (I used the Harmony One icons from here).  Looks much prettier at the client end.

I also added a user (xbmc/xbmc) in TvHeadEnd as without it you can't access the admin panel (at :9981) or the TCP TV feed itself (:9982)

The Client Ends, & How It Works

All this is part of preparing to move to Eden for me so I have been trying Eden builds with the clients.  This is where they can be found:
 
Linux Client - http://packages.pulse-eight.net/OpenELEC/stable/
Windows Client - http://www.scintilla.utwente.nl/~marcelg/xbmc/prebuild.html

Install one of those, go into add ons and enable and configure the TvHeadEnd Client Add on.  And then go into XBMC main settings and set up the Live TV settings.  Now back to the main menu and click on Live TV and with any luck you'll see a nice channel map screen...

First test was using the Windows.  And WOW!  it works.  And it works so well!

Instant connection to the TvHeadEnd server.  Live TV @1080 streaming (over wireless 11g!! no less) - to my laptop.  And the PVR stuff in XBMC is *slick*.  Way, way better (and more stable) than I expected.  Very nice.  The build is not super current so there are some noticeable Eden bugs but overall it's remarkably good...and I am sure more builds will follow as Eden heads towards stable release.  It's easily good enough for testing though.

Next I installed the Openelec version on to my ASRock Ion 330 box (from one usb key to another, not yet willing to over-write my very stable Openelec 1.0.2 Dharma install on that machine).

Here's what you get.....

Basic viewing & Channel changes - takes maybe 2-3 seconds - about the same speed as my 50" pana plasma really, maybe marginally slower, but way quicker than most TV tuner PC things I have seen.  All channels work well, in a window or fullscreen, and changing is very easy using up/down channel, or popping up the mini window (I will need to do some dedicated remote mapping for this section!).  There's seems to be no speed or quality trade off streaming the TV over Ethernet at all.  It works so well it really shocked me.

Pause TV works?! - which I was not expecting as I kept reading TvHeadEnd does not support time shifting.  So I guess this is some clever XBMC/instant record implementation....I have not tested thus for long periods yet but pausing for a few seconds certainly works just fine...

Recording & Scheduling works, and is all done back at the server end - you can trigger the record from the client end or the server end, but I mean the actual recording is done on the server machine so visible to all the clients and you don't need to have big hard drives in your clients.


EPG works well - It picked up my EPG from the digital TV feed (no need for xml grabbers thankfully) and this info is all in place at both the client and server ends - although once I have seen it get lost on one of the clients, I just went into the settings and dumped the EPG and it got it all back..it probably happened because I tweaked the channels on the server end.


Summary & Results

It's all amazingly good.

I really had no idea how well this would all work.  Honestly, apart from the alpha nature of xbmc-pvr and Eden itself, I had real concerns about the virtual machine offering enough performance.  But in reality, the virtual machine approach is very neat and doesn't seem to place undue load on our home server (a fairly beefy Core i7 with 16GB machine offering about 20TB of total storage).  It very happily streams at least two channels to two client at 1080 simultaneously with no issues at all.  XBMC-PVR startup initially takes maybe 5 seconds after XBMC itself starts, as it re-loads channels and EPG at the beginning, and after that it's really just like a super super slick PVR fully inegrated into XBMC.

Recordings I have re-directed from the Virtual Machine virtual HD to a shared folder on my main server, so these are accessible both directly through the PVR interface and more manually via the 'video files' view in XBMC.  This means the tiny virtual HD won't fill up.


I have snapshotted the virtual machine in this working state and can now experiment with moving some of my other server stuff acorss, like SABnzbd, sickbeard etc, to isolate this virtual machine into a very easily re-installable thing if I upgrade the home server, or a friend wants a pre-made copy of all this stuff etc.  And basically, the snapshots form a very easy backup - I will probably even copy the entire VM over the net to my work server so it's completely backed up even if the computer dies/gets stolen or whatever.

Nothing techy has impressed me so much since XBMC itself.  Most of the difficulties were from my own unfamiliarity with Linux and virtual machines, although obviously this isn't for the faint of heart in all.  It does take a fair bit of work to get going like this.  If you has a Linux server in place already, it would doubtless be much quicker....

I will say working with the UI in Oncelot is truly truly hideous (Unity, they call it - under the VM I seem to only have Unity 2D working) - I think I would just whack Lucid on if I could be bothered doing it again as the old Ubuntu interface is a lot nicer.  I presume Unity will be much improved over time although its such a bad base one can hardly see where it can go from here that would help.  But to be honest, I mainly use the terminal when I am working in the VM, and Firefox for downloading things etc, so I won't spend much time in there.

XBMC is very slick, we all know that - and XBMC + PVR really is the ultimate media centre.  All the goodness of regular XBMC (gorgeous skins, great meta info, easy navigation and high WAF factor, extensive customisation, add ons etc etc) - with a very slick and easy PVR interface bolted on.  Just beautiful.  You can use cheap, quiet, simple client machines, with all the load on the server, way off in the inaudible distance.