Rhapsody Web Services

Server Scalability through URL affinity

This isn't really about webservices, but it is about building cool websites.  Sometimes developers face that classic "good problem" that their site is more popular than their server(s) can handle.  Some of you might have noticed that we ran into this recently issue recently and I'd like to share with you what we did about it. 

First, a little background on our server architecture. www.rhapsody.com points to a load balancer that distributes page requests amongst a set of identical app servers.  The app servers try to serve content out of their local caches and if the content they need isn't there, they look it up in a shared database.  All very standard.

Our particular issues start with the fact that the entire rhapsody catalog is too big to fit into the cache space that our app servers can use.  It takes an order of magnitude or two longer to serve pages out of the database than from one of the local caching layers.  The result has been that popular content which is in cache gets served pretty fast and less popular content gets served much slower.  Adding more app servers to the farm helps to a certain extent, but at some point the database gets maxed out and adding more app servers doesn't help at all or even makes things worse.  We could add more read-only database instances and replicate, but there's a better solution.

Load balancers often are configured to use "session affinity" whereby a particular user/browser-session always gets routed to the same server so that server-side session information is preserved.  In our case, there is very little user-specific information on our pages -- the catalog content is the vast majority of the pages and also the performance bottleneck.  So instead of session affinity we enabled a kind of load balancing that seems obvious, but based on my reading of the literature I don't think is very common.  You could call it "content affinity" or I think a more descriptive term is "URL affinity."  Instead of binding a session to a particular server, we bind URLs to servers.

This means that every time somebody asks for the hoobastank page, it goes to the same server.  So now instead of every server on the farm trying to cache hoobastank's data, we have just one place.  (Actually a couple -- I'm simplifying things a bit.)  Even better, because of our friendly URL mechanism, it's easy for the load balancer to send all hoobastank traffic to the same server -- their artist pages, album pages, etc.  So when we cache the hoobastank object on the app server, we get to use that for all of their pages.  Thus each of our app-server caches is responsible for less content, and gets more efficient for the content it is responsible for.  Moreover, as we add more servers to the farm, the cache efficiency increases.  As our caches get more efficient, database load goes down, as does page load time.  Ahh, scalability.  It's beautiful.

I'm posting this primarily because I'm hoping some of you will find it useful.  I've read a number of books on the subject, and none of the ones I've found talk about this kind of content caching problem.  I'm looking forward to reading Cal Henderon's new book on scalable web sites as I bet he's had to address these issues on flickr.

I'm not posting asking for advice on how we should have done things or what's wrong with this system.  I understand full-well that this isn't ideal for a lot of reasons.  But it works pretty well for what we need right now.  A more sophisticated architecture would do essentially the same thing but with a dedicated data caching tier behind the web application tier.  That gives you the ability to scale multiple types of content, e.g. catalog content and user-specific content.  But we're actually going to move that functionality down to the browser.  Stick around and you'll see what I mean.

-Leo Parker Dirac

RSS vs XSPF

Why does the world need XSPF?  Playlists are linear lists of items.  I think RSS covers this need pretty well, and is already supported by a huge range of apps.  Especially when you consider the custom data fields in RSS 2.0, what is the value of XSPF?  Are these extra metadata so important that we need a whole new XML format?  Can't we just standardize on some extra RSS tags?  What do you think?

Simple.js: A Javascript Interface for Playing Music

I'm happy to share with you a new API for playing music on your site.  We call it simple.js, because that's where you find it:  http://www.rhapsody.com/simple.js.  This API lets your web pages trigger music playback without requiring your visitors to leave your site.  In places where you can include javascript, simple.js will start a track or album playing in the web popup player window without navigating off your page.  Much like a RhapLink, it will re-use an existing web popup player window if it's already open, or open a new one if needed.  Plugin detection/install and login are all handled for you.  As the name implies, a goal of simple.js is to be as simple to use as possible.  Consider this example:

<script src="http://www.rhapsody.com/simple.js" />
<a href="javascript:RhapsodyPlayer.playTrack(2899985);"> Listen to Cake's "Stickshifts and Safetybelts" on Rhapsody.com</a>

Which renders as:

    Listen to Cake's "Stickshifts and Safetybelts" on Rhapsody.com

Pretty straightforward, eh?  The only tricky part is finding the RCID value of the track, in this case 2899985.  We've been working hard to remove obscure database IDs from the site, and they're almost completely gone fom the URLs.  But you can find them in the RSS feeds.  In this case, open http://feeds.rhapsody.com/cake/fashionnugget/tracks.rss and look at the <guid> field -- that's the RCID value you need to pass to simple.js.  We've got some other ways to find RCIDs in the works that you'll hear about soon.

As always, feedback on the API is most welcome.

-Leo Parker Dirac

A Modest Audioscrobbler Proposal

Posted by Dave

I've noticed some discussion over on the LastFM forum about wanting a Rhapsody playback API so a plugin can be built.  Jim posted a solution that I have yet to try out, which uses SQL to pull data from the Rhapsody desktop client's data cache.  Jim's idea seems like it will work for the client, but what about a solution that leverages the Rhapsody listening history feeds we're offering in private beta?  These feeds contain all listening history from both Rhapsody.com and Rhapsody 3.0+ clients, by track, album, and artist.  An Audioscrobbler plugin could simply hit the feed, and forward the track info on to Audioscrobbler.  In fact, it could be set up as a web service that doesn't even require an install. 

What do you think?

Check out my recent tracks feed and see if the information is sufficient.

What are the bloggers listening to?

How often do your friends tell you about some cool new music they discovered?  Do you have any really plugged-in friends that you're always bugging to tell you what music they're into right now?  Wouldn't it be great to just know what your friends are listening to without having to keep asking them? 

I've started playing with a new Rhapsody Web Services feature that lets you do exactly that, and I'll say that it really is great.  Look in the upper left corner of our blog and you can see what music I've been listening to most recently.  It's a list of the last several albums that I heard tracks from.  Click any of the links and the whole album will start playing back for you in either Rhapsody.com or the Rhapsody Jukebox Client.

How does it work?  This blog is loading an RSS feed from Rhapsody.com that hooks up to my listening history.  The feature is currently being beta tested, so you're not going to find it by poking around the site.  But if you'd like to share your listening history with your friends, drop us a note and we'll tell you how to get your own feeds.  You can put the feed in your reader, on your blog, on your personalized homepage or lots of other places.  Let us know how you're using them.

-Leo Parker Dirac

Neat little iTunes apple script for the Mac users

This Applescript allows you to search Rhapsody for any track in your iTunes library.
Click here to see a screenshot.

Go to you home directory -> Library -> iTunes and create a directory called Scripts.
Place the script into the Scripts directory.
Restart iTunes and notice “Scripts” has been added to your menu bar.
Play or highlight a track and then pick “Search In Rhapsody” in the Scripts menu.
Click here for a screenshot of the menu.
A dialogue will give you the choice of searching on the artist, album or track.
Pick one and watch Safari start up and run your search for you.

Works on MacOS 10.3+ and any version iTunes.