focused data-mining

[cellio]

There is a service that I wish someone -- Amazon is an obvious candidate -- would provide to me, and I wonder how realistic it is. (It seems like it should be easy given the data, and they've surely got the data.)

Here's an example: I've bought three CDs from a particular artist. I really like two of them, and I don't care for the third. She has more CDs. Which (if any) am I likely to like? If I could find ratings for those CDs from people who liked the two I liked and didn't like the one I didn't like, that would be informative. Overall ratings of those products are irrelevant, because I don't know anything the raters. Reviews can be more informative but, again, I don't know how the reviewers' tastes match my own and most reviews are not well-written. User-submitted "if you like this buy that" links can be helpful but are under-used; they're also not annotated. Knowing that people also bought them isn't helpful; I bought the one I didn't like, too. I want to tap into the actual ratings (not purchase history) from a subset of raters -- the ones who match my own profile. Ratings are more reliable than the other options available because Amazon uses them to push things at you, so you're motivated to get them right.

I'm casting this in terms of Amazon because they're doing part of the problem already, but in theory this could be done by anyone who can mine the data.

Is anyone providing a service like this (with a large-enough user base to be relevant)? Does any Amazonian reading this want to run with it? :-)

Comments

[geekosaur]

Last.FM, supposedly, but you need to drink their koolaid.

[filkerdave.livejournal.com]

Who's the artist, BTW?

[cellio]

Thanks for the pointer. Looks like I need to visit them when their site is less busy. :-)

(In an ideal universe I'd also be able to perform the same kind of analysis with books, or movies, or anything that can garner sufficient ratings.)

[cellio]

Neshama Carlebach. I very much like "Journey" (her newest CD) and "Soul" (fairly old, and a lot of her father's songs). I recently bought "Ani Shelach", which I didn't care for overall (though it's got some good tracks), and am wondering whether I would like the other CD Amazon lists for her. (I thought there was a fifth, but I don't see it now.) There are no sound clips for any except "Journey".

[xthlcm.livejournal.com]

Hmm. Well, there are a number of group-rating systems out there, but none I know of that will actually recommend particular CDs. I've become a huge Pandora junkie lately, although because I'm picky my "custom radio stations" tend to have only a small set of songs in rotation. :p

Sadly, I don't work at Amazon anymore.

[ralphmelton.livejournal.com]

I have such an artist of my own. (Nine Inch Nails. The first album was well-nigh perfect for me; the later albums have not pleased me nearly so much. I would love to find more albums like the first one.)

[dmnsqrl.livejournal.com]

Launchcast? Mind, reccomendations tend to be more on a song-by-song basis than album-by-album... but the thing is that you rate songs, artists, albums, and they make reccomendations based on things liked by people who had similar sets of ratings

I can't find it

[(Anonymous)]

On my blog (quite possibly before Christmas, back before the latest reboot (sigh), there was talk of a computer system that analyzed music based on computer analysis. For once, no humans were involved. The idea is that, being able to analyze large numbers of songs, the computer might find good stuff that's being overlooked.

As I remember, though, it wasn't ready for prime time. I also remember something about it using neural nets to do the analysis. That's actually problematic. Neural nets can find connections between things that humans don't care about. For example, a data set comparing images with and without tanks worked fine in training, but presented with new images not in either data set, the system couldn't determine if there was a tank in the picture. It seems that all the non-tank images were taken on sunny days and all the tank images on cloudy days. The computer completely missed the tanks and paid attention to sunny vs. non-sunny. Oops.

Rob of UnSpace (http://www.unspace.net/)

PS: Jupiter has a new red spot! (http://www.scienceblog.com/cms/jupiters_got_a_new_red_spot_10142.html) Wheee! Ok, so I get excited over really stupid stuff....

[cellio]

Thanks for the pointer to Pandora. It looks like it could be useful.

[cellio]

Sometimes first albums are strong because the artist has been saving up prime material for years, with the second album representing the delta -- which might be shorter than the time to the first album. And sometimes, instead, later albums are stronger than first offerings becuase the artist is getting better both at making music and at recording it (a very different set of skills from just performing).

In this case, oddly, the earliest and latest albums I own are strong, and one in the middle is weak. So I now look with some suspicion on the other "middle" album that's out there.

[cellio]

Thanks; I'll check it out. Ratings of individual songs can be rolled up into ratings of ablums, at least.

I'm hoping to find a fairly narrow profile. Amazon, for instance, will already make recommendations based on my entire rating/buying history, but I don't think there's necessarily any correlation between SF books, DVD sets of certain TV shows, and Jewish music, for instance. But, near as I can tell, with Amazon it's all or nothing; I can look at the recommendations based on the hundreds of varied things I've rated or not, but I can't limit it to particular domains. Launchcast, being specifically about music, at least helps narrow the scope somewhat; I can narrow it more by being careful about my profile (using multiple profiles if I want to collect what I think are unlrelated recommendations).

Re: I can't find it

[cellio]

I remember that post. I, too, had the impression that this was early research, but hey, maybe in several years ordinary folks like us will be able to make use of it.

Re: I can't find it

[(Anonymous)]

Well, I suspect that this might have been a comment from me (Gclectic), posting about Music Magic Mixer, which is still one of my favorite programs for generating playlists from the songs I already own. They also do provide services for recommending new albums based upon similarity to the acoustic characteristics of any album you specify, matching against the databases of either Amazon or , though this feature is indeed a little bit less reliable than their mainline playlist-generation feature.

For my full, in-depth, gushing review, you can check out this post (http://gclectic.typepad.com/gclectic/2005/08/my_favorite_sof.html) on my blog.

Re: I can't find it

[cellio]

Thanks for the pointer! This might get me to actually sit down and import a bunch of my music. (I'm still mostly in the world of physical media, I'm afraid.)

[gclectic.livejournal.com]

I tried writing a small essay explaining all of the reasons why this sort of collaborative filtering problem is hard in general, and got bogged down. Suffice it to say that (a) you are right that this would be really nifty; and (b) there are a lot of tricky details in creating an appropriate dataset for such a query.

What I find most interesting about your suggestion is the desire to take a broad user profile and emphasize a selected small portion of it in order to produce a query. This is, indeed, an idea that seems incredibly obvious and useful -- and is entirely new to me. Maybe someone will pick up on it and run with it.