Friday, January 9, 2009

re:hippies, part 1 revisited

I was just going to post a comment to the comment string here, but instead I would like to revisit/restate my initial query. From my understanding (and I always must disclaim that I don't truly understand a lot of this, but I am trying to work through it), current applications use crowd-sourcing or some type of large data set to analyze the tastes of people and use that information to suggest "what" else might be purchased, listened to, watched, etc. Lots of consumer-oriented spaces do this, both online and in the real world.

For example, a program will report that out of 600 people who bought X Grateful Dead album, 500 of them also bought Y Phish album; therefore if I buy X, the program will suggest Y. That is a "what" comparison.

What I am interested is whether a "why" could be established instead of only a "what".

For example, out of 600 songs that I rate as excellent, could a program analyze the metadata (rhythm, speed, tonal range, etc.) of those songs and make a recommendation based on a "why"?

Or, using Melissa's example, could a program analyze my yearly shopping lists and not just recommend similar products (based on other shoppers' patterns), but instead analyze the ingredients, colors, texture, convenience (and other such metadata) and make recommendations based on "why" certain products were bought.

Make sense?

If programs could begin determining patterns of "why"s, it seems that a true AI would be much closer to realization. Not just what people are Googling, but why they are Googling it. Or in my case, not that Matt hates and loves hippies, but why he does so.


To be faced with such a program, more advanced than us (unless you can already figure yourself out), would truly be remarkable, scary, and evolutionarily progressive (none of the adjectives, by the way, that I ascribe to hippies).

shanti,
mjh

1 comment:

Anonymous said...

I think you hit on it a little at the end. Can you answer the question about why you like a certain song?

If it's purely about the music, there have been some advancements in the way computers can make suggestions based off that. Check out the Music Genome Project which is used by Pandora.

Of course, one of your favorite songs may be completely outside the style you normally listen to and be a favorite for reasons having nothing to do with the actual music content. Perhaps a song that your grandfather always played, or one that a friend in college loved and had going constantly and reminds you have a great time in your life.

There are a huge number of variables that enter into. Not to mention that the actual number of variables changes. So, a moving target sitting on a moving target.

The flip side of this approach is Amazon. They really don't care about the "why", but they are very good at mining prior sales data and user generated wish lists to come up with stuff you might like. All their recommendations won't be hits, but they don't have to be. As long as a high enough percentage is at least semi interesting, it'll keep you looking. With the cost of a few "wrong" items being displayed being effectively nothing it's easy for them to experiment with tweaking what they show you to see if it makes a difference.

These type of recommendation engines are hugely valuable for the companies that run them. Check out Netflix Prize where Netflix is offering $1,000,000 to whoever can improve their service %10. Follow that up with this article on the Napoleon Dynamite problem that competitors are running into for a little insight into how difficult this stuff is.