Some thoughts on fandom stats "web app" kind of projects

[eve11]

Wanted to write down a notebook here just to remind myself of some things I was interested in doing.

Teaspoon doesn't list who has favorited a story (I should amend: you can bring up an author page and get a list of their favorites but you cannot bring up a story page and get who has favorited that story) but you can get to comments on that story and then find the favorites of the authors of the comments pretty easily. So what I'm thinking of is a "People who liked X also liked" kind of "app". Where you can plug in a story (for example, the latest calufrax rec), and then do a diagram of favorites. It might have a condition like "must have at least 2 different authors favoriting" so that it wouldn't eg, take every favorite of authors like me (I have 300+ favorites on Teaspoon and I comment a lot). It would be nice to have the diagram of the favorites be a kind of cluster thing, where similar stories are closer together... maybe like a wheel with different levels you could spin, surrounding the current story? Line up the different categories (era, word count characters, etc) and the wheel will point to stories that others (who have liked this story) have favorited. Right click on the wheel level to gray it out (ignore). You could have a string of differentely sized circles (size corresponding to similarity or to the number of people who favorited the story) and mouseover text that has the story title and description and a list of who favorited it from the list of commenters on the story you're at.

The data seeding this would be the list of favorites of commenters on the story you're looking at. Could also be augmented by the list of favorites of the author of the story too, and that author's other stories.

I think d3 could be used for this. Just some ideas I'm hurling out there so I don't forget.

Comments

[a-phoenixdragon.livejournal.com]

Ohhh, that would be fantastic!! That way you don't have to scroll endlessly for what you like! Ohh, that's marvelous!

*HUGS*

[greenpear.livejournal.com]

That's why I friended you. You like making stats for fandom a reality. I'm still in the static visualizing mode but hope to get my d3 up and running soon.

[betawho.livejournal.com]

That sounds fantastic! It's always hard to find stories you might enjoy, this would make it loads easier, and allow people to find authors or older stories they might not find otherwise.

That would be wonderful.

[nonelvis]

I'm definitely curious to see how this would work, and if you need design help, you know where to find me! What I'd be particularly interested in, though this would be a massive pain in the ass to deal with, is whether there's any difference between the stories favorited on Teaspoon vs. Teaspoon stories favorited on social bookmarking sites, since I doubt I'm the only person who still posts bookmarks to del.icio.us/pinboard instead of using Teaspoon's bookmarks. (I know, I know, I run the site and don't use that part of it, but I already had those other accounts set up long before I was a mod!)

[eve11]

Hm, that would be a data collection/scraping issue more so than an analytic issue. Do you have a list of 2-4 common social bookmarking sites and can they be searched via a script/query by the story URL? I think if that's the case you could do it with regular expression fu. You've got a unique Identifier in the story URL to link them together. So, plug in the story URL, scrape the account IDs where that story shows up, for each of those do a grep for the URL of Teaspoon stories (www.whofic.com/viewstory? etc except keeping in mind about those pesky ampersands). It would have to figure out a way to get the "splash info" from whatever link it had, but you can do that by going to the author page and then searching for the table entry that matches the story URL (that is kind of convoluted, but should work. Famous last words, I know...).

So short answer: once the analytic/viz part was in place, to add social bookmarking sites you would need to basically build the scraper that looked for Teaspoon story URLs and then you could go from there?

ETA: thinking waaaay too far ahead, with social bookmarking sites you might also be able to build in cross-references to FF.net and AO3 for example, based on the URLs for their stories.

[nonelvis]

The two most commonly used by fandom, AFAIK, are del.icio.us and pinboard.in, both of which have publically available APIs (Delicious' API; Pinboard's). I don't know if there's an easy way to use either to scrape entire sites, though.

[eve11]

Thanks!

I also realized in the "Getting ahead of myself" bit that trying to incorporate other fic archive links (AO3, Lj, FF.net) would be difficult for figuring out keywords and how they match up. So... not too far ahead of myself yet then! Anyhow, I might have to write the guts/backend of this first and then have it spit out boring sorted tables before getting to the actual viz part. The fun thing would be to make it something that wouldn't rely on the snapshot of the database I have, but instead be able to compile the most recent data on the fly when it gets fed a "seed" story.