Twist my arm

[eve11]

And encourage my fandom nerdiness. A friend and coworker (a software engineer) stopped by my office today asking me a statistics question. "It's not work-related" he told me. "I have a database with three variables for each record and I want to know how they are related."


Me: Numeric?
Him: Yeah
Me: Are they continuous variables? Or discrete?
Him: Okay, I will just tell you. They're follows, hits, and review counts for fanfiction stories.
Me: OMG can you actually scrape me that data???
Him: Yeah, I have a script that feeds a SQL Lite data base and can grab everything on fanfic.net within a category. It gets all of the info from the title blurbs. It fills in 0s for when favorites and reviews don't exist.
Me: That is awesome. Can you scrape all of Doctor Who for me?

We know each other well enough to know we both intersect with the fanfic world. He is an avid reader of long anime-related stories and apparently his profile on ff.net has "A list of common grammatical errors." :D I have often wanted to collect the data but my script-fu is lacking. I always stop at the "build a script to scrape the websites" step. He has the data and is stumbling at the "make pretty pictures and describe relationships" step. He's curious about relationships between follows, favorites and time with respect to completion and popularity, etc.

It could be the start of a beautiful friendship!

He also says he can probably scrape all of Teaspoon for me too, as I showed him the format and he said, "Oh yeah, I'm familiar with that format". Also cool. What would be a little more time consuming would be to scrape everyone's favorites and match them up so we can look at it like a big graph.

A colleague alerted me to this web-based javascript viz library called DS3: http://anna.ps/talks/fel/ Might be interesting to see if we can figure out who, empirically, is the "BNF" and to see who the cross-era people are or how insular different niches are, etc.

Comments

[ladymercury-10.livejournal.com]

This is a very interesting experiment. It would be cool to compare information for the same stories from different sites, if you found ones that were cross-posted, to see how they fared with different audiences.

[a-phoenixdragon.livejournal.com]

Ohhh! When you do find them...let me know. This is an interesting study!!

You should do FF.Net, AO3 and Teaspoon. There are differences in hit counts, Kudos, Favs and Reviews (each site has its own tastes and followers), but all three probably hold the same numbers for certain people (or at least ratio of/type depending on design, etc).

I'm sorry...this made sense when I thought of it. UGH. *is a moron*

[thisbluespirit]

Heh, sounds fascinating! :-)

[singeaddams.livejournal.com]

Interesting! I'd like a bar-graph with the longest bar being MINE.

*fantasizes*

[eve11]

Ha! I will see what I can do about that, as long as I'm free to make the Y-axis whatever count I wish...

[eve11]

It's quite exciting for a data nerd like me. I have to remember to do my work first and not get side-tracked.

[eve11]

It certainly makes sense. I think I'd want to explore around one site before trying to link to others. I think the easiest comparison to start off making would be Doctor Who folks who post at ff.net and Teaspoon.

[eve11]

That would be cool, though I think it's in "step 2" of getting to know the data, "step 1" being get a handle on how to summarize and visualize a set in and of itself.

[a-phoenixdragon.livejournal.com]

Oh, thank goodness!! And those are good sites to start off with...

[nonelvis]

As long as your friend can scrape the data without creating undue load on Teaspoon's server, I'd be very curious to see the results.

[eve11]

He can probably cron something to get everything gradually over a couple of days or weeks or something. Do you have a set of bandwidth limits or anything via the hosting ISP?

[nonelvis]

I don't know our bandwidth limits, actually, and would have to look into that. If he does things gradually, though, I doubt there'd be an issue, especially since we're serving low-bandwidth text, not graphics.

I can check on things once I'm back from Gally.

[eve11]

Cool, thanks for the info. I would like to explore the data, but not to crash the site while gathering it. It would just be title info, not all the stories (meaning not all of the text). But what might be harder to do (as I noted) is because Teaspoon doesn't list who has favorited a story from that story link, I'd also want to scrape the authors' favorites lists. So that might take a while longer than grabbing all 34K stories.

[nonelvis]

Okay, I've finally checked our bandwidth limits, and in the past few months, we've never exceeded more than about 75% of our limit. One well-behaved script scraping things gradually shouldn't have any significant impact on our bandwidth.

ETA: if your friend still wants to go ahead with this project, I'd like to alert the rest of the mod team. Do you mind if I mention what's going on? I don't want to break your f'lock without your permission.

[eve11]

No problem. I will unlock this post so they can look at it. I don't want to cause any kerfluffles or anything; I'm just curious about the data.

I think actually that Teaspoon might have fewer kinks in the data than ff.net, on account of the modding for "things that are not really stories", several of which I've already come across in the ff.net data.

You can link them to this post for an example. I'm also working on a post about character listings that should go up sometime today.

[nonelvis]

Thanks! I'm curious about the data myself, and I'm pretty sure my fellow mods will be, too.

[eve11]

I made a "fandom statistics" tag for my data-related posts, if people want to look at what little is there as of yet.

[nonelvis]

Cool! Thanks for letting me know.

[eve11]

Coworker is almost done the python code for scraping Teaspoon; he says he's going to try it out later this week and then probably hand it over to me so I can run it on my own. Shout really loud if we break something. (I am usually quickly reachable via lj or my lj email address)