Source
r/TheoryOfReddit•
blackstar9000

2010 is the year that Reddit stopped focusing on finding what's best on the Internet, and started focusing on itself.

I wrote a script to scrape every instance of http://reddit.com/r/all/new on archive.org's Wayback Machine and count the instances of terms in the domain field for each of the submissions on those days. The result is a data set that shows, in broad terms, where the site's users were being directed by the links submitted to Reddit at each of those data points. I included four categories:

  1. Internal: That is, links to other pages on reddit.com or redd.it.
  2. Self: Text-only posts that do not directly link to any other page.
  3. Imgur: As reddit's de facto substitute for image-only self posts.
  4. External: Which, in this case, means every link that does not fall in one of the above categories.

Here's the graph and the data set. Wayback Machine snapshots captured anywhere from 24-28 new submissions; to compensate, the chart tracks each category type as a percentage of the total submissions for each snapshot.

As you can plainly see, external links dominated the site until 2010, undergoing a precipitous decline thereafter. In 2011, they were completely overtaken by imgur links, though they climbed back to ~50% during 2012.

Self posts are currently the single most frequently submitted format. External links consistently make up less than half of the content submitted to the site.

Some notes

  • The format on "snapshot time" is 14-digit: YYYYMMDDhhmmss. So 20080218085901 would be February 18, 2008 at 8:59am and 1 second.
  • The distribution of the data across dates was entirely determined by the caprices of the Wayback Machine's mechanism for collecting snapshots. So while the graph may give the appearance of regular intervals, the dates posted at the bottom are just there for reference, not as a measurement of the x axis.
  • Some of the snapshots returned no data. I'm not sure why—it's possible that the Wayback Machine caught Reddit during some outages—but in either case, I've removed null sets. That took relatively large spans of time out of several years, most notably 2011 and 2012. That doesn't seem to have affected the overall pattern of domain occurrence.
  • I have no idea what caused the spike in self posts on Nov. 5th 2008. My first guess was that it had something to do with self.posts and karma, but I dug up the date that the admins changed how self.posts handle karma, and it was well before.
  • Because it has traditionally been the most popular, imgur.com is the only image hosting site counted separately from other technically external links. Especially since r/adviceanimals made it into the default set, though, various meme generator sites have also functioned as a kind of image-based self.post for redditors. It's highly probably that if I had excluded meme sites, external links would rank even lower in the later years

Update

It never ceases to amaze me how many people will rush to dismiss any data that might suggests there was a time when images weren't so dominant on Reddit—and to dismiss it without so much as a gesture toward producing a better data set.

Since, far and away, the biggest objection in the comments so far is that the original test didn't account for other image hosts, I've run the script again, this time with a filter for imgur and another filter for non-imgur image hosts. Included in that second set are:

  • photobucket;
  • flickr;
  • 500px;
  • tinypic;
  • yfrog;
  • imageshack;
  • and any domain name containing the word "meme".

The result? Prior to Feb 2009 (when Imgur was released) the image hosting group peaked above 10% only twice. External links (excluding the above listed image hosts) only dipped below 75% once during for the same period.

For about 3 months after the launch of Imgur, other image hosts maintained consistent rates. After June, they bottomed out, and Imgur began to pick up their traffic. Early June is also the second time external links dropped below 75%.

In March 2010, external links dropped decisively below 75% and continued to drop. By December, links to images had reached an all-time high of 28%. All of those were links to Imgur; other image hosts held at 0% the entire year. External links were below 50% and would stay there for more than a year.

Imgur hovered above 50% for most of 2011. In other words, Imgur by itself counted for more of the links submitted to the site than any of the other categories: not just other image hosts, but also self.posts, internal links, and all other external links.

Other image hosts began to creep back in during 2012. Spot-checking on a few snapshots in that time period, this would appear to be the rise of meme hosts—any domain with "meme" in the name. At the same time, external links begin to stabilize around 50%, and self.posts start to claim a slightly bigger share of the new queue. Imgur falls back below external links, but images as a whole (Imgur + other image hosts) remain the most submitted content type.

If 2010 was the year of Imgur, then 2013 has been the year of the self.post. For the first time, self.posts consistently beat out external links, and sometimes topped 50%. External links dropped to below 25% for much of the year

Now it's possible that I've missed one or more significant domains that should have been included in that test. If so, let me know in the comments, and when we've got all the bases covered, I'll run it one last time.

Assuming, though, that I've covered most of the major alternatives, the results seem fairly conclusive. Imgur did take over the traffic of other image hosts, but prior to 2010, that didn't amount to much—usually no more than 4% of the queue, i.e. 1 post per archived snapshot. When the site's demographics shifted hard in 2010 and 2011, the number of hosted images submitted to the site rose drastically, and Imgur was the primary beneficiary of those links.

Updated chart with "less chart junk".

Another update

... because, why not? This one was prompted by this comment suggesting that looking at /r/all, rather than /r/all/new, might be more revealing. It occurred to me that looking at them in tandem might relate back to a previous experiment. And sure enough...

Well, just look at the chart. The place where Imgur suddenly spikes above 50%? That's February 2010.

Up to that point, the pattern matches the previous charts pretty closely. Unlike the other charts, though, Imgur never really comes down from that spike. At a certain point, Imgur submissions consistently claim more than 50% of /r/all and continues to do so up to the present.

So for the period after 2011, Imgur posts made up roughly 15-35% of new submissions to /r/all/new, but between ~50-80% of /r/all. That's pretty strong confirmation of the conclusions I drew after the simulations I ran 6 months ago.

On the other hand, the earlier charts shows self.posts becoming dominant in 2013—at least on the new page. On /r/all, though, they consistently rank below 25%.

Posted Jan 15, 2014, 6:24 PM UTC · edited

464114
insomniaclyric•

I have no idea what caused the spike in self posts on Nov. 5th 2008.

Barack Obama was elected President on Nov. 4.

116
Deimorz•

I've done a terrible job at continuing to keep the site updated for almost a year now, but stattit's "time machine" is still pretty useful for looking at old instances of r/all (and individual subreddits). So if you're interested in checking out some areas of time to see what the site actually looked like around then, give it a shot. For example, to look at the days around that date, you could try:

Just change the date on the end around (YYYY-MM-DD) to see what r/all looked like for various other dates.

42
ManWithoutModem•

Have you ever thought about getting someone to keep it more up-to-date? I'm sure you would have volunteers from some great redditors/programmers. I can see how being an admin can definitely take away time from those projects.

Side note: how do you feel about http://redditmetrics.com starting to replace stattit a bit? Maybe collab with the guy behind that site to do something sweet/combine both projects into one? Not sure.

16
Deimorz•

Honestly, I think that if anyone was really interested in keeping a stattit-like site up to date, they'd most likely already be doing it (like the guy behind redditmetrics). The large majority of stattit was put together over a weekend, it's literally just a few scripts that scrape and combine public data through reddit's API, and pages that display charts/graphs from that data using easily-available free libraries. Anyone with the interest and ability to do it could duplicate the site extremely quickly, there's just not much to it at all.

Realistically, I should probably just shut it down and point people to redditmetrics, it's doing a lot of the same stuff that stattit was doing anyway, but better. I could likely supply him with some older data that he doesn't have and let him take over.

17
ManWithoutModem•

I don't want to say it and be that guy...but: What are your thoughts on integrating parts of stattit into reddit possibly using the admin data? It could be pretty cool to see subreddit stats like users online in each sub, largest subreddits by size, fastest growing subreddits, charts/graphs for tons of stuff all on one page, etc.

And iirc, u/stuck_in_the_matrix is the one who runs redditmetrics.com. I think the main feature it is missing compared to stattit is mod stats, which is a good and bad thing honestly.

7
[deleted]•

[deleted]

1
rhiever•

Here's a much easier to interpret view of the data: http://i.imgur.com/lF4qXvD.png

I used the data from my evolution of reddit post and performed the same classification as OP did, on a monthly basis.

I don't really agree with OP's classification of all image files as "inward focused" content, but you can see a clear trend of increased self posts and image posts since late 2009.

15
blackstar9000OP•

For what it's worth, I wouldn't classify all images as "inward focused."

Part of the rationale for concentrating on image hosting domains, rather than scraping the links for common image file types, is that a) hosting an image frequently implies inward directed behavior—like repacking a found image for Reddit's audience rather than pointing directly to the original source, and b) discovered images are more likely to be on domains that weren't set up specifically for hosting—like the creator's website.

In other words, part of what I'm hoping to capture here are links that were hosted specifically in order to post them to Reddit, whether or not they were created by Redditors. And for that, the hosting site of choice appears to have been Imgur. That strikes me as a form of facing inward.

For example, if you follow r/mrw, many of the gifs posted are deliberate reposts/rehosts. The content is technically "discovered" (in the sense I've contrasted to "user-generated") and the user's contribution is mostly in the form of a personalized title. But in those cases, the resubmission of old content is inward facing to the extent that it has been rehosted with the intent of reposting it with a clever new title.

In any case, the submission of that sort of post is not about sharing something newly discovered on the Web at large, but rather about interacting with the Reddit community.

11
rhiever•

Interesting, that rationale makes sense. Hope the visualization I made is useful for your analysis. It's from all posts in the time range shown (100s of thousands).

1
blackstar9000OP•

That's probably the cause. I suppose people rushed to share their feelings in self.posts. That's weird to me—why did so many people think to use Reddit as a bulletin board?—but it's the most likely explanation.

7
itty53•

I can only speak for myself, but to me, reddit really served as a replacement for social-networking sites like Facebook. I dropped all of them about a year ago because I realized what I enjoyed about them was real discussion with people, and I hated literally everything else it brought with it. On these social sites, you get an almost 'acted out' discussion with people you know because they're putting up fronts for everyone else who might potentially read (and potentially meet face to face on a regular basis). It's the anonymous aspect of reddit's discussion that allows it to be a bit more real, and thus more valuable (again, to me).

Especially with things like politics: many people might think twice about their eloquent rant about Obama on Facebook, but wouldn't be bothered by posting it to reddit. When you care about a thing deeply, you basically just have to say it and nevermind to whom. So when you're forced to mind the whom because your mother/boss/brother/ex/current/etc is on Facebook, you post it to an anonymous place. Because you still care deeply, you want it to be seen by many. reddit would be the internet's place to do that fairly easily.

40
ryecurious•

It could have been a decision by a few subs (or perhaps a very large sub) to remove linking ability, making themselves self post only temporarily to allow easier moderation. I know there are some subs that do this during predictably massive surges in their reader-base, like TV show subreddits. I'm fairly certain r/community has done this a few times when episodes air.

3
blackstar9000OP•

I could be mistaken, but I don't think that was a moderation preference until later on. Earlier today, I tried (unsuccessfully) to find a changelog entry for when that feature was introduced, and the best I could find was that it happened sometime before 2011.

1
insomniaclyric•

I really can't say, as I wasn't a regular lurker until a few months later.

2
karmanaut•

It's hard to sort imgur links as internal content. Some of them clearly are; r/AdviceAnimals would basically be "self posts on images." But I would bet that the majority of imgur links submitted to, say, r/funny or r/pics would not be original content.

24
andytronic•

I would bet that the majority of imgur links submitted to, say, r/funny or r/pics would not be original content.

Prolly the same with r/aww, too.

11
blackstar9000OP•

Again, I doubt it. One (not entirely reliable) way to test this would be to scrape title submissions from /r/aww/new and look for first and second person language, e.g. "my" or "meet." Based on a cursory scan of the front page of the sub, though, it looks like the majority of submissions at least present themselves as user-uploaded content.

0†
andytronic•

One (not entirely reliable) way to test this would be to scrape title submissions from r/aww/new and look for first and second person language, e.g. "my" or "meet."

You may be over-estimating the honesty of some redditors! There are lots of them that make-up stories from images that were originally posted by others. That might not be a significant % of posts, though.

8
blackstar9000OP•

As I said: not entirely reliable. But if we don't rely on some version of self-reporting, then the only way to test that is through a time-consuming process of checking every image with tools like Tineye, Karma Decay and Google image search. Even then, we can't be 100% sure that the image didn't come from somewhere else.

One thing we can be reasonably sure about, though, is that the introduction and growing popularity of Imgur contributed to a change in the percentage of user-generated content relative to external links to content that was user-discovered. The precise degree of that change is really the only thing in question.

7
blackstar9000OP•

I'm not so sure of that. I just looked at the front page of /r/funny, and more than half of the imgur links I clicked at least presented themselves as OC. Even those that clearly weren't images made by the submitter involve an element of user intervention—e.g. gifs broken into separate images. Ditto for /r/pics.

The other frequent use I see for imgur is rehosting, either a) because the submitter originally found it on a semi-private platform like Facebook, b) because they want to repost it, or c) to avoid linking directly to the original source. The overall contribution of those methods is to make Reddit function more like a social media network, rather than as a platform for directing traffic out to content where it lives on the Web.

5
ManWithoutModem•

I always wondered what would happen if AA made people use self posts only and used image macros for their memes so they couldn't karmawhore?

3
hansjens47•

That would be interesting. I suspect much fewer submissions, much less traffic and a lot of hate would be immediate results. But how the remaining submissions would be, that would be interesting.

3
coloicito•

You already sort-of tried it for the last April's Fools. It didn't went well IIRC.

2
crshbndct•

What do you mean? Like two text boxes and a selection box for people to choose the image?

1
ManWithoutModem•

Check the comment section of r/braveryjerk, pretty much if people were required to use legitimate versions of those image macros in self posts.

1
[deleted]•

I joined this site in 2010 as part of the Digg exodus. So you guys can blame me.

20
blackstar9000OP•

I think it's entirely likely that the influx of Digg users contributed to the dominance of Imgur links in 2011, but other external domains were on a sharp downward slope even before the Digg migration.

19
Fuzznuggets•

Fantastic data you have presented... I think this is actually a bit sad. I think perhaps what could help with this is abolishing Karma. It really only drives people to post for popularity rather than because they found an item interesting. I know that doesn't have anything to do with the graph, but I think it could be a solution

16
karmanaut•

I think perhaps what could help with this is abolishing Karma.

It's kind of the opposite. His graph shows a large rise in self posts despite the fact that they got no karma, whereas link posts do get karma.

23
splattypus•

How much is that skewed by r/askreddit, though, which thanks to its default status has remained one of the biggest and most active subs on reddit?

Not to mention the rise of r/askscience, which was another self-only default sub for a while.

Would self-posts around the rest of reddit continued to rise in numbers were it not for the fact that 2 self-only subs were defaults?

11
blackstar9000OP•

How much is that skewed by r/askreddit, though, which thanks to its default status has remained one of the biggest and most active subs on reddit?

Even assuming that AskReddit is heavily represented in the pages I scraped, how would that be "skewing" the results? If people post more frequently to AskReddit than to other subs, then isn't that part of what we're out to measure?

Would self-posts around the rest of reddit continued to rise in numbers were it not for the fact that 2 self-only subs were defaults?

Hard to say. What I would say is fairly certain is that we wouldn't have so many Ask subreddits if it weren't for people discovering that AskReddit isn't particularly good for getting answers to certain kinds of question.

But again, I see that as a cause of the effects I'm isolating here, not a factor that somehow "skews" the conclusions.

5
iamanevilgenius•

Can't speak for /askscience, but compared to other defaults, I've found that /askreddit/new seems to move faster than say /funny/new. That is, more posts are submitted to askreddit than to funny--likely because a simple question requires less effort than an image. This is only subjective, personal experience as someone who's gone through /new.

1
crshbndct•

If people post more frequently to AskReddit than to other subs, then isn't that part of what we're out to measure?

People Karmawhore for comment karma too, and AskReddit is the best place to get it.

1
blackstar9000OP•

Exactly; and the question is, why? Self.post rates stayed fairly steady for about 2 years after the admins made them karma-less. There's a pretty clear growth spurt leading into 2011, which was likely related to an upswing in the number of a new subscribers. After an initial spike, even that stayed fairly constant for another 2 years, at which point the ratio of self.posts to links grew to current levels, with self.posts outperforming other categories individually and routinely making up more than half of new submissions.

One explanation is a cultural shift. I incline in that direction, and I think it has a lot to do with a growing perception that Reddit is discussion venue similar to social media platforms like Facebook and Twitter.

At the same time, I suspect one feature in particular helped solidify the growth of self.posts: the ability of moderators to make their subs self.post-only, link only, or both. That essentially confirmed the impression that large swaths of the site were specifically set up for discussion, and helped subs like /r/AskReddit and /r/IAMA grew become the most visible faces of the site.

6
darknecross•

It's not about the "karma" as a tracked and collected metric. It's always been about the attention, and having more votes on a submission (self post or not) is a reflection of all the attention they received.

Nobody keeps tracks of total Likes on Facebook, but that doesn't mean people don't try to get them as a form of validation.

2
blackstar9000OP•

Nobody keeps tracks of total Likes on Facebook, but that doesn't mean people don't try to get them as a form of validation.

Individuals might not keep track of Likes, but as someone who's worked for companies with social media ambitions, I can assure that you someone is watching them very closely.

4
Deimorz•

I think perhaps what could help with this is abolishing Karma. It really only drives people to post for popularity rather than because they found an item interesting.

This sort of thing is said often, but I really don't think it's true. It's not the karma points that people care about, it's the popularity of the post itself. The part that most people find exciting out of a popular submission is seeing their post rise up through the ranks, get lots of comments, etc. Very, very few users overall care about how many points are going to get added to their score out of it. That's just a side effect, not the actual goal.

Take a look through a lot of the top posts in subreddits like r/pics, r/aww, etc. at any time, and you'll find that most of the users really don't have that much karma. In many cases, they only have one or two popular submissions over years of being a reddit user. There are certainly some users that have basically made it a game to see how much karma they can accumulate, but they're far from the average case.

11
ManWithoutModem•

This sort of thing is said often, but I really don't think it's true.

The amount of times that people get a popular self post in one of my subs and then they modmail wondering where their karma is for their submission has been rising lately. So I think some users do care about karma for self posts but just don't know how reddit works. Most don't care or even know what karma is if they are newish probably.

1
Gamiac•

I would love to see an experimental feature which would allow subs to rank posts based on which thread was commented on most recently. The results would be interesting to watch.

4
Yiin•

Seems like 4chan, with the thread limit being 1000 threads.

2
Gamiac•

Yeah, but there would be a few significant differences, though. Namely, that normal users can't just create new boards out of nowhere on 4chan, and it would be on a per-sub basis. You also don't have anonymous commenting, again, unlike 4chan.

You could also theoretically implement it as a sorting option to be chosen by the user if that's what they want.

2
ryecurious•

You missed the obvious difference that comments are sortable by users, unlike 4chan which limits comment order to chronological.

3
Drunken_Economist•

It's kind of useless data considering imgur didn't exist before then. You're taking the "external links" and taking out a huge chunk of them that isn't taken out of the previous years.

14
blackstar9000OP•

You're taking the "external links" and taking out a huge chunk of them that isn't taken out of the previous years.

That doesn't really make sense. If there was no Imgur, then links to Imgur are excluded by default.

As it stands, Imgur was launched in February of 2009, so it's represented in 4 out of the 5 years covered by the data. It didn't overtake over external links for a full 2 years after it launched, which is, in itself, interesting and potentially useful information.

One hypothesis I'd suggest (and this appears to be supported by the pre-2010 snapshots I've looked at directly) is that direct link to images didn't make up a substantial portion of total submitted links prior to Imgur's rise as the major platform for image-sharing on Reddit.

15
Nimos•

he means that before there was imgur, image links were just hosted on something like photobucket or whatever was trendy at that time

13
blackstar9000OP•

I know what he means. Looking directly at the Wayback Machine snapshots of the period before 2011, though, it seems clear to me that wasn't the case.

4†
c_albicans•

Sure, but that isn't represented in your data. I'm not saying you're wrong, but your figures should tell the story accurately.

1
slapdashbr•

Before imgur there wasn't a good single website for sharing pictures. I mean seriously have you seen a photobucket link? ugh

1
BrowsOfSteel•

I think the point is: what about the various hosting sites that were serving the same purpose of Imgur?

With the current breakdown, it’s possible that there were just as many image links before and all the chart documents is a shift from other image hosts to Imgur.

2
blackstar9000OP•

And part of what I'm saying is: there weren't. Here's a new page chosen at random from the period before external links started to decline. There's one imgur link and one other image host (flickr). That's less than 10% of the links on that page. And of the earlier pages I've seen, that's about standard. By 2011, on the other hand, imgur links alone made up more than half of the links on the new page.

In other words, all of the evidence I've seen points to the popularity of imgur as a cause of the increasing use of Reddit as an image aggregator, and not vice versa.

If that really worries anyone, though, I'll be glad to run the script again, with a fuller batter of image hosts. Just compile a list of all of the image hosting domains you want me to include. I'll try to upload a revised chart later on today.

7
doctorpwn•

Do that. If the effect you're noticing is there, it'll be smaller. It would still be interesting and then we could have a proper discussion.

3
Phantom_Pizza•

Would would the use of RES be enough to filter content towards sites that are more compatible with that program?

However, the former question assumes that RES is extensively used.

2
blackstar9000OP•

See the update. I've re-run the test with an additional category for those other image hosts. Doing so did very little to change the results on the early end, though it did sharpen then decline for other types of external links for the 2012-2013 period.

2
Drunken_Economist•

Yes, but the existence of imgur didn't prompt a bunch of new posts that wouldn't have been made otherwise. They were just on other hosts previously.

1†
blackstar9000OP•

They weren't, though. See this comment for details.

7
blackstar9000OP•

See the update. I've run the test again, with an additional category for image hosts other than imgur. Short version: Nothing much changes.

4
Drunken_Economist•

Ohhh, I like that data a lot more. Interesting stuff

2
[deleted]•

[deleted]

1
Measure76•

There are only X number of new web pages worthy of visiting on any given day. While that X grows over time, the number of users of reddit is growing at a larger pace. So all these redditors with time on their hands talk about themselves or each other on reddit instead.

In other words, there are fewer worthy external links per reddit user now than there were several years ago.

Also, your graph pretty clearly shows IMGUR taking a big chunk of external traffic away. There was no single imgur replacement before imgur, people used to share links from hundreds of different sites that are now consolidated under the imgur banner. As soon as imgur started up, people would steal content from other sites and host it on imgur, making the external links go down, and imgur go up.

9
irontail•

There are only X number of new web pages worthy of visiting on any given day. While that X grows over time, the number of users of reddit is growing at a larger pace.

This is a bold claim. What data are you basing it on?

6
Measure76•

Which part of the claim do you feel is bold?

2
irontail•

The claim that the reddit user base is growing more quickly than the number of worthy web pages. This might be true, but I don't think it's obvious. Thus my request for a little evidence.

Considering all of the niche-interest blogs, tumblrs, celebrity twitter posts, etc., I think there is more content being generated on the web than you're allowing for. You may not consider a lot of it "worthy," but that's subjective, and there may be a significant population of reddit users who disagree.

7
blackstar9000OP•

Exactly; if nothing else, the growth of easy-to-maintain microblogging sites might mean that the Web is actually growing at a more rapid rate than it did during Reddit's early days. Just as an example, Tumblr was hosting 152 million blogs as of November 2013. If even 99.99% of those are crap, that still leaves 15,200 worthwhile blogs, producing content to which redditors could link. If each of those blogs produced only a single post each year, it would still take more than 40 redditors posting a link a day to cover them all.

Short of some compelling evidence to suppose otherwise, I'd say the more reasonable conclusion is simply that Reddit misses a lot of "worthy" content.

9
slappymcnutface•

It would be great to get data for this hypothesis.
My personal view is that the specific population that was initially drawn to reddit is different from the majority of new redditors. I mean, the people I knew as redditors before r/AdviceAnimals was a big thing and before r/politics and r/atheism were taken off the front page were interested primarily in content. Well, I mean, they were essentially a more nerdy class of people. Obviously you had brevity of wit and the occasional frat bros.

But it really seemed that for every thread discussing some social situation, the overwhelming attitude was "Oh my god I'm so bad at talking to people, how do I girlfriend? I have no friends pls what do?"

And now practically every thread dealing with boyfriends/girlfriends, parties, friends, etc. is filled by interesting "normal" people sharing their stories. The old reddit social-anxiety whining had effectively disappeared a number of years ago.

To me this speaks volumes about the kind of user that joins reddit now. Put simply, the majority of the introspective nerd community has already joined. Now most new users are the regular internet users.

As an angry nerd who dislikes sociable people, this is essentially what I blame the frontpage drop in quality on, the dirty plebian masses overwhelming our stronghold.

A bit dramatic and anecdotal I will admit. And of course the way subreddits work any effective user can escape these kind of problems.

But in terms of frontpage analysis like the data set OP has brought, I am quite sure that a good digging would show new users are more sociable than the earlier crowd. Reddit has more self-posts now and is more "introspective" only in that it has enough people to talk to each other. Threads are now people sharing stories and conversations like a bar. Before reddit was
"here is the subject: (2006 election/new technology/internet migration patterns/comic artist) let us discuss it."

According to OP's data, more posts are about ourselves than the rest of the world.

3
blackstar9000OP•

In other words, there are fewer worthy external links per reddit user now than there were several years ago.

I doubt that matters as much as you think it does. At the start of the time period (January 2008), there were only about a dozen subreddits. Since then, the number of subs has grown to several hundred thousand. So even if we assume that the number of Web pages hasn't grown in proportion to the number of Reddit users, the number of places redditors have to post links to those pages has grown at a more proportional rate.

So take a site like Daily Kos. They probably publish around 20 posts per day. It's possible that they published about the same number in 2008. But in 2008, there was only one political-themed subreddit on the site, so it didn't matter how many redditors there were, Daily Kos wasn't likely to get more than 20 submissions to the site on any given day.

Today, on the other hand, even going only by the related subs page of /r/politics' wiki, there are around 200 political-themed subs. Even assuming that any given Kos post would be off-topic for some substantial portion of those, that's hundreds of opportunities for the Kos domain to make it into r/all/new each day. While the site has a mechanism to discourage reposting a link to a sub where it's already been posted, there's nothing to prevent redditors from posting it to a relevant sub where it hasn't already been posted.

What you'll find if you look around, though, is that, for the most part, they don't. [This is the top post in r/politics for the last 24 hours](http://www.reddit.com/r/politics/comments/1v9xxb/lets_not_forget_that_a_mere_40_years_ago_we_had_a/). Despite relevance to several dozen subreddits across the site, it's been posted to three. There are daily exceptions, of course, but that's pretty typical—there are plenty of links that never get posted to Reddit, and those that do only get posted to a fraction of the relevant subreddits.

So it's certainly not the case that there are fewer opportunities for linking to external domains other than imgur. If anything, there are far more opportunities now than there were 6 years ago. Even if we assume that the number of potential links has stayed constant, there are thousands more opportunities to post each link, and several million redditors who could be posting them. The fact that they don't strikes me as the sort of change you can only explain as a change in the culture of the site.

1
Measure76•

It is not that there isn't opportunity to post external links. Again, I'm talking about the ratio of worthy external links to the number of reddit users.

Reddit has been growing much faster than the internet as a whole over its lifespan, and has not yet reached a 'peak userbase'.

Ever since subreddits were opened for creation be all users, an infinite number of links to the same external content has been possible. This has not changed that the number of worthy external links has increased at a smaller rate than the size of the reddit user base over that time.

1
blackstar9000OP•

Again, I'm talking about the ratio of worthy external links to the number of reddit users.

I might be talked about the same thing. After all, my example was the top post in one of the site's most popular subs. That's about as close to a falsifiable model of "worthiness" as you're likely to find. If it can attract a score of several thousand, then surely there are enough people who think it worthy that we ought to expect it to crop up in more subs. That it doesn't suggests to me that very large numbers of redditors fail to submit even links that they themselves deem worthy.

At any rate, if you want to pursue that line of argument, you're going to need to be a great deal more specific about your criteria for "worthiness."

Reddit has been growing much faster than the internet as a whole over its lifespan, and has not yet reached a 'peak userbase'.

I'd be surprised if you could produce evidence in support of the first claim. People forget that the internet is still growing quickly outside of the major industrialized countries. As for "peak userbase," Reddit will probably continue to grow, but it looks to me as though the rate has slowed significantly since the major gains won in 2010 and 2011.

1
Measure76•

The redditors "Fail to submit" because they've already seen it on reddit. Redditors who feel a particular link is worthy are extremely likely to be subscribed to the largest reddits that their favorite content are posted to.

Those that delete the default reddits are in the extreme minority. The admins have said that only a small percentage of users create accounts, and only a small percentage of those with accounts customize subscriptions. When you get to a small percent of a small percent, that is tiny.

My criteria for worthiness is what people submit to reddit, and what gets upvoted. I would suspect that the total number of external links submitted to the site would grow slowly, at a rate similar to the overall growth of the internet. At the same time, percentage of total internet users on reddit has grown.

So I would suspect that your data would show only a slight increase in external links over time while also showing a large increase in internal links.

However, since you are taking certain categories of external links out of the 'external links' catgory, you can pretty much get the data to show what you want instead of the truth.

0
Doomed•

If photos get reposted all the time on Reddit, then why don't interesting websites get reposted as often?

1
Measure76•

I would say that this is because there are several orders of magnitude more photos on the internet than there are websites.

1
Cheimon•

That's very interesting. I'm not sure the focus on reddit in itself is necessarily a good one. There's a lot of good content out there that does go unread and it can be hard to find compilations for it. When websites like this stop filling the gap it's a shame.

5
[deleted]•

The great inward turn of reddit.

5
rhiever•

Can you please remake this as a 100% stacked area chart? I can't tell what's going on with all the noise.

5
Kensin•

What domains that were not image hosts were the most popular? I guess what I'm asking is, if reddit used to be primarily links to non-image external content, what was that content? What are we not seeing now that used to be so popular?

3
hansjens47•

I would guess content elsewhere on the web like articles. Some of the larger article-based subreddits have higher historical post volume on a place like statit where the data is older, than they do today.

Images as a whole seem to be on the rise, that is short-attention content muscling out content that takes longer to digest. That may partially be due to the rise in mobile redditors.

2
exizt•

I really wish this chart would have less chartjunk in it. Could you at least remove the shadows, perhaps?

2
sara1138•

Interesting post. Nice job addressing the imgur-didn't-exist complaints.

2
penguinv•

Meme linkss might be imcluded in internel links.

1
blackstar9000OP•

Why would meme links be included in internal links?

1
penguinv•

Becauae the poster wrote them so it is essentially a self post.

1
blackstar9000OP•

In this case, "internal links" means links to other reddit pages—for example, the way that r/bestof and r/depthhub link to other comments on Reddit. Sorry if I wasn't clear about that in the OP.

2
[deleted]•

[deleted]

1
drivers9001•

r/all/new doesn't seem like a good representation of reddit. Maybe r/all because then it's what most people actually looked at.

1
blackstar9000OP•

What makes you think "what people looked at" is a better representation of Reddit than "what people submitted"? Experiments like this one aren't really about who's on Reddit—nor could they be, really, save in the most indirect way. Rather, they're about the sort of information that makes it into an information system structured the way Reddit is structured.

At any rate, I could re-run the test, pointing the script at r/all instead of r/all/new. It wouldn't really change my conclusions, since the possibilities of r/all are limited by what appears in r/all/new, but it might show something interesting about how the voting process filters those possibilities.

1
drivers9001•

Because, I think "what people looked at" is a better indication of what reddit is like as an experience.

The top 10 on r/all right now are 8 imgur links and 2 gifs hosted on other sites.

The top 10 on r/all/new right now is: 1 wikipedia, SEVEN self.posts, 1 livememe, and 1 imgur.

3
blackstar9000OP•

What Reddit is like as an experience will vary widely depending on the person and how they approach the site. It isn't the sort of thing you could capture with a script.

But anyway, take another look at the OP. I've added another update with data culled from /r/all. The major changes I noticed are a) Imgur posts benefit from a competitive advantage, and b) the dominance of self.posts on last year's new queue doesn't at all translate into more self.posts on the front page.

1
drivers9001•

So for the period after 2011, Imgur posts made up roughly 15-35% of new submissions to r/all/new, but between ~50-80% of r/all. That's pretty strong confirmation of the conclusions I drew after the simulations I ran 6 months ago.

On the other hand, the earlier charts shows self.posts becoming dominant in 2013—at least on the new page. On r/all, though, they consistently rank below 25%.

Yeah that sounds about right. Nice work.

1
rhiever•

Do I understand correctly that you sampled the ~25 posts from r/all/new for each day?

If so, holy moly that's not proper sampling at all. It's not only a highly biased, non-random sample, but it's also an incredibly small sample. No wonder there's so much noise in your plots. There are tens of thousands (if not more) submissions to reddit every day. 25 posts is not nearly enough to be a representative sample.

2
blackstar9000OP•

I actually didn't do the sampling. All I did was scrape the information that was stored by the Wayback Machine. Find a way to open up a larger data source, and I'll glad rerun the test on that pool, but for this test I was working with the information I knew how to work with.

So I'll concede that it's a small sample, relative to the total volume of submissions coming through the site. I don't, however, see how it's a biased or non-random.

1
rhiever•

It's a biased and non-random sample because you're taking 25 posts that were posted one after another at some point in the day whenever the wayback machine made its snapshot. For it to be a random and unbiased sample, you'd need to take 25 random posts out of random time points in each day.

u/drivers9901 also makes a great point about how sampling r/all/new isn't the best idea: http://www.reddit.com/r/TheoryOfReddit/comments/1vaiea/2010_is_the_year_that_reddit_stopped_focusing_on/ceqrqoz

2
[deleted]•

[deleted]

1
[deleted]•

[deleted]

1
[deleted]•

[deleted]

1
djloreddit•

this post is cool as fuck op. someone submit this to r/bestof

1
blackstar9000OP•

You're someone. I believe in you.

1
deltree711•

Is it just me, or is there a significant amount of irony in this post?777777

1
doctorpwn•

I'm sorry but these statistics are garbage, for the simple reason that you're considering imgur to be the only image hosting site. A link to photobucket or livememe, for these purposes, is exactly the same as a link to imgur. However, you are grouping all of the photobucket, livememe, quickmeme, tinypic, etc links as external sites even though they are all really the exact same fucking thing as imgur. So before imgur became dominant, all those other image hosts would be counted as external links.

2010 wasn't the year reddit began looking inward, it was the year imgur became dominant.

0†
blackstar9000OP•

See the update. I've re-run the test with an additional category covering non-Imgur image hosts. The short version is that you're wrong, though I'm sure you can dredge up another reason for dismissing the data as "garbage."

4
darknecross•

You could write a whole new ToR post about this very exchange -- how quickly redditors make it a mission to point out any and every flaw from a post or article without actually discussing the main points. Missing the forest in the trees, as they say.

4
blackstar9000OP•

Don't think I haven't thought about it.

And, actually, I don't mind someone pointing out when an untested assumption threatens to skew the data. I'd just prefer that it arrive as a constructive suggestion for how to make the experiment better, rather than a skeptical dismissal.

3
doctorpwn•

I didn't mean to come off as so dismissive. It just seemed like a pretty massive flaw. Sorry about that.

1
Zoklar•

Another thing I didn't see was that many of the "self posts" may include other links. Many subreddits that are self post only still see links in the self post with little explanation. Whether it works depends on the sub, but i didn't see that variable discussed.

2