RaiderBDev
u/RaiderBDev
https://academictorrents.com/details/5d0bf258a025a5b802572ddc29cde89bf093185c
- subreddit about pages and metadata
- includes description, subscriber count, nsfw flag, icon urls, and more
- 22 million subreddits
- subreddit metadata only
- subreddits that could not be retrieved, but at some point appeared in the pushshift or arctic shift data dumps
- metadata includes number of posts+comments and the date of the first post+comment
- 1.6 million subreddits
- subreddit rules
- posting/commenting rules of subreddits that go beyond the site wide rules
- 345k subreddits
- subreddit wiki pages
- wiki text contents of URLs that can be found in the pushshift or arctic shift data dum
Downloads: https://github.com/ArthurHeitmann/arctic_shift/releases/tag/2024_01_subreddits
This contains the names, ids, descriptions, etc. of 18 million subreddits.
Of those, 2 million were no longer available (private, banned, quarantined, etc.). Those are separate in a separate file and only contain the name, id, potentially subscribers and statistics.
Statistics contain aggregate information from the pushshift and arctic shift datasets: date of earliest post & comment, number of posts & comments and when that data was last updated.
Not sure yet, at which frequency I'll be redoing this. Maybe once a year or so.
Just wanted to share this, for those people who use reddit IDs for statistics or data collection.
Today from 00:10:28 (1706227828) until 00:45:01 (1706229901) (UTC) reddit seems to have had some issue, during which no posts where posted. After the issue was resolved, the post ids jumped from 19fnzox to 1ab550a. So over 50 million. Not sure if someone let a bot loose or reddit did something on their end. I'd guess the latter.
For those that don't know, a short introduction. I'm the person who's been archiving new reddit data and releasing the new reddit dumps, since pushshift no longer can.
So far almost all content has been retrieved less than 30 seconds after it was created. Some people have noticed that the "score" and "num_comments" fields are always 1 or 0. This can make judging the importance of a post/comment more difficult.
For this reason I've now started retrieving posts and comments a second time, with a 36 hour delay. I don't want to release almost the same data twice. No one has that much storage space. But I can add some potentially useful info
TLDR: Downloads and instructions are available here.
This release contains a new version of the July files, since there were some small issues with them. Changes compared to the previous version:
- The objected are sorted by
["created_utc", "id"] &,<,>have been replaced with&,<and>(thanks to Watchful1 for noticing that)- Removed trailing new line characters
If you encoder any other issues, please let me know.
In addition, about 30 million unavailable, partially deleted or fully deleted comments were recovered with data from before the reddit blackouts. Big thank you to FlyingPackets for providing that
First off, I'm not associated with pushshift. Yet, mods please don't delete this :)
For downloads and usage instructions, visit the GitHub page.
How is this possible under reddits new rate limit rules?
Over the last month almost 300 million post and comments were created. That's about 6,500 per minute. With one API request you can fetch 100 posts/comments. So you need to make about 65 requests per minute. Now, what are the new rate limits? 100 request per minute. That leaves enough room to handle peaks and for retrieving older content.
There's a small catch though. The dumps use a slightly different file format, than the one pushshift use
Tomorrow Reddits API changes come into effect. How have the subreddit protests developed so far and where are they now? [OC]
reddit.com/gallery/14n9bszVisualizing the scale of the reddit protests from the top 1500 subreddits
blackout.photon-reddit.com/So we're less then a month away from one of the biggest API changes that will impact majority of clients, bots and tools. And yet:
- developers.reddit.com the supposedly new place for managing your reddit applications is still in waitlist only mode.
- We have no way of accurately seeing how many API requests our applications are making and to which endpoints. This is kinda a very important information, since you don't have full control over what happens in distributed apps.
- Even if we want to pay, there's no way to setup payments right now, to guarantee a smooth transition.
If Reddit was actually serious about being "enterprise", this would have all been
I want to make a desktop app which is basically just one big XML editor. The data looks kinda like this (I didn't choose the format):
- File
- File
- [...]
- File
- Property 1
- Property 2
- Entry
- [...]
- Entry
- [More properties]
- [More nested lists and objects]
The data can be edited from different widgets and might be changed externally. I want a widget implementation for almost every XML tag. Each widget should of course only have access to it's own data. And since there will be a lot of different widget classes, there should be as little boilerplate code as possible.
I've looked at quite a few tutorials for different st
If you open the links below on the reddit website you'll see that they have text below the link or image. Boost currently only displays the link/image and you don't even know that you're missing context.
Link with text: https://www.reddit.com/r/FlutterDev/comments/wx0uit/flutter_flame_simplest_optimization_techniques/
Image with text: https://www.reddit.com/r/revancedapp/comments/wwhnvm/quick_question_about_microg/
Surprisingly enough this new features works with the public reddit api. The post type doesn't change but the selftext property now also has text. So implementing this shouldn't be relatively simple.
If you have any other problems, let me know.
I noticed some weird behavior in my web app, where posts wouldn't refresh. Turns out you get different cache-control headers depending on whether you have a user access token or an implicit grant (anonymous user) access token.
Different cache-control headers:
normal user: cache-control: private, s-maxage=0, max-age=0, must-revalidate, no-store
implicit grant: cache-control: private, max-age=3600
I don't know if this has always been the case or not, but I noticed it today. The caching is of course only client side, but still might be good to know.