Source
r/programming•
codev

Here you are reddit: a bot to post imgur caches of craigslist posts - it's called ric

I noticed that a lot of craigslist.org posts have cached images in case the page gets pulled and for redditors who work somehwere craigslist is blocked. So I wrote a quick script which I'm running on a server to post an imgur link every time it finds a craigslist post. The tool is in python, is GPL'd and the code is available here:

http://github.com/codev/ric

Obviously you shouldn't run it as it is because I'm already doing that (you can run it in demo mode to just generate the images without posting comments) but the code might be useful for similar tools. Improvements are very welcome, as the license implies.

It has three parts which you can use in your own projects:

  • a bit of an imgur api lib (just for uploading images - needs a lot more functionality adding)
  • a bit of a reddit api (posting comments and stories, searching stories)
  • a webpage renderer based on WebKit.

I only spent a couple of nights on it but some improvements might be:

  • check if the page has no images and post a text cache instead
  • check if there is already a cache link and not post again
  • extend the sites cached (it doesn't cover Canadian craigslist for example) and restrict others (forums and bestof)

Other feature suggestions are welcome. Enjoy!

EDIT: Corrections in italics

Posted Dec 9, 2009, 6:12 PM UTC · edited

4017
gjs278•

http://garyshood.com/craigslist/mirror.phps

my version. grabs text from craigslist, posts it back on reddit. look at process_post for the regex for grabbing from craigslist.

5
codevOP•

Sounds useful but the link doesn't work

3
gjs278•

oh whoops. sorry about that, check again.

1
vsuontam•

Does it convert craiglist.org articles to images, or did I get this wrong?

Why not just fetch the text, and host it in some other place, github for example? Save some CO2 man :)

2
codevOP•

It does, and posts them to reddit as comments - here's an example.

I suggest in the post:

  • check if the page has no images and post a text cache instead

as a possible addition (i.e. just post the text directly as a comment, no need to use github or other text hosting sites) but I haven't done it yet - it would need to detect if there are images in the craigslist post as then an image cache makes more sense. I'm not sure the CO2 savings are quantifiable either way but it would be a nice addition :)

3
vsuontam•

Nice. Seems like you use Qt's QWebKit components to get the image of the web page.

How is pyqt working btw? Have thought of using it someday.

1
codevOP•

It does use pyqt and the QWebKit component that's in recent versions (not sure when it was added but it doesn't work in the version you get with Ubuntu 8.04). There's a better (many more features) image renderer also based on QWebKit at:

http://cutycapt.sourceforge.net/

which I found while I was trying to figure out how to get it to run on a headless server (the trick is to use xvfb).

PyQt is pretty straightforward, I'm used to Qt in C++ so the transition is fine - it's GPL licensed so you have that restriction when using it (everything you write has to be GPL) but you can buy a license for commercial projects as well.

2
mouseit•

Augh! I've been looking for something to take pictures of webpages for ages.

1
[deleted]•

Couldn't you just search for <img> tags, and save any images separately?

1
[deleted]•

[deleted]

1
simucal•

Good work!

I particularly like reddit.py. That could be useful in other projects as well.

2
wtfrara•

Mmmm reddit pie.... gluaaaaaahhhhaahaaaaa

3
codevOP•

Yeah it could be useful, hopefully someone (or a few people) can take it and extend it - there's no offical reddit API (that I know of) so you just have to debug the ajax calls to figure out how to perform actions.

2
FlySwat•

I wonder if spammers use something similar.

1
codevOP•

I'm sure they do which might be why there isn't an official API. You could cut off API keys that were used to post spam if there was one but I guess that's only the same as banning accounts that post spam.

1
cag_ii•

There is an official API, It's just not documented very well. Refer to the source.

2
moss•

Based on the title, I assumed that this was a bot to just submit a random Craigslist post to Reddit every few minutes. It seems horribly plausible that such a thing already exists.

Glad to see this is something better.

2
[deleted]•

[deleted]

1
[deleted]•

[deleted]

1
you_do_realize•

Awesome.

1