Public Stash Tab API changing river over time?

Question

For a side project I have been indexing the Public Stash Tab API river for a while. Between December 2020 and February 2021 I've collected around 600 GiB of stash data. For testing purposes I restarted my indexing a few days ago with one of my first change IDs from around mid December and it seems like my indexer has already reached the end of the river. The problem is, during these few days I've only collected around 50 GiB - without any code changes of course, no filtering whatsoever.

My question being, is there a known reason for this discrepancy in dataset size? Am I not actually receiving the same data when following old chunks of the river? Or do I have to assume there was an error in my end?

Thank you very much.

Posted Mar 18, 2021, 12:11 AM UTC

114
klayveR•

The API has no historical data, it only has the most up to date state of every stash tab somewhere along the river.

https://www.reddit.com/r/pathofexiledev/comments/48i4s1/information_on_the_new_stash_tab_api/d0ka7ym/

3
MaximumStockOP•

Ahh, thank you very much for that link. Seems like I skipped too fast over that last night.

1
eulennatzer•

If you want archived stash page data, I am running an indexer 24/7 on a cheap vps and keep the last 400GB stored there. If you want access, just hit me up anytime. (all zipped, not raw)

For reference, since Ritual started I accumulated around 1.5TB of data, only missing transactions that happened in the "5 min delay" window.

2
normie1990•

Can you say what server are you running, from which provider?

1
eulennatzer•

I got a 50% off deal from time4vps.com for the most cheapass 500GB vps. (the 3.99€ storage vps one)

You can expect around 20GB or data per day when people are actively playing.

Another option is to run the indexer on a Raspberry PI, but I didnt like leeching bandwidth in my home network.

1
normie1990•

Does the CPU handle it well? For that kind of money it must be a credit based system I think. When I tried my parser on the cheapest AWS lightsail it ran for like 15 minutes before running out of credits. And what kind of database do you use?

1
eulennatzer•

yeah well, by indexer i just meant the download.

I gave up on running the database on a vps, because there is a huuuge bottleneck with disc access. You basically need a big ssd, which is already quite expensive to rent by itself. Well at least for my model.

So I am just downloading and archiving the data on the vps in realtime and then build the database locally from archives.

I parse the data with some python and then insert it in a temporary table unnormalized.After that I normalize the data with just sql in Postgres, which is actually quite performant and has way less overhead then doing it outside the dbms.

My personal observation is that around 1-2 cpu cores can handle indexing everything, but you need an ssd and some big memory to keep it performant. My old pc with 8 GB of memory is slowing down quite a bit after the database reaches around 150-200GB, if I am tracking items with ids (quite a big id table).

If you just want to track currencies and some specific item types you might even be able to run everything on a Raspberry PI. The big bottleneck will still be disc space and disc access.

1
[deleted]•

[deleted]

1
eulennatzer•

Tinkering around with machine learning and evaluating some markets that are hard to evaluate otherwise. ;)

1
[deleted]•

[deleted]

1
eulennatzer•

Well, I dont have the funds to run the indexer/database on a webserver. Also my whole project already went down, because I don't want to deal with annoying "you need an impressum, even if you run the website without profit" issue that is expected from you here in my country.

My original idea was to run an extensive ninja like website with prices evaluating more often then the 1hour they do at ninja.

And then the data is not whitelisted, so every trade < 5 min is not registered and you dont see which players are online, therefore you dont know which trades are actually available and which are not.

Also we are talking about 20GB, which is 100GB of raw data per day in early weeks. Even cutting the overhead and tracking items, it is a lot of stuff that is happening.

So I got the deal with 2€ per month for a server to just get the data and do stuff locally with it. If I even do something, because parsing the data takes a lot of time and 200-300Wh for my old pc isn't cheap either (if run 24/7).

1
MaximumStockOP•

Oh, I never got around to answering your comment. Thank you very much for offering up your archive. In don't have good use for it right now though. Are you still working on anything with it?

1
eulennatzer•

Nah, atm I am working on a filter generator, because I am unhappy with Neversink for a long time. ;)

So technically for something I can always use the data (getting all item bases for example). ;)

2
MaximumStockOP•

That's nice to hear, good luck going forward!

1