Source
r/algotrading•
rymarc

Historical Options Data (CBOE Datashop vs Algoseek vs ivolatility)

Data

I've seen a few posts on here where people are asking for free historical options data, but not much discussion regarding purchasing data from a reputable dealer.

Based on my naive research, the main players in this space appear to be CBOE with their Datashop, Algoseek and IVolatility.

I am not looking for super detailed data. 5 to 15 minute time slices of trade data, with daily open interest, for full options chains on a few underlyings.

I have two main questions:

1.) Does anyone have any experience with these vendors and/or recommendations for other data sources?

2.) Is there any significant difference in the data that will be received from these vendors, or is it all coming from the same faucet?

Many thanks for any help or guidance.

Happy Trading!

Posted May 1, 2021, 5:07 PM UTC

4735
baconcodpiece•

I've bought data from all three of those vendors. In theory the data should be the same, as the resellers are licensed to sell the same data that the sources are also selling. I eventually moved over to Cboe since I figured the source of the data should have the fewest errors of all (the telephone game). It is more expensive compared to IVolatility though. It's been so long since I bought from Algoseek that I don't remember how much it would be compared to Cboe, but I'm sure Cboe is more expensive than them too.

One thing that really annoys me about Cboe is that the data format is inconsistent depending on the age of the data. For example, they switched time zones after a certain date, which you definitely need to be aware of when backtesting. But what's even worse is sometimes they'll do something like use a two-digit year, and then switch to the four-digit version. Like why??? Why on Earth wouldn't you make it consistent for every year you have the data? And the icing on the cake is it's totally common for them to not use the YYYY-MM-DD format, but instead month/day/year. It needlessly adds complexity to parsing it. I wish they just formatted everything the same, using the same standards that the rest of the free world uses.

Edit: I should add that it was index and futures data that gave me the most trouble when parsing. Their options data was pretty good and consistently followed the YYYY-MM-DD format.

18
rymarcOP•

Wow, very helpful information regarding issues with date formats. It's almost impossible to find reviews for these services on the internet, so this information is great.

I started asking for quotes last week and IVolatility has actually been the most expensive, so far...

Have you made any customer support requests with any of the vendors? Are any of them more/less likely to help you with questions or issues you find in the data?

7
baconcodpiece•

That's interesting about the cost. Maybe I bought different data compared to you or the prices have changed since. I just realized that you're talking about options data only, whereas I've also bought index and futures data in addition to options. The historical options data from Cboe was at least formatted better when it came to dates, so I think you'll be able to avoid the formatting headaches I had to deal with for the other types of data.

I've had issues with all three that required contacting them.

  • Algoseek: Noticed that I was missing some days when I bought options data. Their customer service was pretty bad at being responsive and I had to follow up on them multiple times. I eventually received a partial refund for the missing data. They left me with the impression that it was a one-man shop. This was a few years ago and things may have changed since.

  • IVolatility: Pretty responsive, although they are in a European time zone I think. Way different than the US time zone, so something to be aware of for responses. Bought options data with Greek/IV calculations and noticed some impossible values, so they had to fix that and regenerate the data. Overall pretty responsive and helpful. I think only once I had to follow up. However, their site leaves me with the impression that their security isn't the greatest (I forget exactly what it was, they might have emailed me my password when I couldn't remember it), and it would not surprise me in the least if they aren't PCI compliant when it comes to handling your credit card data.

  • Cboe: Also very responsive to email, looks like they use JIRA behind the scenes for support. I've contacted them multiple times, once for missing data. They definitely have some ancient legacy systems that they need to upgrade though. I'm subscribed to get EOD futures data, and no joke, after RTH end it takes them over eight hours to make a CSV file that has less then 70 lines of data available via their SFTP. It's awful because I wanted to fetch it that night and review it manually, but given how late it is I'm already in bed by then, and by that time I might as well wait for them to post it on their public website the following morning. I'm going to have to find another provider for that specific data.

6
rymarcOP•

Thanks again for the info about the Greeks/IV calculations. I don't think algoseek offers Greek/IV calculations but CBOE and IVolatility charge a premium for including them in a dataset.

I have been trying to figure out if the Greeks/IV are worth the price premium or I should just calculate my own. It kind of sounds like I might spend the same amount of time double checking their calculations as I would just generating my own.

1
wsb202009•

CBOE, do you get VIX futures data, esp pre-market live data?

1
baconcodpiece•

I've bought their historical VIX futures tick data. I was going to buy their VIX futures OHLC data but they post it on their website for free.

I wish I could get live VIX futures data. I've talked to them about it and they don't have anything for retail right now. They're supposed to be working on an AWS version with a streaming feed but it's still in PoC phase.

I've scoured the Internet searching for a provider but everything offered is really expensive because it's marketed towards institutional investors.

2
wsb202009•

Thank you for the details.

1
stackstistics•

I think QuantConnect uses Algoseek

7
rymarcOP•

Yes, I believe you are correct. There are a few services that utilize Algoseek as source data.

QuantConnect and QuantGo both pull their data from Algoseek, but part of their terms of use is that the data remains on their platform and can not be exported for personal use.

I don't want to be tied into a platform like QuantConnect or QuantGo and will be utilizing the data in my own development environment.

4
jaredbroad•

With LEAN (as any FOSS) you're not tied to QuantConnect. IMHO misleading claiming QC ties you to anything when our using `pip install lean` CLI is effortless now.

We just shipped fully open-source total margin modeling, i.e. spreads margin modeling with dynamic real-time position solving.. in the open-source. It is worth taking a look.

Additionally, we're about to open up the sale of our data in a LEAN format so you won't even have to do the 680 CPU cluster processing of 400TB data that we do to make it digestible.

3
rymarcOP•

Thanks for the information. I wasn't trying to make a comment about the capabilities of QuantConnect as a platform. I'm certain it is extremely useful for many people.

I just wanted to make it clear that historical options data available on QuantConnect, which is sourced from Algoseek, is not meant to leave the platform. Is that correct?

1
jaredbroad•

is sourced from Algoseek, is not meant to leave the platform.

Yes-correct; sourced from AlgoSeek,
And soon; will be able to leave the platform,

1
kayaking_is_fun•

We buy data from both algoseek and ivolatility for 1 minute data. Both offerings are broadly similar - algoseek has CME Future Options data which is helpful, and IMO has slightly better data structures, but I don’t think there’s a massive difference.

If you have the cash to do both it’s always helpful to double test algos. We do see results that differ by 10-20% (from each other)

5
rymarcOP•

I would rather not spend the cash to buy from two places.

10-20% difference is a pretty huge margin, which I know is just arbitrary performance metric of your algo. Are there any specific discrepancies between the different sources that you've found?

2
kayaking_is_fun•

For us it’s looking at delta hedged options. Most of the discrepancies come from snapshot times as there is no guarantee that a given instrument will be snapshotted at the exact same second as the rest, so there can be small inconsistencies if things are snapshotted at start or end of timestamps. This also makes underlying price matching difficult.

In our experience both providers will occasionally have poor data historically due to systems failure on their end - but rarely a problem and neither seems to be better for it. Talking about maybe 1 day every 1-2 years.

To be clear, when I say 10-20% it’s e.g 18% return vs. 22% return - so it’s different, but in the context of a sharpe 2 strategy feels not huge.

Hope that helps!! Basically to summarize, I don’t see either as better.

2
rymarcOP•

Gotcha, that makes a lot more sense with the snapshot/time-slice datapoints. Makes me feel better about just purchasing from one source, those kinds of discrepancies shouldn't matter for my personal use case.

1
boeggles•

Check out https://uselantern.com/api

disclaimer: I'm the founder.

we're currently just a mobile app but building up our API offering so should be out soon.

some of the stuff we have right now: historical prices & greeks (computed and real), open interest, volume, and money flow. all on tick basis for the last few months.

3
rymarcOP•

Looks slick! But I need data going back a minimum of 5 years.

Any idea what your pricing structure is going to look like?

4
boeggles•

Hey no worries! That level of historical data will def be around in by EOY.

Our pricing isn’t exact yet but definitely way more optimal than the other providers.

If you’re interested in checking it out, we’ll be opening a private beta for our initial waitlist.

2
penagwin•

The more options the better, can’t wait to check it out!

2
boeggles•

Thanks! We’re really looking forward to releasing this soon

1
mjr2015•

Do you also track ivr?

2
boeggles•

Yup!

1
Thetaticians•

For research purposes (and personal trading use), I've been running a script to collect live quotes on the SP500 option chains every few minutes for the past month using TDA's API (not tick-change as that would easily overwhelm any non-enterprise storage plan). I've already processed the responses into Numpy arrays that contain estimated greeks, bid, ask, volume, OI, etc for every strike and expiration. So far the uncompressed dataset is about 30 GB.

I was planning on open sourcing this dataset + my API for loading and querying these Numpy tensors but haven't found the motivation/demand to justify hosting it yet. At the very least I'm planning to continue collecting this data for the next few years so that there's an open source dataset of historical option data for academic (and beginner algo) purposes

3
flashcrash7•

I am TD user too. I was wondering about this. I presume you meant, you are running your query every few minutes so you can go back into the past a bit further each time?

I can host this if you would allow me to, with credit attributed to you. I think this is a great idea. We can continue to keep it updated.

Let me know what you think.

3
Thetaticians•

Yep, throughout the trading session I continuously send REST calls to their API on the entire SP, vol ETFs, sector etfs, and leveraged bull/bear indices.

I was planning to publish the dataset once I've collected a quarter worth of data since the usefulness of said dataset increases with size. I'll keep you in mind once I get to the stage of publishing it.

2
rymarcOP•

I also utilize the TD API for capturing and storing live data. I've been doing that for a little more than 2 years, for a few different underlying.

Unfortunately, because of various technical issues over those two years. The dataset is definitely rough around the edges. My project has advanced to the point where I need high quality historical data going back at least 5 years. Seems like these data dealers are the only realistic option.

As with most things in life, you get what you pay for!

1
ArcticLights•

Theta Data (thetadata.net) is launching soon and is providing this data for a fraction of the price of what you could get at CBOE. They are offering minute intervals historical data with a 15-minute delay. The data is from a direct feed so there is no game of telephone.

​

Disclosure: I work here.

2
destroyer1134•

I've used data shop a bit and really liked it. It might be more than what you need but it gives you the chance to get more analytical if the needs arises.

1
amnezzia•

There is also orats. I only had enough money for daily snapshot, so don't know much about intraday quality.

1
[deleted]•

I think ivolatility adjusts option prices for corporate actions / early exercises while others send raw data. Worth double checking with them.

1
allwaysontarget•

What is the cost?

1
rymarcOP•

Depends on what you want.

CBOE Data Shop gives quotes on their website for whatever data you want. The other websites are quote by request.

I asked them all for 6 years of options interval data on one underlying (Prices, Volume & Open Interest). Prices ranged from ~$500 - $1100, but the datasets they were offering were not totally equivalent.

1
thecheese27•

I bought CBOE 30 minute interval options data with calculations and it was delivered promptly and very clean. Granted, it was only a month's worth of data, but I more or less bought it as a trial. I plan on purchasing 10 year's worth of daily SPY options quotes very soon. Btw, if you're a student, you can ask a faculty member to apply for accreditation so students can receive discounts. I believe it's 50% off on any order above $500.

1
allwaysontarget•

Looking for for 10 year historical patterns. Where can I find this source?

0