Best way to store historical data

[Adam G]

I'm working on a fun little project here at home and was wondering what you guys use to store historical data? I've never worked with databases, but now seems like a good time to start. I'm storing minute by minute historical data on a handful of stocks and their options contracts. Right now I'm just pulling the data off IB's historical servers and storing it in an Excel spreadsheet. This isn't going to scale up that easily. The goal is to retrieve the data and possibly run some intermediate calculations on it and graph the raw or calculated data.

Would an Access database be sufficient for my needs or should I lean more towards SQL?

 

[satyag]

If it is just storage and retrial, I would suggest using BerkelyDB. Searches will be limited to single key that you define. It lets you store raw structures but the api is also raw.
If you need SQL access and transactional integrity, but dont like to deal with installing and maintaining RDBMS process, I recommend SQLLite.
Both are embedded databases - meaning you dont need an external database process running. Both are portable and have interface bindings available for most of the commonly used languages.

 

[Andy Nguyen]

You may want to refer to Richard's post on how tick data is stored and retrieved.
https://www.quantnet.com/threads/analyzing-ultra-high-frequency-data-sets.2442/

 

[alain]

If you want to stored historical data as in Time Series data, a column oriented database seems to be the way to go. However, you can use BDB as well.

Here is a link to a post to an article posted by somebody who uses BDB and Esper to stored tick data.

http://www.jroller.com/mrettig/entry/time_series_database_with_berkeley