Storing EOD Market Data

I've been working on extending my own market scanner, and I've revamped the way I store massive amounts of "end of day" historical stock data.

Previously I was using an organized hierarchy of compressed binary files containing a symbol's quotes. Since I'm starting to work solely with Python/SciPy/NumPy, I decided to look at PyTables. PyTables is a hierarchical database package to efficiently manage large amounts of data using HDF5. HDF5 is basically just a nice file format for storing a lot of scientific data.

I'm currently maintaining different PyTables files for the NASDAQ, NYSE, AMEX, and CME. I'm grouping stocks by sector, and creating an individual PyTables table in each group for each stock. The setup has been working immensely well, and I would highly recommend it.

OpenTick Python API

I use OpenTick as a nice source for historic and real time market data when testing any of my new trading agents or systems. I've recently started doing a lot of work in Python, and I was a bit frustrated to see that OpenTick did not provide a Python API. I searched on the OpenTick forums and discovered that RussT programmed a nice wrapper for the provided OpenTick C++ dll.

The wrapper he distributed was compiled for Python23, so I took it upon myself to recompile everything for Python25. I figured this would be of interest to other developers, so you can grab the binaries here:

http://hoo.la/f/sQTZ8_opentick-python.zip

The archive contains a nice README, the original credits, and a good example. The original source archive can be found here:

http://mysite.verizon.net/rla3rd/python-api.zip

It's alive.

I've resurrected this blog once again! I'll try my best to post regularly and provide some informative content.