Boston Linux & UNIX was originally founded in 1994 as part of The Boston Computer Society. We meet on the third Wednesday of each month at the Massachusetts Institute of Technology, in Building E51.

BLU Discuss list archive


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Discuss] No-SQL Database Recommendation?



Reporting back results...

As I mentioned earlier, I decided to try SQLAlchemy on top of Postgres, 
and I finally found some time today to try it out. After some googling 
and playing around, it looks like it is going to work. I have cobbled 
together hunks of demo code that do all the key things I need.

I also put together a little test program that creates a million records 
and then does a few of my little limited queries (where my criteria 
match effectively the entire table, but are count-limited to just a few).

It is fast. Queries are pretty much constant time (that is, independent 
of table size) taking well under a millisecond on a cheap x86 box. 
Building my table of fake data I discovered that doing a commit every 4k 
records is pretty fast and doesn't use terribly much memory, and that 
averages well under a millisecond each, also. Looking at top while 
building my table of fake data, the CPU is split 60:40 with my python 
program taking the bigger chunk of the time. There are certainly ways to 
improve my code, but with this crude program I am keeping postgres 
fairly busy, all from the comfort of python. Not bad.

At the moment I am waiting on building a table that is bigger than RAM, 
then I'll see what the penalty is to hop around in it with my queries, 
but so far SQLAlchemy looks good and is already much faster than was 
mongodb.

Thanks for all the help,

-kb, the Kent who wishes his weekends were longer, but he has hopes the 
coming Boston snowstorm with give him some bonus time.




BLU is a member of BostonUserGroups
BLU is a member of BostonUserGroups
We also thank MIT for the use of their facilities.

Valid HTML 4.01! Valid CSS!



Boston Linux & Unix / webmaster@blu.org