Recently I wanted to simulate “slow” requests against a web server in order to investigate how high latency requests impacted the system overall. After some initial experimentation with writing my own code (in python), I started looking around for other solutions. I wanted to be able to use Multi-Mechanize, one of my favorite tools for [...]
Just a heads up that I finished a piece about batch acknowledgements with the pipeline pattern with zeromq on the Aggregate Knowledge blog.
Just dropping a quick note that Aggregate Knowledge, where I work as Service Delivery Data Architect, has released our first open source release this week – zeromq input and output plugins for rsyslog. Give them a spin if it’s something you’re interested in! You can find them at the Aggregate Knowledge ZeroMQ Rsyslog Plugin repository [...]
Today I had to do some fast analysis on a data warehouse to see how much the database was growing each day. The PostgreSQL database was date partitioned in a fairly standard way: each partitioned table had tables with the naming convention “tablename_YYYY_MM_DD”, and these tables were located in a schema named “partitions”. There were [...]
I worked up one more example, in this case, using PL/Ruby rather than PL/PgSQL… and then using yaml to serialize the NEW record returned by the trigger, send it to the client, and reconstitute it as a hash. The PL/Ruby stored proc: And, the test client…
The LISTEN / NOTIFY functionality of PostgreSQL received a large boost to functionality in the 9.0 release. Since I’ve lately been experimenting with JRuby, I decided to write up some example code of utilizing LISTEN / NOTIFY via JDBC. LISTEN / NOTIFY basically provides a publish / subscribe message bus within PostgreSQL. Clients can register [...]
Awhile back, I had the pleasure of contributing a small amount of code to Corey Goldberg’s excellent multi-mechanize web performance and load testing framework. Since I have been experimenting with using zeromq in place of the python Queue object for communication between processes and threads, I decided that multi-mechanize would be an excellent code base [...]
I spent part of the day playing with eventlet today. Eventlet recently added zeromq support. Here’s a very simple message forwarder that sends line terminated messages sent over TCP to a zeromq push socket, using the zeromq hub:
As a quick follow up to my previous post, here’s a look at the performance of passing messages between two python processes using the Queue class vs using 0mq push / pull connections. As a quick test, we will pass 10 million messages between two processes, first using Queue, then using 0mq. Multiprocessing test with [...]
Recently, I’ve begun exploring the depths of ZeroMQ, and the pyzmq bindings. This post is not an introduction to ZeroMQ, but for a basic rundown the “0MQ in A Hundred Words” blurb from the project site suffices: ØMQ (ZeroMQ, 0MQ, zmq) looks like an embeddable networking library but acts like a concurrency framework. It gives you [...]
Aggregate Knowledge Blog
