Background

Despite the dramatic decrease in data storage costs in recent years, the explosion in demand for high-speed data transmission has been even more dramatic. In short, compression technology is more important than ever, and time series data will continue to garner a hefty share of the attention.

My company, Stability Systems LLC, develops highly optimized commercial time series compression technology used in enterprise solutions for business and finance. When RandomWalkCodec (RWC) was originally created the goal was to use it in-house to achieve the best possible space and time performance possible for correlated time series data.

The evolution of RWC was organic as opposed to deliberate. It started as a relatively simple algorithm to save space on cramped server SCSI drives. After a number years of reliable service I revisited the code to see what improvements could be made. Eventually this led to significant performance enhancements.

Recently, while preparing related patent applications, I discovered simple patterns in the code that might be relevant and useful to other developers. After scouring the web to see if those patterns were already in widespread use for similar purposes, I was quite surprised to find absolutely nothing.

That is when I decided that a simple open source framework might be a worthwhile contribution.

After refactoring the code to get at the boilerplate “scaffolding”, what was left showed promise as a standalone framework for building alternative algorithms. In fact, it was so useful that I decided to rework RWC so that it depends on the framework instead of the other way around!



Last edited Jun 10, 2015 at 8:45 AM by bstabile, version 9