[metrics-user] Reservoir implementation using HdrHistogram

If you’re using Histograms to capture latency information as many of us no doubt are, you may be interested in my Reservoir implementation backed by HdrHistogram:
https://bitbucket.org/marshallpierce/hdrhistogram-metrics-reservoir

The short version for why you want this: if your measurements are measuring something that doesn’t follow a normal distribution (e.g. latency in computer systems), then the built-in reservoir types don’t do a good job capturing that behavior. For the gory details, see this thread:https://groups.google.com/forum/#!msg/mechanical-sympathy/I4JfZQ1GYi8/ocuzIyC3N9EJ



One of issues I have had with HdrHistogram is it not being thread safe, and the synchronized version is very slow. Looking at the source I don't see anything that provides thread safety or atomicity (like the metrics lib one).  Has this been tested with multiple thread access? Would be interesting to take the jmh benchmark and set groups to have more threads reading/writing.




You're correct that HdrHistogram is itself not thread-safe; that's why this implementation uses a Recorder. This supports concurrent writes as new values are recorded: http://hdrhistogram.github.io/HdrHistogram/JavaDoc/org/HdrHistogram/Recorder.html

There is a plain (not thread safe) HdrHistogram used in the Reservoir impl in addition to the Recorder described above, but it is only accessed within a synchronized method that is only hit when taking a snapshot. Thus, the fast path (recording new values) does not hit the lock at all, and we don't really care that much about snapshot performance since it's far less frequent. We need a lock of some sort here since we're swapping the intervalHistogram, so we use the same lock to manage the running totals histogram as well.

There are multi-threaded benchmarks already inhttps://bitbucket.org/marshallpierce/metrics-reservoir-benchmark; feel free to twiddle the thread numbers, especially if you have access to a machine with lots of cores!

Awesome! Is this safe for production use? And which versions of Metrics does it support?

Looking forward to trying this out....

It's built against metrics 3.1.0, but the only types it uses from there are Reservoir and Snapshot which haven't changed in a while, so you could probably use it against 3.0 without problems.

While I haven't deployed it anywhere yet, I would feel comfortable doing so when the opportunity arises (${dayjob}'s metrics rollout isn't very far yet). I don't know of any reason why you couldn't use it in production. Take a look at the tests and see if you can come up with anything that isn't sufficiently covered. :)