From 2d2a43f200b88627253f2906fbae87cef7c1e8ce Mon Sep 17 00:00:00 2001 From: Franck Cuny Date: Thu, 4 Aug 2016 11:12:37 -0700 Subject: Mass convert all posts from markdown to org. --- posts/2013-01-10-carbons-manhole.org | 69 ++++++++++++++++++++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 posts/2013-01-10-carbons-manhole.org (limited to 'posts/2013-01-10-carbons-manhole.org') diff --git a/posts/2013-01-10-carbons-manhole.org b/posts/2013-01-10-carbons-manhole.org new file mode 100644 index 0000000..003d529 --- /dev/null +++ b/posts/2013-01-10-carbons-manhole.org @@ -0,0 +1,69 @@ +We're rolling out Graphite and statsd at [[http://saymedia.com][work]], +and I've spend some time debugging our setup. Most of the time, the only +thing I need is =tcpdump= to verify that a host is sending correctly the +various metrics. + +But today, thanks to a +[[http://if.andonlyif.net/blog/2013/01/the-case-of-the-disappearing-metrics.html][stupid +reason]], I've learned about another way to debug +[[http://graphite.readthedocs.org/en/latest/carbon-daemons.html][carbon]]: +the manhole. The idea of the manhole is to give you a access to a REPL +attached to the live process. When my boss told me about it, I was at +first surprised to see this in a Python application. I've already been +exposed to this kind of debugging thanks to Clojure, where it's not +uncommon to connect a REPL to your live application (for example, Heroku +[[https://devcenter.heroku.com/articles/debugging-clojure][document how +to connect to a remote live REPL in your application]]). When I first +heard of that I was very skeptical (give access to a /live/ environment, +and let the developer mess with the process ?!). But I've learned to +love it and I feel naked when I'm working in an environment where this +is not available. So I was happy to jump and take a look at that +feature. + +Since it's not very well documented and I had a hard time finding some +information, let me share here the basics. + +First you'll need to configure Carbon's to allow the connection: + +#+BEGIN_EXAMPLE + ENABLE_MANHOLE = True # by default it's set to False + MANHOLE_INTERFACE = 127.0.0.1 + MANHOLE_PORT = 7222 + MANHOLE_USER = admin + MANHOLE_PUBLIC_KEY = +#+END_EXAMPLE + +Now you can restart carbon, and connect to the Python shell with +=ssh admin@127.0.0.1 -p7222=. This manhole is useful to get an idea of +the data structure your process is handling, or to get an idea of what's +going on (is there a lot of keys being held in memory? Is the queue size +for one metric huge? etc). + +From here, you can execute Python code to examine the data of the +process: + +#+BEGIN_SRC python + >>> from carbon.cache import MetricCache + >>> print MetricCache['PROD.apps.xxx.yyy.zzz] + [(1357861603.0, 93800.0), (1357861613.0, 98200.0), (1357861623.0, 91900.0)] +#+END_SRC + +The +[[https://github.com/graphite-project/carbon/blob/master/lib/carbon/cache.py#L19][=MetricCache=]] +class is a Python dictionary where you can access your keys. You can +also list all the metrics with the size of their queue with +=MetricCache.counts()=. + +Or even force the daemon to write to disk all the data points: + +#+BEGIN_SRC python + >>> from carbon.writer import writeCachedDataPoints + >>> writeCachedDataPoints() +#+END_SRC + +Before doing any of that, I would recommend to read the code of carbon. +It's pretty short and quiet straight forward, especially the code of the +[[https://github.com/graphite-project/carbon/blob/master/lib/carbon/writer.py][writer]]. + +Of course, you have to know what you're doing when you're executing code +from a REPL in a live environment. -- cgit v1.2.3