summaryrefslogtreecommitdiff
path: root/posts/2013-01-10-carbons-manhole.org
diff options
context:
space:
mode:
authorFranck Cuny <franck.cuny@gmail.com>2016-08-10 14:33:04 -0700
committerFranck Cuny <franck.cuny@gmail.com>2016-08-10 20:17:56 -0700
commit8d7d02f42c3947f756c18cb4d37d9d97fbd0d27d (patch)
treea6cecddaaea7e87d901a6c28bebe3a531438f24b /posts/2013-01-10-carbons-manhole.org
parentMerge branch 'convert-to-org' (diff)
downloadlumberjaph-8d7d02f42c3947f756c18cb4d37d9d97fbd0d27d.tar.gz
convert back to md
Diffstat (limited to 'posts/2013-01-10-carbons-manhole.org')
-rw-r--r--posts/2013-01-10-carbons-manhole.org69
1 files changed, 0 insertions, 69 deletions
diff --git a/posts/2013-01-10-carbons-manhole.org b/posts/2013-01-10-carbons-manhole.org
deleted file mode 100644
index 003d529..0000000
--- a/posts/2013-01-10-carbons-manhole.org
+++ /dev/null
@@ -1,69 +0,0 @@
-We're rolling out Graphite and statsd at [[http://saymedia.com][work]],
-and I've spend some time debugging our setup. Most of the time, the only
-thing I need is =tcpdump= to verify that a host is sending correctly the
-various metrics.
-
-But today, thanks to a
-[[http://if.andonlyif.net/blog/2013/01/the-case-of-the-disappearing-metrics.html][stupid
-reason]], I've learned about another way to debug
-[[http://graphite.readthedocs.org/en/latest/carbon-daemons.html][carbon]]:
-the manhole. The idea of the manhole is to give you a access to a REPL
-attached to the live process. When my boss told me about it, I was at
-first surprised to see this in a Python application. I've already been
-exposed to this kind of debugging thanks to Clojure, where it's not
-uncommon to connect a REPL to your live application (for example, Heroku
-[[https://devcenter.heroku.com/articles/debugging-clojure][document how
-to connect to a remote live REPL in your application]]). When I first
-heard of that I was very skeptical (give access to a /live/ environment,
-and let the developer mess with the process ?!). But I've learned to
-love it and I feel naked when I'm working in an environment where this
-is not available. So I was happy to jump and take a look at that
-feature.
-
-Since it's not very well documented and I had a hard time finding some
-information, let me share here the basics.
-
-First you'll need to configure Carbon's to allow the connection:
-
-#+BEGIN_EXAMPLE
- ENABLE_MANHOLE = True # by default it's set to False
- MANHOLE_INTERFACE = 127.0.0.1
- MANHOLE_PORT = 7222
- MANHOLE_USER = admin
- MANHOLE_PUBLIC_KEY = <your public SSH key, the string, not the path to the key>
-#+END_EXAMPLE
-
-Now you can restart carbon, and connect to the Python shell with
-=ssh admin@127.0.0.1 -p7222=. This manhole is useful to get an idea of
-the data structure your process is handling, or to get an idea of what's
-going on (is there a lot of keys being held in memory? Is the queue size
-for one metric huge? etc).
-
-From here, you can execute Python code to examine the data of the
-process:
-
-#+BEGIN_SRC python
- >>> from carbon.cache import MetricCache
- >>> print MetricCache['PROD.apps.xxx.yyy.zzz]
- [(1357861603.0, 93800.0), (1357861613.0, 98200.0), (1357861623.0, 91900.0)]
-#+END_SRC
-
-The
-[[https://github.com/graphite-project/carbon/blob/master/lib/carbon/cache.py#L19][=MetricCache=]]
-class is a Python dictionary where you can access your keys. You can
-also list all the metrics with the size of their queue with
-=MetricCache.counts()=.
-
-Or even force the daemon to write to disk all the data points:
-
-#+BEGIN_SRC python
- >>> from carbon.writer import writeCachedDataPoints
- >>> writeCachedDataPoints()
-#+END_SRC
-
-Before doing any of that, I would recommend to read the code of carbon.
-It's pretty short and quiet straight forward, especially the code of the
-[[https://github.com/graphite-project/carbon/blob/master/lib/carbon/writer.py][writer]].
-
-Of course, you have to know what you're doing when you're executing code
-from a REPL in a live environment.