diff options
| author | Franck Cuny <franckcuny@gmail.com> | 2016-07-02 20:06:31 -0700 |
|---|---|---|
| committer | Franck Cuny <franckcuny@gmail.com> | 2016-07-02 20:06:31 -0700 |
| commit | 4b8e43f75b394a4e6169884fbfb4c606865c6a22 (patch) | |
| tree | 48cae6b8e8f9b68cae29676d8a15cb3ddbfcccda /content/post | |
| parent | Stop using Jekyll. (diff) | |
| download | lumberjaph-4b8e43f75b394a4e6169884fbfb4c606865c6a22.tar.gz | |
Import migration from Jekyll to Hugo.
All the posts were converted, and the layout is created. This looks like
it works just fine.
Diffstat (limited to '')
99 files changed, 7159 insertions, 0 deletions
diff --git a/content/post/2008-06-14-how-to-use-vim-as-a-personal-wiki.md b/content/post/2008-06-14-how-to-use-vim-as-a-personal-wiki.md new file mode 100644 index 0000000..b847e1c --- /dev/null +++ b/content/post/2008-06-14-how-to-use-vim-as-a-personal-wiki.md @@ -0,0 +1,48 @@ +--- +date: 2008-06-14T00:00:00Z +summary: In which I describe how I use vim as a personal wiki. +title: On how to use vim as a personal wiki +--- + +There is different reasons to want a personal wiki on your machine: + +* privacy +* having it everywhere + +I've tested a few wikis engines, like [tiddlywiki](http://tiddlywiki.com/), but I've found nothing that was really what I wanted. The main inconveniance is the need to use a webbrowser. A browser is not a text processor, so it's really painfull to use them for writing. + +I've started to try to use vim as wiki. Why would I want to use something like vim for this ? well, it's plain text (easy to grep, or to write script for manipulating data), application independent, it's a real text processor, you can customize it, and most importantly, I know how to use it, ... + +I've got a **wiki** directory in my home directory, with all my files in it. I use git to track versions of it (you can use svn if you prefer, there is no difference for this usage). In my .vimrc, i've added this instruction: `set exrc`. + +In my wiki directory, i've got another .vimrc with some specific mapping: + +```vim +map ,I <esc>:e index.mkd <cr> +map ,T <esc>:e todo.mkd <cr> +map ,S <esc>:e someday.mkd <cr> +map ,c <esc>:s/^ /c/<cr> +map ,w <esc>:s/^ /w/<cr> +map ,x <esc>:s/^ /x/<cr> +map gf :e <cfile>.mkd<cr> " open page +map <backspace> :bp<cr> +imap \date <c-R>=strftime("%Y-%m-%d")<cr> +set tabstop=2 " Number of spaces <tab> counts for. +set shiftwidth=2 " Unify +set softtabstop=2 " Unify +``` + +I organize my files in directory. I've got a **work**, **lists**, **recipes**, **misc**, ... and I put my files in this directory. + +I've got an index page, with links to main section. I don't have wikiword in camelcase or things like that, so if i want to put a link to a page, I just wrote the link this way **dir_name/page_name**, then, i juste have to hit `gf` on this link to open the page. I also use this place as a todo list manager. I've got one paragrah per day, like this : + + 2008-06-14 + - [@context] task 1 + - [@context] task 2 + ... + +and a bunch of vim mapping for marking complete (`,c`), work in progress (`,w`) or canceled (`,x`). + +If i don't have a deadline for a particular task, I use a 'someday' file, where the task is put with a context. + +The good things with markdown, is that the syntax is easy to use, and it's easy to convert to HTML. diff --git a/content/post/2008-06-17-vim-function-for-creating-new-task.md b/content/post/2008-06-17-vim-function-for-creating-new-task.md new file mode 100644 index 0000000..535c6ec --- /dev/null +++ b/content/post/2008-06-17-vim-function-for-creating-new-task.md @@ -0,0 +1,22 @@ +--- +date: 2008-06-17T00:00:00Z +summary: In which I add a few functions for my vim wiki. +title: Vim function for creating new task +--- + +I've added a new function to my .vimrc for creating quickly a new task: + +```vim +function! CreateTask() + let context = input("Enter context: ") + exe ":set noautoindent" + exe "normal 0" + exe "normal o \<tab>- [@".context."] " + exe ":set autoindent" + exe ":startinsert" +endfunction +``` + +and then this mapping: `map ct <esc>:call CreateTask()<cr>` + +Now, I've just to hit `,n` and type my context. A new line will be inserted and I just have to create my task. diff --git a/content/post/2008-06-18-keep-your-zshrc-simple.md b/content/post/2008-06-18-keep-your-zshrc-simple.md new file mode 100644 index 0000000..f5c1293 --- /dev/null +++ b/content/post/2008-06-18-keep-your-zshrc-simple.md @@ -0,0 +1,35 @@ +--- +date: 2008-06-18T00:00:00Z +summary: In which I explain how I maintain my zsh configuration. +title: keep your zshrc simple +--- + +Keep your .zshrc simple. Mine looks like this : + +```vim +autoload -U compinit zrecompile +zsh_cache=${HOME}/.zsh_cache +mkdir -p $zsh_cache +compinit -d $zsh_cache/zcomp-$HOST +for f in ~/.zshrc $zsh_cache/zcomp-$HOST; do + zrecompile -p $f && rm -f $f.zwc.old +done +setopt extended_glob +for zshrc_snipplet in ~/.zsh.d/S[0-9][0-9]*[^~] ; do + source $zshrc_snipplet +done +function history-all { history -E 1 } +``` + +and then, in my **.zsh.d** directory, I've got: + + S10_zshopts + S20_environment + S30_binds + S40_completion + S50_aliases + S60_prompt + S71_ssh + S72_git + +All my aliases are in the same file, it's much easier to search/find/add. diff --git a/content/post/2008-06-20-mirror-cpan.md b/content/post/2008-06-20-mirror-cpan.md new file mode 100644 index 0000000..492eb9d --- /dev/null +++ b/content/post/2008-06-20-mirror-cpan.md @@ -0,0 +1,30 @@ +--- +date: 2008-06-20T00:00:00Z +summary: In which I setup a mirror of CPAN using minicpan. +title: Mirror cpan +--- + +For the last 10 months, I've been living with no internet connection at home (not on purpose, but this is another story), so I've tried to be as much as possible independent from the web. I've started to use git for being able to work off-line, I use Vim as a wiki on my computer, my blog engine for writing post off-line, ... + +As as perl developer, I use a lot the CPAN. So, I've start to mirror the CPAN on my computer. Here is how: + +First, you will need the minicpan: `cpan CPAN::Mini`. + +Then, edit a **.minicpanrc** file and add the following: + +```sh +local: /path/to/my/mirror/cpan +remote: ftp://ftp.demon.co.uk/pub/CPAN/ +``` + +And to finish, add this in your crontab: + +```sh +5 14 * * * /usr/local/bin/minicpan > /dev/null 2>&1 +``` + +Everyday, at 14h05, your cpan will be updated. + +Now use the CPAN cli: `sudo cpan` and execute the following command `cpan[1]> o conf urllist unshift file:///path/to/my/mirror/cpan` + +And voilà, I've got my own minicpan on my computer, so I can install everything when I need it, being off-line or not. diff --git a/content/post/2008-06-21-debug-your-dbix-class-queries.md b/content/post/2008-06-21-debug-your-dbix-class-queries.md new file mode 100644 index 0000000..7270d82 --- /dev/null +++ b/content/post/2008-06-21-debug-your-dbix-class-queries.md @@ -0,0 +1,21 @@ +--- +date: 2008-06-21T00:00:00Z +summary: In which I explain how to see SQL queries generated for DBIx::Class. +title: debug your DBIx::Class queries +--- + +If you use DBIx::Class and want to see what the SQL generated looks like, you can set the environment variable **DBIC_TRACE**. + +```sh +% DBIC_TRACE=1 my_programme.pl +``` + +And all the SQL will be printed on **STDERR**. + +If you give a filename to the variable, like this + +```sh +% DBIC_TRACE="1=/tmp/sql.debug" +``` + +all the statements will be printed in this file. diff --git a/content/post/2008-06-24-ack.md b/content/post/2008-06-24-ack.md new file mode 100644 index 0000000..ec836f5 --- /dev/null +++ b/content/post/2008-06-24-ack.md @@ -0,0 +1,24 @@ +--- +date: 2008-06-24T00:00:00Z +summary: In which I share my settings for ack. +title: Ack +--- + +> Ack is designed as a replacement for 99% of the uses of grep. + +[Ack](https://metacpan.org/module/App::Ack) is a really nice tool for searching your source code. It's faster than grep because he already knows what you want : searching your sources files :) + +By default it will not search in SCM files (.git, .svn, ...), backups files (source.pl~, source.pl.bak, ...). You can specify what kind of files you want (`--perl`, `--cc`, ...), make it match some regex with `--match`, ... + +And you can set some defaults configuration in a .ackrc file ! Mine looks like this: + +```sh +--sort-files +--color +--context=1 +--follow +``` + +Check also: [vim with ack integration](http://use.perl.org/use.perl.org/_Ovid/journal/36430.html). + +Oh, and it's the only program with `--thpppt` option! diff --git a/content/post/2008-06-26-git-branch-everywhere.md b/content/post/2008-06-26-git-branch-everywhere.md new file mode 100644 index 0000000..5667e67 --- /dev/null +++ b/content/post/2008-06-26-git-branch-everywhere.md @@ -0,0 +1,38 @@ +--- +date: 2008-06-26T00:00:00Z +summary: In which I share a snippet of code to display a git branch in vim. +title: Git branch everywhere +--- + +The current trend is to have the name of the current git branch everywhere. Personnaly I display it in my vim's status bar, and in my zsh prompt. + +Here is my vimrc configuration for this (I'm not the author of this function, and can't remember where I saw it first): + +```vim +set statusline=%<[%n]%m%r%h%w%{'['.(&fenc!=''?&fenc:&enc).':'.&ff}%{g:gitCurrentBranch}%{']'}%y\ %F%=%l,%c%V%8P +autocmd BufEnter * :call CurrentGitBranch() + +let g:gitCurrentBranch = '' +function! CurrentGitBranch() + let cwd = getcwd() + cd %:p:h + let branch = matchlist(system('/usr/local/git/bin/git branch -a --no-color'), '\v\* (\w*)\r?\n') + execute 'cd ' . cwd + if (len(branch)) + let g:gitCurrentBranch = '][git:' . branch[1] . '' + else + let g:gitCurrentBranch = '' + endif + return g:gitCurrentBranch +endfunction +``` + +and my zshrc: + +```vim +local git_b +git_b='$(get_git_prompt_info '%b')' +PROMPT="%(?..%U%?%u:) $git_b %40>...<%/%(#.%U>%u.%B>%b) " +``` + +with the following script [S55_git](http://www.jukie.net/~bart/conf/zsh.d/S55_git). diff --git a/content/post/2008-06-27-dotfiles-and-scm.md b/content/post/2008-06-27-dotfiles-and-scm.md new file mode 100644 index 0000000..c6b32e1 --- /dev/null +++ b/content/post/2008-06-27-dotfiles-and-scm.md @@ -0,0 +1,45 @@ +--- +date: 2008-06-27T00:00:00Z +summary: In which I share how I manage my dotfiles. +title: Dotfiles and SCM +--- + +All my dotfiles are stored in a SCM. Most of the time I'm on my main computer, but I can be working on a server or a different workstation. In this case, I like to have all my configurations for zsh, vim, screen, etc. + +So, instead of copying my files over different computers, I put everything in a private repostiroy, and when I'm on a new computer, I just have to checkout it. If I do a modification on a machine, I just need to commit it, and I can have the modification everywhere else. + +I've got a $HOME/dotfiles directory, which is versionned (with git in my case). All my configurations file are stored here. + +In this directory, as I'm avery lazy person, I've created a Makefile. Each time I create a new file, I add it to the makefile at the same time. The content of the Makefile is the following: + +```make +DOTFILES := $(shell pwd) +all: shell code perl plagger web +shell: + ln -fs $(DOTFILES)/zshrc ${HOME}/.zshrc + ln -fns $(DOTFILES)/zsh.d ${HOME}/.zsh.d + ln -fs $(DOTFILES)/inputrc ${HOME}/.inputrc + ln -fs $(DOTFILES)/screenrc ${HOME}/.screenrc + ln -fns $(DOTFILES)/screen ${HOME}/.screen + ln -fs $(DOTFILES)/profile ${HOME}/.profile + ln -fs $(DOTFILES)/gnupg ${HOME}/.gnupg code: + ln -fs $(DOTFILES)/vimrc ${HOME}/.vimrc + ln -fs $(DOTFILES)/gvimrc ${HOME}/.gvimrc + ln -fns $(DOTFILES)/vim ${HOME}/.vim + ln -fs $(DOTFILES)/ackrc ${HOME}/.ackrc + ln -fs $(DOTFILES)/gitignore ${HOME}/.gitignore + ln -fs $(DOTFILES)/gitconfig ${HOME}/.gitconfig + ln -fs $(DOTFILES)/psqlrc ${HOME}/.psqlrc perl: + ln -fs $(DOTFILES)/proverc ${HOME}/.proverc + ln -fs $(DOTFILES)/pause ${HOME}/.pause + ln -fs $(DOTFILES)/perltidyrc ${HOME}/.perltidyrc + ln -fns $(DOTFILES)/module-starter ${HOME}/.module-starter plagger: + ln -fns $(DOTFILES)/plagger ${HOME}/.plagger web: + ln -fns $(DOTFILES)/irssi ${HOME}/.irssi + ln -fns $(DOTFILES)/vimperator ${HOME}/.vimperator + ln -fs $(DOTFILES)/vimperatorrc ${HOME}/.vimperatorrc + ln -fs $(DOTFILES)/flickrrc ${HOME}/.flickrrc + ln -fs $(DOTFILES)/rtorrent.rc ${HOME}/.rtorrent.rc +``` + +So next time I want to deploy my dotfiles on a new computer, I can run `make all` or `make perl code vim` and I can start coding some perl with vim. diff --git a/content/post/2008-06-30-upgrading-to-perl-5.10.md b/content/post/2008-06-30-upgrading-to-perl-5.10.md new file mode 100644 index 0000000..b4ad719 --- /dev/null +++ b/content/post/2008-06-30-upgrading-to-perl-5.10.md @@ -0,0 +1,25 @@ +--- +date: 2008-06-30T00:00:00Z +summary: In which we upgrade to Perl 5.10. +title: Upgrading to perl 5.10 +--- + +Get the list of your installed 5.8 modules: + +```sh +% perl -MExtUtils::Installed -e'print join("\n", new ExtUtils::Installed->modules)' > module.list +``` + +then install Perl 5.10: + +```sh +% wget http://www.cpan.org/src/perl-5.10.0.tar.gz +% tar xzf perl-5.10.0.tar.gz +% cd perl-5.10.0 +% sh Configure -de -Dprefix=/opt/perl -Duserelocatableinc +% make && make test +% sudo make install +% /opt/perl/bin/perl -e 'use feature qw(say); say "hi"' +``` + +and then re-install your modules with <code>cpan \`cat module.list\`</code>. diff --git a/content/post/2008-08-08-customize-your-mysql-prompt.md b/content/post/2008-08-08-customize-your-mysql-prompt.md new file mode 100644 index 0000000..801a7fe --- /dev/null +++ b/content/post/2008-08-08-customize-your-mysql-prompt.md @@ -0,0 +1,14 @@ +--- +date: 2008-08-08T00:00:00Z +summary: In which we customize our MySQL prompt +title: Customize your MySQL prompt +--- + +To customize your MySQL prompt, create a .my.cnf file in your $HOME then add the following: + +```sh +[mysql] +prompt="\\u [\\d] >" +``` + +Your prompt will now looks like this: `username [dabatases_name] >` diff --git a/content/post/2008-08-19-offlineimap-on-osx.md b/content/post/2008-08-19-offlineimap-on-osx.md new file mode 100644 index 0000000..61d01ba --- /dev/null +++ b/content/post/2008-08-19-offlineimap-on-osx.md @@ -0,0 +1,28 @@ +--- +date: 2008-08-19T00:00:00Z +summary: In which I have to patch offline imap to make it work on OS X. +title: offlineimap on osx +--- + +If you are using offlineimap on leopard, on an imap connection with ssl (like GMail) and it keep crashing because of the following error: + +```sh +File "/Library/Python/2.5/site-packages/offlineimap/imaplibutil.py", line 70, in _read +return self.sslsock.read(n) +MemoryError +``` + +you can fix it with this fix: + +```sh +sudo vim /Library/Python/2.5/site-packages/offlineimap/imaplibutil.py +70 +``` + +then, comment line 70 and add this line + +```python +return self.sslsock.read(min(n, 16384)) +#return self.sslsock.read(n) +``` + +you can read a description of the bug <a href="http://bugs.python.org/issue1389051">here</a>. diff --git a/content/post/2008-12-05-vim-and-git.md b/content/post/2008-12-05-vim-and-git.md new file mode 100644 index 0000000..ac09231 --- /dev/null +++ b/content/post/2008-12-05-vim-and-git.md @@ -0,0 +1,43 @@ +--- +date: 2008-12-05T00:00:00Z +summary: In which I share another snippet of code for vim. +title: vim and git +--- + +idea from [Ovid's journal](http://use.perl.org/use.perl.org/_Ovid/journal/37966.html) (ovid is full of really good ideas for vim): + +to get a quick git diff in my vim session, put this in your .vimrc + +```vim +map ,gh :call SourceDiff() " gh for git history + +function! SourceDiff() + let filename = bufname("%") + let command = 'git log -5 --pretty=format:"%h - (%ar) %an - %s" "'.filename.'"' + let result = split( system(command), "\n" ) + + if empty(result) + echomsg("No past revisions for " . filename) + return + endif + + " get the list of files + let revision = PickFromList('revision', result) + + if strlen(revision) + let items = split(revision, " ") + execute '!git diff ' . items[0] . ' -- "' . filename .'" | less' + endif +endfunction +``` + +the output looks like this: + + Choose a revision: + 1: ea0bb4d - (3 days ago) franck cuny - fix new_freq + 2: a896ac7 - (5 weeks ago) franck cuny - fix typo + 3: c9bc5fd - (5 weeks ago) franck cuny - update test + 4: e9de4be - (5 weeks ago) franck cuny - change the way we rewrite and check an existing url + 5: 3df1fd6 - (7 weeks ago) franck cuny - put id category + +You choose the revision you want to check the diff against, and you got a (colorless) diff in your vim buffer. diff --git a/content/post/2009-02-17-tidify-a-json-in-vim.md b/content/post/2009-02-17-tidify-a-json-in-vim.md new file mode 100644 index 0000000..9e1cbf3 --- /dev/null +++ b/content/post/2009-02-17-tidify-a-json-in-vim.md @@ -0,0 +1,15 @@ +--- +date: 2009-02-17T00:00:00Z +summary: In which we tidify a JSON in vim. +title: tidify a json in vim +--- + +If you have to edit json files from vim, you may want to make them more readable, here is how you can do this: + +start by installing the JSON::XS perl module from the CPAN by running `sudo cpan JSON::XS`, then, edit your .vimrc and add the following + +```vim +map <leader>jt <Esc>:%!json_xs -f json -t json-pretty<CR> +``` + +now while editing a json file, you can hit `,jt` (or whatever your leader is set to) and tidify a json. diff --git a/content/post/2009-03-08-belgian-perl-workshop-09.md b/content/post/2009-03-08-belgian-perl-workshop-09.md new file mode 100644 index 0000000..975e89b --- /dev/null +++ b/content/post/2009-03-08-belgian-perl-workshop-09.md @@ -0,0 +1,21 @@ +--- +date: 2009-03-08T00:00:00Z +summary: In which I went to the Belgian Perl Workshop. +title: Belgian Perl Workshop 09 +--- + +last weekend my co-workers and I went to the [Belgian Perl Workshop 09](http://conferences.mongueurs.net/bpw2009/). I attended the following presentations: + + * [KiokuDB](http://conferences.mongueurs.net/bpw2009/talk/1720), by nothingmuch. Slides are available [here](http://www.iinteractive.com/kiokudb/talks/bpw2009.xul). We were able to talk with him during the afternoon, we might we use it at [$work](http://rtgi.fr). + + * [Painless XSLT with Perl](http://conferences.mongueurs.net/bpw2009/talk/1740), by andrew shitov. Was interesting, even if I don't do any XSLT anymore. Again, some ideas might be used for work. + + * [What are you pretending to be ?](http://conferences.mongueurs.net/bpw2009/talk/1792), by liz. That's a hell of a hack. The module is available on the [CPAN](http://search.cpan.org/~elizabeth/persona/). + + * [Regular Expressions and Unicode Guru](http://conferences.mongueurs.net/bpw2009/event/473), by abigail. Feel better to know that I'm not the only one suffering with unicode in Perl ;). Learn some stuff like how to create a custom character classe, etc. + + * [Catalyst](http://conferences.mongueurs.net/bpw2009/event/474), by matt trout. Ok, we're using Catalyst at work for our webservices. So we allready know about catalyst, but we were curious. And as i was hoping, we learn some nice tweaks. We discovered [Catalyst::Model::Adaptor](http://search.cpan.org/perldoc?Catalyst::Model::Adaptor), so we don't have to do some horrible stuff in our Controller any more, and some other interesting stuff were put in this talk. And matt is a really good speaker, manage to keep an audiance amused and interested. + + * [Catalyst & AWS](http://conferences.mongueurs.net/bpw2009/event/476), by matt trout. Once again, a really good talk by matt. Some good advices, and a lot of fun. + +We didn't stay long for the social event in the evening; we had booked a hotel in bruxelles. But i'm glad that we were able to get to this perl workshop, it was well-organised, good talks, meet nice people, and learn some stuff. All in all, a good day :) diff --git a/content/post/2009-04-14-git-and-prove.md b/content/post/2009-04-14-git-and-prove.md new file mode 100644 index 0000000..e7585f6 --- /dev/null +++ b/content/post/2009-04-14-git-and-prove.md @@ -0,0 +1,31 @@ +--- +date: 2009-04-14T00:00:00Z +summary: In which I add a hook to git to run my tests. +title: Git and prove +--- + +A little trick to force you to run your tests before a commit: + +in a repositorie, create the following file **.git/hooks/pre-commit** with this content: + +```sh +#!/bin/sh +if [ -d t ]; then + res=`prove t` + if [ $? -gt 0 ]; then + echo "tests fails" + exit 1 + fi +fi +if [ -d xt ]; then + res=`prove xt` + if [ $? -gt 0 ]; then + echo "tests fails" + exit 1 + fi +fi +``` + +and don't forget to chmod with +x. + +Now, when you will do your next commit, your test suit will be executed. If the tests fails, the commit will be rejected. diff --git a/content/post/2009-04-25-controll-xmms2-from-vim.md b/content/post/2009-04-25-controll-xmms2-from-vim.md new file mode 100644 index 0000000..fccdc8a --- /dev/null +++ b/content/post/2009-04-25-controll-xmms2-from-vim.md @@ -0,0 +1,17 @@ +--- +date: 2009-04-25T00:00:00Z +summary: In which I control xmms2 from vim. +title: controll xmms2 from vim +--- + +a really basic way to controll xmms2 from your vim session: + +```vim +map <leader>xn <Esc>:!xmms2 next<CR><CR> +map <leader>xb <Esc>:!xmms2 previous<CR><CR> +map <leader>xP <Esc>:!xmms2 pause<CR><CR> +map <leader>xp <Esc>:!xmms2 play<CR><CR> +map <leader>xs <Esc>:!xmms2 stop<CR><CR> +``` + +now, type `,xn` in vim, and xmms2 will start to play the next track from your playlist. diff --git a/content/post/2009-04-27-a-simple-feed-aggregator-with-modern-perl-part-1.md b/content/post/2009-04-27-a-simple-feed-aggregator-with-modern-perl-part-1.md new file mode 100644 index 0000000..7226d0d --- /dev/null +++ b/content/post/2009-04-27-a-simple-feed-aggregator-with-modern-perl-part-1.md @@ -0,0 +1,137 @@ +--- +date: 2009-04-27T00:00:00Z +summary: In which I write a feed aggregator in Perl. +title: A simple feed aggregator with modern Perl - part 1 +--- + +Following [Matt's post](http://www.shadowcat.co.uk/blog/matt-s-trout/iron-man/) about people not blogging enough about Perl, I've decided to try to post once a week about Perl. So I will start by a series of articles about what we call **modern Perl**. For this, I will write a simple feed agregator (using [Moose](https://metacpan.org/pod/Moose), [DBIx::Class](http://search.cpan.org/perldoc?DBIx::Class), [KiokuDB](http://search.cpan.org/perldoc?KiokuDB), some tests, and a basic frontend (with [Catalyst](http://search.cpan.org/perldoc?Catalyst)). This article will be split in four parts: + + * the first one will explain how to create a schema using **DBIx::Class** + * the second will be about the aggregator. I will use **Moose*** and **KiokuDB** + * the third one will be about writing tests with **Test::Class** + * the last one will focus on **Catalyst** + +The code of these modules will be available on [my git server](http://git.lumberjaph.net/) at the same time each article is published. + +> I'm not showing you how to write the perfect feed aggregator. The purpose of this series of articles is only to show you how to write a simple aggregator using modern Perl. + +### The database schema + +We will use a database to store a list of feeds and feed entries. As I don't like, no, wait, I *hate* SQL, I will use an ORM for accessing the database. For this, my choice is **DBIx::Class**, the best ORM available in Perl. + +> If you never have used an ORM before, ORM stands for Object Relational Mapping. It's a SQL to OO mapper that creates an abstract encapsulation of your databases operations. **DBIx::Class**' purpose is to represent "queries in your code as perl-ish as possible. + +For a basic aggregator we need: + + * a table for the list of feeds + * a table for the entries + +We will create these two tables using *DBIx::Class*. For this, we first create a Schema module. I use *Module::Setup*, but you can use **Module::Starter** or whatever you want. + +```bash +% module-setup MyModel +% cd MyModel +% vim lib/MyModel.pm +``` + +```perl +package MyModel; +use base qw/DBIx::Class::Schema/; +__PACKAGE__->load_classes(); +1; +``` + +So, we have just created a schema class. The **load_classes** method loads all the classes that reside under the **MyModel** namespace. We now create the result class **MyModel::Feed** in **lib/MyModel/Feed.pm**: + +```perl +package MyModel::Feed; +use base qw/DBIx::Class/; +__PACKAGE__->load_components(qw/Core/); +__PACKAGE__->table('feed'); +__PACKAGE__->add_columns(qw/ feedid url /); +__PACKAGE__->set_primary_key('feedid'); +__PACKAGE__->has_many(entries => 'MyModel::Entry', 'feedid'); +1; +``` + +Pretty self explanatory: we declare a result class that uses the table feed, with two columns: **feedid** and **url**, **feedid** being the primary key. The **has_many** method declares a one-to-many relationship. + +Now the result class **MyModel::Entry** in **lib/MyModel/Entry.pm**: + +```perl +package MyModel::Entry; +use base qw/DBIx::Class/; +__PACKAGE__->load_components(qw/Core/); +__PACKAGE__->table('entry'); +__PACKAGE__->add_columns(qw/ entryid permalink feedid/); +__PACKAGE__->set_primary_key('entryid'); +__PACKAGE__->belongs_to(feed => 'MyModel::Feed', 'feedid'); +1; +``` + +Here we declare **feed** as a foreign key, using the column name **feedid**. + +You can do a more complex declaration of your schema. Let's say you want to declare the type of your fields, you can do this: + +```perl +__PACKAGE__->add_columns( + 'permalink' => { + 'data_type' => 'TEXT', + 'is_auto_increment' => 0, + 'default_value' => undef, + 'is_foreign_key' => 0, + 'name' => 'url', + 'is_nullable' => 1, + 'size' => '65535' + }, +); +``` + +**DBIx::Class** also provides hooks for the deploy command. If you are using MySQL, you may need a InnoDB table. In your class, you can add this: + +```perl +sub sqlt_deploy_hook { + my ($self, $sqlt_table) = @_; + $sqlt_table->extra( + mysql_table_type => 'InnoDB', + mysql_charset => 'utf8' + ); +} +``` + +Next time you call deploy on this table, the hook will be sent to **SQL::Translator::Schema**, and force the type of your table to InnoDB, and the charset to utf8. + +Now that we have a **DBIx::Class** schema, we need to deploy it. For this, I always do the same thing: create a **bin/deploy_mymodel.pl** script with the following code: + +```perl +use strict; +use feature 'say'; +use Getopt::Long; +use lib('lib'); +use MyModel; + +GetOptions( + 'dsn=s' => \my $dsn, + 'user=s' => \my $user, + 'passwd=s' => \my $passwd +) or die usage(); + +my $schema = MyModel->connect($dsn, $user, $passwd); +say 'deploying schema ...'; +$schema->deploy; + +say 'done'; + +sub usage { + say + 'usage: deploy_mymodel.pl --dsn $dsn --user $user --passwd $passwd'; +} +``` + +This script will deploy for you the schema (you need to create the database first if using with mysql). + +Executing the following command `perl bin/deploy_mymodel.pl --dsn dbi:SQLite:model.db` will generate a **model.db** database so we can work and test it. Now that we got our (really) simple **MyModel** schema, we can start to hack on our aggregator. + +[The code is available on my git server](http://git.lumberjaph.net/p5-ironman-mymodel.git/). + +> while using **DBIx::Class**, you may want to take a look at the generated queries. For this, export `DBIC_TRACE=1` in your environment, and the queries will be printed on STDERR. diff --git a/content/post/2009-04-28-a-simple-feed-aggregator-with-modern-perl-part-2.md b/content/post/2009-04-28-a-simple-feed-aggregator-with-modern-perl-part-2.md new file mode 100644 index 0000000..dd99f27 --- /dev/null +++ b/content/post/2009-04-28-a-simple-feed-aggregator-with-modern-perl-part-2.md @@ -0,0 +1,251 @@ +--- +date: 2009-04-28T00:00:00Z +summary: In which we continue to write a feed aggregator in Perl. +title: A simple feed aggregator with modern Perl - part 2 +--- + +> I've choose to write about a feed aggregator because it's one of the things I'm working on at [RTGI](http://rtgi.eu/) (with web crawler stuffs, gluing datas with search engine, etc) + +For the feed aggregator, I will use **Moose**, **KiokuDB** and our **DBIx::Class** schema. Before we get started, I'd would like to give a short introduction to Moose and KiokuDB. + +Moose is a "A postmodern object system for Perl 5". Moose brings to OO Perl some really nice concepts like roles, a better syntax, "free" constructor and destructor, ... If you don't already know Moose, check [it here](http://www.iinteractive.com/moose/) for more information. + +KiokuDB is a Moose based frontend to various data stores [...] Its purpose is to provide persistence for "regular" objects with as little effort as possible, without sacrificing control over how persistence is actually done, especially for harder to serialize objects. [...] KiokuDB is meant to solve two related persistence problems: + +* Store arbitrary objects without changing their class definitions or worrying about schema details, and without needing to conform to the limitations of a relational model. +* Persisting arbitrary objects in a way that is compatible with existing data/code (for example interoperating with another app using **CouchDB** with **JSPON** semantics). + +I will store each feed entry in KiokuDB. I could have chosen to store them as plain text in JSON files, in my DBIx::Class model, etc. But as I want to show you new and modern stuff, I will store them in Kioku using the DBD's backend. + +### And now for something completely different, code! + +First, we will create a base module named **MyAggregator**. + +```bash +% module-setup MyAggregator +``` + +We will now edit **lib/MyAggregator.pm** and write the following code: + +```perl +package MyAggregator; +use Moose; +1; +``` + +As you can see, there is no `use strict; use warnings` here: Moose automatically turns on these pragmas. We don't have to write the new method either, as it's provided by Moose. + +For parsing feeds, we will use **XML::Feed**, and we will use it in a Role. If you don't know what roles are: + +> Roles have two primary purposes: as interfaces, and as a means of code reuse. Usually, a role encapsulates some piece of behavior or state that can be shared between classes. It is important to understand that roles are not classes. You cannot inherit from a role, and a role cannot be instantiated. + +So, we will write our first role, **lib/MyAggregator/Roles/Feed.pm**: + +```perl +package MyAggregator::Roles::Feed; +use Moose::Role; +use XML::Feed; +use feature 'say'; + +sub feed_parser { + my ($self, $content) = @_; + my $feed = eval { XML::Feed->parse($content) }; + if ($@) { + my $error = XML::Feed->errstr || $@; + say "error while parsing feed : $error"; + } + $feed; +} +1; +``` + +This one is pretty simple. It will read a content, try to parse it, and return a XML::Feed object. If it can't parse the feed, the error will be shown, and the result will be set to undef. + +Now, a second role will be used to fetch the feed, and do basic caching, **lib/MyAggregator/Roles/UserAgent.pm**: + +```perl +package MyAggregator::Roles::UserAgent; +use Moose::Role; +use LWP::UserAgent; +use Cache::FileCache; +use URI; + +has 'ua' => ( + is => 'ro', + isa => 'Object', + lazy => 1, + default => sub { LWP::UserAgent->new(agent => 'MyUberAgent'); } +); +has 'cache' => ( + is => 'rw', + isa => 'Cache::FileCache', + lazy => 1, + default => + sub { Cache::FileCache->new({namespace => 'myaggregator',}); } +); + +sub fetch_feed { + my ($self, $url) = @_; + + my $req = HTTP::Request->new(GET => URI->new($url)); + my $ref = $self->cache->get($url); + if (defined $ref && $ref->{LastModified} ne '') { + $req->header('If-Modified-Since' => $ref->{LastModified}); + } + + my $res = $self->ua->request($req); + $self->cache->set( + $url, + { ETag => $res->header('Etag') || '', + LastModified => $res->header('Last-Modified') || '' + }, + '5 days', + ); + $res; +} +1; +``` + +This role has 2 attributes: **ua** and **cache**. The **ua** attribute is our UserAgent. 'lazy' means that it will not be constructed until I call `$self->ua->request`. + +I use **Cache::FileCache** for doing basic caching so I don't fetch or parse the feed if it's unnecessary, and I use the Etag and Last-Modified header to check the validity of my cache. + +The only method of this role is **fetch_feed**. It will fetch an URL if it's not already in the cache, and return a **HTTP::Response** object. + +Now, I create an Entry class in **lib/MyAggregator/Entry.pm**: + +```perl +package MyAggregator::Entry; +use Moose; +use Digest::SHA qw(sha256_hex); +has 'author' => (is => 'rw', isa => 'Str'); +has 'content' => (is => 'rw', isa => 'Str'); +has 'title' => (is => 'rw', isa => 'Str'); +has 'id' => (is => 'rw', isa => 'Str'); +has 'date' => (is => 'rw', isa => 'Object'); +has 'permalink' => ( + is => 'rw', + isa => 'Str', + required => 1, + trigger => sub { + my $self = shift; + $self->id(sha256_hex $self->permalink); + } +); +1; +``` + +Here the **permalink** has a trigger attribute: each entry has a unique **ID**, constructed with a sha256 value from the **permalink**. So, when we fill the **permalink** accessor, the **ID** is automatically set. + +We can now change our **MyAggregator** module like this: + +```perl +package MyAggregator; +use feature ':5.10'; +use MyModel; +use Moose; +use MyAggregator::Entry; +use KiokuDB; +use Digest::SHA qw(sha256_hex); +with 'MyAggregator::Roles::UserAgent', 'MyAggregator::Roles::Feed'; + +has 'context' => (is => 'ro', isa => 'HashRef'); +has 'schema' => ( + is => 'ro', + isa => 'Object', + lazy => 1, + default => sub { MyModel->connect($_[0]->context->{dsn}) }, +); +has 'kioku' => ( + is => 'rw', + isa => 'Object', + lazy => 1, + default => sub { + my $self = shift; + KiokuDB->connect($self->context->{kioku_dir}, create => 1); + } +); + +sub run { + my $self = shift; + + my $feeds = $self->schema->resultset('Feed')->search(); + while (my $feed = $feeds->next) { + my $res = $self->fetch_feed($feed->url); + if (!$res || !$res->is_success) { + say "can't fetch " . $feed->url; + } + else { + $self->dedupe_feed($res, $feed->id); + } + } +} + +sub dedupe_feed { + my ($self, $res, $feed_id) = @_; + + my $feed = $self->feed_parser(\$res->content); + return if (!$feed); + foreach my $entry ($feed->entries) { + next + if $self->schema->resultset('Entry') + ->find(sha256_hex $entry->link); + my $meme = MyAggregator::Entry->new( + permalink => $entry->link, + title => $entry->title, + author => $entry->author, + date => $entry->issued, + content => $entry->content->body, + ); + + + $self->kioku->txn_do( + scope => 1, + body => sub { + $self->kioku->insert($meme->id => $meme); + } + ); + $self->schema->txn_do( + sub { + $self->schema->resultset('Entry')->create( + { entryid => $meme->id, + permalink => $meme->permalink, + feedid => $feed_id, + } + ); + } + ); + } +} +1; +``` + + +* the with function composes roles into a class. So my MyAggregator class has a fetch\_feed and parse\_feed methods, and all the attributes of our roles +* context is a HashRef that contains the configuration +* schema is our MyModel schema + * kioku is a connection to our kiokudb backend + +Two methods in this object: `run` and `dedupe`. + +The `run` method gets the list of feeds (line 28, via the `search`). For each feed return by the search, we try to fetch it, and if it's successful, we dedupe the entries. To dedupe the entries, we check if the permalink is alread in the database (line 45, via the `find`). If we already have this entry, we skip this one, and do the next one. If it's a new entry, we create a **MyAggregator::Entry** object, with the content, date, title, ... we store this object in kiokudb (line 55, we create a transaction, and do our insertion in the transaction), and create a new entry in the MyModel database (line 61, we enter in transaction too, and insert the entry in the database). + +And to run this, a little script: + +```perl +use strict; +use MyAggregator; +use YAML::Syck; +my $agg = MyAggregator->new(context => LoadFile shift); +$agg->run; +``` + +so we can run our aggregator like this `perl bin/aggregator.pl conf.yaml` + +And it's done :) We got a really basic aggregator now. If you want to improve this one, you would like to improve the dedupe process, using the permalink, the date and/or the title, as this one is too much basic. In the next article we will write some tests for this aggregator using Test::Class. + +big thanks to [tea](http://bunniesincyberspace.wordpress.com/) and [blob](http://code.google.com/p/tinyaml/) for reviewing and fixing my broken english in the first 2 parts. + +[The code is available on git server](http://git.lumberjaph.net/p5-ironman-myaggregator.git/). + +Part 3 and 4 next week. diff --git a/content/post/2009-05-04-rtgi-and-perl-conferences.md b/content/post/2009-05-04-rtgi-and-perl-conferences.md new file mode 100644 index 0000000..cb73700 --- /dev/null +++ b/content/post/2009-05-04-rtgi-and-perl-conferences.md @@ -0,0 +1,7 @@ +--- +date: 2009-05-04T00:00:00Z +summary: In which I go to a few conferences. +title: RTGI and Perl conferences +--- + +<a href="http://rtgi.fr">RTGI</a> will be one of the sponsors of the <a href="http://conferences.mongueurs.net/fpw2009/">French Perl Workshop 2009</a>. I (or Camille, not sure yet) will also give a <a href="http://conferences.mongueurs.net/fpw2009/talk/1934">talk</a> about the Perl and CPAN community on the web. Camille (for sure this time) will also give do this talk at the <a href="http://yapceurope2009.org/ye2009/">YAPC::EU</a> at Lisbon this summer <a href="http://yapceurope2009.org/ye2009/talk/2061">(and in english this time)</a>. diff --git a/content/post/2009-05-06-a-simple-feed-aggregator-with-modern-perl-part-3.md b/content/post/2009-05-06-a-simple-feed-aggregator-with-modern-perl-part-3.md new file mode 100644 index 0000000..c260318 --- /dev/null +++ b/content/post/2009-05-06-a-simple-feed-aggregator-with-modern-perl-part-3.md @@ -0,0 +1,259 @@ +--- +date: 2009-05-06T00:00:00Z +summary: In which we continue to write our feed aggregator. +title: A simple feed aggregator with modern Perl - part 3 +--- + +Now that we have our aggregator, we have to write our tests. For this I will use Test::Class. Ovid have wrote an [excellent](http://www.modernperlbooks.com/mt/2009/03/organizing-test-suites-with-testclass.html) [serie](http://www.modernperlbooks.com/mt/2009/03/reusing-test-code-with-testclass.html) [of](http://www.modernperlbooks.com/mt/2009/03/making-your-testing-life-easier.html) [articles](http://www.modernperlbooks.com/mt/2009/03/using-test-control-methods-with-testclass.html) [about Test::Class](http://www.modernperlbooks.com/mt/2009/03/working-with-testclass-test-suites.html). You should really read this, because I will not enter in details. + +We have two things to test: + + * roles + * aggregator + +### Roles + +For this, we create the following files: + + * t/tests/Test/TestObject.pm + * t/tests/Test/MyRoles.pm + * t/tests/Test/MyAggregator.pm + * t/run.t + +We will write our **run.t**: + +```perl +use lib 't/test'; +use Test::MyRoles; +Test::Class->runtests; +``` + +this test load our tests and run them. + +This is a just a class for the tests, that load our 2 roles. + +now the roles' tests: + +```perl +package Test::MyRoles; + +use strict; +use warnings; +use base 'Test::Class'; +use Test::Exception; +use Test::More; + +sub class {'Test::TestObject'} + +sub url {"http://lumberjaph.net/blog/index.php/feed/"} + +sub startup : Tests(startup => 1) { + my $test = shift; + use_ok $test->class, "use ok"; + `rm -rf /tmp/FileCache/myaggregator/`; +} + +sub constructor : Tests(1) { + my $test = shift; + can_ok $test->class, 'new'; +} + +sub fetch_feed : Tests(5) { + my $test = shift; + can_ok $test->class, 'fetch_feed'; + + ok my $obj = $test->class->new(), '... object is created'; + my $res = $obj->fetch_feed($test->url); + is $res->code, "200", "... fetch is a success"; + like $res->content, qr/lumberjaph/, "... and content is good"; + + # now data should be in cache + my $ref = $obj->cache->get($test->url); + ok defined $ref, "... url is now in cache"; + + $res = $obj->fetch_feed($test->url); + is $res->code, "304", "... already in cache"; +} + +sub feed_parser : Tests(3) { + my $test = shift; + can_ok $test->class, 'feed_parser'; + + my $ua = LWP::UserAgent->new; + my $res = $ua->get($test->url); + ok my $obj = $test->class->new(), "... object is created"; + my $feed = $obj->feed_parser(\$res->content); + isa_ok $feed, "XML::Feed::Format::RSS"; +} + +1; +``` + +As you can see, some methods have an attribute, which indicate this method as a test method. + +The startup method is run as the first method each time the tests are executed. In our case, we test if we can load our class, and we delete the cache of the aggregator. + +We have a "constructor" test, that check we can do a new on our class. + +Now we have to tests our 2 methods from the roles. We will test the fetch_feed method first. + +First, we indicate the number of tests that will be executed (6 in our case). Then we can write the test in the method: + +* create an object +* fetch an url, and test the HTTP code of the response +* check if the content look like something we want +* now the data should be in cache, and the a new fetch of the url should return a 304 HTTP code + +The second method to test is feed_parser. This method will do 3 tests. + +* create an object +* we manually fetch the content from a feed +* send this content to feed_parser +* the result should return a XML::Feed::Format::RSS object + +When you run the tests now `prove t/run.t` + +the following result is produced: + +```perl +t/run.t .. +1..11 +ok 1 - use Test::TestObject; +# +# Test::MyRoles->constructor +ok 2 - Test::TestObject->can('new') +# +# Test::MyRoles->feed_parser +ok 3 - Test::TestObject->can('feed_parser') +ok 4 - ... object is created +ok 5 - The object isa XML::Feed::Format::RSS +# +# Test::MyRoles->fetch_feed +ok 6 - Test::TestObject->can('fetch_feed') +ok 7 - ... object is created +ok 8 - ... fetch is a success +ok 9 - ... and content is good +ok 10 - ... url is now in cache +ok 11 - ... already in cache +ok +All tests successful. +Files=1, Tests=11, 3 wallclock secs ( 0.03 usr 0.01 sys + 0.66 cusr 0.09 csys = 0.79 CPU) +Result: PAS +``` + +### Aggregator + +As we have our tests for the roles, we can write the tests for the aggregator now. First, we add a new line in **t/run.t**. + +```perl +use Test::MyAggregator +``` + +We edit our **t/tests/Test/MyAggregator.pm**: + +```perl +package Test::MyAggregator; + +use strict; +use warnings; +use base 'Test::Class'; +use Test::Exception; +use Test::More; + +sub class {'MyAggregator'} + +sub context { + { dsn => 'dbi:SQLite:dbname=/tmp/myaggregator.db', + kioku_dir => 'dbi:SQLite:/tmp/mykioku.db', + }; +} + +sub startup : Tests(startup => 2) { + my $test = shift; + use_ok $test->class, "use ok"; + `touch /tmp/myaggregator.db`; + my $context = $test->context; + my $dsn = $context->{dsn}; + my $schema = MyModel->connect($dsn); + $schema->deploy; + + ok $schema->resultset('Feed')->create( + { feedid => 1, + url => 'http://lumberjaph.net/blog/index.php/feed/', + } + ), + "... insert one feed in the db"; +} + +sub shutdown : Tests(shutdown => 2) { + my $test = shift; + ok unlink '/tmp/myaggregator.db', '... unlink db test'; + ok unlink '/tmp/mykioku.db', '... unlink kioku test'; +} + +sub constructor : Tests(1) { + my $test = shift; + can_ok $test->class, 'new'; +} + +sub dedupe_feed : Tests(4) { + my $test = shift; + + my $context = $test->context; + my $ua = LWP::UserAgent->new; + my $res = $ua->get("http://lumberjaph.net/blog/index.php/feed/"); + + ok my $obj = $test->class->new(context => $context), + "... MyAggregator created"; + + $obj->dedupe_feed($res, 1); + + my $schema = MyModel->connect($context->{dsn}); + is $schema->resultset('Entry')->search()->count, 10, + '... 10 entries in the db'; + + my $first = $schema->resultset('Entry')->search()->first; + my $res_kiokudb; + $obj->kioku->txn_do( + scope => 1, + body => sub { + $res_kiokudb = $obj->kioku->lookup($first->id); + } + ); + + ok $res_kiokudb, '... got an object'; + is $res_kiokudb->permalink, $first->permalink, '... content is valid'; +} + +1; +``` + +The startup test create a database from our model, and insert a feed. The shutdown test remove the 2 database that we will use (MyModel and kiokudb). + +The dedupe_feed is really simple. We create a MyAggregator object, fetch a rss feed manually, and dedup the result. Now we check the result in the MyModel database: do we have 10 entries ? if it's the case, we are good. We fetch a result from this db, and check if it's also present in KiokuDB, and if the permalink is the same for the two. So with 4 tests, we do a simple check of our class. + +Execute the tests (you can comment the roles' tests in run.t): + +```perl +t/run.t .. +1..9 +ok 1 - use MyAggregator; +ok 2 - ... insert one feed in the db +# +# Test::MyAggregator->constructor +ok 3 - MyAggregator->can('new') +# +# Test::MyAggregator->dedupe_feed +ok 4 - ... MyAggregator created +ok 5 - ... 10 entries in the db +ok 6 - ... got an object +ok 7 - ... content is valid +ok 8 +ok 9 +ok +All tests successful. +Files=1, Tests=9, 3 wallclock secs ( 0.01 usr 0.01 sys + 1.39 cusr 0.12 csys = 1.53 CPU) +Result: PASS +``` + +We have our tests, so next step is the Catalyst frontend. As for the precedents parts, [the code is available on my git server](http://git.lumberjaph.net/p5-ironman-myaggregator.git/). diff --git a/content/post/2009-05-13-a-simple-feed-aggregator-with-modern-perl-part-4.md b/content/post/2009-05-13-a-simple-feed-aggregator-with-modern-perl-part-4.md new file mode 100644 index 0000000..129a3b6 --- /dev/null +++ b/content/post/2009-05-13-a-simple-feed-aggregator-with-modern-perl-part-4.md @@ -0,0 +1,185 @@ +--- +date: 2009-05-13T00:00:00Z +summary: In which we reach the conclusion on how to write a feed aggregator. +title: A simple feed aggregator with modern Perl - part 4 +--- + +We have the model, the aggregator (and some tests), now we can do a basic frontend to read our feed. For this I will create a webapp using [Catalyst](http://www.catalystframework.org). + +[Catalyst::Devel](http://search.cpan.org/perldoc?Catalyst::Devel) is required for developping catalyst application, so we will install it first: + +```perl +% cpan Catalyst::Devel +``` + +Now we can create our catalyst application using the helper: + +```perl +% catalyst.pl MyFeedReader +``` + +This command initialise the framework for our application **MyFeedReader**. A number of files are created, like the structure of the MVC directory, some tests, helpers, ... + +We start by creating a view, using [TTSite](http://search.cpan.org/perldoc?Catalyst::View::TT). TTSite generate some templates for us, and the configuration for this template. We will also have a basic CSS, a header, footer, etc. + +```bash +cd MyFeedReader +perl script/myfeedreader_create.pl view TT TTSite +``` + +TTSite files are under **root/src** and **root/lib**. A **MyAggregator/View/TT.pm** file is also created. We edit it to make it look like this: + +```perl +__PACKAGE__->config({ + INCLUDE_PATH => [ + MyFeedReader->path_to( 'root', 'src' ), + MyFeedReader->path_to( 'root', 'lib' ) + ], + PRE_PROCESS => 'config/main', + WRAPPER => 'site/wrapper', + ERROR => 'error.tt2', + TIMER => 0, + TEMPLATE_EXTENSION => '.tt2', +}); +``` + +Now we create our first template, in **root/src/index.tt2** + +```html +to <a href="/feed/">your feeds</a> +``` + +If you start the application (using `perl script/myfeedreader_server.pl`) and point your browser on http://localhost:3000/, this template will be rendered. + +We need two models, one for KiokuDB and another one for MyModel: + +**lib/MyFeedReader/Model/KiokuDB.pm** + +```perl +package MyFeedReader::Model::KiokuDB; +use Moose; +BEGIN { extends qw(Catalyst::Model::KiokuDB) } +1; +``` + +we edit the configuration file (**myfeedreader.conf**), and set the dsn for our kiokudb backend + +```xml + <Model KiokuDB> + dsn dbi:SQLite:../MyAggregator/foo.db + </Model> +``` + +**lib/MyFeedReader/Model/MyModel.pm** + +```perl +package MyFeedReader::Model::MyModel; +use base qw/Catalyst::Model::DBIC::Schema/; +1; +``` + +and the configuration: + +```xml +<Model MyModel> + connect_info dbi:SQLite:../MyModel/model.db + schema_class MyModel +</Model> +``` + +We got our view and our model, we can do the code for the controller. We need 2 controller, one for the feed, and one for the entries. The Feed controller will list them and display entries titles for a given feed. The Entry controller will just display them. + +**lib/MyFeedReader/Controller/Feed.pm** + +```perl +package MyFeedReader::Controller::Feed; +use strict; +use warnings; +use parent 'Catalyst::Controller'; + +__PACKAGE__->config->{namespace} = 'feed'; + +sub index : Path : Args(0) { + my ( $self, $c ) = @_; + $c->stash->{feeds} + = [ $c->model('MyModel')->resultset('Feed')->search() ]; +} + +sub view : Chained('/') : PathPart('feed/view') : Args(1) { + my ( $self, $c, $id ) = @_; + $c->stash->{feed} + = $c->model('MyModel')->resultset('Feed')->find($id); +} + +1; +``` + +The function `index` list the feeds, while the function `view` list the entries for a give feed. We use the chained action mechanism to dispatch this url, so we can have urls like this **/feed/\*** + +We create our 2 templates (for index and view): + +**root/src/feed/index.tt2** + +```html +<ul> + [% FOREACH feed IN feeds %] + <li><a href="/feed/view/[% feed.id %]">[% feed.url %]</a></li> + [% END %] +</ul> +``` + +**root/src/feed/vew.tt2** + +```html +<h1>[% feed.url %]</h1> + +<h3>entries</h3> +<ul> + [% FOREACH entry IN feed.entries %] + <li><a href="/entry/[% entry.id %]">[% entry.permalink %]</a></li> + [% END %] +</ul> +``` + +If you point your browser to http://localhost:3000/feed/ you will see this: + +<img src='/imgs/list_feed.webp' alt='list feeds'> + +Now the controller for displaying the entries: + +```perl +package MyFeedReader::Controller::Entry; +use strict; +use warnings; +use MyAggregator::Entry; +use parent 'Catalyst::Controller'; + +__PACKAGE__->config->{namespace} = 'entry'; + +sub view : Chained('/') : PathPart('entry') : Args(1) { + my ( $self, $c, $id ) = @_; + $c->stash->{entry} = $c->model('KiokuDB')->lookup($id); +} + +1; +``` + +The function **view** fetch an entry from the kiokudb backend, and store it in the stash, so we can use it in our template. + +**root/src/entry/view.tt2** + +```html +<h1><a href="[% entry.permalink %]">[% entry.title %]</a></h1> +<span>Posted [% entry.date %] by [% entry.author %]</span> +<div id="content"> + [% entry.content %] +</div> +``` + +If you point your browser to an entry (something like **http://localhost:3000/entry/somesha256value**), you will see an entry: + +<img src='/imgs/show_entry.webp' alt='show entry'> + +Et voila, we are done with a really basic feed reader. You can add methods to add or delete feed, mark an entry as read, ... + +[The code is available on my git server](http://git.lumberjaph.net/p5-ironman-myfeedreader.git/). diff --git a/content/post/2009-05-18-a-simple-feed-aggregator-with-modern-perl-part-4.1.md b/content/post/2009-05-18-a-simple-feed-aggregator-with-modern-perl-part-4.1.md new file mode 100644 index 0000000..5486ec3 --- /dev/null +++ b/content/post/2009-05-18-a-simple-feed-aggregator-with-modern-perl-part-4.1.md @@ -0,0 +1,156 @@ +--- +date: 2009-05-18T00:00:00Z +summary: In which we had one more article on how to write a feed aggregator. +title: A simple feed aggregator with modern Perl - part 4.1 +--- + +You can thanks [bobtfish](http://github.com/bobtfish) for being such a pedantic guy, 'cause now you will have a better chained examples. He forked my repository from GitHub and fix some code that I'll explain here. + +### lib/MyFeedReader.pm + +```perl + package MyFeedReader; ++use Moose; ++use namespace::autoclean; + +-use strict; +-use warnings; +- +-use Catalyst::Runtime '5.70'; ++use Catalyst::Runtime '5.80'; + +-use parent qw/Catalyst/; ++extends 'Catalyst'; +``` + +You can see that he use [Moose](http://search.cpan.org/perldoc?Moose), so we can remove + +```perl +use strict; +use warnings; +``` + +and have a more elegant way to inherit from [Catalyst](http://search.cpan.org/perldoc?Catalyst) with + +```perl +extends 'Catalyst'; +``` + +instead of + +```perl +use parent qw/Catalyst/; +``` + +He also have updated the **Catalyst::Runtime** version, and added **namespace::autoclean**. The purpose of this module is to keep imported methods out of you namespace. Take a look at the "documentation":http://search.cpan.org/perldoc?namespace::autoclean, it's easy to understand how and why it's usefull. + +### lib/MyFeedReader/Controller/Root.pm + +```perl +-use strict; +-use warnings; +-use parent 'Catalyst::Controller'; ++use Moose; ++use namespace::autoclean; ++BEGIN { extends 'Catalyst::Controller' } + +-sub index :Path :Args(0) { ++sub root : Chained('/') PathPart() CaptureArgs(0) {} ++ ++sub index : Chained('root') PathPart('') Args(0) { + my ( $self, $c ) = @_; + + # Hello World + $c->response->body( $c->welcome_message ); + } + +-sub default :Path { ++sub default : Private { + my ( $self, $c ) = @_; + $c->response->body( 'Page not found' ); + $c->response->status(404); +``` + +A new method, `root`, that will be the root path for our application. All our methods will be chained from this action. If start you catalyst server and go to **http://localhost:3000/** you will be served with the Catalyst's welcome message as before. + +### lib/MyFeedReader/Controller/Entry.pm + +```perl +-use warnings; ++use Moose; + use MyAggregator::Entry; +-use parent 'Catalyst::Controller'; +- +-__PACKAGE__->config->{namespace} = 'entry'; ++use namespace::autoclean; ++BEGIN { extends 'Catalyst::Controller'; } + +-sub view : Chained('/') : PathPart('entry') : Args(1) { ++sub view : Chained('/root') : PathPart('entry') : Args(1) { + my ( $self, $c, $id ) = @_; + + $c->stash->{entry} = $c->model('KiokuDB')->lookup($id); + } + +-1; +- ++__PACKAGE__->meta->make_immutable; +``` + +We extends the **Catalyst::Controller** in a Moose way, and the `make_immutable` instruction is a Moose recommanded best practice (you can alsa add `no Moose` after the make_immutable). + +### lib/MyFeedreader/Controller/Feed.pm + +```perl ++use Moose; ++use namespace::autoclean; ++BEGIN { extends 'Catalyst::Controller' } + +-use strict; +-use warnings; +-use parent 'Catalyst::Controller'; ++sub feed : Chained('/root') PathPart('feed') CaptureArgs(0) {} + +-__PACKAGE__->config->{namespace} = 'feed'; +- +-sub index : Path : Args(0) { ++sub index : Chained('feed') PathPart('') Args(0) { + my ( $self, $c ) = @_; + + $c->stash->{feeds} + = [ $c->model('MyModel')->resultset('Feed')->search() ]; + } + +-sub view : Chained('/') : PathPart('feed/view') : Args(1) { ++sub view : Chained('feed') : PathPart('view') : Args(1) { + my ( $self, $c, $id ) = @_; + + $c->stash->{feed} + = $c->model('MyModel')->resultset('Feed')->find($id); + } + +-1; ++__PACKAGE__->meta->make_immutable; +``` + +We got `feed` which is chained to root. `index` is chained to feed, and take no arguments. This method display the list of our feeds. And we got the `view` method, chained to feed too, but with one argument, that display the content of an entry. + +If you start the application, you will see the following routes: + + + .-------------------------------------+--------------------------------------. + | Path Spec | Private | + +-------------------------------------+--------------------------------------+ + | /root/entry/* | /root (0) | + | | => /entry/view | + | /root/feed | /root (0) | + | | -> /feed/feed (0) | + | | => /feed/index | + | /root/feed/view/* | /root (0) | + | | -> /feed/feed (0) | + | | => /feed/view | + | /root | /root (0) | + | | => /index | + '-------------------------------------+--------------------------------------' + +I hope you got a better idea about chained action in catalyst now. And again, thanks to bobtfish for the code. diff --git a/content/post/2009-05-22-modules-i-like---module-setup.md b/content/post/2009-05-22-modules-i-like---module-setup.md new file mode 100644 index 0000000..a9e7cf3 --- /dev/null +++ b/content/post/2009-05-22-modules-i-like---module-setup.md @@ -0,0 +1,75 @@ +--- +date: 2009-05-22T00:00:00Z +summary: In which I share my enthusiasm for Module::Setup. +title: modules I like Module::Setup +--- + +[Module::Setup](http://search.cpan.org/perldoc?Module::Setup) by [Yappo](http://blog.yappo.jp/) is a really nice module. I don't like [Module::Starter](http://search.cpan.org/perldoc?Module::Starter), it's not easy to create template to make it do what you need. With Module::Setup you can create flavors for any type of modules you want. Most of the modules I create for work use Moose, and I like to use Test::Class too. I've created a Moose flavor for creating this kind of modules. + +### Creating a Moose flavor for Module::Setup + +First, you tell it to init a new flavor: + +```bash +module-setup --init moose +``` + +Module::Setup ask what is your favorite SCM. For me, it's git. It will create files in *$HOME/.module-setup/flavors/moose/*. + +Start by editing *$HOME/.module-setup/flavors/moose/template/lib/____var-module_path-var____.pm* to make it look like this + +```perl +- use strict; +- use warnings; ++ use Moose; +``` + +Add **requires 'Moose'** in **Makefile.PL**. Create a **t/tests/Test/____var-module_path-var____.pm** file with the following content: + +```perl +package Test :: [%module %]; + +use strict; +use warnings; +use base 'Test::Class'; +use Test::Exception; +use Test::More; + +sub class {'[% module %]'} + +sub startup : Tests(startup => 1) { + my $test = shift; + use_ok $test->class, "use ok"; +} + +sub shutdown : Tests(shutdown) { + my $test = shift; +} + +sub constructor : Tests(1) { + my $test = shift; + can_ok $test->class, 'new'; +} + +1; +``` + +You will have a Test::Class test ready with basic tests already implemented. + +If you want to share this template at $work, easy: + +```bash +module-setup --pack DevMoose moose > DevMoose.pm +``` + +You just have to send DevMoose.pm to who need it, and he will be able to import it with + +```bash +module-setup --init --flavor-class=+DevMoose moose +``` + +Now you can create a new module + +```bash +module-setup MY::AWESOME::MODULE moose +``` diff --git a/content/post/2009-05-30-catalystx-dispatcher-asgraph.md b/content/post/2009-05-30-catalystx-dispatcher-asgraph.md new file mode 100644 index 0000000..211ba51 --- /dev/null +++ b/content/post/2009-05-30-catalystx-dispatcher-asgraph.md @@ -0,0 +1,27 @@ +--- +date: 2009-05-30T00:00:00Z +summary: In which I wrote a module to visualize routes in Catalyst. +title: CatalystX::Dispatcher::AsGraph +--- + +This morning I saw [this post](http://marcus.nordaaker.com/awesome-route-graph-with-mojoxroutesasgraph/) from Marcus Ramberg about [MojoX::Routes::AsGraph](http://search.cpan.org/perldoc?MojoX::Routes::AsGraph). I liked the idea. But as I Catalyst instead of Mojo, I thought I could give a try and do the same thing for Catalyst dispatcher, and I've coded CatalystX::Dispatcher::AsGraph. For the moment only private actions are graphed. + +<img src='/imgs/routes-300x249.webp' alt='routes'> + +You use it like this: `perl bin/catalyst_graph_dispatcher.pl --appname Arkham --output routes.png` + +You can create a simple script to output as text if you prefer: + +```perl +#!/usr/bin/perl -w +use strict; +use CatalystX::Dispatcher::AsGraph; + +my $graph = CatalystX::Dispatcher::AsGraph->new_with_options(); +$graph->run; +print $graph->graph->as_txt; +``` + +The code is on [my git server](http://git.lumberjaph.net/p5-catalystx-dispatcher-asgraph.git/) for the moment. + +For thoses who are interested by visualization, I'll publish soon some (at least I think) really nice visualisations about CPAN, Perl, and his community, that we have created at [$work](http://rtgi.fr). diff --git a/content/post/2009-06-06-modules-i-like-web-scraper.md b/content/post/2009-06-06-modules-i-like-web-scraper.md new file mode 100644 index 0000000..ba383d1 --- /dev/null +++ b/content/post/2009-06-06-modules-i-like-web-scraper.md @@ -0,0 +1,100 @@ +--- +date: 2009-06-06T00:00:00Z +summary: In which I talk about Web::Scraper. +title: Modules I like Web::Scraper +--- + +For [$work](http://rtgi.fr) I need to write scrapers. It used to be boring and painful. But thanks to [miyagawa](http://search.cpan.org/~miyagawa/), this is not true anymore. [Web::Scraper](http://search.cpan.org/perldoc?Web::Scraper) offer a nice API: you can write your rules using XPath, you can chaine rules, a nice and simple syntax, etc. + +I wanted to export my data from my last.fm account but there is no API for this, so I would need to scrap them. All the data are available [as a web page](http://www.last.fm/user/franckcuny/tracks) that list your music. So the scraper need to find how many pages, and find the content on each page to extract a list of your listening. + +For the total of pages, it's easy. Let's take a look at the HTML code and search for something like this: + +```html +<a class="lastpage" href="/user/franckcuny/tracks?page=272">272</a> +``` + +the information is in a class **lastpage**. + +Now we need to find our data: I need the artist name, the song name and the date I played this song. + +All this data are in a **table**, and each new entry is in a **td**. + +```html +<tr id="r9_1580_1920248170" class="odd"> +[...] + <td class="subjectCell"> + <a href="/music/Earth">Earth</a> + <a href="/music/Earth/_/Sonar+and+Depth+Charge">Sonar and Depth Charge</a> + </td> +[...] +<td class="dateCell last"> + <abbr title="2009-05-13T15:18:25Z">13 May 3:18pm</abbr> +</td> +``` + +It's simple: information about a song are stored in **subjectcell**, and the artist and song title are each in a tag **a**. The date is in a **dateCell**, and we need the **title** from the **abbr** tag. + +The scraper we need to write is + +```perl +my $scrap = scraper { + process 'a[class="lastpage"]', 'last' => 'TEXT'; + process 'tr', 'songs[]' => scraper { + process 'abbr', 'date' => '@title'; + process 'td[class="subjectCell"]', 'song' => scraper { + process 'a', 'info[]' => 'TEXT'; + }; + } +}; +``` + +The first rule extract the total of page. The second iter on each **tr** and store the content in an array named **songs**. This **tr** need to be scraped. So we look the the **abbr** tag, and store in **date** the property **title**. Then we look for the song and artitst information. We look for the **td** with a class named **subjectCell**, a extract all links. + +Our final script will look like this: + +```perl +#!/usr/bin/perl -w +use strict; +use feature ':5.10'; + +use Web::Scraper; +use URI; +use IO::All -utf8; + +my $username = shift; +my $output = shift; + +my $scrap = scraper { + process 'a[class="lastpage"]', 'last' => 'TEXT'; + process 'tr', 'songs[]' => scraper { + process 'abbr', 'date' => '@title'; + process 'td[class="subjectCell"]', 'song' => scraper { + process 'a', 'info[]' => 'TEXT'; + }; + } +}; + +my $url = "http://www.last.fm/user/" . $username . "/tracks?page="; +scrap_lastfm(1); + +sub scrap_lastfm { + my $page = shift; + my $scrap_uri = $url . $page; + say $scrap_uri; + my $res = $scrap->scrape(URI->new($scrap_uri)); + my $lastpage = $res->{last}; + foreach my $record (@{$res->{songs}}) { + my $line = join("\t", @{$record->{song}->{info}}, $record->{date}); + $line . "\n" >> io $output; + } + $page++; + scrap_lastfm($page) if $page <= $lastpage; +} +``` + +You can use this script like this: + +```bash +% perl lastfmscraper.pl franckcuny store_data.txt +``` diff --git a/content/post/2009-06-12-shape-of-cpan.md b/content/post/2009-06-12-shape-of-cpan.md new file mode 100644 index 0000000..e467646 --- /dev/null +++ b/content/post/2009-06-12-shape-of-cpan.md @@ -0,0 +1,50 @@ +--- +date: 2009-06-12T00:00:00Z +summary: In which I talk about the shape of the CPAN +title: The shape of the CPAN +--- + +My talk at the [FPW](http://conferences.mongueurs.net/fpw2009/) this year is about the shape of the Perl and CPAN community. This talk was prepared by some [$coworkers](http://labs.rtgi.eu/) and me. + +<img src='/imgs/draft_cpan_prelimsmall.webp' alt='map of the Perl community on the web' align='right'> + +We generated two maps (authors and modules) using the CPANTS' data. For the websites, we crawled a seed generated from the CPAN pages of the previous authors. + +Each of this graphs are generated using a [force base algorithm](http://en.wikipedia.org/wiki/Force-based_algorithms), with the help of [Gephi](http://gephi.org/). + +All the map are available in PDF files, in creative common licence. The slides are in french, but I will explain the three maps here. + + * [slides (french)](http://labs.rtgi.eu/fpw09/resources/slides/) + * [authors map](http://labs.rtgi.eu/fpw09/resources/pdf/cpan_authors_core_march2009.pdf) + * [modules map](http://labs.rtgi.eu/fpw09/resources/pdf/cpan_packages_core_march2009.pdf) + * [community maps](http://labs.rtgi.eu/fpw09/resources/pdf/cpan-web-may2009-poster.pdf) + * [community map (flash version)](http://labs.rtgi.eu/fpw09/map/) + * [cpan-explorer.org](http://cpan-explorer.org/) + +### CPAN's modules + +The first map is about the modules available on the CPAN. We selected a list of modules which are listed as dependancies by at least 10 others modules, and the modules who used them. This graph is composed of 7193 nodes (or modules) and 17510 edges. Some clusters are interesting: + + * LWP and URI are really the center of the CPAN + * a lot of web modules (XML::*, TemplateToolkit, HTML::Parser, ...) + * TK is isolated from the CPAN + * Moose, DBIx::Class and Catalyst are forming a group. This data are from march, we will try to do a newer version of this map this summer. This one will be really interesting as Catalyst have switched to Moose + +### The CPAN's authors + +This map is about the authors on the CPAN. There is about 700 authors and their connections. Each time an author use a module of another author, a link is created. + + * Modern Perl, constitued by Moose, Catalyst, DBIx::Class. Important authors are Steven, Sartak, perigin, jrockway, mstrout, nothingmuch, marcus ramberg + * Slaven Rezi? and others TK developpers are on the border + +### Web map + +We crawled the web using the seed generated using the CPAN's authors pages. + + * again, the "modern group", on the top of the map, with Moose/Catalyst/DBIx::Class developpers + * some enterprises, like shadowcat and iinteractive in the middle of the "modern Perl", Booking in the middle of the YAPC's websites (they are a major sponsor of this events), 6apart, ... + * perl.org is the reference for the Perl community (the site is oriented on their sides) + * cpan.org is the reference for the open source community + * github is in the center of the Perl community. It's widely adopted by the Perl developpers. It offers all the "social media" features that are missing on the CPAN + +I hope you like this visualisations, have fun analyzing them :) diff --git a/content/post/2009-06-17-xmobar-on-debian-sid.md b/content/post/2009-06-17-xmobar-on-debian-sid.md new file mode 100644 index 0000000..e10fb26 --- /dev/null +++ b/content/post/2009-06-17-xmobar-on-debian-sid.md @@ -0,0 +1,34 @@ +--- +date: 2009-06-17T00:00:00Z +summary: In which I path xmobar for Debian SID +title: xmobar on debian SID +--- + +If you are using [xmonad](http://www.xmonad.org/) and [xmobar](http://projects.haskell.org/xmobar/) on [Debian SID](http://www.debian.org/) on a laptop, and don't see any battery information, you may have test this [solution](http://5e6n1.wordpress.com/2009/03/30/xmobar-battery-plugin-using-sysfs-not-procfs/). + +If this didn't solve your problem, try this patch on SysfsBatt.hs : + +```diff +52c52 +< let path = sysfsPath ++ p ++ "/charge_full" +--- +> let path = sysfsPath ++ p ++ "/energy_full" +62c62 +< let path = sysfsPath ++ p ++ "/charge_now" +--- +> let path = sysfsPath ++ p ++ "/energy_now" +74c74 +< else showWithColors (printf "%.2f%%") stateInPercentage +--- +> else showWithColors (printf "Batt: %.2f%%") stateInPercentage +``` + +Then as before: + +```bash +% runhaskell Setup.lhs configure --user +% runhaskell Setup.lhs build +% runhaskell Setup.lhs install --user +``` + +Battery information should be visible. diff --git a/content/post/2009-06-22-modules-i-like-getopt-long-and-moosex-getopt.md b/content/post/2009-06-22-modules-i-like-getopt-long-and-moosex-getopt.md new file mode 100644 index 0000000..10cb1b1 --- /dev/null +++ b/content/post/2009-06-22-modules-i-like-getopt-long-and-moosex-getopt.md @@ -0,0 +1,129 @@ +--- +date: 2009-06-22T00:00:00Z +summary: In which I talk about GetOpt::Long and Moosex::Getopt +title: Modules I like Getopt::Long and MooseX::Getopt +--- + +## Getopt::Long + +[Getopt::long](http://search.cpan.org/perldoc?Getopt::Long) is a useful module to parse command line arguements. + +A basic usage is something like this: + +```perl +#!/usr/bin/perl -w +use strict; +use YAML::Syck; +use Getopt::Long; + +GetOptions('config=s' => \my $cfg_file,); + +my $config = LoadFile $cfg_file +``` + +In **GetOptions**, we require a value for config with **config=s**. If we wante an integer, we replace 's' with 'i', and for a floating point, with 'f'. + +Call your script : + +```bash +% script.pl --config=file.yml #this one works +% script.pl --config file.yml #this one too! +% script.pl -c file.yml #and this one too +``` + +The three syntaxes are understood. + +A good practices is to combine this module with [Pod::Usage](http://search.cpan.org/perldoc?Pod::Usage). Let's do some modifications on the example: + +```perl +#!/usr/bin/perl -w +use strict; +use YAML::Syck; +use Getopt::Long; +use Pod::Usage; + +GetOptions('config=s' => \my $cfg_file,) or pod2usage(2); +pod2usage(2) unless @ARGV > 0; + +my $config = LoadFile $cfg_file + +__END__ +=head1 NAME + +uberscript + +=head1 SYNOPSIS + +uberscript [options] + +Options: + + --config config file + +=head1 Options + +=over 4 + +=item B<config> + +Path to the config file +``` + +then + +```bash +% perl uberscript + +Usage: + uberscript [options] + + Options: + --config config file +``` + +From now if we call our script without argument, the POD will be printed on STDIN. + +## MooseX::Getopt + +[MooseX::Getopt](http://search.cpan.org/perldoc?MooseX::Getopt) is a Role that add a `new_with_options` to your object. We create a basic Object : + +```perl +package OurShinyObject; + +use Moose; +with qw/MooseX::Getopt/; + +has 'config' => (isa => 'Str', is => 'ro', required => 1); +has 'context' => ( + isa => 'HashRef', + is => 'rw', + lazy => 1, + traits => ['NoGetopt'], + default => sub { LoadFile shift->config } +); + +... +``` + +create a script to call this object + +```perl +#!/usr/bin/perl -w +use strict; +use OurShinyObject; + +my $obj = OurShinyObject->new_from_options(); + +``` + +```sh +% script.pl --config file.yml +``` + +The role will set our attribute **context** using the value from the argument set on the command line. + +The `traits => ['NoGetopt']` indicate that this attributes will be not be read from the command line. An alternate way to do this is to prefix the attributes with **_**. + +## conclusion (?) + +When you write a script, even if you're sure you will never need to have more than one argument, or that you never will have to update the code, *please* consider to use of **Getopt::Long** instead of a `shift @ARGV`, because we all know that you will at a certain point update this script and you will more than one argument :). diff --git a/content/post/2009-06-25-how-to-prevent-some-components-to-be-loaded-by-catalyst.md b/content/post/2009-06-25-how-to-prevent-some-components-to-be-loaded-by-catalyst.md new file mode 100644 index 0000000..6b23767 --- /dev/null +++ b/content/post/2009-06-25-how-to-prevent-some-components-to-be-loaded-by-catalyst.md @@ -0,0 +1,24 @@ +--- +date: 2009-06-25T00:00:00Z +summary: In which I show how to disable some components in Catalyst. +title: How to prevent some components to be loaded by Catalyst +--- + +Let's say you have a large [Catalyst](http://search.cpan.org/perldoc?Catalyst) application, with a lot of compoments. When you deploy your application, or when you want to test it while your developping, you may not want to have some of thoses components loaded (you don't have all the dependencies, they are incompatible, etc...). Catalyst use [Module::Pluggable](http://search.cpan.org/perldoc?Module::Pluggable) to load the components, so you can easily configure this. In your application's configuration, add: + +```yaml +setup_components: + except: + - MyApp::Model::AAAA + - MyAPP::Model::BBBB::REST + ... +``` + +Module::Pluggable have some other interesting features. You may have a second Catalyst application, and want to use one or more components from this one. You can easily do this: + +```yaml +setup_components: + search_path: + - MyApp + - MyOtherApp::Model +``` diff --git a/content/post/2009-06-30-private-and-protected-methods-with-moose.md b/content/post/2009-06-30-private-and-protected-methods-with-moose.md new file mode 100644 index 0000000..a182a31 --- /dev/null +++ b/content/post/2009-06-30-private-and-protected-methods-with-moose.md @@ -0,0 +1,48 @@ +--- +date: 2009-06-30T00:00:00Z +summary: In which I show how to write dummy private methods for Moose +title: Private and protected methods with Moose +--- + +Yesterday, one of our interns asked me a question about private method in <a href="http://www.iinteractive.com/moose/">Moose</a>. I told him that for Moose as for Perl, there is no such things as private method. By convention, methods prefixed with '_' are considered private. + +But I was curious to see if it would be something complicated to implement in Moose. First, I've started to look at how the 'augment' keyword is done. I've then hacked Moose directly to add the private keyword. After asking advice to <a href="http://blog.woobling.org/">nothingmuch</a>, he recommended me that I implement this in a MooseX::* module instead. The result is <a href="http://git.lumberjaph.net/p5-moosex-methodprivate.git/">here</a>. + +From the synopsis, MooseX::MethodPrivate do: + +```perl +package Foo; +use MooseX::MethodPrivate; + +private 'foo' => sub { +}; + +protected 'bar' => sub { +}; + + +my $foo = Foo->new; +$foo->foo; # die, can't call foo because it's a private method +$foo->bar; # die, can't call bar because it's a protected method + +package Bar; +use MooseX::MethodPrivate; +extends qw/Foo/; + +sub baz { + my $self = shift; + $self->foo; #die, can't call foo because it's a private method + $self->bar; # ok, can call this method because we extends Foo and + # it's a protected method +} +``` + +I was surprised to see how easy it's to extend Moose syntax. All I've +done was this: + +```perl + Moose::Exporter->setup_import_methods( + with_caller => [qw( private protected )],); +``` + +and write the `private` and `protected` sub. I'm sure there is some stuff I can do to improve this, but for a first test, I'm happy with the result and still amazed how easy it was to add this two keywords. diff --git a/content/post/2009-07-07-cpan-and-auto-install.md b/content/post/2009-07-07-cpan-and-auto-install.md new file mode 100644 index 0000000..b877c84 --- /dev/null +++ b/content/post/2009-07-07-cpan-and-auto-install.md @@ -0,0 +1,9 @@ +--- +date: 2009-07-07T00:00:00Z +summary: In which I show how to auto answer questions asked by cpan +title: CPAN and auto-install +--- + +When you install a module from the [CPAN](http://search.cpan.org), and this module requires other modules, the cpan shell will ask you if you want to install them. When you are installing [Catalyst](http://www.catalystframework.org/), it may take a while, and you may not want to spend your afternoon in front of the prompt answering "yes" every 5 seconds. + +The solution is to set the environment variable **PERL_MM_USE_DEFAULT**. Next time you want to install a big app: `PERL_MM_USE_DEFAULT=1 cpan Catalyst KiokuDB` and your done. diff --git a/content/post/2009-07-16-cpanhq-and-dependencies-graph.md b/content/post/2009-07-16-cpanhq-and-dependencies-graph.md new file mode 100644 index 0000000..04d92cb --- /dev/null +++ b/content/post/2009-07-16-cpanhq-and-dependencies-graph.md @@ -0,0 +1,17 @@ +--- +date: 2009-07-16T00:00:00Z +summary: In which I graph dependencies for CPAN modules +title: CPANHQ and dependencies graph +--- + +CPANHQ is a new project that "aims to be a community-driven, meta-data-enhanced alternative to such sites as [search.cpan.org](http://search.cpan.org) and [kobesearch.cpan.org](http://kobesearch.cpan.org/). + +I believe that a good vizualisation can help to have a better understanding of datas. One of the missing thing on the actual search.cpan.org is the lack of informations about a distribution's dependencies. So my first contribution to the CPANHQ project was to add such informations. + +<img src='/imgs/cpanhq-dep.webp' alt='cpanhq deps' align=left'> + +For each distributions, a graph is generated for the this distribution. For this, I use [Graph::Easy](http://search.cpan.org/perldoc?Graph::Easy) and data available from the CPANHQ database. I alsa include a simple list of the dependencies after the graph. + +Only the first level dependencies are displayed, as the distribution's metadata are analysed when the request is made. I could follow all the dependencies when the request is made, but for some distribution it could take a really long time, and it's not suitable for this kind of services. + +**edit**: you can find [CPANHQ on GitHub](http://github.com/bricas/cpanhq/tree/master). diff --git a/content/post/2009-07-26-apply-a-role-to-a-moose-object.md b/content/post/2009-07-26-apply-a-role-to-a-moose-object.md new file mode 100644 index 0000000..7f105c5 --- /dev/null +++ b/content/post/2009-07-26-apply-a-role-to-a-moose-object.md @@ -0,0 +1,38 @@ +--- +date: 2009-07-26T00:00:00Z +summary: In which I show how to apply a role to a Moose's object +title: Apply a role to a Moose object +--- + +You can apply a role to a Moose object. You can do something like + +```perl +#!/usr/bin/perl -w +use strict; +use feature ':5.10'; + +package foo; +use Moose::Role; +sub baz { + say 'i can haz baz'; +} + +package bar; +use Moose; +1; + +package main; + +my $test = bar->new; +say "i can't haz baz" if !$test->can("baz"); + +foo->meta->apply($test); +$test->baz; +``` + +with the following output: + +``` +i can't haz baz +i can haz baz +``` diff --git a/content/post/2009-07-26-cpan-explorer.md b/content/post/2009-07-26-cpan-explorer.md new file mode 100644 index 0000000..a68eec2 --- /dev/null +++ b/content/post/2009-07-26-cpan-explorer.md @@ -0,0 +1,9 @@ +--- +date: 2009-07-26T00:00:00Z +summary: In which I share an update to CPAN Explorer. +title: CPAN Explorer +--- + +We ([RTGI](http://rtgi.fr)) have been working to update the [cpan-explorer](http://cpan-explorer.org). A new version will be available this week, before YAPC::EU. Three new maps have been created, using different informations than the previous one, and you will be able to search and pinpoint the browsable maps. + +<img src='/imgs/authorsmap.webp' alt='authors map'> diff --git a/content/post/2009-07-28-cpan-explorer-update-and-three-new-maps.md b/content/post/2009-07-28-cpan-explorer-update-and-three-new-maps.md new file mode 100644 index 0000000..8afb06d --- /dev/null +++ b/content/post/2009-07-28-cpan-explorer-update-and-three-new-maps.md @@ -0,0 +1,29 @@ +--- +date: 2009-07-28T00:00:00Z +summary: In which CPAN explorer got updated. +title: CPAN Explorer update and three new maps +--- + +The site [cpan-explorer](http://cpan-explorer.org/) have been update with three new maps for the [YAPC::EU](http://yapceurope2009.org/ye2009/). This three maps are different from the previous one. This time, instead of having a big photography of the distributions and authors on the CPAN, Task::Kensho have been used to obtain a representation of what we call the **modern Perl**. + +## distributions map + +<img src='/imgs/moosedist.webp' alt='moose'> + +Task::Kensho acted as the seed for this map. Task::Kensho contains a list of modules recommended to do modern Perl development. So we extracted the modules that have a dependancie toward one of these modules, and create the graph with this data. + +## authors map + +The authors listed on this map are the one from the previous map. There is a far less authors thant the previous authors map, but it's more readable. A lot of informations are on the map : label size, node size, edge size, color of the node. + +## web communities map + +This map look a lot like the previous one, as we used nearly the same data. The seed have been extended with a few websites only. + +## cpan-explorer + +CPAN Explorer is now hosted on a wordpress, so you can leave comments or suggestions for new maps you would like to see (a focus on web development modules, tests::* module, etc ...). All the new maps are also searchable, and give you a permalink for you search ([I'm here](http://cpan-explorer.org/2009/07/28/new-web-communities-map-for-yapceu/#dist%3Dlumberjaph.net) and [here](http://cpan-explorer.org/2009/07/28/version-of-the-authors-graph-for-yapceu/#author%3Dfranck)) + +I will give a talk at the [YAPC::EU](http://yapceurope2009.org/ye2009/talk/2061) about this work. Also, each map have been printed, and will be given for the auction. + +This work is a collective effort from all the guys at [RTGI](http://rtgi.fr/) (antonin created the template for wordpress, niko the js and the tools to extract information from the SVG for the searchable map, julian helped me to create the graphs and extract valuable informations, and I got a lot of feedback from others coworkers), thanks guys!. diff --git a/content/post/2009-08-23-perl-5.10.1-released.md b/content/post/2009-08-23-perl-5.10.1-released.md new file mode 100644 index 0000000..c33f066 --- /dev/null +++ b/content/post/2009-08-23-perl-5.10.1-released.md @@ -0,0 +1,32 @@ +--- +date: 2009-08-23T00:00:00Z +summary: In which I install Perl 5.10.1 +title: Perl 5.10.1 released +--- + +[Perl 5.10.1](http://www.cpan.org/modules/by-authors/id/D/DA/DAPM/perl-5.10.1.tar.bz2) [has been released](http://www.nntp.perl.org/group/perl.perl5.porters/2009/08/msg150172.html). You can download it from the CPAN, or if you can't wait, [here](http://www.iabyn.com/tmp/perl-5.10.1.tar.bz2). + +Next, you need to build it: + +```bash +mkdir ~/perl/5.10.1 +cd ~/code/build/perl-5.10.1 +perl Configure -de -Dprefix=${HOME}/perl/5.10.1/ +make +make test +make install +``` + +Add the path to your environment + +```bash +export PATH=${HOME}/perl/5.10.1/bin:$PATH +``` + +and install your CPAN's modules: + +```bash +PERL_MM_USE_DEFAULT=1 cpan Bundle::CPANxxl Task::Kensho +``` + +Now you can start to play with it :) diff --git a/content/post/2009-08-23-perl5lib-and-zsh.md b/content/post/2009-08-23-perl5lib-and-zsh.md new file mode 100644 index 0000000..6899efc --- /dev/null +++ b/content/post/2009-08-23-perl5lib-and-zsh.md @@ -0,0 +1,18 @@ +--- +date: 2009-08-23T00:00:00Z +summary: In which I show how I manage my $PERL5LIB. +title: $PERL5LIB and zsh +--- + +in my .zsh.d/S80_perl + +```bash +BASE_PATH=~/code/work/rtgi +for perl_lib in $(ls $BASE_PATH); do + if [ -f $BASE_PATH/$perl_lib/Makefile.PL ]; then + PERL5LIB=${PERL5LIB:+$PERL5LIB:}$BASE_PATH/$perl_lib/lib + fi +done +export PERL5LIB + +``` diff --git a/content/post/2009-08-31-osdc.fr.md b/content/post/2009-08-31-osdc.fr.md new file mode 100644 index 0000000..cb46b0d --- /dev/null +++ b/content/post/2009-08-31-osdc.fr.md @@ -0,0 +1,11 @@ +--- +date: 2009-08-31T00:00:00Z +summary: In which I promote the OSDC.fr conference +title: OSDC.fr +--- + +The 2nd and 3rd of October, will be held in Paris the <a href="http://act.osdc.fr/osdc2009fr/">Open Source Developers Conference</a>. The purpose of this conference is to gather developers from various open source communities (Perl, Ruby, Python, ...) during two days. + +It's still possible de submit a talk about the subject of your choice, and it's important to note that this conference is free. + +<a href="http://rtgi.fr">RTGI</a> is proud to be one of the sponsor of this event :). diff --git a/content/post/2009-10-03-teh-batmoose-at-osdc.fr.md b/content/post/2009-10-03-teh-batmoose-at-osdc.fr.md new file mode 100644 index 0000000..6c9e2e1 --- /dev/null +++ b/content/post/2009-10-03-teh-batmoose-at-osdc.fr.md @@ -0,0 +1,13 @@ +--- +date: 2009-10-03T00:00:00Z +summary: In which I share the batmoose +title: teh batmoose at osdc.fr +--- + +Today I presented a talk about Moose at [OSDC.fr](http://osdc.fr). The slides are available [here](http://franck.lumberjaph.net/blog/slides/Introduction_a_Moose.pdf) + +<img src='/imgs/batmoose_1024cut-300x225.webp' alt='teh batmoose'> + +And big thanks to my friend [Morgan](http://www.bwoup.com) for his illustration of the batmoose :) + + diff --git a/content/post/2009-11-09-modules-i-like-devel-declare.md b/content/post/2009-11-09-modules-i-like-devel-declare.md new file mode 100644 index 0000000..fdae319 --- /dev/null +++ b/content/post/2009-11-09-modules-i-like-devel-declare.md @@ -0,0 +1,248 @@ +--- +date: 2009-11-09T00:00:00Z +summary: In which I share my enthusiasm for Devel::Declare. +title: Modules I like Devel::Declare +--- + +For [$work](http://linkfluence.net/), I've been working on a job queue system, using Moose, Catalyst (for a REST API) and DBIx::Class to store the jobs and some meta (yeah I know, there is not enough job queue system already, the world really needs a new one ...). + +Basicaly, I've got a XXX::Worker class that all the workers extends. This class provide methods for fetching job, add a new job, mark a job as fail, retry, ... + +The main loop in the XXX::Worker class look like this: + +```perl +# $context is a hashref with some info the job or method may need +while (1) { + my @jobs = $self->fetch_jobs(); + foreach my $job (@jobs) { + my $method = $job->{funcname}; + $self->$method($context, $job); + } + $self->wait; +} +``` + +and the worker look like this + +```perl +package MyWorker; +use Moose; +extends 'XXX::Worker'; + +sub foo { + my ($self, $context, $job) = @_; + + # do something + $self->job_success(); +} +``` + +But as I'm using Moose, I want to add more sugar to the syntax, so writing a new worker would be really more easy. + +Here comes [Devel::Declare](http://search.cpan.org/perldoc?Devel::Declare). + +The syntax I want for my worker is this one: + +```perl +work foo { + $self->logger->info("start to work on job"); + # do something with $job +}; + +work bar { + # do something with $job +}; + +success foo { + $self->logger->info("woot job success"); +}; + +fail bar { + $self->logger->info("ho noez this one failed"); +}; +``` + +Where with `work` I write the code the writer will execute on a task, `success`, a specific code that will be executed after a job is marked as successfull, and `fail` for when the job fail. + +I will show how to add the `work` keyword. I start by writing a new package: + +```perl +package XXX::Meta; + +use Moose; +use Moose::Exporter; +use Moose::Util::MetaRole; + +use Devel::Declare; + +use XXX::Meta::Class; +use XXX::Keyword::Work; + +Moose::Exporter->setup_import_methods(); + +sub init_meta { + my ($me, %options) = @_; + + my $for = $options{for_class}; + + XXX::Keyword::Work->install_methodhandler(into => $for,); + + Moose::Util::MetaRole::apply_metaclass_roles( + for_class => $for, + metaclass_roles => ['XXX::Meta::Class'], + ); + +} + +1; +``` + +The `init_meta` method is provided by Moose: (from the POD) + +> The `init_meta` method sets up the metaclass object for the class specified by `for_class`. This method injects a a meta accessor into the class so you can get at this object. It also sets the class's superclass to base_class, with Moose::Object as the default. + +So I inject into the class that will use XXX::Meta a new metaclass, XXX::Meta::Class. + +Let's take a look to XXX::Meta::Class: + +```perl +package XXX::Meta::Class; + +use Moose::Role; +use Moose::Meta::Class; +use MooseX::Types::Moose qw(Str ArrayRef ClassName Object); + +has work_metaclass => ( + is => 'ro', + isa => Object, + builder => '_build_metaclass', + lazy => 1, +); + +has 'local_work' => ( + traits => ['Array'], + is => 'ro', + isa => ArrayRef [Str], + required => 1, + default => sub { [] }, + auto_deref => 1, + handles => {'_add_work' => 'push',} +); + +sub _build_metaclass { + my $self = shift; + return Moose::Meta::Class->create_anon_class( + superclasses => [$self->method_metaclass], + cache => 1, + ); +} + +sub add_local_method { + my ($self, $method, $name, $code) = @_; + + my $method_name = $method . "_" . $name; + my $body = $self->work_metaclass->name->wrap( + $code, + original_body => $code, + name => $method_name, + package_name => $self->name, + ); + + my $method_add = "_add_" . $method; + $self->add_method($method_name, $body); + $self->$method_add($method_name); +} + +1; +``` + +Here I add to the `->meta` provided by Moose `local_work`, which is an array that contains all my `work` methods. So each time I do something like + +```perl +work foo {}; + +work bar {}; +``` + +in my worker, I add this method to **->meta->local_work**. + +And the class for our keyword work: + +```perl +package XXX::Keyword::Work; + +use strict; +use warnings; + +use Devel::Declare (); +use Sub::Name; + +use base 'Devel::Declare::Context::Simple'; + +sub install_methodhandler { + my $class = shift; + my %args = @_; + { + no strict 'refs'; + *{$args{into} . '::work'} = sub (&) { }; + } + + my $ctx = $class->new(%args); + Devel::Declare->setup_for( + $args{into}, + { work => { + const => sub { $ctx->parser(@_) } + }, + } + ); +} + +sub parser { + my $self = shift; + $self->init(@_); + + $self->skip_declarator; + my $name = $self->strip_name; + $self->strip_proto; + $self->strip_attrs; + + my $inject = $self->scope_injector_call(); + $self->inject_if_block( + $inject . " my (\$self, \$content, \$job) = \@_; "); + + my $pack = Devel::Declare::get_curstash_name; + Devel::Declare::shadow_sub( + "${pack}::work", + sub (&) { + my $work_method = shift; + $pack->meta->add_local_method('work', $name, $work_method); + } + ); + return; +} + +1; +``` + +The `install_methodhandler` add the `work` keyword, with a block of code. This code is sent to the parser, that will add more sugar. With the inject_if_block, I inject the following line + +```perl +my ($self, $context, $job) = @_; +``` + +as this will always be my 3 arguments for a work method. + +Now, for each new worker, I write something like this: + +```perl +package MyWorker; +use Moose; +extends 'XXX::Worker'; +use XXX::Meta; + +work foo {}; +``` + +The next step is too find the best way to reduce the first four lines to two. + +(some of this code is ripped from other modules that use Devel::Declare. The best way to learn what you can do with this module is to read code from other modules that use it) diff --git a/content/post/2009-11-17-sd-the-peer-to-peer-bug-tracking-system.md b/content/post/2009-11-17-sd-the-peer-to-peer-bug-tracking-system.md new file mode 100644 index 0000000..12b1d3b --- /dev/null +++ b/content/post/2009-11-17-sd-the-peer-to-peer-bug-tracking-system.md @@ -0,0 +1,167 @@ +--- +date: 2009-11-17T00:00:00Z +summary: In which I write about SD. +title: sd the peer to peer bug tracking system +--- + +<a href="http://syncwith.us/sd/">SD</a> is a peer to peer bug tracking system build on top of <a href="http://syncwith.us/">Prophet</a>. Prophet is <strong> A grounded, semirelational, peer to peer replicated, disconnected, versioned, property database with self-healing conflict resolution</strong>. SD can be used alone, on an existing bug tracking system (like RT or redmine or github) and it plays nice with git. + +Why should you use SD ? Well, at <a href="http://linkfluence.net/">$work</a> we are using <a href="http://www.redmine.org/">redmine</a> as our ticket tracker. I spend a good part of my time in a terminal, and checking the ticket system, adding a ticket, etc, using the browser, is annoying. I prefer something which I can use in my terminal and edit with my <a href="http://www.vim.org/">$EDITOR</a>. So if you recognize yourself in this description, you might want to take a look at SD. + +> In the contrib directory of the SD distribution, you will find a SD ticket syntax file for vim. + +## how to do some basic stuff with sd + +We will start by initializing a database. By default + +```bash +% sd init +``` + +will create a *.sd* directory in your $HOME. If you want to create in a specific path, you will need to set the SD_REPO in your env. + +```bash +% SD_REPO=~/code/myproject/sd sd init +``` + +The init command creates an sqlite database and a config file. The config file is in the same format as the one used by git. + +Now we can create a ticket: + +```bash +% SD_REPO=~/code/myproject/sd ticket create +``` + +This will open your $EDITOR, the part you need to edit are specified. After editing this file, you will get something like this: + +> Created ticket 11 (437b823c-8f69-46ff-864f-a5f74964a73f) +> Created comment 12 (f7f9ee13-76df-49fe-b8b2-9b94f8c37989) + +You can view the created ticket: + +```bash +% SD_REPO=~/code/myproject/sd ticket show 11 +``` + +and the content of your ticket will be displayed. + +You can list and filter your tickets: + +```bash +% SD_REPO=~/code/myproject/sd ticket list +% SD_REPO=~/code/myproject/sd search --regex foo +``` + +You can edit the SD configuration using the config tool or editing directly the file. SD will look for three files : /etc/sdrc, $HOME/.sdrc or the config file in your replica (in our exemple, ~/code/myproject/sd/config). + +For changing my email address, I can do it this way: + +```bash +% SD_REPO=~/code/myproject/sd config user.email-address franck@lumberjaph.net +``` + +or directly + +```bash +% SD_REPO=~/code/myproject/sd config edit +``` + +and update the user section. + +## sd with git + +SD provides a script for git: *git-sd*. + +Let's start by creating a git repository: + +```bash +% mkdir ~/code/git/myuberproject +% cd ~/code/git/myuberproject +git init +``` + +SD comes with a git hook named "git-post-commit-close-ticket" (in the contrib directory). We will copy this script to <strong>.git/hooks/post-commit</strong>. + +now we can initialize our sd database + +```bash +% git-sd init +``` + +git-sd will try to find which email you have choosen for this project using git config, and use the same address for it's configuration. + +Let's write some code for our new project + +```perl +#!/usr/bin/env perl +use strict; +use warnings; +print "hello, world\n"; +``` + +then + +```bash +% git add hello.pl +% git commit -m "first commit" hello.pl +``` + +now we can create a new entry + +```bash +% git-sd ticket create # create a ticket to replace print with say +``` + +We note the UUID for the ticket: in my exemple, the following output is produced: + +> Created ticket 11 (92878841-d764-4ac9-8aae-cd49e84c1ffe) +> Created comment 12 (ddb1e56e-87cb-4054-a035-253be4bc5855) + +so my UUID is <strong>92878841-d764-4ac9-8aae-cd49e84c1ffe</strong>. + +Now, I fix my bug + +```bash +#!/usr/bin/env perl +use strict; +use 5.010; +use warnings; +say "hello, world"; +``` + +and commit it + +```bash +% git commit -m "Closes 92878841-d764-4ac9-8aae-cd49e84c1ffe" hello.pl +``` + +If I do a + +```bash +% git ticket show 92878841-d764-4ac9-8aae-cd49e84c1ffe +``` + +The ticket will be marked as closed. + +## sd with github + +Let's say you want to track issues from a project (I will use <a href="http://plackperl.org/">Plack</a> for this exemple) that is hosted on github. + +```bash +% git clone git://github.com/miyagawa/Plack.git +% git-sd clone --from "github:http://github.com/miyagawa/Plack" +# it's the same as +% git-sd clone --from "github:miyagawa/Plack" +# or if you don't want to be prompted for username and password each time +% git-sd clone --from github:http://githubusername:apitoken@github.com/miyagawa/Plack.git +``` + +It will ask for you github username and your API token, and clone the database. + +Later, you can publish your sd database like this: + +```bash +% git-sd push --to "github:http://github.com/$user/$project" +``` + +Now you can code offline with git, and open/close tickets using SD :) diff --git a/content/post/2009-12-13-riak-perl-and-kiokudb.md b/content/post/2009-12-13-riak-perl-and-kiokudb.md new file mode 100644 index 0000000..a4fdd5c --- /dev/null +++ b/content/post/2009-12-13-riak-perl-and-kiokudb.md @@ -0,0 +1,153 @@ +--- +date: 2009-12-13T00:00:00Z +summary: In which I write about Riak Perl and KiokuDB. +title: Riak, Perl and KiokuDB +--- + +As I was looking for a system to store documents at <a href="http://linkfluence.net">$work</a>, <a href="http://www.basho.com/Riak.html">Riak</a> was pointed to me by one of my coworkers. I'm looking for a solution of this type to store various types of documents, from HTML pages to json. I need a system that is distributed, faul tolerant, and that works with Perl. + +So Riak is a document based database, it's key value, no sql, REST, and in Erlang. You can read more about it <a href="http://riak.basho.com/nyc-nosql/">here</a> or watch an introduction <a href="http://vimeo.com/6973519">here</a>. Like <ahref="http://couchdb.apache.org/">CouchDB</a>, Riak provides a REST interface, so you don't have to write any Erlang code. + +One of the nice things with Riak it's that it let you defined the N, R and W value for each operation. This values are: + +* N: the number of replicas of each value to store +* R: the number of replicas required to perform a read operation +* W: the number of replicas needed for a write operation + +Riak comes with library for python ruby PHP and even javascript, but not for Perl. As all these libraries are just communicating with Riak via the REST interface, I've <a href="http://git.lumberjaph.net/p5-anyevent-riak.git/">started to write one</a> using AnyEvent::HTTP, and <a href="http://git.lumberjaph.net/p5-kiokudb-backend-riak.git/">also a backend for KiokuDB</a>. + +## Installing and using Riak + +If you interested in Riak, you can install it easily. First, you will need the Erlang VM. On debian, a simple + +```bash +% sudo aptitude install erlang +``` + +install everything you need. Next step is to install Riak: + +```bash +% wget http://hg.basho.com/riak/get/riak-0.6.2.tar.gz +% tar xzf riak-0.6.2.tar.gz +% cd riak +% make +% export RIAK=`pwd` +``` + +Now, you can start to use it with + +```bash +% ./start-fresh config/riak-demo.erlenv +``` + +or if you want to test it in cluster mode, you can write a configuration like this: + +```erlang + {cluster_name, "default"}. + {ring_state_dir, "priv/ringstate"}. + {ring_creation_size, 16}. + {gossip_interval, 60000}. + {storage_backend, riak_fs_backend}. + {riak_fs_backend_root, "/opt/data/riak/"}. + {riak_cookie, riak_demo_cookie}. + {riak_heart_command, "(cd $RIAK; ./start-restart.sh $RIAK/config/riak-demo.erlenv)"}. + {riak_nodename, riakdemo}. + {riak_hostname, "192.168.0.11"}. + {riak_web_ip, "192.168.0.11"}. + {riak_web_port, 8098}. + {jiak_name, "jiak"}. + {riak_web_logdir, "/tmp/riak_log"}. +``` + +Copy this config on a second server, edit it to replace the riak\_hostname and riak\_nodename. On the first server, start it like show previously, then on the second, with + +```bash +% ./start-join.sh config/riak-demo.erlenv 192.168.0.11 +``` + +where the IP address it the address of the first node in your cluster. + +Let's check if everything works: + +```bash +% curl -X PUT -H "Content-type: application/json" \ + http://192.168.0.11:8098/jiak/blog/lumberjaph/ \ + -d "{\"bucket\":\"blog\",\"key\":\"lumberjaph\",\"object\":{\"title\":\"I'm a lumberjaph, and I'm ok\"},\"links\":[]}" + +% curl -i http://192.168.0.11:8098/jiak/blog/lumberjaph/ +``` + +will output (with the HTTP blabla) + +```javascript + {"object":{"title":"I'm a lumberjaph, and I'm ok"},"vclock":"a85hYGBgzGDKBVIsbGubKzKYEhnzWBlCTs08wpcFAA==","lastmod":"Sun, 13 Dec 2009 20:28:04 GMT","vtag":"5YSzQ7sEdI3lABkEUFcgXy","bucket":"blog","key":"lumberjaph","links":[]} +``` + +## Using Riak with Perl and KiokuDB + +I need to store various things in Riak: html pages, json data, and objects using KiokuDB. I've started to write a client for Riak with AnyEvent, so I can do simple operations at the moment, (listing information about a bucket, defining a new bucket with a specific schema, storing, retriving and deleting documents). To create a client, you need to + +```perl +my $client = AnyEvent::Riak->new( + host => 'http://127.0.0.1:8098', + path => 'jiak', +); +``` + +As Riak exposes to you it's N, R, and W value, you can also set them in creation the client: + +```perl +my $client = AnyEvent::Riak->new( + host => 'http://127.0.0.1:8098', + path => 'jiak', + r => 2, + w => 2, + dw => 2, +); +``` + +where: + +* the W and DW values define that the request returns as soon as at least W nodes have received the request, and at least DW nodes have stored it in their storage backend. +* with the R value, the request returns as soon as R nodes have responded with a value or an error. You can also set this values when calling fetch, store and delete. By default, the value is set to 2. + +So, if you wan to store a value, retrieve it, then delete it, you can do: + +```perl +my $store = + $client->store({bucket => 'foo', key => 'bar', object => {baz => 1},}) + ->recv; +my $fetch = $client->fetch('foo', 'bar')->recv; +my $delete = $client->delete('foo', 'bar')->recv; +``` + +If there is an error, the croak method from AnyEvent is used, so you may prefer to do this: + +```perl +use Try::Tiny; +try { + my $fetch = $client->fetch('foo', 'baz')->recv; +} +catch { + my $err = decode_json $_; + say "error: code => " . $err->[0] . " reason => " . $err->[1]; +}; +``` + +The error contains an array, with the first value the HTTP code, and the second value the reason of the error given by Riak. + +At the moment, the KiokuDB backend is not complete, but if you want to start to play with is, all you need to do is: + +```perl +my $dir = KiokuDB->new( + backend => KiokuDB::Backend::Riak->new( + db => AnyEvent::Riak->new( + host => 'http://localhost:8098', + path => 'jiak', + ), + bucket => 'kiokudb', + ), +); + +$dir->txn_do(sub { $dir->insert($key => $object) }); +``` diff --git a/content/post/2009-12-20-moosex-net-api.md b/content/post/2009-12-20-moosex-net-api.md new file mode 100644 index 0000000..7fc00ca --- /dev/null +++ b/content/post/2009-12-20-moosex-net-api.md @@ -0,0 +1,110 @@ +--- +date: 2009-12-20T00:00:00Z +summary: In which I introduce MooseX::Net::API. +title: MooseX::Net::API +--- + +## Net::Twitter + +I've been asked for [$work](http://linkfluence.net) to write an API client for [backtype](http://www.backtype.com/), as we plan to integrate it in one of our services. A couple of days before I was reading the [Net::Twitter](http://search.cpan.org/perldoc?Net::Twitter) source code, and I've found interesting how [semifor](http://blog.questright.com/) wrote it. + +Basically, what Net::Twitter does is this: for each API method, there is a `twitter_api_method` method, where the only code for this method is an API specification of the method. Let's look at the public timeline method: + +```perl +twitter_api_method home_timeline => ( + description => <<'', +Returns the 20 most recent statuses, including retweets, posted by the +authenticating user and that user's friends. This is the equivalent of +/timeline/home on the Web. + + path => 'statuses/home_timeline', + method => 'GET', + params => [qw/since_id max_id count page/], + required => [], + returns => 'ArrayRef[Status]', +); +``` + +The `twitter_api_method` method is exported with Moose::Exporter. It generates a sub called `home_timeline` that is added to the class. + +## MooseX::Net::API + +As I've found this approch nice and simple, I thought about writing a [little framework](http://git.lumberjaph.net/p5-moosex-net-api.git/) to easily write API client this way. I will show how I've write a [client for the Backtype API](http://git.lumberjaph.net/p5-net-backtype.git/) using this (I've wrote some other client for private API at works too). + +## Backtype API + +First we defined our class: + +```perl +package Net::Backtweet; +use Moose; +use MooseX::Net::API; +``` + +MooseX::Net::API export two methods: `net_api_declare` and `net_api_method`. The first method is for all the paramters that are common for each method. For Backtype, I'll get this: + +```perl +net_api_declare backtweet => ( + base_url => 'http://backtweets.com', + format => 'json', + format_mode => 'append', +); +``` + +This set + +* the base URL for the API +* the format is JSON +* some API use an extension at the name of the method to determine the format. "append" do this. + +Right now three formats are supported: xml json and yaml. Two modes are supported: `append` and `content-type`. + +Now the `net_api_method` method. + +```perl +net_api_method backtweet_search => ( + path => '/search', + method => 'GET', + params => [qw/q since key/], + required => [qw/q key/], + expected => [qw/200/], +); +``` + +* path: path for the method (required) +* method: how to acces this resource (GET POST PUT and DELETE are supported) (required) +* params: list of parameters to access this resource (required) +* required: which keys are required +* expected: list of HTTP code accepted + +To use it: + +```perl +my $backtype = Net::Bactype->new(); +my $res = + $backtype->backtweet_search(q => "http://lumberjaph.net", key => "foo"); +warn Dump $res->{tweets}; +``` + +## MooseX::Net::API implementation + +Now, what is done by the framework. The `net_api_declare` method add various attributes to the class: + +* api\_base_url: base URL of the API +* api_format: format for the query +* api\_format_mode: how the format is used (append or content-type) +* api_authentication: if the API requires authentication +* api_username: the username for accessing the resource +* api_password: the password +* api_authentication: does the resource requires to be authenticated + +It will also apply two roles, for serialization and deserialization, unless you provides your own roles for this. You can provides your own method for useragent and authentication too (the module only do basic authentication). + +For the `net_api_method` method, you can overload the authentication (in case some resources requires authentication). You can also overload the default code generated. + +In case there is an error, an MooseX::Net::API::Error will be throw. + +## Conclusion + +Right now, this module is not finished. I'm looking for suggestions (what should be added, done better, how I can improve stuff, ...). I'm not aiming to handle all possibles API, but at least most of the REST API avaible. I've uploaded a first version of +[MooseX::Net::API](http://search.cpan.org/perldoc?MooseX::Net::API) and [Net::Backtype](http://search.cpan.org/perldoc?Net::Backtype) on CPAN, and [the code](http://git.lumberjaph.net/p5-net-backtype.git/) is also [available on my git server](http://git.lumberjaph.net/p5-moosex-net-api.git/). diff --git a/content/post/2009-12-21-tatsumaki-or-how-to-write-a-nice-webapp-in-less-than-two-hours.md b/content/post/2009-12-21-tatsumaki-or-how-to-write-a-nice-webapp-in-less-than-two-hours.md new file mode 100644 index 0000000..c1d6a83 --- /dev/null +++ b/content/post/2009-12-21-tatsumaki-or-how-to-write-a-nice-webapp-in-less-than-two-hours.md @@ -0,0 +1,102 @@ +--- +date: 2009-12-21T00:00:00Z +summary: In which I write about Tatsumaki. +title: Tatsumaki, or how to write a nice webapp in less than two hours +--- + +Until today, I had a script named "lifestream.pl". This script was triggered via cron once every hour, to fetch various feeds from services I use (like <a href="http://github.com/">github</a>, <a href="http://identi.ca/">identi.ca</a>, ...) and to process the result through a template and dump the result in a HTML file. + +Today I was reading <a href="http://github.com/miyagawa/Tatsumaki">Tatsumaki's code</a> and some examples (<a href="http://github.com/gugod/Social">Social</a> and <a href="http://github.com/miyagawa/Subfeedr">Subfeedr</a>). Tatsumaki is a "port" <a href="http://www.tornadoweb.org/">tornado</a> (a non blocking server in Python), based on Plack and AnyEvent. I though that using this to replace my old lifestream script would be a good way to test it. Two hours later I have a complete webapp that works (and the code is available <a href="http://git.lumberjaph.net/p5-lifestream.git/">here</a>). + +The code is really simple: first, I define an handler for my HTTP request. As I have only one things to do (display entries), the handler is really simple: + +```perl +package Lifestream::Handler; +use Moose; +extends 'Tatsumaki::Handler'; + +sub get { + my $self = shift; + my %params = %{$self->request->params}; + $self->render( + 'lifestream.html', + { memes => $self->application->memes($params{page}), + services => $self->application->services + } + ); +} +1; +``` + +For all the get request, 2 methods are called : <strong>memes</strong> and <strong>services</strong>. The <strong>memes</strong> get a list of memes to display on the page. The services get the list of the various services I use (to display them on a sidebar). + +Now, as I don't want to have anymore my lifestream.pl script in cron, I will let Tatsumaki do the polling. For this, I add a service to my app, which is just a worker. + +```perl +package Lifestream::Worker; +use Moose; +extends 'Tatsumaki::Service'; +use Tatsumaki::HTTPClient; + +sub start { + my $self = shift; + my $t; + $t = AE::timer 0, 1800, sub { + scalar $t; + $self->fetch_feeds; + }; +} + +sub fetch_feeds { + my ($self, $url) = @_; + Tatsumaki::HTTPClient->new->get( + $url, + sub { + #do the fetch and parsing stuff + } + ); +} +``` + +From now, every 60 minutes, feeds will be checked. Tatsumaki::HTTPClient is a HTTP client based on AnyEvent::HTTP. + +Let's write the app now + +```perl +package Lifestream; + +use Moose; +extends "Tatsumaki::Application"; + +use Lifestream::Handler; +use Lifestream::Worker; + +sub app { + my ($class, %args) = @_; + my $self = $class->new(['/' => 'Lifestream::Handler',]); + $self->config($args{config}); + $self->add_service(Lifestream::Worker->new(config => $self->config)); + $self; +} + +sub memes { +} + +sub services { +} +``` + +The <strong>memes</strong> and <strong>services</strong> method called from the handler are defined here. In the app method, I "attch" the "/" path to the handler, and I add the service. + +and to launch the app + +```perl +my $app = Lifestream->app(config => LoadFile($config)); +require Tatsumaki::Server; +Tatsumaki::Server->new( + port => 9999, + host => 0, +)->run($app); +``` + +And that's it, I now have a nice webapp, with something like only 200 LOC. I will keep playing with <a href="http://www.slideshare.net/miyagawa/tatsumaki">Tatsumaki</a> as I have more ideas (and probably subfeedr too). Thanks to <a href="http://bulknews.typepad.com/">miyagawa</a> for all this code. diff --git a/content/post/2010-01-31-dancer-1.130.md b/content/post/2010-01-31-dancer-1.130.md new file mode 100644 index 0000000..ca7a2c7 --- /dev/null +++ b/content/post/2010-01-31-dancer-1.130.md @@ -0,0 +1,43 @@ +--- +date: 2010-01-31T00:00:00Z +summary: In which I announce Dancer 1.130 +title: Dancer 1.130 +--- + +[Alexis](http://www.sukria.net/) ([sukria](http://search.cpan.org/~sukria/)) released [Dancer](http://search.cpan.org/perldoc?Dancer) 1.130 this weekend. Dancer is a small and nice web framework based on ruby's [sinatra](http://www.sinatrarb.com/). + +Dancer have few dependancies (and it doesn't depends anymore on CGI.pm). The path dispatching is done using rules declared with HTTP methods (get/post/put/delete), and they are mapped to a sub-routine which is returned as the response to the request. Sessions are supported, and two template engines (one of them is Template Toolkit) comes with the Core. Dancer::Template::MicroTemplate is also available on CPAN if you need a light template engine. + +You can easily test it with a simple script + +```perl +#!/usr/bin/env perl +use Dancer; + +get '/' => sub { + return "dancer"; +}; + +get '/:name' => sub { + return params->{name} . " is dancing"; +}; + +dance; +``` + +and execute this script, point your browser to http://127.0.0.1:3000, and voila. + +Dancer provides also a small helper to write a new application: `dancer -a MyApplication` + +If you create an application with this script, an **app.psgi** file will be created. You can now execute `plackup --port 8080` + +(which comes with [Plack](http://search.cpan.org/perldoc?Plack) the [Perl Web Server](http://plackperl.org/)) and test if everything works fine: `curl http://localhost:8080`. + +This release remove some components from the core and they are now available as differents CPAN distributions. Two new keyword have also been added, **header** and **prefix**. + +If you want to read more about Dancer: + + * [Dancer's documentation](http://search.cpan.org/perldoc?Dancer) + * [review by xsawyerx](http://blogs.perl.org/users/sawyer_x/2010/01/i-gotz-me-a-dancer.html) + * [gugod's review](http://gugod.org/2009/12/dancer.html) + * [sukria's blog](http://www.sukria.net/fr/archives/tag/dancer/) diff --git a/content/post/2010-02-03-sd-and-github.md b/content/post/2010-02-03-sd-and-github.md new file mode 100644 index 0000000..6c207af --- /dev/null +++ b/content/post/2010-02-03-sd-and-github.md @@ -0,0 +1,20 @@ +--- +date: 2010-02-03T00:00:00Z +summary: In which I share how to use SD with GitHub +title: SD and github +--- + +If you are using the version of <a href="http://syncwith.us/">SD</a> hosted on <a href="http://github.com/bestpractical/sd">GitHub</a>, you can now clone and pull issues very easily. First, + +```sh +$ git config --global github.user franckcuny +$ git config --global github.token myapitoken +``` + +This will set in your <strong>.gitconfig</strong> your github username and api token. Now, when you clone or pull some issues using sd: + +```sh +$ git sd clone --from github:sukria/Dancer +``` + +sd will check your .gitconfig to find your credentials. diff --git a/content/post/2010-03-07-github-explorer-a-preview.md b/content/post/2010-03-07-github-explorer-a-preview.md new file mode 100644 index 0000000..0d57796 --- /dev/null +++ b/content/post/2010-03-07-github-explorer-a-preview.md @@ -0,0 +1,19 @@ +--- +date: 2010-03-07T00:00:00Z +summary: In which I share a preview of GitHub Explorer. +title: GitHub Explorer - a preview +--- + +> You may want to see the final version here: [GitHub Explorer](/github-explorer/). + +For the last weeks, I've been working on the successor of [CPAN Explorer](http://cpan-explorer.org/). This time, I've decided to create some visualizations (probably 8) of the various communities using [GitHub](http://github.com/). I'm happy with the result, and will soon start to publish the maps (statics and interactives) with some analyses. I'm publishing two previews: the Perl community and the european developers. These are not final results. The colors, fonts, and layout may change. But the structure of the graphs will be the same. All the data was collected using the [GitHub API](http://developer.github.com/). + +<a href='/imgs/4413528529_8d6b8dca1c_o.webp'><img src='/static/imgs/github-perl-preview.webp' alt='the Perl community on GitHub' /></a> + +Each node on the graph represents a developer. When a developer "follows" another developer on GitHub, a link between them is created. The color on the Perl community map represent the countries of the developer. One of the most visible things on this graph is that the japanese community is tighly connected and shares very little contact with the foreign developers. miyagawa obviously acts as a glue between japanese and worldwide Perl hackers. + +<a href='/imgs/4414287310_20094fe6bc_o.webp'><img src='/static/imgs/github-europe-preview.webp' alt='European developers on GitHub' /></a> + +The second graph is a little bit more complex. It represents the European developers on GitHub. Here the colors represent the languages used by the developers. It appears that ruby is by far the most represented language on GitHub, as it dominates the whole map. Perl is the blue cluster at the bottom of the map, and the green snake is... Python. + +Thanks to [bl0b](http://code.google.com/p/tinyaml/) for his suggestions :) diff --git a/content/post/2010-03-19-easily-create-rest-interface-with-the-dancer-1.170.md b/content/post/2010-03-19-easily-create-rest-interface-with-the-dancer-1.170.md new file mode 100644 index 0000000..5218e2c --- /dev/null +++ b/content/post/2010-03-19-easily-create-rest-interface-with-the-dancer-1.170.md @@ -0,0 +1,161 @@ +--- +date: 2010-03-19T00:00:00Z +summary: In which we see that it's easy to create REST interface with Dancer. +title: Easily create REST interface with the Dancer 1.170 +--- + +This week, with [Alexi](http://www.sukria.net/fr/)'s help, [I've been working on](http://github.com/perldancer/Dancer) to add auto-(de)serialization to Dancer's request. This features will be available in the next [Dancer](http://perldancer.org/) version, the 1.170 (which will be out before April). + +The basic idea was to provides to developer a simple way to access data that have been send in a serialized format, and to properly serialize the response. + +At the moment, the supported serializers are : + +* Dancer::Serialize::JSON +* Dancer::Serialize::YAML +* Dancer::Serialize::XML +* Dancer::Serialize::Mutable + +## Configuring an application to use the serializer + +To activate serialization in your application: + +```perl +set serializer => 'JSON'; +``` + +or in your configuration file: + +```yaml +serializer: "JSON" +``` + +## A simple handler + +Let's create a new dancer application (you can fetch the source on [my git server](http://git.lumberjaph.net/p5-dancer-rest.git/) : + +```bash +% dancer -a dancerREST +% cd dancerREST +% vim dancerREST.pm +``` + +then + +```perl +package dancerREST; +use Dancer ':syntax'; + +my %users = (); + +post '/api/user/' => sub { + my $params = request->params; + if ($params->{name} && $params->{id}) { + if (exists $users{$params->{id}}) { + return {error => "user already exists"}; + } + $users{$params->{id}} = {name => $params->{name}}; + return {id => $params->{id}, name => $params->{name}}; + } + else { + return {error => "name is missing"}; + } +}; + +true; +``` + +We can test if everything works as expected: + +```bash +% plackup app.psgi & +% curl -H "Content-Type: application/json" -X POST http://localhost:5000/api/user/ -d '{"name":"foo","id":1}' +# => {"name":"foo","id":"1"} +``` + +Now we add a method to fetch a list of users, and a method to get a +specific user: + +```perl +# return a specific user +get '/api/user/:id' => sub { + my $params = request->params; + if (exists $users{$params->{id}}) { + return $users{$params->{id}}; + } + else { + return {error => "unknown user"}; + } +}; + +# return a list of users +get '/api/user/' => sub { + my @users; + push @users, {name => $users{$_}->{name}, id => $_} + foreach keys %users; + return \@users; +}; +``` + +If we want to fetch the full list: + +```sh +curl -H "Content-Type: application/json" http://localhost:5000/api/user/ +# => [{"name":"foo","id":"1"}] +``` + +and a specific user: + +```sh +curl -H "Content-Type: application/json" http://localhost:5000/api/user/1 +# => {"name":"foo"} +``` + +## The mutable serializer + +The mutable serializer will try to load an appropriate serializer guessing from the **Content-Type** and **Accept-Type** header. You can also overload this by adding a **content_type=application/json** parameter to your request. + +While setting your serializer to mutable, your let your user decide which format they prefer between YAML, JSON and XML. + +## And the bonus + +Dancer provides now a new method to the request object : `is_ajax`. Now you can write something like + +```perl +get '/user/:id' => sub { + my $params = request->params; + my $user = $users{$params->{id}}; + my $result; + if (!$user) { + _render_user({error => "unknown user"}); + } + else { + _render_user($user); + } +}; + +sub _render_user { + my $result = shift; + if (request->is_ajax) { + return $result; + } + else { + template 'user.tt', $result; + } +} +``` + +If we want to simulate an AJAX query: + +```bash +% curl -H "X-Requested-With: XMLHttpRequest" http://localhost:5000/user/1 +``` + +and we will obtain our result in JSON. But we can also test without the X-Requested-With: + +```bash +% curl http://localhost:5000/user/1 +``` + +and the template will be rendered. + +Hope you like this new features. I've also been working on something similar for [Tatsumaki](http://github.com/miyagawa/tatsumaki). diff --git a/content/post/2010-03-25-github-explorer.md b/content/post/2010-03-25-github-explorer.md new file mode 100644 index 0000000..40a9ace --- /dev/null +++ b/content/post/2010-03-25-github-explorer.md @@ -0,0 +1,138 @@ +--- +date: 2010-03-25T00:00:00Z +summary: In which I write about GitHub Explorer. +title: GitHub explorer +--- + +> *More informations about the poster are available on [this post](http://lumberjaph.net/graph/2010/04/02/github-poster.html)* + +Last year, with help from my coworkers at [Linkfluence](http://linkfluence.net/), I created two sets of maps of the [Perl](http://perl.org) and [CPAN](http://search.cpan.org/)'s community. For this, I collected data from CPAN to create three maps: + + * [dependencies between distributions](http://cpan-explorer.org/2009/07/28/new-version-of-the-distributions-map-for-yapceu/) + * [which authors wre important in term of reliability](http://cpan-explorer.org/2009/07/28/version-of-the-authors-graph-for-yapceu/) + * [and how the websites theses authors are structured](http://cpan-explorer.org/2009/07/28/new-web-communities-map-for-yapceu/) + +I wanted to do something similar again, but not with the same data. So I took a look at what could be a good subject. One of the things that we saw from the map of the websites is the importance [GitHub](http://github.com/) is gaining inside the Perl community. GitHub provides a [really good API](http://develop.github.com/), so I started to play with it. + +> This graph will be printed on a poster, size will be [A2](http://en.wikipedia.org/wiki/A2_paper_size) and [A1](http://en.wikipedia.org/wiki/A1_paper_size). Please, contact me franck.cuny [at] linkfluence.net if you will be interested by one. + +<img class="img_center" src="/imgs/general.webp" alt="github explorer global" title="github explorer global" /> + +<img alt="github explorer zoom" class="img_center" src="/imgs/zoom.webp" title="github explorer zoom" /> + +This time, I didn't aim for the Perl community only, but the whole github communities. I've created several graphs: + +> all the graph are available "on my flickr account":http://www.flickr.com/photos/franck_/sets/72157623447857405/ + +* [a graph of all languages](http://www.flickr.com/photos/franck_/4460144638/) +* [a graph of the Perl community](http://www.flickr.com/photos/franck_/4456072255/in/set-72157623447857405/) +* [a graph of the Ruby community](http://www.flickr.com/photos/franck_/4456914448/) +* [a graph of the Python community](http://www.flickr.com/photos/franck_/4456118597/in/set-72157623447857405/) +* [a graph of the PHP community](http://www.flickr.com/photos/franck_/4456830956/in/set-72157623447857405/) +* [a graph of the European community](http://www.flickr.com/photos/franck_/4456862434/in/set-72157623447857405/) +* [a graph of the Japan community](http://www.flickr.com/photos/franck_/4456129655/in/set-72157623447857405/) + +I think a disclaimer is important at this point. I know that github doesn't represent the whole open source community. With these maps, I don't claim to represent what the open source world looks like right now. This is not a troll about which language is best, or used at large. It's **ONLY** about GitHub. + +Also, I won't provide deep analysis for each of these graphs, as I lack insight about some of those communities. So feel free to [re-use the graphs](http://franck.lumberjaph.net/graphs.tgz) and provide your own analyses. + +## Methodology + +I didn't collect all the profiles. We (with [Guilhem](http://twitter.com/gfouetil) decided to limit to peoples who are followed by at least two other people. We did the same thing for repositories, limiting to repositories which are at least forked once. Using this technique, more than 17k profiles have been collected, and nearly as many repositories. + +For each profile, using the github API, I've tried to determine what the main language for this person is. And with the help of the [geonames](http://www.geonames.org), find the right country to attach the profile to. + +Each profile is represented by a node. For each node, the following attributes are set: + +* name of the profile +* main language used by this profile, determined by github +* name of the country +* follower count +* following count +* repository count + +An edge is a link between two profiles. Each time someone follows another profile, a link is created. By default, the weight of this link is 1. For each project this person forked from the target profile, the weight is incremented. + +As always, I've used [Gephi](http://gephi.org/) (now in version 0.7) to create the graphs. Feel free to download the various graph files and use them with Gephi. + +## Github + +> properties of the graph: 16443 nodes / 130650 edges + +<a href="http://www.flickr.com/photos/franck_/4460144638/" alt="GitHub - All" title="Github - All - by languages by franck.cuny, on Flickr"><img class="img_center" src="http://farm5.static.flickr.com/4027/4460144638_48e7d83e80.jpg" width="482" height="500" alt="Github - All - by languages" /></a> + +The first map is about all the languages available on github. This one was really heavy, with more than 17k nodes, and 130k edges. The final version of the graph use the 2270 more connected nodes. + +You can't miss Ruby on this map. As github uses Ruby on Rails, it's not really surprising that the Ruby community has a particular interest on this website. The main languages on github are what we can expect, with PHP, Python, Perl, Javascript. + +Some languages are not really well represented. We can assume that most Haskell projects might use darcs, and therefore are not on github. Some other languages may use other platforms, like launchpad, or sourceforge. + +## Perl + +> properties of the graph: 365 nodes / 4440 edges + +<a href="http://www.flickr.com/photos/franck_/4456842344/" title="Perl community on Github by franck.cuny, on Flickr"><img src="http://farm5.static.flickr.com/4002/4456842344_06f39127a8.jpg" class="img_center" width="500" height="437" alt="Perl community on Github" /></a> + +The Perl community is split into two parts. On the left side, there is the occidental community, driven by people like "Florian":http://github.com/rafl, "Yuval":http://github.com/nothingmuch, "rjbs":http://github.com/rjbs, ... The second part are the japanese Perl hackers, with <a href="http://github.com/tokuhirom">Tokhuirom</a>, <a href="http://github.com/typester">Typester</a>, <a href="http://github.com/yappo">Yappo</a>, ... And in between them, <a href="http://github.com/miyagawa">Miyagawa</a> acts as a glue. This map looks a lot like the previous map of the CPAN. We can see that this community is international, with the exception of Japan that don't mix with others. + +There is no main project on github that gathers people, even though we can see a fair amount of <strong>MooseX::</strong> projects. Most of the developers will work on different modules, that may not have the same purpose. Lately we have seen a fair amount of work on various <a href="http://plackperl.org/">Plack</a> stuff, mainly middleware, but also HTTP servers (twiggy, starman, ...) and web framework (<a href="http://perldancer.org/">dancer</a>). + +One important project that is not (deliberately) represented on this graph is the <a href="http://github.com/gitpan">gitpan</a>, <a href="http://github.com/schwern">Schwern</a>'s project. The gitpan is an import of all the CPAN modules, with complete history using the Backpan. + +To conclude about Perl, there are only 365 nodes on this graph, but no less than 4440 edges. That's nearly two times the number of edges compared to the Python community. Perl is a really well structured community, probably thanks to the CPAN, which already acted as hub for contributors. + +## Python + +> properties of the graph: 532 nodes / 2566 edges + +<a href="http://www.flickr.com/photos/franck_/4456118597/" title="Python community, by country, on Github by franck.cuny, on Flickr"><img src="http://farm3.static.flickr.com/2676/4456118597_9d39f8d413.jpg" class="img_center" width="470" height="500" alt="Python community, by country, on Github" /></a> + +The Python community looks a lot like the Perl community, but only in the structure of the graph. If we look closely, <a href="http://www.djangoproject.com/">Django</a> is the main project that represent Python on Github, in contrast with Perl where there is no leader. Some small projects gather small community of developers. + +## PHP + +> properties of the graph: 301 nodes / 1071 edges + +<a href="http://www.flickr.com/photos/franck_/4456830956/" title="PHP community on Github by franck.cuny, on Flickr"><img src="http://farm5.static.flickr.com/4033/4456830956_ef0e8f3587.jpg" class="img_center" width="500" height="372" alt="PHP community on Github" /></a> + +PHP is the only community that is structured this way on Github. We can clearly see that people are structured based on a project where they mainly contribute. + +<a href="http://cakephp.org/">CakePHP</a> and <a href="http://www.symfony-project.org/">Symphony</a> are the two main projects. Nearly all the projects gather an international community, at the exception of a few japanese-only projects + +## Ruby + +> properties of the graph: 3742 nodes / 24571 edges + +<a href="http://www.flickr.com/photos/franck_/4456914448/" title="Ruby community, by country, on Github by franck.cuny, on Flickr"><img src="http://farm5.static.flickr.com/4012/4456914448_8089c3acca.jpg" class="img_center" width="500" height="469" alt="Ruby community, by country, on Github" /></a> + +As for the Github graph, we can clearly see that some countries are isolated. On the right side, we have: the Japan community is at the bottom; the Spanish at the top. Australian are represented on the upper right corner, while on the left side we got the Brazilians. + +The main projects that gather most of the hackers are <a href="http://rubyonrails.org/">Rails</a> and <a href="http://sinatrarb.com/">Sinatra</a>, two famous web frameworks. + +## Europe + +> properties of the graph: 2711 nodes / 11259 edges + +<a href="http://www.flickr.com/photos/franck_/4456862434/" title="Europe community on Github by franck.cuny, on Flickr"><img src="http://farm5.static.flickr.com/4062/4456862434_324e7b2c75.jpg" class="img_center" width="500" height="450" alt="Europe community on Github" /></a> + +This one shows interesting features. Some countries are really isolated. If we look at Spain, we can see a community of Ruby programmers, with an important connectivity between them, but no really strong connection with any foreign developers. We can clearly see the Perl community exists as only one community, and is not split by country. The same is true for Python. + +## Japanese hackers community + +> properties of the graph: 559 nodes / 5276 edges + +<a href="http://www.flickr.com/photos/franck_/4456129655/" title="Japan community on github by franck.cuny, on Flickr"><img src="http://farm3.static.flickr.com/2800/4456129655_8c6f7f20a0.jpg" class="img_center" width="500" height="410" alt="Japan community on github" /></a> + +This community is unique on github. In 2007, <a href="http://github.com/yappo">Yappo</a> created <a href="http://coderepos.org/share/">coderepos.org</a>, a repository for open source developers in Japan. It was a subversion repository, with Trac as an HTTP front-end. It gathered around 900 developers, with all kind of projects (Perl, Python, Ruby, Javascript, ...). Most of these users have switched to github now. + +Three main communities are visible on this graph: Perl; Ruby; PHP. As always, the Javascript community as a glue between them. And yes, we can confirm that Perl is big in Japan. + +We have seen in the previous graph that the Japanese hackers are always isolated. We can assume that their language is an obstacle. + +This is a really well-connected graph too. + +## Conclusions and graphs + +I may have not provided a deep analysis of all the graph. I don't have knowledge of most of the community outside of Perl. Feel free to <a href="http://franck.lumberjaph.net/blog/graphs.tgz">download the graph</a>, to load them in <a href="http://gephi.org/">Gephi</a>, experiment, and provides your own thoughts. + +I would like to thanks everybody at <a href="http://twitter.com/linkfluence">Linkfluence</a> (<a href="http://twitter.com/gfouetil">guilhem</a> for his advices, <a href="http://twitter.com/cmaussan">camille</a> for giving me time to work on this, and antonin for the amazing poster), who have helped me and let me use time and resources to finish this work. Special thanks to <a href="http://code.google.com/p/tinyaml/">blob</a> for reviewing my prose and <a href="http://twitter.com/cdlm">cdlm</a> for the discussion :) diff --git a/content/post/2010-04-02-github-poster.md b/content/post/2010-04-02-github-poster.md new file mode 100644 index 0000000..ff70d4d --- /dev/null +++ b/content/post/2010-04-02-github-poster.md @@ -0,0 +1,26 @@ +--- +date: 2010-04-02T00:00:00Z +title: Github Poster +--- + +The "Github poster":http://fr.linkfluence.net/posters/ is available as a PDF on the "linkfluence atlas":http://fr.linkfluence.net/insights-2-0/atlas/. + +It's distributed under a "Attribution-Noncommercial-No Derivative Works 3.0 Unported":http://creativecommons.org/licenses/by-nc-nd/3.0/ Creative Commons license, and you are free to print it. + +It's optimized for a "A2":http://en.wikipedia.org/wiki/A2_paper_size size. You shouldn't have any problem to print it at a bigger size though, as it's vectorial. + +We ("linkfluence":http://linkfluence.net/) really wanted to print and send the poster to people who were interested. But this is too complicated and too much work to handle by ourselves. If you do print a poster, please send us a notice, for we would love to know where it may end up :) + +However, for lazy/busy people, I plan to attend the following Perl conferences : + + * "French Perl Workshop":http://journeesperl.fr/fpw2010/ + * "Belgian Perl Workshop":http://www.perlworkshop.be/bpw2010/ + * "YAPC::EU":http://conferences.yapceurope.org/ye2010/ + +so if you are interested in buying a poster and contact me early enough, I'll print it and bring it with me. The cost should be between 35 and 50 euros per poster (this is the raw cost). + +I would like to thank all the people who emailed me, and I'm really sorry I'm unable to provide each of you a poster. + +<center> +<a href="http://fr.linkfluence.net/posters/"><img style="border: 1px solid #000;" src="http://fr.linkfluence.net/wp-content/images/atlas/github_thumb.png" alt="poster" /></a> +</center> diff --git a/content/post/2010-04-03-more-fun-with-tatsumaki-and-plack.md b/content/post/2010-04-03-more-fun-with-tatsumaki-and-plack.md new file mode 100644 index 0000000..abfcf15 --- /dev/null +++ b/content/post/2010-04-03-more-fun-with-tatsumaki-and-plack.md @@ -0,0 +1,166 @@ +--- +date: 2010-04-03T00:00:00Z +summary: In which I have fun with Tatsumaki. +title: More fun with Tatsumaki and Plack +--- + +Lately I've been toying a lot with [Plack](http://plackperl.org/) and two Perl web framework: [Tatsumaki](http://search.cpan.org/perldoc?Tatsumaki) and [Dancer](http://search.cpan.org/perldoc?Dancer). I use both of them for different purposes, as their features complete each other. + +## Plack + +If you don't already know what Plack is, you would want to take a look at the following Plack resources: + +* [Plack (redesigned) website](http://plackperl.org) +* [Plack documentation](http://search.cpan.org/perldoc?Plack) +* [Miyagawa's screencast](http://bulknews.typepad.com/blog/2009/11/plack-and-psgi-screencast-and-feedbacks.html) +* [Plack advent calendar](http://advent.plackperl.org/) + +> As [sukria](http://www.sukria.net/) is planning to talk about [Dancer](http://perldancer.org) during the [FPW 2010](http://journeesperl.fr/fpw2010/index.html), I will probably do a talk about Plack. + +After reading some code, I've started to write two middleware: the first one add ETag header to the HTTP response, and the second one provides a way to limit access to your application. + +### Plack::Middleware::ETag + +This middleware is really simple: for each request, an [ETag](http://en.wikipedia.org/wiki/HTTP_ETag) header is added to the response. The ETag value is a sha1 of the response's content. In case the content is a file, it works like apache, using various information from the file: inode, modified time and size. This middleware can be used with [Plack::Middleware::ConditionalGET](http://search.cpan.org/perldoc?Plack::Middleware::ConditionalGET), so the client will have the ETag information for the page, and when he will do a request next time, it will send an "if-modified" header. If the ETag is the same, a 304 response will be send, meaning the content have not been modified. This module is [available on CPAN](http://search.cpan.org/perldoc?Plack::Middleware::ETag). + +Let's see how it works. First, we create a really simple application (we call it app.psgi): + +```perl +#!/usr/bin/env perl +use strict; +use warnings; +use Plack::Builder; + +builder { + enable "Plack::Middleware::ConditionalGET"; + enable "Plack::Middleware::ETag"; + sub { + ['200', ['Content-Type' => 'text/html'], ['Hello world']]; + }; +}; +``` + +Now we can test it: + +```bash +% plackup app.psgi& +% curl -D - http://localhost:5000 +HTTP/1.0 200 OK +Date: Sat, 03 Apr 2010 09:31:43 GMT +Server: HTTP::Server::PSGI +Content-Type: text/html +ETag: 7b502c3a1f48c8609ae212cdfb639dee39673f5e +Content-Length: 11 + +% curl -H "If-None-Match: 7b502c3a1f48c8609ae212cdfb639dee39673f5e" -D - http://localhost:5000 +HTTP/1.0 304 Not Modified +Date: Sat, 03 Apr 2010 09:31:45 GMT +Server: HTTP::Server::PSGI +ETag: 7b502c3a1f48c8609ae212cdfb639dee39673f5e +``` + +### Plack::Middleware::Throttle + +[With this middleware](http://git.lumberjaph.net/p5-plack-middleware-throttle.git/), you can control how many times you want to provide an access to your application. This module is not yet on CPAN, has I want to add some features, but you can get the code from git. There is four methods to control access: + +* Plack::Middleware::Throttle::Hourly: how many times in one hour someone can access the application +* P::M::T::Daily: the same, but for a day +* P::M::T::Interval: which interval the client must wait between two query +* by combining the three previous methods + +To store sessions informations, you can use any cache backend that provides `get`, `set` and `incr` methods. By default, if no backend is provided, it will store informations in a hash. You can easily modify the defaults throttling strategies by subclassing all the classes. + +Let's write another application to test it: + +```perl +#!/usr/bin/env perl +use strict; +use warnings; +use Plack::Builder; + +builder { + enable "Plack::Middleware::Throttle::Hourly", max => 2; + sub { + ['200', ['Content-Type' => 'text/html'], ['Hello world']]; + }; +}; +``` + +then test + +```bash +% curl -D - http://localhost:5000/ +HTTP/1.0 200 OK +Date: Sat, 03 Apr 2010 09:57:40 GMT +Server: HTTP::Server::PSGI +Content-Type: text/html +X-RateLimit-Limit: 2 +X-RateLimit-Remaining: 1 +X-RateLimit-Reset: 140 +Content-Length: 11 + +Hello world + +% curl -D - http://localhost:5000/ +HTTP/1.0 200 OK +Date: Sat, 03 Apr 2010 09:57:40 GMT +Server: HTTP::Server::PSGI +Content-Type: text/html +X-RateLimit-Limit: 2 +X-RateLimit-Remaining: 0 +X-RateLimit-Reset: 140 +Content-Length: 11 + +Hello world + +% curl -D - http://localhost:5000/ +HTTP/1.0 503 Service Unavailable +Date: Sat, 03 Apr 2010 09:57:41 GMT +Server: HTTP::Server::PSGI +Content-Type: text/plain +X-RateLimit-Reset: 139 +Content-Length: 15 + +Over rate limit +``` + +Some HTTP headers are added to the response : + +* **X-RateLimit-Limit**: how many request can be done +* **X-RateLimit-Remaining**: how many requests are available +* **X-RateLimit-Reset**: when will the counter be reseted (in seconds) + +This middleware could be a very good companion to the [Dancer REST stuff](http://www.sukria.net/fr/archives/2010/03/19/let-the-dancer-rest/) [added recently](/easily-create-rest-interface-with-the-dancer-1.170/). + +## another Tatsumaki application with Plack middlewares + +To demonstrate the use of this two middleware, [I wrote a small application](http://git.lumberjaph.net/p5-feeddiscovery.git/) with Tatsumaki. This application fetch a page, parse it to find all the feeds declared, and return a JSON with the result. + +```bash +% GET http://feeddiscover.tirnan0g.org/?url=http://lumberjaph.net/blog/ +``` + +will return + +```javascript +% [{"href":"http://lumberjaph.net/blog/index.php/feed/","type":"application/rss+xml","title":"i'm a lumberjaph RSS Feed"}] +``` + +This application is composed of one handler, that handle only **GET** request. The request will fetch the url given in the **url** parameter, scrap the content to find the links to feeds, and cache the result with Redis. The response is a JSON string with the informations. + +The interesting part is the app.psgi file: + +```perl +my $app = Tatsumaki::Application->new(['/' => 'FeedDiscovery::Handler'],); + +builder { + enable "Plack::Middleware::ConditionalGET"; + enable "Plack::Middleware::ETag"; + enable "Plack::Middleware::Throttle::Hourly", + backend => Redis->new(server => '127.0.0.1:6379',), + max => 100; + $app; +}; +``` + +The application itself is really simple: for a given url, the Tatsumaki::HTTPClient fetch an url, I use [Web::Scraper](http://search.cpan.org/perldoc?Web::Scraper) to find the **link rel="alternate"** from the page, if something is found, it's stored in Redis, then a JSON string is returned to the client. diff --git a/content/post/2010-04-14-presque-a-redis-tatsumaki-based-message-queue.md b/content/post/2010-04-14-presque-a-redis-tatsumaki-based-message-queue.md new file mode 100644 index 0000000..ef3463c --- /dev/null +++ b/content/post/2010-04-14-presque-a-redis-tatsumaki-based-message-queue.md @@ -0,0 +1,87 @@ +--- +date: 2010-04-14T00:00:00Z +summary: In which I introduce presque +title: presque, a Redis / Tatsumaki based message queue +--- + +[presque](http://github.com/franckcuny/presque/tree/) is a small message queue service build on top of [redis](http://code.google.com/p/redis/) and [Tatsumaki](http://search.cpan.org/perldoc?Tatsumaki). It's heavily inspired by [RestMQ](http://github.com/gleicon/restmq) and [resque](http://github.com/defunkt/resque). + +* Communications are done in JSON over HTTP +* Queues and messages are organized as REST resources +* A worker can be writen in any language that make a HTTP request and read JSON +* Thanks to redis, the queues are persistent + +## Overview + +resque need a configuration file, writen in YAML that contains the host and port for the Redis server. + +```yaml +redis: + host: 127.0.0.1 + port: 6379 +``` + +Let's start the server: + +```bash +% plackup app.psgi --port 5000 +``` + +The applications provides some HTTP routes: + +* **/**: a basic HTML page with some information about the queues +* **/q/**: REST API to get and post job to a queue +* **/j/**: REST API to get some information about a queue +* **/control/**: REST API to control a queue (start or stop consumers) +* **/stats/**: REST API to fetch some stats (displayed on the index page) + +Queues are created on the fly, when a job for an unknown queue is inserted. When a new job is created, the JSON send in the POST will be stored "as is". There is no restriction on the schema or the content of the JSON. + +Creating a new job simply consist to : + +```bash +% curl -X POST "http://localhost:5000/q/foo" -d '{"foo":"bar", "foo2":"bar" }' +``` + +and fetching the job: + +```bash +% curl "http://localhost:5000/q/foo" +``` + +When a job is fetched, it's removed from the queue. + +## A basic worker + +I've also pushed [presque::worker](http://git.lumberjaph.net/p5-presque-worker.git/). It's based on [AnyEvent::HTTP](http://search.cpan.org/perldoc?AnyEvent::HTTP) and [Moose](http://search.cpan.org/perldoc?Moose). Let's write a basic worker using this class: + +```perl +use strict; +use warnings; +use 5.012; # w00t + +package simple::worker; +use Moose; +extends 'presque::worker'; + +sub work { + my ($self, $job) = @_; + say "job's done"; + ...; # yadda yadda! + return; +} + +package main; +use AnyEvent; + +my $worker = + simple::worker->new(base_uri => 'http://localhost:5000', queue => 'foo'); + +AnyEvent->condvar->recv; +``` + +A worker have to extends the presque::worker class, and implement the method *work*. When the object is created, the class check if this method is avalaible. You can also provide a `fail` method, which will be called when an error occur. + +## The future + +I plan to add support for [websocket](http://en.wikipedia.org/wiki/WebSocket), and probably [XMPP](http://en.wikipedia.org/wiki/Xmpp). More functionalities to the worker too: logging, forking, handling many queues, ... I would like to add priorities to queue also, and maybe scheluding job for a given date (not sure if it's feasable with Redis). diff --git a/content/post/2010-04-19-the-dancer-ecosystem.md b/content/post/2010-04-19-the-dancer-ecosystem.md new file mode 100644 index 0000000..c31145f --- /dev/null +++ b/content/post/2010-04-19-the-dancer-ecosystem.md @@ -0,0 +1,107 @@ +--- +date: 2010-04-19T00:00:00Z +summary: In which we look at Dancer's ecosystem. +title: The Dancer Ecosystem +--- + +Even though it's still a young project, an active community is starting to emerge around <a href="http://search.cpan.org/perldoc?Dancer">Dancer</a>. Some modules start to appear on CPAN and github to add functionalities, or to extend existing ones. + +## Templates + +By default, Dancer comes with support for two templating systems: <a href="http://search.cpan.org/dist/Template-Toolkit/">Template Toolkit</a> and Dancer::Template::Simple, a small templating engine written by <a href="http://www.sukria.net/">sukria</a>. But support for other templating systems are available: + + + * <a href="http://search.cpan.org/perldoc?Dancer::Template::Tenjin">Dancer::Template::Tenjin</a> by ido + * <a href="http://search.cpan.org/perldoc?Dancer::Template::Sandbox">Dancer::Template::Sandbox</a> by Sam Graham + * <a href="http://search.cpan.org/perldoc?Dancer::Template::Tiny">Dancer::Template::Tiny</a> by Sawyer + * <a href="http://search.cpan.org/perldoc?Dancer::Template::MicroTemplate">Dancer::Template::MicroTemplate</a> by me + * <a href="http://search.cpan.org/perldoc?Dancer::Template::Mason">Dancer::Template::Mason</a> by Yanick Champoux + * <a href="http://search.cpan.org/perldoc?Dancer::Template::Haml">Dancer::Template::Haml</a> by David Moreno + +## Logger + +Out of the box, Dancer only has a simple logging system to write to file, but more logging sytems are available: + +<ul> +<li><a href="http://search.cpan.org/perldoc?Dancer::Logger::Syslog">Dancer::Logger::Syslog</a> by sukria</li> +<li><a href="http://search.cpan.org/perldoc?Dancer::Logger::LogHandler">Dancer::Logger::LogHandler</a> by me</li> +<li><a href="http://git.lumberjaph.net/p5-dancer-logger-psgi.git/">Dancer::Logger::PSGI</a> by me</li> +</ul> + +The last one is for writing directly your log message via <a href="http://search.cpan.org/perldoc?Plack">Plack</a>. You can use a middleware like <a href="http://search.cpan.org/~miyagawa/Plack-0.9932/lib/Plack/Middleware/LogDispatch.pm">P::M::LogDispatch</a> or <a href="http://search.cpan.org/~miyagawa/Plack-0.9932/lib/Plack/Middleware/Log4perl.pm">P::M::Log4perl</a> to handle logs for your application. Even better, if you use <a href="http://github.com/miyagawa/Plack-Middleware-ConsoleLogger">P::M::ConsoleLogger</a>, you can have logs from your Dancer application in your javascript console. + +## Debug + +To debug your application with Plack, you can use the awesome <a href="http://search.cpan.org/perldoc?Plack::Middleware::Debug">Plack::Middleware::Debug</a>. I've writen <a href="http://git.lumberjaph.net/p5-dancer-debug.git/">Dancer::Debug</a> (which requires my fork of <a href="http://github.com/franckcuny/Plack-Middleware-Debug">P::M::Debug</a>), a middleware that add panels, with specific informations for Dancer applications. + +<img src="/imgs/4535496880_37e5e68a57_z.webp" alt="Dancer::Debug middleware" /> + +To activate this middleware, update your app.psgi to make it look like this: + +```perl +my $handler = sub { + my $env = shift; + my $request = Dancer::Request->new($env); + Dancer->dance($request); +}; +$handler = builder { + enable "Debug", panels => [ + qw/Dancer::Settings Dancer::Logger Environment Memory + ModuleVersions Response Session Parameters Dancer::Version / + ]; + $handler; +}; +``` + +## Plugins + +Dancer has support for plugins since a few version. There is not a lot of plugins at the moment, but this will soon improve. Plugins support is one of the top priorities for the 1.2 release. + +### Dancer::Plugin::REST + +<a href="http://github.com/sukria/Dancer-Plugin-REST">This one is really nice</a>. This plugin, used with the serialization stuff, allow you to write easily REST application. + +```perl +resource user => get => sub { # return user where id = params->{id} }, + create => sub { # create a new user with params->{user} }, + delete => sub { # delete user where id = params->{id} }, + update => sub { # update user with params->{user} }; +``` + +And you got the following routes: + + * GET /user/:id + * GET /user/:id.:format + * POST /user/create + * POST /user/create.:format + * DELETE /user/:id + * DELETE /user/:id.:format + * PUT /user/:id + * PUT /user/:id.:format + +### Dancer::Plugin::Database + +<a href="http://github.com/bigpresh/Dancer-Plugin-Database">This plugin</a>, by bigpresh, add the <strong>database</strong> keyword to your app. + +```perl +use Dancer; +use Dancer::Plugin::Database; + +# Calling the database keyword will get you a connected DBI handle: +get '/widget/view/:id' => sub { + my $sth = database->prepare('select * from widgets where id = ?', + {}, params->{id}); + $sth->execute; + template 'display_widget', {widget => $sth->fetchrow_hashref}; +}; +``` + +### Dancer::Plugin::SiteMap + +<a href="http://search.cpan.org/perldoc?Dancer::Plugin::SiteMap">With this plugin</a>, by James Ronan, a <a href="http://en.wikipedia.org/wiki/Sitemap">sitemap</a> of your application is created. + +<blockquote>Plugin module for the Dancer web framwork that automagically adds sitemap routes to the webapp. Currently adds /sitemap and /sitemap.xml where the former is a basic HTML list and the latter is an XML document of URLS.</blockquote> + +## you can help! :) + +There is still a lot of stuff to do. Don't hesitate to come on #dancer@irc.perl.org to discuss ideas or new features that you would like. diff --git a/content/post/2010-06-10-moosex-net-api-update.md b/content/post/2010-06-10-moosex-net-api-update.md new file mode 100644 index 0000000..161daa7 --- /dev/null +++ b/content/post/2010-06-10-moosex-net-api-update.md @@ -0,0 +1,133 @@ +--- +date: 2010-06-10T00:00:00Z +summary: In which I write an update about MooseX::Net::API +title: Moosex::Net::API - update +--- + +[MooseX::Net::API](http://git.lumberjaph.net/p5-moosex-net-api.git/) is a module to help writing clients for RESTful (and even non-RESTful) WebServices: + +```perl +package my::api; +use MooseX::Net::API; + +net_api_declare myapi => ( + api_base_url => 'http://api....', + api_format => 'json', +); + +net_api_method users => ( + method => 'GET', + path => '/users/:country', + description => 'fetch a list of users', + params => [qw/country/], + expected => [qw/200 404/], +); +``` + +We've been using this module at work for the last few months on various internal APIs, and I'm pretty pleased with the result so far. + +Lately I've started to rework the core. I've tried to split most of the functionalities into roles, and rework the code that generates the various methods. I've also added methods to access miscellaneous information : + +```perl +my $client = my::api->new; + +# to get a list of API methods +$client->meta->get_all_net_api_methods(); + +# find an API method +my $method = $client->meta->find_net_api_method_by_name('users'); + +# and now informations about the method +say $method->documentation; + +name: users +description: fetch a list of useres +method: GET +path: /users/:country +arguments: country +expected: 200, 404 +``` + +It's not yet complete, but a new version will be available soon on CPAN. Here is a list of some more features I plan to add quickly: + +* better internal API +* better authorization support (OAuth!) +* add more methods to provide better introspection +* better unserialization +* more tests and better documentation +* generate POD via a PODWeaver plugin ? +* plugins ? +* renaming ? (not sure it really fits in the MooseX:: namespace) + +## http-console + +I've also started [Net::HTTP::Console](http://git.lumberjaph.net/p5-net-http-console.git/). It's inspired by [http-console](http://github.com/cloudhead/http-console). It relies on MX::Net::API, and can use any libraries written with MX::Net::API, as well as any **raw** RESTful API. As an example, let's use it on twitter. + +```bash +% http-console --url http://api.twitter.com --format json + +http://127.0.0.1:5984> GET /1/statuses/public_timeline +[ + { + "source" : "web", + "favorited" : false, + "geo" : null, + "coordinates" : null, + "place" : null, + ... + } +] + +http://127.0.0.1:5984> show headers +cache-control: no-cache, no-store, must-revalidate, pre-check=0, post-check=0 +last-modified: Mon, 07 Jun 2010 15:27:12 GMT +x-transaction: 1275924432-94882-31146 +x-ratelimit-reset: 1275925258 +... +``` + +You can call any method from the twitter API (at the exception of the ones that require authentication: it's not supported yet). + +You can also use it with any library that uses MX::Net::API: + +```bash +% http-console --lib Net::Backtweet + +http://api.backtweet.com> help command +available commands: +- tweets\_by_url +- stats\_by_url +- good\_tweets_by_url + +http://api.backtype.com> help command tweets\_by_url +name: tweets_by_url +description: Retrieve tweets that link to a given URL, whether the links are shortened or unshortened. +method: GET +path: /tweets/search/links + +http://api.backtype.com> stats\_by_url {"q":"http://lumberjaph.net","key":s3kr3t"} +{ + "tweetcount" : 388 +} +``` + +Arguments to the methods are serialized in JSON format. Not sure if it's the best idea I will see if it needs improvement while using it. You can also perform POST and PUT with content. + +```bash + http://localhost:5984> POST /test_rtgi_fetcher {"foo":"bar"} + { + "ok" : true, + "rev" : "1-fe67006eb0e02e5f0057b5b2a6672391", + "id" : "fe3175615a34eb28153479307c000f26" + } +``` + +It's far from being complete at the moment, but I will extend it quickly. Right now, you can define global headers, and get help for all methods in your MX::Net::API library. Authentication is on top of my priority list, as is alias creation, so instead of doing (on a non-moosex::net::api lib): `GET /users/` you will do: + +> alias users/:country as users + +then: + +> users {"country":"france"} + +(and yes, I've switched from wordpress to [blawd](http://github.com/perigrin/blawd)). diff --git a/content/post/2010-06-13-fpw2010-summary.md b/content/post/2010-06-13-fpw2010-summary.md new file mode 100644 index 0000000..00601c0 --- /dev/null +++ b/content/post/2010-06-13-fpw2010-summary.md @@ -0,0 +1,29 @@ +--- +date: 2010-06-13T00:00:00Z +summary: In which I summarize my FPW experience. +title: FPW 2010 summary +--- + +First, no more [welsh](http://en.wikipedia.org/wiki/Welsh_rarebit). Ever. + +Even if Calais was not the easiest destination for every one, it was a really fun and intersting two days. I met nice folks, had great discussions and drink good beers. + +For those who missed this workshop, a short summary for you: + + * had excellent discussions with [sukria](http://www.sukria.net/fr/) about the future of [Dancer](http://github.com/perldancer/dancer) + * fun facts about time zone and unicode with [maddingue](http://twitter.com/maddingue), [rgs](http://twitter.com/octoberequus) and [fperrad](http://github.com/fperrad) + * interesting chat with fperrad about lua and parrot + * convinced more people that Plack *is* the future for Perl Web development + * beers, and more beers with sukria, jerome, [Arnaud](http://twitter.com/ephoz), rgs, [Camille](http://twitter.com/cmaussan) and [Stephane](http://twitter.com/straux). + * talked with [Marc](http://www.tinybox.net/) about Plack, Dancer, and other stuff + +!http://farm5.static.flickr.com/4045/4695068097_1193f8c4d6.webp(diner)! + +My slides are available online (in french): + + * [GitHub explorer](http://franck.lumberjaph.net/blog/slides/github-explorer.pdf) + * [Introduction to plack](http://franck.lumberjaph.net/slides/introduction_a_plack.pdf) + +And to finish, two other summaries: [sukria's one](http://www.sukria.net/fr/archives/2010/06/12/french-perl-workshop-2010-report/) and [twitter's timeline](http://twitter.com/#search?q=%23fpw2010); and [some pictures on flickr](http://www.flickr.com/photos/franck_/sets/72157624263416548/). + +**Thanks to Laurent and Sebastien for their hard work on organizing this conference.** diff --git a/content/post/2010-06-20-dancer-meeting.md b/content/post/2010-06-20-dancer-meeting.md new file mode 100644 index 0000000..eeef14a --- /dev/null +++ b/content/post/2010-06-20-dancer-meeting.md @@ -0,0 +1,31 @@ +--- +date: 2010-06-20T00:00:00Z +summary: In which I share some notes from June's Dancer meeting. +title: Monthly Dancer meeting +--- + +I've been contributing to Dancer for a few months now, and the discussions occurs mainly on IRC (irc.perl.org, #dancer) or on [the Mailing List](http://lists.perldancer.org/cgi-bin/listinfo/dancer-users). + +Last weekend, I had the occasion to meet sukria during the French Perl Workshop. This has been really productive, we had the occasion to talk about Plack, the templating system, websocket, ... and I really think we should have met before. It was also the occasion to meet another contributor, eiro, with whom I've been able to share some knowledge about Plack. + +During the workshop, I made a talk about Plack and the Middlewares. The direct result of this is [the last feature added by sukria](http://github.com/sukria/Dancer/commit/5ee83a5206e08256d7326f92c2f2f62c5e035ba9#L0R440): middlewares can be set in the configuration file of your Dancer application, and will be loaded for you. + +The next release of Dancer won't generate an **app.psgi** file anymore, so you will only need to edit your environment file (like **deployement.yaml**), and add the following configuration: + +```yaml +warnings: 1 +auto_reload: 1 +plack_middlewares: + Debug: + - panels + - + - Response + - Dancer::Version + - Dancer::Settings +``` + +and your application will load some [Plack::Middleware::Debug](http://search.cpan.org/perldoc?Plack::Middleware::Debug) and [Dancer::Debug](http://search.cpan.org/dist/Dancer-Debug/) panels. + +Sukria has suggested a monthly drinkup meeting for people in/near Paris, to talk about Dancer and Plack, in a pub or another place where we can bring a laptop, have some beers and share idea/codes and other exchange technicals thoughts. + +I hope to meet more Dancer developers and users in a near future (sawyer at Pisa maybe ?). diff --git a/content/post/2010-06-25-presque-new_features.md b/content/post/2010-06-25-presque-new_features.md new file mode 100644 index 0000000..98cbad2 --- /dev/null +++ b/content/post/2010-06-25-presque-new_features.md @@ -0,0 +1,41 @@ +--- +date: 2010-06-25T00:00:00Z +summary: In which I show some new features for presque. +title: presque +--- + +I've added a few new features to [presque](http://github.com/franckcuny/presque). + +[presque](/presque-a-redis-tatsumaki-based-message-queue/) is a persistant job queue based on [Redis](http://github.com/antirez/redis) and [Tatsumaki](http://github.com/miyagawa/Tatsumaki). + +A short list of current features implemented: + +* jobs are JSON object +* possibility to stop/start queues +* jobs can be delayed to run after a certain date in the future +* workers can register themself, doing this, you can know when a worker started, what he have done, ... +* statistics about queue, jobs, and workers +* possible to store and fetch jobs in batch +* a job can be unique + +The REST interface is simple, and there is only a few methods. It's fast (I will provide numbers soon from our production environment), and workers can be implemented in any languages. + +There have been a lot of refactoring lately. The main features missing right now are a simple HTML interface that will display various informations, pulling the data from the REST API (hint : if someone want to help to design this one ... :) ), websocket (sending a message to all workers). + +There is a Perl client to the REST API: [net::presque](http://git.lumberjaph.net/p5-net-presque.git/), that you can use with [net::http::console](http://git.lumberjaph.net/p5-net-http-console.git/): + +```bash +% perl bin/http-console --api_lib Net::Presque --url http://localhost:5000 +http://localhost:5000> fetch_job {"queue_name":"twitter_stream"} +{ + "text" : "Australias new prime minister - julia gillard is our 27th prime minister.", + "user" : "Lov3LifeAlways" +} +``` + +I've also wrote a better [worker for Perl](http://git.lumberjaph.net/p5-presque-worker.git/). It's a Moose::Role that you apply to your class. You need to write a **work** method, and your done. This worker handle retries, provide a logger, ... As for [resque](http://github.com/defunkt/resque), there is two dispatcher: + +* normal : the worker grab a job, process it, then ask for the next job +* fork : the worker grab a job, fork, let the child do the job and exit, while the parent ask for the next job. As resque says, "Resque assumes chaos". And me too, I like (ordered) chaos + +I hope to finish the documentation and to writes one or two more workers as example (maybe in Python and javascript/node.js) soon to be able to tag a first version, and to collect some info about how many jobs have been processed at work (we use it to do url resolution and collect twitter data among few other things). Although I'm not sure I will release it to CPAN. diff --git a/content/post/2010-06-30-github-poster-to-ship.md b/content/post/2010-06-30-github-poster-to-ship.md new file mode 100644 index 0000000..d33a1bd --- /dev/null +++ b/content/post/2010-06-30-github-poster-to-ship.md @@ -0,0 +1,14 @@ +--- +date: 2010-06-30T00:00:00Z +summary: In which I try to see if anyone is interested in buying a poster. +title: Github Communities Posters for shipping +--- + +I've finally found a printer that can do international shipping for reasonable costs. The price will be 65€, and a paypal account will be setup soon. + +<center> +<a href="http://fr.linkfluence.net/posters/"><img style="border: 1px solid #000;" alt="poster" src="http://fr.linkfluence.net/wp-content/images/atlas/github_thumb.png" /></a> +</center> + +So, if you're interested by a poster in A1 size, send me a mail. I'll need at least 15 persons to be able to do this. So contact me, and I will keep you informed. + diff --git a/content/post/2010-09-10-dancer-summer-of-code.md b/content/post/2010-09-10-dancer-summer-of-code.md new file mode 100644 index 0000000..4ebbe09 --- /dev/null +++ b/content/post/2010-09-10-dancer-summer-of-code.md @@ -0,0 +1,128 @@ +--- +date: 2010-09-10T00:00:00Z +summary: In which I talk about Dancer's Summer of Code +title: Dancer's Summer of Code +--- + +### Middleware + +After the [French Perl Workshop](http://journeesperl.fr/fpw2010/), we decided to focus our efforts on bringing middleware into Dancer. As the .psgi script is now obsolete, we wanted to simplify the middleware configuration for users not familiar with Plack. + +It's now possible to load a middleware by adding it to your configuration: + +```yaml +plack_middlewares: + Debug: + - panels + - + - DBITrace + - Memory + - Timer +``` + +### YAPC::Eu 2010 + +During YAPC::EU, I've been happy to meet with [squeeks](http://github.com/squeeks), [sawyer](http://blogs.perl.org/users/sawyer_x/) and [Martin Berends](http://github.com/mberends). Sadly, we didn't have much time to talk and do some coding. + +I had already met Martin during the FPW, where he started to port Dancer to Perl6. His first objective is to have [HTTP::Server::Simple](http://github.com/mberends/http-server-simple) work. I was really impressed with his works; if I manage to find some spare time soon, I will join his effort. + +### Dancer's application + +In august, [Alexis](http://www.sukria.net/) brought a big effort to refactor the core of Dancer, to add the possibility to "plug" components to your application. This required a lot of rewriting, but in the meantime, we added more tests to ensure nothing would break. + +With this feature, you can do the following: + +```perl +package myapp::forum; +use Dancer ':syntax'; + +before => sub { + ... +}; + +get '/' => sub { + ... +}; + +package myapp:blog; +use Dancer ':syntax'; + +load_app 'myapp::forum', prefix => '/forum'; + +before => sub { + ... +}; + +get '/' => sub { + ... +}; +``` + +Now you can request **/** and **/forum**. The before filter declared in the package **myapp::forum** will be executed when the **/forum** path is matched, and the filter in **myapp::blog** will be executed for **/**. + +### QA + +The weekend following the YAPC::EU, we held a small hackaton/QA day on irc. Not many people were present, but we managed to achieve some results: + + * reached the 1K tests + * documentation cleanup + * added Hooks + * improved our code coverage + +Today our code average is over 92%, and we have more than 1200 tests. + +With the new hook system, two new keywords have been added: **before_template** and **after**. They work as the **before** keyword, except the **before_template** is executed before sending the tokens to the template, so you can modify them (a small example can be found in the [Dancer::Plugin::i18n](http://git.lumberjaph.net/p5-dancer-plugin-18n.git/)). The **after** is executed before the response is sent to the user. + +Sukria has also set up an autobuild system for our two main branches. Every 15 minutes, the test suite is executed when there is a new commit, and builds a report. Code coverage is also measured, so we can always know the state of our various development cycles. + +### WebSocket + +This is the question that came back from time to time: when/will Dancer support websocket ? + +We investigated various ways to do this: + + * new async. handler + * writing our own implementation + * ... + +I didn't want to write a websocket implementation for Dancer, as the spec are not yet final and it's not easy to do. Thanks to [clkao](http://github.com/clkao), we didn't have to care about all this, as he already wrote a Plack middleware for this: [Web::Hippie](http://search.cpan.org/perldoc?Web::Hippie::Pipe). + +So, what we did, is to use this middleware and add some syntactic sugar so people can use it easily in their applications. A small application is available [here](http://git.lumberjaph.net/p5-dancer-chat.git/). + +This is not yet complete, it's only available in the 'devel' branch, and subject to change. A small code sample: + +```perl +websocket '/new_listener' => sub { + my $env = request->env; + my $room = $env->{'hippie.args'}; + my $topic = $env->{'hippie.bus'}->topic($room); + $env->{'hippie.listener'}->subscribe($topic); +}; + +websocket '/message' => sub { + my $env = request->env; + my $room = $env->{'hippie.args'}; + my $topic = $env->{'hippie.bus'}->topic($room); + + my $msg = $env->{'hippie.message'}; + $msg->{time} = time; + $msg->{address} = $env->{REMOTE_ADDR}; + $topic->publish($msg); +}; +``` + +As you can see, a lot of stuff can be improved quite easily in terms of syntax. + +### Deployment + +We're also in the process of reworking our current Deployment documentation. Lots of people are trying to deploy Dancer using various configurations, and not all are well documented, or don't work as expected. If you use Dancer, and have deployed an application in a way not documened in our Deployement documentation, please join us on irc (#dancer on irc.perl.org) or contact us on the mailing list, or even better, send us a patch, so we can improve this part. + +### Future + +There is also our next Dancer's meeting meeting to organize, at the end of Septembre. + +In October will take place the 2nd OSDC.fr, where I will talk about Plack, and alexis will present Dancer. + +I want to thank my company ([Linkfluence](http://linkfluence.net)) and my boss ([Camille](http://twitter.com/cmaussan)) for giving me time to code on Dancer at work. + +As always, thanks to blob for reviewing my (slightly improving) english :) diff --git a/content/post/2010-09-17-spore.md b/content/post/2010-09-17-spore.md new file mode 100644 index 0000000..9046325 --- /dev/null +++ b/content/post/2010-09-17-spore.md @@ -0,0 +1,165 @@ +--- +date: 2010-09-17T00:00:00Z +summary: In which I introduce SPORE. +title: SPORE +--- + +## Specification to a POrtable Rest Environment + +More and more web services offer [ReST API](http://en.wikipedia.org/wiki/Representational_State_Transfer). Every time you want to access one of these services, you need to write a client for this API, or find an existing one. Sometimes, you will find the right library that will do exactly what you need. Sometimes you won't, and you end up writing your own. + +Some parts of an API client are always the same: + +* build an uri +* make a request +* handle the response +* ... + +With SPORE, I propose a better solution for this. SPORE is composed of two parts: + +* a specification that describes an API +* a "framework" for clients (think this as [WSGI](http://www.python.org/dev/peps/pep-0333/), [Plack](http://plackperl.org/), [Rack](http://rack.rubyforge.org/), [JSGI](http://jackjs.org/jsgi-spec.html), ...) + +## API Specifications + +I know, at this point, you're thinking "what ? isn't it just what [WSDL](http://en.wikipedia.org/wiki/Web_Services_Description_Language) does ?". Well, yes. But it's (in my opinion) simpler to write this than to write a WSDL. And when you say "WSDL" people think "[SOAP](http://en.wikipedia.org/wiki/SOAP_(protocol))", and that's definitly not a good thing. + +The first part is the specification to API. A ReST request is mainly : + +* a **HTTP method** +* a **path** +* some **parameters** + +This is **easy to describe**. For this example, I will use the [twitter API](http://dev.twitter.com/doc/get/statuses/public_timeline). So, if i want to describe the user timeline method, we will get something like this: + +```yaml +public_timeline: + method: GET + path: /statuses/public_timeline.:format + params: + - trim_user + - include_entities + required: + - format +``` + +Whatever your language of choice is, you'll always need this informations. The idea with the API description, is that it can be reused by every language. If the API provider publishes this file, everyone can easily use it. It's very similar to a documentation (for the twitter description, all I needed to do was to copy/paste the documentation, it's really that simple) but it can be used by a framework to generate a client. + +The specifications should be in JSON (I've written the example in YAML for the sake of readability). The complete description of the specifications are available [here](https://github.com/SPORE/specifications). + + +There is many advantages to do it this way: + +* if you have a client in Perl and Javascript, the names of the methods are the same in both langages, and the names of parameters too +* if the API changes some endpoints, you don't have to change your code, you only need to update the description file + +I've started to write some specifications for a few services ([twitter](https://github.com/SPORE/api-description/blob/master/services/twitter.json), [github](https://github.com/SPORE/api-description/blob/master/services/github.json), [backtype](https://github.com/franckcuny/spore/blob/master/services/backtype.json), [backtweet](https://github.com/franckcuny/spore/blob/master/services/backtweet.json), ...) and applications ([couchdb](https://github.com/franckcuny/spore/blob/master/apps/couchdb.json), [presque](https://github.com/franckcuny/spore/blob/master/apps/presque.json)). They are not complete yet, so you're welcomed to [fork the repository](https://github.com/franckcuny/spore), add missings methods, and add your own specifications! :) + +## Client Specification + +Now that we have a simple description for the API, we want to have [an easy solution to use it](https://github.com/franckcuny/net-http-spore/blob/master/spec/spore_implementation.pod). I will describe [the Perl implementation](https://github.com/franckcuny/net-http-spore), but there is also one for Ruby (will be published soon), and a early version for [Clojure](http://github.com/ngrunwald/clj-spore) and [Python](http://github.com/elishowk/pyspore). + +This kind of thing is really easy to implement in dynamic languages, and still doable in others. + +The client is composed of two parts: core and middlewares. + +The core will create the appropriate functions using the previous description. Thanks to metaprogramming, it's very easy to do it. If we use [Moose](http://search.cpan.org/perldoc?Moose), all I need to do, is to extend the [Moose::Meta::Method](http://search.cpan.org/perldoc?Moose::Meta::Method) and add new attributes like: + +* path +* method +* params +* authentication +* ... + +For each method declared in the description, I build a new Moose method I will attach to my class. Basicaly, the code looks like this: + +```perl +foreach my $method_name ( keys %$methods_spec ) { + $class->meta->add_spore_method( + "user_timeline", + path => '/statuses/public_timeline.:format', + required => [qw/format/], + params => [qw/trim_user include_entities/] + ); +} +``` + +The code of the `user_timelime` method will be generated via the `add_spore_method`. + +Middlewares are the nice part of it. By default, the core only creates a request, executes it, and gives you the result. Nothing more. By adding middlewares, you can handle the following stuff: + +* headers manipulation +* authentication (basic, OAuth, ) +* (de)serialization (JSON, XML, YAML, CSV, ...) +* caching +* proxying +* ... + +<img src='/imgs/chart.webp' alt='schema'> + +The most obvious middleware is the one that handles the format. When you load the middleware Format::JSON, it will set various headers on your request. In case of a GET method, the **Accept** header will be set to **application/json**. For a POST, the **Content-Type** will be also set. Before returning the result to the client, the content will be transformed from JSON to a Perl structure. + +For twitter, I can have a client with this few lines: + +```perl +my $client = Net::HTTP::Spore->new_from_spec('twitter.json'); +$client->enable('Format::JSON'); + +my $timeline = $client->public_timeline( format => 'json', include_rts => 1 ); +my $tweets = $timeline->body; +foreach my $tweet (@$tweets) { + say $tweet->{user}->{screen_name} . " says " . encode_utf8($tweet->{text}); +} +``` + +Now, I want to use my friends timeline, which requires OAuth ? easy + +```perl +$client->enable( + 'Auth::OAuth', + consumer_key => $consumer_key, + consumer_secret => $consumer_secret, + token => $token, + token_secret => $token_secret, +); +my $friends_timeline = $client->friends_timeline(format => 'json', include_rts => 1); +my $tweets = $friends_timeline->body; +foreach my $tweet (@$tweets) { + print $tweet->{user}->{screen_name} . " says " . encode_utf8($tweet->{text}) . "\n"; +} +``` + +Middlewares are easy to write. They should implement a *call* method, which receive a request object as argument. The middleware can return: + +* nothing: the next middleware will be executed +* a callback: it will be executed when the request is done, and will receive a response object +* a response object: no more middlewares will be executed + +A simple middleware that add a runtime header to the response object will look like this: + +```perl +sub call { + my ( $self, $req ) = @_; + + my $start_time = [Time::HiRes::gettimeofday]; + + $self->response_cb( + sub { + my $res = shift; + my $req_time = sprintf '%.6f', + Time::HiRes::tv_interval($start_time); + $res->header( 'X-Spore-Runtime' => $req_time ); + } + ); +} +``` + +I've tried to mimic as much as possible Plack's behavior. The result of a request is a Response object, but you can also use it as an arrayref, with the following values [http\_code, [http\_headers], body]. + +## Conclusion + +The real target for this are not API developers (even if it's useful to have this when you write your own API, I will show some examples soon), neither client developers (even it's really easier to do with this), but people who want to play immediatly with an API to fetch data, without the coding skill or knowledge of what an HTTP request is, how to define headers, what is OAuth, ... As it was suggested to me, an implementation for [R](http://en.wikipedia.org/wiki/R_(programming_language)) would be really usefull to a lot of people. + +Right now, I'm looking for people interested by this idea/project, and to work on completing the specification. I'm pretty happy with the current status, as it works with most API I've encountered. + +I will present SPORE and its implementations [during OSDC.fr](http://act.osdc.fr/osdc2010fr/) next month. diff --git a/content/post/2010-09-27-jitterbug.md b/content/post/2010-09-27-jitterbug.md new file mode 100644 index 0000000..d512ec1 --- /dev/null +++ b/content/post/2010-09-27-jitterbug.md @@ -0,0 +1,49 @@ +--- +date: 2010-09-27T00:00:00Z +summary: In which I introduce Jitterbug. +title: jitterbug - a first step to continuous integration for Perl modules on GitHub +--- + +> If you'd like to be a jitter bug, +> First thing you must do is get a jug, +> Put whiskey, wine and gin within, +> And shake it all up and then begin. + +Earlier this month sukria set up a [shell script](http://github.com/sukria/capsule) to run the Dancer's tests suit. The script is simple, but does what we want: + +* every 15 minutes, it pulls from the github repository +* for every perl installed via perlbrew, it executes the tests suite +* it stores the results in text file +* finally, it creates a code coverage report for the master branch + +The only problem I had with this is that it wasn't really easy to find the appropriate test report you might want to check. + +That's when I decided to write an interface to this: [jitterbug](https://github.com/franckcuny/jitterbug). + +## Demo + +[You can check Dancer's version](http://jitterbug.perldancer.org/). The interface is really simple: there's a list of repositories, the status of the last build, and a link to a list of all the project's builds. + +<center>!/imgs/jitterbug.webp(jitterbug)!</center> + +## How it works + +For each project you want to use with jitterbug, you set the url of the HTTP hook in the administration page of your project. Each time a push is detected, GH will send you a notification (you can see details of the content). If the project doesn't already exist in your setup, it will be created, so you don't need to maintain or update a configuration file each time you create a new repository. The notification creates a task in a queue, and a script pulls the task and executes it. + +> If you don't want anyone to use your setup for building their modules, you can protect the **/hook/** path with a .htaccess, and use the following kind of url : http://user:pass@.../ + +The script (`builder.pl`, provided with JB) will clone the repository, switch to the given commit and run the tests. It works both with classic distribution having a Makefile.PL and Dist::Zilla setups. The output is stored on the disk and some information about the project will be updated. [You can see the result for Dancer](http://jitterbug.perldancer.org/project/Dancer). + +This script relies on two important things: [perlbrew](http://github.com/gugod/App-perlbrew) and [cpanminus](http://github.com/miyagawa/cpanminus). So your tests will be executed against every Perl version you have installed and thanks to cpanminus, all the deps will be installed too. + +If the test fails, a mail is sent to the author. + +## What's next + +A list of some features I want to have: + +* IRC notifications (something like: [author's name]: link to the commit - commit message - link to the build) +* customizable builder script (so you can add your own hook to building a package, having private notifications, etc.) +* simple administration (restart a build, etc.) + +(thanks to [sawyer](http://github.com/xsawyerx) and [sukria](http://github.com/sukria/)). diff --git a/content/post/2010-10-04-how-to-contribute-to-dancer.md b/content/post/2010-10-04-how-to-contribute-to-dancer.md new file mode 100644 index 0000000..a3a5e33 --- /dev/null +++ b/content/post/2010-10-04-how-to-contribute-to-dancer.md @@ -0,0 +1,38 @@ +--- +date: 2010-10-04T00:00:00Z +summary: In which I explain how to contribute to Dancer. +title: How to contribute to Dancer +--- + +For our development projects, we rely a lot on Github. Lately, more and more people started contributing to Dancer, but not all of them are familiar with Github or git. Here is a little step-by-step guide on how to contribute. You don't need to be a Perl expert to contribute, you can provide help by correcting documentation error, or adding a new recipe to our cookbook. + +## the code + +The main repository is hosted [here](http://github.com/perldancer/dancer). There are two main branches: + +* master +* devel + +In the master branch we accept only bug fixes and doc fixes/updates. The current master branch will be the future 1.2 version. + +The devel branch is where we add new features, or improve existing features. + +## contributing + +First, go to [github.com/perldancer/dancer](http://github.com/perldancer/dancer) and click on the "fork" button. Now, here is a little tutorial on how to fetch the repository, list the local and remote branches, and track the remote devel branch. + +Now that you know what the purpose of each branch is, you can decide to work on master or devel (`git checkout devel` to switch branch). + +## sending your patch + +As I've previously stated, we rely a lot on the github features and interface. So now you've written your patch. First, be sure to provide one or more tests, and to run the test suite (with `make test` or `prove -r t/`). If all the tests pass, you can send a pull request. For this, you go on your own fork on github (http://github.com/$user/dancer), and you click on the "Pull Request" button. + +You can at any time see all the commits done by others that have not yet been merged into one of our branches at [this url](http://github.com/perldancer/Dancer/forkqueue). + +## reporting and/or fixing bugs + +We prefer to use the github issue tracker instead of RT. So if you want to report a bug, go [there](http://github.com/perldancer/dancer/issues). + +If your commit fixes a bug reported there, please add in your commit message something like 'fixing GH #xxx" where xxx is the bug id. + +Thank you to everyone who have contributed so far! diff --git a/content/post/2010-10-12-osdcfr.md b/content/post/2010-10-12-osdcfr.md new file mode 100644 index 0000000..929f9b6 --- /dev/null +++ b/content/post/2010-10-12-osdcfr.md @@ -0,0 +1,107 @@ +--- +date: 2010-10-12T00:00:00Z +summary: In which I summarize my OSDC.fr experience +title: OSDC.fr report +--- + +This weekend I went to the second edition of the OSDC.fr conference. This conference is organized by the Python, Ruby, PHP and Perl french organizations. This edition was really really good, and well organized (kudo to the whole team!). + +The first day of the conference, we had two excellents talk about Git. The first one by [mojombo](http://github.com/mojombo), about [advanced git usages](http://git-tips.heroku.com/#1). I've managed to get him to sign my copy of the [GitHub poster](http://lumberjaph.net/graph/2010/03/25/github-explorer.html). The second one by BooK was about his module [Git::Repository](http://search.cpan.org/perldoc?Git::Repository) (which I use for [jitterbug](http://github.com/franckcuny/jitterbug)). He show us how he used git to graph his familly tree. + +<img src="/imgs/github-sig-small.webp" alt="github poster" /> + +Germain did an [introduction to Riak](http://www.slideshare.net/franckcuny/riak-a-file-system-for-internet), and [Julian](http://twitter.com/flngr) did a talk about [Gephi](http://gephi.org/), about how it relies on the netbeans platform, and why a tool to visualize graphs is useful. + +I've talked about Plack in the afternoon, and [sukria](http://sukria.net) talked about Dancer right after me. I think both our talks went well, and his one was a nice demonstration of Dancer, since he used [Broadway](http://github.com/sukria/broadway) to write his slides. I planned to do some demos during my talk, but a problem with my laptop prevented me to do this. Anyway, if you attended my talk and want to try them, here they are : + +```perl +use strict; +use warnings; +use Plack::Builder; + +my $app = sub { + return [ + 200, + [ 'Content-Type' => 'text/html' ], + [ '<html><body>Hello World</body></html>' ] + ]; +}; + +builder { + enable 'Debug'; + $app; +}; +``` + +```perl +use strict; +use warnings; +use Plack::Builder; + +my $app1 = sub {[200, ['Content-Type' => 'text/html'], ['hello from app1']]}; +my $app2 = sub {[200, ['Content-Type' => 'text/html'], ['hello from app2']]}; + +builder { + mount "/app1" => $app1; + mount "/app2" => $app2; +}; +``` + +```perl +use strict; +use warnings; +use Plack::Builder; + +my $app = sub { [ 200, [ 'Content-Type' => 'text/html' ], ['hello world'] ] }; + +my $middleware = sub { + my $env = shift; + my $res = $app->($env); + $res->[2]->[0] =~ s/world/OSDC.fr/; + return $res; +}; +``` + +```perl +use strict; +use warnings; +use Plack::Builder; + +my $app = sub { + die "die"; + [ 200, [ 'Content-Type' => 'text/html' ], ['hello world'], ]; +}; + +builder { + enable "StackTrace"; + $app; +}; +``` + +```perl +use strict; +use warnings; +use Plack::Builder; + +my $app = sub { return [ '200', [ 'Content-Type' => 'text/html' ], ['hello world'] ] }; + +builder { + enable "Throttle::Hourly", max => 1; + $app; +}; +``` + +The evening GitHub offered the beers (yeah!) in a pub, and I had an intersting talk with [fperrad](http://github.com/fperrad) about SPORE. Francois surprised me earlier last week when he contacted me to announce me he add write a Lua version of SPORE. He had some questions and suggestions for the current spec and API, and we managed to handle all the issues. + +The sunday I talked about [SPORE](http://github.com/franckcuny/spore). Even if the talk went not so well, I managed to interest some people, since I had some questions and positive feedback. I've not seen much talk the sunday, as I've spent some time with sukria and others to discuss about code. But I managed to see the talk about Redis which was good, and gave me some ideas for [presque](http://github.com/franckcuny/presque). + +The ligthning talks were also interesting. [Bruno Michel](http://twitter.com/brmichel) talked about [EventMachine](http://rubyeventmachine.com/), [htty](http://github.com/htty) and [mongrel2](http://mongrel2.org/home), [dolmen](http://search.cpan.org/~dolmen/) showed us a small application to graph a svn repository, and [BooK](http://search.cpan.org/~book/) told us about his work on [Devel::TraceUse](http://search.cpan.org/perldoc?Devel::TraceUse). + +I wish I would have been able to attend [oz](http://twitter.com/ephoz) presentation of [nodejs](http://cyprio.net/nodejs_osdc.pdf), and I've also missed the talks given by carl masak, since I was working with sukria on Dancer at that time. + +All the slides for my talks are available: + + * [Plack](https://docs.google.com/presentation/d/1Ty8df3IG44rwHRuBnpgriTHiKREqGN_aIJcABddFIKM/present#slide=id.i0) + * [SPORE](https://docs.google.com/presentation/d/1JpDz3F9K41KVFQ878IEFAnHQZ05BpGJz6dpdPi21tjU/present#slide=id.i0) + * [jitterbug](https://docs.google.com/presentation/d/1-fpByFQ4OJoe2NE8lMkwl-zbK8OWuLqGA_Y4B4YmFhA/present#slide=id.i0) + * [presque](https://docs.google.com/presentation/d/1Y9TZh9A4an9ctC6mEZcONkx_9z9b66TJ5Lg4awVIoiA/present#slide=id.i0) diff --git a/content/post/2010-10-20-spore-update.md b/content/post/2010-10-20-spore-update.md new file mode 100644 index 0000000..5f9cc91 --- /dev/null +++ b/content/post/2010-10-20-spore-update.md @@ -0,0 +1,329 @@ +--- +date: 2010-10-20T00:00:00Z +summary: In which I share some update on SPORE. +title: SPORE update +--- + +As I've said [in my OSDC report](http://lumberjaph.net/conference/2010/10/12/osdcfr.html), after I [presented SPORE](http://www.slideshare.net/franckcuny/spore) I've had some positive feedback. In the last ten days, I've created a [google group](http://groups.google.com/group/spore-rest) to discuss the current specification and implementations, a [SPORE account on github](http://github.com/SPORE) to hold the implementation specifications and the API descriptions files, and more importantly, we have some new implementations: + +* [Ruby](http://github.com/sukria/Ruby-Spore) +* [node.js](http://github.com/francois2metz/node-spore) +* [Javascript](http://github.com/nikopol/jquery-spore) (client side) +* PHP (not published yet) + +in addition to the already existing implementations: + +* [Perl](http://github.com/franckcuny/net-http-spore) +* [Lua](http://github.com/fperrad/lua-Spore) +* [Clojure](http://github.com/ngrunwald/clj-spore) +* [Python](http://github.com/elishowk/pyspore) + +In this post, I'll try to show some common usages for SPORE, in order to give a better explanation of why I think it's needed. + +## Consistency + +```lua +require 'Spore' + +local github = Spore.new_from_spec 'github.json' + +github:enable 'Format.JSON' +github:enable 'Runtime' + +local r = github:user_information{format = 'json', username = 'schacon'} + +print(r.status) --> 200 +print(r.headers['x-runtime']) --> 126ms +print(r.body.user.name) --> Scott Chacon +``` + +```perl +use Net::HTTP::Spore; + +my $gh = Net::HTTP::Spore->new_from_spec('github.json'); + +$gh->enable('Format::JSON'); +$gh->enable('Runtime'); + +my $r= $gh->user_information( format => 'json', username => 'schacon' ); + +say "HTTP status => ".$r->status; # 200 +say "Runtime => ".$r->header('X-Spore-Runtime'); # 128ms +say "username => ".$r->body->{user}->{name}; # Scott Chacon +``` + +```ruby + +require 'spore' + +gh = Spore.new(File.join(File.dirname(__FILE__), 'github.json')) + +gh.enable(Spore::Middleware::Runtime) # will add a header with runtime +gh.enable(Spore::Middleware::Format::JSON) # will deserialize JSON responses + +# API call +r = gh.user_information( :format => 'json', :username => 'schacon' ) + +puts "HTTP status => ".r.code # 200 +puts "Runtime => ".r.header('X-Spore-Runtime') # 128ms +puts "username => ".r.body['user']['name'] # Scott Chacon +``` + +As you can see in the previous example, I do the same request on the GitHub API: fetch informations from the user "mojombo". In the three languages, the API for the client is the same: + +* you create a client using the github.json API description +* you enable some middlewares +* you execute your request: the method name is the same, the argument names are the same! +* you manipulate the result the same way + +## Easy to switch from a language to another + +You can switch from a language to another without any surprises. If you must provide an API client to a third-party, you don't have to care about what languages they use, you only need to provide a description. Your methods call will be the same between all the languages, so it's easy to switch between languages, without the need to chose an appropriate client for your API (if one exists), to read the documentation (when there is one), and having the client implementation going in your way. + +## Better maintanability + +What annoys me the most when I want to use an API, is that I have to choose between two, three, or more clients that will communicate with this API. I need to read the documentations, the code, and test thoses implementations to decide which one will best fit my needs, and won't go in my way. And what if I need to do caching before the content is deserialized ? And what if the remote API changes it's authentication method (like twitter, from basic auth to OAuth) and the maintainer of the client doesn't update the code ? + +With SPORE, you don't have to maintain a client, only a description file. Your API changes, all you have to do is to update your description, and all the clients, using any language, will be able to use your new API, without the need to release a new client specific for this API in javascript, Perl, Ruby, ... + +## Easy to use with APIs that are compatible + +If you want to use the Twitter public timeline: + +```perl +use Net::HTTP::Spore; + +my $client = Net::HTTP::Spore->new_from_spec('twitter.json'); + +$client->enable('Format::JSON'); + +my $r = $client->public_timeline( format => 'json' ); + +foreach my $t ( @{ $r->body } ) { + my $username = Encode::encode_utf8( $t->{user}->{name} ); + my $text = Encode::encode_utf8( $t->{text} ); + say $username . " says " . $text; +} +``` + +And now on statusnet: + +```perl +use Net::HTTP::Spore; + +my $client = Net::HTTP::Spore->new_from_spec( + 'twitter.json', + base_url => 'http://identi.ca/api' +); + +$client->enable('Format::JSON'); + +my $r = $client->public_timeline( format => 'json' ); + +foreach my $t ( @{ $r->body } ) { + my $username = Encode::encode_utf8( $t->{user}->{name} ); + my $text = Encode::encode_utf8( $t->{text} ); + say $username . " says " . $text; +} +``` + +easy, right ? As both APIs are compatible, the only thing you need to do is change the argument **base_url** when you create your new client. + +## It's easy to write a description + +It's really easy to write a description for your API. Let's take a look at the +one for github: + +```json +{ + "base_url" : "http://github.com/api/v2/", + "version" : "0.2", + "methods" : { + "follow" : { + "required_params" : [ + "user", + "format" + ], + "path" : "/:format/user/follow/:user", + "method" : "POST", + "authentication" : true + } + }, + "name" : "GitHub", + "authority" : "GITHUB:franckcuny", + "meta" : { + "documentation" : "http://develop.github.com/" + } +} +``` + +The important parts are the basic API description (with a name, a base url for the API) and the list of available methods (here I've only put the 'follow' method). + +More descriptions are available on [GitHub](http://github.com/SPORE/api-description), as well as and the [full specification](http://github.com/SPORE/specifications/blob/master/spore_description.pod). + +We also have [a schema](http://github.com/SPORE/specifications/blob/master/spore_validation.rx) to validate your descriptions. + +## Middlewares + +By default, your SPORE client will only do a request and return a result. But it's easy to alter the default behavior with various middlewares. The most obvious one is the deserialization for a response, like the previous example with github and the middleware Format::JSON. + +### Control your workflow + +The use of middlewares allow you to control your workflow as with Plack/Rack/WSGI. You can easily imagine doing this: + +* check if the request has already been made and cached +* return the response if the cache is still valid +* perform the request +* send the content to a remote storer in raw format +* cache the raw data locally +* deserialize to json +* remove some data from the response +* give the response to the client + +Or to interrogate a site as an API: + +* send a request on a web page +* pass the response to a scraper, and put the data in JSON +* return the JSON with scraped data to the client + +### Creating a repository on GitHub + +In this example, we use a middleware to authenticate on the GitHub API: + +```perl +use Config::GitLike; +use Net::HTTP::Spore; + +my $c = Config::GitLike::Git->new(); $c->load; + +my $login = $c->get(key => 'github.user'); +my $token = $c->get(key => 'github.token'); + +my $github = Net::HTTP::Spore->new_from_spec('github.json'); + +$github->enable('Format::JSON'); +$github->enable( + 'Auth::Basic', + username => $login . '/token', + password => $token, +); + +my $res = $github->create_repo( + format => 'json', + payload => {name => $name, description => $desc} +); +``` + +The middleware Auth::Basic will add the **authorization** header to the request, using the given tokens. + +### SPORE + MooseX::Role::Parameterized + +I really like [MooseX::Role::Parameterized](http://search.cpan.org/perldoc?MooseX::Role::Parameterized). This module allows you to build dynamically a Role to apply to your class/object: + +```perl +package My::App::Role::SPORE; + +use MooseX::Role::Parameterized; +use Net::HTTP::Spore; + +parameter name => ( isa => 'Str', required => 1, ); + +role { + my $p = shift; + my $name = $p->name; + + has $name => ( + is => 'rw', + isa => 'Object', + lazy => 1, + default => sub { + my $self = shift; + my $client_config = $self->context->{spore}->{$name}; + my $client = Net::HTTP::Spore->new_from_spec( + $client_config->{spec}, + %{ $client_config->{options} }, + ); + foreach my $mw ( @{ $client_config->{middlewares} } ) { + $client->enable($mw); + } + }, + ); +}; + +1; + +# in your app + +package My::App; + +use Moose; + +with 'My::App::Role::SPORE' => { name => 'couchdb' }, + 'My::App::Role::SPORE' => { name => 'url_solver' }; + +1; +``` + +This Role will add two new attributes to my class: **couchdb** and **url_solver**, reading from a config file a list of middlewares to apply and the options (like base_uri). + +## Testing my application that uses CouchDB + +This is a common case. In your application you use CouchDB to store some information. When you run the tests for this application, you don't know if there will be a couchdb running on the host, if it will be on the default port, on what database should you do your tests, ... + +The Perl implementation of SPORE comes with a Mock middleware: + +```perl +package myapp; + +use Moose; +has couchdb_client => (is => 'rw', isa => 'Object'); + +use Test::More; +use JSON; + +use myapp; + +use Net::HTTP::Spore; + +my $content = { title => "blog post", website => "http://lumberjaph.net" }; + +my $mock_server = { + '/test_database/1234' => sub { + my $req = shift; + $req->new_response( + 200, + [ 'Content-Type' => 'application/json' ], + JSON::encode_json($content) + ); + }, +}; + +ok my $client = + Net::HTTP::Spore->new_from_spec( + '/home/franck/code/projects/spore/specifications/apps/couchdb.json', + base_url => 'http://localhost:5984' ); + +$client->enable('Format::JSON'); +$client->enable( 'Mock', tests => $mock_server ); + +my $app = myapp->new(couchdb_client => $client); + +my $res = + $app->client->get_document( database => 'test_database', doc_id => '1234' ); +is $res->[0], 200; +is_deeply $res->[2], $content; +is $res->header('Content-Type'), 'application/json'; + +done_testing; + +``` + +The middleware catches the request, checks if it matches something defined by the user and returns a response. + +## So ... + +I really see SPORE as something Perlish: a glue. The various implementations are a nice addition, and I'm happy to see some suggestions and discussions about the specifications. + +I'm pretty confident that the current specification for the API description is stable at this point. We still need to write more middlewares to see if we can cover most of the usages easily, so we can decide if the specification for the implementation is valid. + +(as always, thanks to bl0b!). diff --git a/content/post/2010-10-25-perl-moderne.md b/content/post/2010-10-25-perl-moderne.md new file mode 100644 index 0000000..0aa3cc2 --- /dev/null +++ b/content/post/2010-10-25-perl-moderne.md @@ -0,0 +1,27 @@ +--- +date: 2010-10-25T00:00:00Z +summary: In which I do a review of the book 'Moderne Perl' +title: Perl Moderne +--- + +Exercice différent aujourd’hui, puisqu’il s’agit d’une critique d’un livre. + +Il y a quelque jours, j’ai reçu une copie du livre [Perl moderne](http://www.pearson.fr/livre/?GCOI=27440100979970), à paraitre le 29 octobre aux éditions Pearson. Il est intéressant à plus d’un titre : c’est un livre original, pas une traduction d’un énième livre sur Perl ; il se concentre sur le Perl dit “Moderne”, c’est-à-dire les outils comme [Moose](http://search.cpan.org/perldoc?Moose), [DBIx::Class](http://search.cpan.org/perldoc?DBIx::Class), etc ; il est écrit par des personnes impliquées dans la communauté Perl (ce sont des auteurs de modules CPAN, qui organisent des conférences Perl). + +J’en profite pour saluer au passages les auteurs de l’ouvrage: [Maddingue](http://github.com/maddingue), [BooK](http://github.com/book), [jq](http://github.com/jquelin) et [dams](http://github.com/dams). + +La première bonne surprise est la taille du livre : au format poche. Je trouve ça pratique pour le transporter dans son sac (usager du métro, bonjour) et pour le laisser traîner sur le bureau sans qu’il prenne de la place. La seconde bonne surprise est le nombre de sujets abordés : une introduction solide aux base de Perl ; la programmation objet; les expressions régulières; les bases de données ; les manipulations de fichier XML ; et les outils pour travail sur le web. + +Un des points très positif du livre est sa partie sur la programmation objet. Les auteurs ont fait le choix de présenter Moose comme étant “Le” modèle objet à utiliser. Ils n’abordent pas du tout le modèle objet de base de Perl (pas d’explication de bless, etc), mais ça me semble un choix judicieux dans le cadre de l’apprentissage de Perl. Si la personne vient d’un autre langage objet, elle se retrouve tout de suite avec des bases qu’elle connaît (accesseurs, méthodes, héritages, etc) et des paradigmes probablement nouveaux, comme les Roles. Si la personne connaît déjà Perl, pas de redites. + +N’aimant pas particulièrement le SQL, j’ai trouvé agréable d’avoir une bonne introduction à DBIx::Class, qui vient compléter la partie sur DBI. Les bases de données dites “NoSQL” sont également présentées, avec notamment CouchDB. + +Enfin, la dernière partie est celle consacrée au web, avec la présentation des outils pour manipuler du HTML, réccuperer du contenu, ou encore automatiser des comportements sur des pages avec WWW::Mechanize. + +Par ailleurs, si après la lecture de cet ouvrage il vous vient à l’idée de vouloir récupérer du contenu sur des pages web à l’aide d’expression régulières, il est fort probable que vous ayez lu ce livre à l’envers, ou c’est par pur goût de la provocation envers les auteurs. + +A, et un point négatif, me diriez vous ? Bien sûr que j’en ai un. A mon grand regret, nulle part dans l’ouvrage il n’est fait référence à [Dancer](http://github.com/perldancer/dancer). Tant pis ! + +Dans l’ensemble c’est un bon livre pour qui veut découvrir Perl en 2010. Tous les outils que l’on s’attend à utiliser au quotidien sont présentés. L’organisation du livre, et le fait d’avoir de nombreux exemples, seront pratiques pour les débutants. Je pense en commander quelques exemplaires pour le travail, afin de le mettre à disposition de nos (futurs) stagiaires. + +[Perl Moderne](http://www.pearson.fr/livre/?GCOI=27440100979970) est publié par [Pearson](http://www.pearson.fr/), parution le 29 octobre. ISBN: [978-2-7440-2419-1](http://www.amazon.fr/Perl-S%C3%A9bastien-Aperghis-Tramoni/dp/2744024198/ref=sr_1_1?ie=UTF8&qid=1288034071&sr=8-1) (22 €). diff --git a/content/post/2010-11-22-vagrant-rocks.md b/content/post/2010-11-22-vagrant-rocks.md new file mode 100644 index 0000000..14549c6 --- /dev/null +++ b/content/post/2010-11-22-vagrant-rocks.md @@ -0,0 +1,172 @@ +--- +date: 2010-11-22T00:00:00Z +summary: In which I share my enthusiasm for Vagrant +title: Vagrant rocks +--- + +## tl;dr + +I've been toying with [vagrant](http://vagrantup.com/) lately, and it **really rocks**. You should definitly give it a try. If you're only looking for some resources to get started with it, go there: + + * [introduction](http://docs.vagrantup.com/v2/why-vagrant/) + * [google group](http://groups.google.com/group/vagrant-up) + +## What is Vagrant + +"Vagrant is a tool for building and distributing virtualized development environments." This sentence summarizes perfectly the project. + +The idea is to use [Chef](http://www.opscode.com/chef) on top of [VirtualBox](http://www.virtualbox.org/) to deploy a VM like you would deploy a server in your production environment. + +I won't go into the details to describe Chef and VirtualBox, but here is a quick reminder. Chef is a framework to deploy infrastructures. It's written in ruby, it uses **cookbooks** to describe how to deploy stuff, and VirtualBox is a virtualization software from Oracle. + +> A little disclaimer. I don't use Chef outside from vagrant, so I may say/do some stupid things. The aim of this tutorial is not about writing a recipe for Chef, but to show what you can do thanks to Chef. So don't hesitate to correct me in the comments if I'm doing some utterly stupid things. + +## The basic + +To install vagrant, you'll need ruby and virtualbox. You have the basic instructions detailed [here](http://docs.vagrantup.com/v2/getting-started/). This will explain how to install vagrant and how to fetch a **base** image. + +### Creating a first project + +You'll probably want to start creating a new project now. For this tutorial, I'll create an image for [presque](https://github.com/franckcuny/presque). + +```sh +% mkdir presque +% vagrant init +``` + +This will create a new image for your project, and create a new file in your directory: **Vagrantfile**. Modify this file to make it look like this: + +```ruby +Vagrant::Config.run do |config| + config.vm.box = "base" + config.vm.provisioner = :chef_solo + config.chef.cookbooks_path = "cookbooks" + config.chef.add_recipe("vagrant_main") + config.vm.forward_port("web", 5000, 8080) +end +``` + +These instructions will: + + * tell vagrant to use the image named **base** (a lucid32 image by default) + * use chef in **solo** mode + * the recipes will be in a directory named **cookbooks** + * the main recipe will be named **vagrant_main** + * forward local HTTP port 4000 to 5000 on the VM + +### My recipes + +Now we need to create or use some recipes. First we create our **cookbooks** directory: + +```sh +% mkdir cookbooks +% mkdir -p cookbooks/vagrant_main/recipes +``` + +We need to add some cookbooks. You will find them on [GitHub](https://github.com/opscode/cookbooks). Copy the following cookbooks inside the **cookbooks** repository: + + * apt: instructions on how to use apt + * ubuntu: this one manages the sources and executes **apt-get update** + * build-essential: installs the build-essential package + * git: installs git + * perl: configures CPAN + * runit: will be used to monitor redis and our web application + +Edit **vagrant_main/recipes/default.rb** to add them: + +```ruby +require_recipe "ubuntu" +require_recipe "git" +require_recipe "perl" +require_recipe "redis" +require_recipe "runit" +``` + +If the VM is already started, you can run `vagrant provision` or `vagrant up`. This will deploy the previous cookbooks on the VM. When it's done, you can log on the VM with `vagrant ssh`. + +You'll need to additional recipes: one for redis; one for presque. You'll find them on my [GitHub account](http://git.lumberjaph.net/chef-cookbooks.git/). Copy the two recipes inside your cookbook directory, and execute `vagrant provision` to install them. + +If everything works fine, you should be able to start using presque. Test this: + +```sh +% curl http://localhost:8080/q/foo/ +{"error":"no job"} + +% curl -X POST -H "Content-Type: application/json" -d '{"foo":"bar"}' http://localhost:8080/q/foo/ + +% curl http://localhost:8080/q/foo/ +{"foo":"bar"} +``` + +If everything is fine, you can shut down the VM with `vagrant halt`. + +### Mounting directories + +Instead of pulling from github, you may prefer to mount a local directory on the VM. For this, you'll need to modifiy the **Vagrantfile** to add this: + +```sh +config.vm.share_folder "v-code", "/deployment/code", "~/code/perl5" +config.vm.share_folder "v-data", "/deployment/data", "~/code/data" +``` + +This will mount your local directories **perl5** and **data** under **/deployment/{code,data}** on the VM. So now you can edit your files locally and they will be automagically updated on the VM at once. + +## and now the awesome part + +If you're like me, you may end up with the need to have multiple VMs which will talk to each other. Common scenarios are a VM with the website, and another one with the DB, or one VM with a bunch of API webservices and another with Workers who need to interact with the VM. Rejoice, this kind of stuff is also handled by vagrant! + +Replace the content of the previous **Vagrantfile** with this: + +```ruby +Vagrant::Config.run do |config| + config.vm.box = "base" + config.vm.provisioner = :chef_solo + + config.chef.cookbooks_path = "cookbooks" + + config.vm.define :presque do |presque_config| + presque_config.chef.add_recipe("vagrant_presque") + presque_config.vm.network("192.168.1.10") + presque_config.vm.forward_port("presque", 80, 8080) + presque_config.vm.customize do |vm| + vm.name = "vm_presque" + end + end + + config.vm.define :workers do |workers_config| + workers_config.chef.add_recipe("vagrant_workers") + workers_config.vm.network("192.168.1.11") + workers_config.vm.customize do |vm| + vm.name = "vm_workers" + end + end +end +``` + +In this configuration, we're creating two VMs, **presque** and **workers**. You'll need to create two new cookbooks, one for each new VM (vagrant_presque, with the same content as vagrant_main, and vagrant_workers, with only the recipe for ubuntu and the instructions to install curl). Once it's done, boot the two VMs: + +```sh +% vagrant up presque +% vagrant up workers +``` + +Now let's log on the worker VM + +```sh +% vagrant ssh workers +vagrant@vagrantup:~$ curl http://192.168.1.10:5000/q/foo +{"error":"no job"} +``` + +and voilà. + +## Conclusion + +I've started to use vagrant for all my new personal projects and for most of my stuff at work. I really enjoy using this, as it's easy to create a cookbook or add one, it's easy to setup a multi VM environment, you can share a configuration amongst your coworkers, etc. + +If you haven't started yet using a VM for your own projects, you really should give it a try, or use a simple VirtualBox setup. If you want to read more on the subject, these two blog posts may be relevant: + + * [Why you should be using virtualisation](http://morethanseven.net/2010/11/04/Why-you-should-be-using-virtualisation.html) + * [nothingmuch setup](http://blog.woobling.org/2010/10/headless-virtualbox.html) + +(oh, and BTW, did you notice that [Dancer 1.2](http://search.cpan.org/perldoc?Dancer) is out ?) diff --git a/content/post/2010-12-06-fpw2O11.md b/content/post/2010-12-06-fpw2O11.md new file mode 100644 index 0000000..7ed5008 --- /dev/null +++ b/content/post/2010-12-06-fpw2O11.md @@ -0,0 +1,13 @@ +--- +date: 2010-12-06T00:00:00Z +summary: In which we start to talk about FPW 2011. +title: French Perl Workshop +--- + +I joined the organization team for the next French Perl Workshop, which should take place in Paris in June. I'll help the existing team (composed of Laurent Boivins, Sebastien DeSeille and [maddingue](http://twitter.com/maddingue)) and also the newcomers ([cmaussan](http://twitter.com/cmaussan) and [eiro](http://github.com/eiro)). + +We've already decided to organize it in Paris, since last year it was in [Calais](http://journeesperl.fr/fpw2010/), and the idea is to organize once every two years in Paris. We don't have yet a venue and a date, but we will try to keep you informed about this. + +The main theme should be "Modern Perl", and we would like to reach a wider audience than previous years: people who are not yet in the Perl community, as well as some foreigners (hey, Paris is a beautiful city). + +I'll try to communicate about our progress at least once a month, and find reasons to motivate you to come! :) diff --git a/content/post/2011-02-20-psgichrome.md b/content/post/2011-02-20-psgichrome.md new file mode 100644 index 0000000..af00fc1 --- /dev/null +++ b/content/post/2011-02-20-psgichrome.md @@ -0,0 +1,75 @@ +--- +date: 2011-02-20T00:00:00Z +summary: In which I show how to log from a PSGI app to Chrome. +title: PSGIChrome +--- + +Earlier this month, I've read about this extension: [chromePHP](http://www.chromephp.com/). + +The principle of this extension is to allow you to log from your PHP application to chrome. You may not be aware, but this is something you already have with every web application if you're using Plack. And not only for Chrome, but every webkit navigator, and Firefox too! + +Let's mimic their page. + +## Installation + +1. install [Plack::Middleware::ConsoleLogger](http://search.cpan.org/perldoc?Plack::Middleware::ConsoleLogger) with `cpanm Plack::Middleware::ConsoleLogger` +2. no step 2 +3. no step 3 +4. write a simple PSGI application and log + +```perl +use strict; +use warnings; + +use Plack::Builder; + +my $app = sub { + my $env = shift; + my $content = "<html><body>this is foo</body></html>"; + foreach my $k ( keys %$env ) { + if ( $k =~ /HTTP_/ ) { + $env->{'psgix.logger'}->({ + level => 'debug', + message => "$k => " . $env->{$k}, + }); + } + } + $env->{'psgix.logger'}->({ + level => 'warn', + message => 'this is a warning', + }); + $env->{'psgix.logger'}->({ + level => 'error', + message => 'this is an error', + }); + return [ 200, [ 'Content-Type' => 'text/html' ], [$content] ]; +}; + +builder { + enable "ConsoleLogger"; + $app; +} +``` + +Load this application with plackup: `plackup chromeplack.pl` + +point your browser to http://localhost:5000, activate the javascript console. + +If this works correctly, you should have a smiliar output in your console: + +<img src="/imgs/plack_chrome.webp" alt="plack chrome" /> + +## Dancer + +I don't know for other framework, but you can also log to your browser with [Dancer](http://perldancer.org/). + +First, you need to install [Dancer::Logger::PSGI](http://search.cpan.org/perldoc?Dancer::Logger::PSGI), then, in your application, you need to edit the environment file. You'll certainly want to change 'development.yml'. + +```yaml +logger: "PSGI" +plack_middlewares: + - + - ConsoleLogger +``` + +Now you can start your application (running in a Plack environment, of course), and next time you'll use 'warning' or 'debug' or any other keyword from Dancer::Logger, the message will end up in your javascript console. diff --git a/content/post/2011-03-06-how_to_use_github_effectively_for_your_project.md b/content/post/2011-03-06-how_to_use_github_effectively_for_your_project.md new file mode 100644 index 0000000..011d3ee --- /dev/null +++ b/content/post/2011-03-06-how_to_use_github_effectively_for_your_project.md @@ -0,0 +1,71 @@ +--- +date: 2011-03-06T00:00:00Z +summary: In which we look at how we can use GitHub effectively for your project. +title: how to use GitHub effectively for your project +--- + +<a href="https://github.com/">GitHub</a> provide an awesome set of tools for opensource developers. For <a href="http://perldancer.org">Dancer</a>, we use them as much as possible. I'll show and explain how we do our development. + +<img class="img_center" src="/imgs/github_dancer.webp" alt="dancer" /> + +## code review + +<a href="https://github.com/perldancer/dancer">Dancer</a>'s development goes fast. We do our best to ship often, we do a lot of refactoring, and we listen our users. This means processing pull request and issues as fast as possible. + +## pull request + +There is five developers with write access to the main repository. Each one is asked to do a review of pending pull request every morning (this is not required, neither enforced, it's a "if you have ten minutes in the morning while drinking your cofee, please, review the PR"). + +When we're reviewing something, most of the time we pull the submited patches into a branch named **review/username**. If the reviewer is happy with the modifications, he will add a comment to the pull request: "approved" (and some other comments if it's required). In case he's not happy, he will comment **disaproved** and give his reasons (tests doesn't passes, or there is no tests for the patch provided, or this is not something we want, etc). If the PR is about something the developer doesn't really knows, or has a doubt, he skip this one, and ask for someone else to take a look. + +In order to merge the branch **review/username**, we need two developers to comment **approved**. Of course, for something simple, like a documentation fix, or changing a few line somewhere, we can merge this without applying this process. + +As the work consists to read the code and comment, it's quiete easy to handle two/three pull request each day for a developer. When one of the developper with access to the repository find some time, he can go through the pull request, and merge the one marqued as approved, since he knows that the people who had already approved them understand the code. + +## issues + +We don't use [RT](http://bestpractical.com/rt/) to manage our issues. It's not that RT is bad or that we don't like it, it's just that GitHub's issues are more integrated with our workflow. GitHub's issues are really simple (one can even say naive), but most of the time it's ok for our usage. We don't need advanced features like "track how much time you've spend on this ticket" (even if I do track my time spent on Dancer, using [orgmode](http://orgmode.org/manual/Clocking-commands.html#Clocking-commands)). + +One of the nice feature of GitHub's issues, is that you can close them with a commit. If the commit's message looks like 'closes GH-123', the issue 123 will be closed, with a link to the commit in the comment ([take a look](https://github.com/perldancer/Dancer/issues/249)). I find this feature really useful, since when refering to a closed issue, you can find the commit inside the ticket. + +Once or twice a week, we try to proceed a **triage**, where we go through the issues, and tag them. + +When someone report something on the mailing list or on irc, we ask them if they can open an issue, since it's easier for us to track the request. + +An issue doesn't need to be about a bug, it could be: + +* a feature request (I want to do x or y with dancer) +* something that need to be refactored +* an issue reported by an user that need feedback from the developers + +## commenting on code + +Another nice feature is the possibility to comment the code. Most of the time you'll do it while reviewing a pull request. But a user can also comment on something. + +Sometimes we push a branch that need some feedback, and [a discussion will be started](https://github.com/perldancer/Dancer/commit/d8e79e0d63d0e1b0e05fd36f9e31c378678fccc3). + +## comparing branches + +You can easily diff two branches. With [this url](https://github.com/perldancer/Dancer/compare/master...devel) you can do a quick diff of the changes between master and devel, see the list of commits, and which files have changed. + +This is usefull when you want to have an overview of the work. + +## gitflow + +We're using [gitflow](https://github.com/nvie/gitflow) to work. Gitflow is a nice tool that help you creating and merging branches. If you don't know gitflow, there is [a good article that explain the reasons behind it](http://nvie.com/posts/a-successful-git-branching-model/). + +Ok, this has nothing to do with GitHub, but I'll explain quickly what we do with it. + +We use the following conventions: + +* **master**: only for release +* **devel**: this is the development branch. This one should *always* work (tests can't be broken) +* **topic/$name**: when we start to develop a new feature, we create a topic branch +* **hotfix/$name**: when we need to release a new version right now, without merging devel before, we create a hofrix branch +* **release/$version**: when we're ready to ship, we create a new branch + +It's very rare that we need to push a hotfix or release branch to GitHub: thoses branches had a really short life span. But **topic** branches can be pushed, when we want feedback from users or other developers. + +## future + +We're already using [jitterbug](https://github.com/franckcuny/jitterbug) to do some continuous integration. When we push to GitHub, a payload is posted to jitterbug, and a build is triggered. In a near future, we will use [git's notes](http://progit.org/2010/08/25/notes.html) with jitterbug. The idea is to store inside a note, for each build, the duration of the build, and the result of the build (failure / success) and maybe the TAP output in case of a failure. This will allow developers to see directly in the logs the status. Of course, GiHub already display them, so this will be very useful for all of us. diff --git a/content/post/2011-04-22-new_job.md b/content/post/2011-04-22-new_job.md new file mode 100644 index 0000000..7a70d1e --- /dev/null +++ b/content/post/2011-04-22-new_job.md @@ -0,0 +1,63 @@ +--- +date: 2011-04-22T00:00:00Z +summary: In which I say goodbye to Linkfluence. +title: new job +--- + +I've worked for nearly four years at [Linkfluence](http://linkfluence.net/). From this summer on, I'm switching to a new job at [Say Media](http://saymedia.com/), San Francisco. + +Working at Linkfluence has been an awesome experience for me. I've learned a lot of things, worked on really interesting projects, and worked with an awesome team. + +## Linkfluence is hiring + +Linkfluence is always looking for developers to join the team. If you want a Perl job in France, and want to work on interesting topics such as: + + * social media monitoring + * web crawling + * data indexation + * data visualization + +Linkfluence is the company you want to join. If you're interested, send your CV to Camille Maussang (CTO at Linkfluence): camille.maussang at linkfluence.net. + +### Technologies at Linkfluence + +We're typically using the following technologies to solve our various problems: + + * Solr and ElasticSearch to index the contents of various social media (blogs, tweeter, etc) + * [Riak](http://labs.linkfluence.net/nosql/2011/03/07/moving_from_couchdb_to_riak.html) to store the contents + * a **lot** of Perl (and not the Perl of your (dear) grandmother, but Catalyst, Moose, DBIx::Class, Dancer, perl-5.12, ...) + * Redis + * PostgreSQL + * [Gephi](http://gephi.org/) + +### Open Source at Linkfluence + +Developers here are encouraged to contribute to open source projects, to publish code on GitHub, and talk about the things they do. + +The company also sends the developers to various open source conferences: + + * YAPC::Europe + * FOSDEM + * Belgium Perl Workshop + * French Perl Workshop + * OSDC.fr + +You're encouraged to talk at those conferences, and you can prepare your talks during work time. Linkfluence is also working closely with the French Perl Mongueurs, and tries to organize events with them (technical meetings, Dancer hackaton, etc). + +### Science at Linkfluence + +Most of the developers in the company are coming from a scientific background (but this is not a requirement). Some of them do publish scientific papers on topics like social media and graphs analysis. + + * [Stabilité globale et diversité globale dans la dynamique des commentaires de Flickr](http://www.liafa.jussieu.fr/~prieur/Publis/TSI2444-RauxPrieurV2.pdf) (Raux, Prieur, 2011) + * [Describing the Web in less than 140 characters](http://www.icwsm.org/2011/index.php) (Raux, Grunwald, Prieur, 2011) + * Essai de géographie de la blogosphère politique française (Cardon, Fouetillou, Lerondeau, Prieur 2011) + * [Two paths of glory - Structural positions and trajectories of websites within their topical territory](http://www.icwsm.org/2011/index.php) (Cardon, Fouetillou, Roth 2011) + * [Clustering based on random graph model embedding vertex features](http://linkinghub.elsevier.com/retrieve/pii/S0167865510000413) (Zanghi, Volant, Ambroise) + * [Strategies for Online Inference of Network Mixture](http://lbbe.univ-lyon1.fr/annexes/franck.../SSB-RR-14-online-estimation.pdf) (Zanghi, Picard, Miele, Ambroise) + * Approches modèles pour la structuration du Web vu comme un graphe (Zanghi) + +If you want to work with smart people on interesting topics like graphs, and help them to implement solutions or experiment with some algorithm, you'll feel at home there. + +## My future + +I'll move to San Francisco this summer to start my new job at Say. I have a lot to do before moving, but I'm very excited with this new opportunity. Sadly, I'll probably miss YAPC::EU, but I should do YAPC::NA next year \o/. diff --git a/content/post/2011-05-08-french_perl_workshop.md b/content/post/2011-05-08-french_perl_workshop.md new file mode 100644 index 0000000..f981529 --- /dev/null +++ b/content/post/2011-05-08-french_perl_workshop.md @@ -0,0 +1,29 @@ +--- +date: 2011-05-08T00:00:00Z +summary: In which I remind you of the French Perl Worksphop. +title: French Perl Workshop 2011 +--- + +The call for paper for the [French Perl Workshop](http://journeesperl.fr/fpw2011/) is open. This event will be held the 24th and 25th of June in Paris. As always, this is a free conference. + +<img align="left" alt="logo" "src="/imgs/affiche_fpw11.webp" /> + +## Where + +The workshop will take place in Cité des Sciences in Paris. This is the same place as for last year OSDC.fr. The venue is easy to access using the subway (line 7), and there is a nice park to take a break or take a lunch. + +## Talks + +Yes, we need more talks :) We've already [accepted](http://journeesperl.fr/fpw2011/talks) a little bit more than 10 talks, but we need **more** talks to fill the schedule. An introduction to Perl will also be organized in addition to the talks. We're also looking for some talks about Perl's success in entreprise. If you're using Perl at work, please, come to share with a captive audiance how you use it and how it's helpful. + +## What + +We will organize two dinners, one the thursday night, and a second the friday night. You should check the [wiki](http://journeesperl.fr/fpw2011/wiki) to find more informations. (BooK is also looking for some people to join him to a concert :)) + +## Who + +You! of course. This event will be nothing without you, your talk, and your presence. The whole organizers team is working hard to make this event a great conference. I also want to thank Laurent Boivin for all his efforts and pushing us to do our tasks. + +If you happen to be in Paris next week during [Solution Linux](http://www.solutionslinux.fr/?lg=en), please, stop by to the French Mongueurs booth. + +So, what are you waiting to register ? diff --git a/content/post/2011-06-20-stargit.md b/content/post/2011-06-20-stargit.md new file mode 100644 index 0000000..3c8785e --- /dev/null +++ b/content/post/2011-06-20-stargit.md @@ -0,0 +1,284 @@ +--- +date: 2011-06-20T00:00:00Z +summary: In which we take a look at StarGit. +title: StarGit +--- + +Last year I did a [small exploration of GitHub](http://lumberjaph.net/graph/2010/03/25/github-explorer.html) to show the various communities using [GitHub](http://github.com) and how they work. I wanted to do it again this year, but I was lacking time and motivation to start over. A couple of months ago, I got a message from [mojombo](https://twitter.com/#!/mojombo) asking me if I was planning to do a new poster. This triggered the motivation to work on it again. + +This time I got help from [Alexis](https://twitter.com/#!/jacomyal) to provide you with an awesome tool: [a real explorer of your graph](http://www.stargit.net), but more on this later ;) + +<img class="img_center" src="/imgs/stargit.webp" title="StarGit" alt="stargit" /> + +And of course, [the poster](http://labs.linkfluence.net). Feel free to print it yourself, the size of the poster is A1. + +<img class="img_center" src="/imgs/github-poster-v2.webp" title="GitHub Poster" alt="GitHub Poster" /> + +## The data + +All the data are available! Last year I got some mails asking me for the dataset. So this time I asked first if I could release the [data](http://maps.startigt.net/dump/github.tgz) with the [code](http://git.lumberjaph.net/p5-stargit.git/) and the poster, and the anwser is yes! So if you're intereseted, you can download it. + +The data are stored in mongodb, so I provide the dump which you can easily use: + +```sh +% wget http://maps.stargit.net/dump/github. +% tar xvzf github.tgz +% cd github +% mongorestore -d github . +``` + +Now you can use mongodb to browse the imported database. There is 5 collections: profiles / repositories / relations / contributions / edges. + +## Methodology + +Last year I did a simple "follower/following" graph. It was already interesting, but it was also *really* too simple. This time I wanted to go deeper in the exploration. + +The various step to process all this data are: + +* using the GitHub API, fetch informations from the profiles. +* when all the profiles are collected, informations about the repositories are fetched. Only forked repositories are kept. +* "simple" relations (followers/following) are kept and used later to add weight to relations. +* tag user with the main programming language they use. Using the GitHub API, I was able to categorize ~40k profiles (about 1/3 of my whole dataset). +* using the GeoNames API, extract the name of the country the user is in. This time, about 55k profiles were tagged. +* fetch contributions for each repositories +* compute a score between the author of the contribution and the owner of the repo +* add a weight to each edges, using the computed score and "+1" if the developer follow the other developer + +For all the graphs, I've used the following colors for: + +* <span style="color:#C40C0F">Ruby</span> +* <span style="color:#4C9E97">JavaScript</span> +* <span style="color:#3F9E16">Python</span> +* <span style="color:#8431C4">C (C++, C#)</span> +* <span style="color:#29519E">Perl</span> +* <span style="color:#9D61C4">PHP</span> +* <span style="color:#C4B646">JVM (Java, Clojure, Scala)</span> +* <span style="color:#90C480">Lisp (Emacs Lisp, Common Lisp)</span> +* <span style="color:#9C9E9C">Other</span> + +## Exploring + +Feel free to do your own analysis in the comments :) For each map, you'll find a PDF of the map, and the graph to explore using gephi (in GEXF or GDF format). + +### but first, some numbers + +I've collected: + +* 123 562 profiles +* 2 730 organizations +* 40 807 repositories + +This took me about a month in order to collect the data and to build the adapted tools. + +### Accounts creations + +The following chart show the number of account created by month. "Everyone" means the total of accounts created. You can also see the numbers for each communities. + +On the "Everyone" graph, you can see a huge pick around April 2008, that's the date GitHub [was launched](https://github.com/blog/40-we-launched). + +For most of the communities, the number of created accounts start to decrease since 2010. I think the reason is that most of the developers from those communities are now on GitHub. + +<script language="javascript" type="text/javascript" src="/js/jquery.js"></script> +<script language="javascript" type="text/javascript" src="/js/jquery.flot.js"></script> + +<div id="placeholder" style="width:800px;height:300px;"></div> + +<ul class="actions"> + <li class="minibutton"><input class="fetchSeries" type="button" value="Everyone" href="/json/global.json"></li> + <li class="minibutton"><input class="fetchSeries" type="button" value="C" href="/json/C.json"></li> + <li class="minibutton"><input class="fetchSeries" type="button" value="JVM" href="/json/JVM.json"></li> + <li class="minibutton"><input class="fetchSeries" type="button" value="JS" href="/json/JavaScript.json"></li> + <li class="minibutton"><input class="fetchSeries" type="button" value="Lisp" href="/json/Lisp.json"></li> + <li class="minibutton"><input class="fetchSeries" type="button" value="Perl" href="/json/Perl.json"></li> + <li class="minibutton"><input class="fetchSeries" type="button" value="PHP" href="/json/PHP.json"></li> + <li class="minibutton"><input class="fetchSeries" type="button" value="Python" href="/json/Python.json"></li> + <li class="minibutton"><input class="fetchSeries" type="button" value="Ruby" href="/json/Ruby.json"></li> + <li class="minibutton"><input class="fetchSeries" type="button" value="Uncategorized users" href="/json/Other.json"></li> + <li class="minibutton"><input class="resetSeries" type="button" value="reset"></li> +</ul> + +<script type="text/javascript"> +$(function () { + var options = { + lines: { show: true }, + points: { show: true }, + xaxis: { mode:"time" } + }; + var data = []; + var placeholder = $("#placeholder"); + + $.plot(placeholder, data, options); + + // fetch one series, adding to what we got + var alreadyFetched = {}; + + $("input.resetSeries").click(function() { + alreadyFetched = {}; + data = []; + $.plot(placeholder, data, options); + }); + + $("input.fetchSeries").click(function () { + var button = $(this); + + // find the URL in the link right next to us + var dataurl = button.attr('href'); + + // then fetch the data with jQuery + function onDataReceived(series) { + // extract the first coordinate pair so you can see that + // data is now an ordinary Javascript object + var firstcoordinate = '(' + series.data[0][0] + ', ' + series.data[0][1] + ')'; + + // let's add it to our current data + if (!alreadyFetched[series.label]) { + alreadyFetched[series.label] = true; + data.push(series); + } + + // and plot all we got + $.plot(placeholder, data, options); + } + + $.ajax({ + url: dataurl, + method: 'GET', + dataType: 'json', + success: onDataReceived + }); + }); +}); +</script> + +### Languages + +(Keep in mind that these numbers are coming from the profiles I was able to tag, roughly 40k) + +* Ruby: 10046 (28%) +* Python: 5403 (15%) +* JavaScript: 5282 (15%) (JavaScript + CoffeeScript) +* C: 5093 (14%) (C, C++, C#) +* PHP: 3933 (11%) +* JVM: 3790 (10%) (Java, Clojure, Scala, Groovy) +* Perl: 1215 (3%) +* Lisp: 348 (0%) (Emacs Lisp, Common Lisp) + +Those numbers doesn't really match "what GitHub gave":https://github.com/languages, but it could be explained by the way I've selected my users. + +### Country + +* United States: 19861 (36%) +* United Kingdom: 3533 (6%) +* Germany: 3009 (5%) +* Canada: 2657 (4%) +* Brazil: 2454 (4%) +* France: 1833 (3%) +* Japan: 1799 (3%) +* Russia: 1604 (2%) +* Australia: 1441 (2%) +* China: 1159 (2%) + +The United States are still the main country represented on GitHub, no suprise here. + +If you are interested in the "geography" of Open Source, you should read these two articles: [Coding Places](http://takhteyev.org/dissertation/) and [Investigating the Geography of Open Source Software through GitHub](http://takhteyev.org/papers/Takhteyev-Hilts-2010.pdf). + +### companies + +Looking at the "company" field on user's profile, here are some stats about which companies has employees using GitHub: + +* ThoughtWorks: 102 +* Google: 66 +* Mozilla: 65 +* Yahoo!: 65 +* Red Hat: 64 +* Globo.com: 55 +* Twitter: 53 +* Facebook: 45 +* Yandex: 43 +* Intridea: 34 +* Microsoft: 33 +* Engine Yard: 32 +* Pivotal Labs: 29 +* MIT: 28 +* Rackspace: 27 +* IBM: 24 +* Caelum: 23 +* Novell: 22 +* GitHub: 22 +* VMware: 22 + +I didn't knew the first company, ThoughtWorks, and I was expecting to see FaceBook or Twitter as the company with most developpers on GitHub. It's also interesting to see Yandex here. + +## Global graph (1628 nodes, 9826 edges) + +([download PDF](http://maps.stargit.net/global/global.pdf, "download GDF":http://maps.stargit.net/global/global.gdf)) + +The main difference with last year, is the android / modders community. They're developing mostly in C and Java. The poster has been created from this map. + +## Ruby (1968 nodes, 9662 edges) + +([download PDF](http://maps.stargit.net/ruby/ruby.pdf), [download GDF](http://maps.stargit.net/ruby/ruby.gdf), [download GEXF](http://maps.stargit.net/ruby/ruby.gexf)) + +This is still the main community on GitHub, even if JavaScript is now [the most popular language](https://github.com/languages/JavaScript). This graph is really dense, it's not easy to read, since there is no real cluster in this one. + +## Python (1062 nodes, 2631 edges) + +([download PDF](http://maps.stargit.net/python/python.pdf), [download GDF](http://maps.stargit.net/python/python.gdf)) + +Here we have some clusters. I'm not familiar with the Python community, so I can't really give any insight. + +## Perl (608 nodes, 2967 edges) + +([download PDF](http://maps.stargit.net/perl/perl.pdf), [download GDF](http://maps.stargit.net/perl/perl.gdf), [download GEXF](http://maps.stargit.net/perl/perl.gexf)) + +I really like this graph since it show (in my opinion) one of the real strength of this community: everybody works with everybody. People working on a webframework will collaborate with people working on Moose, or an ORM, or other tools. It shows that in this community, people are competent in more than one field. + +The Perl community is about the same size as last year. However, we can extract the following informations: + +* the Japaneses Perl Hackers are still a cluster by themselves +* [miyagawa](http://github.com/miyagawa) is still the glue between the Japanese community and the "rest of the world" +* other leaders are: Florian Ragwitz ([rafl](http://github.com/rafl)), Andy Amstrong ([AndyA](http://github.com/andya)), Dave Rolsky ([autarch](http://github.com/autarch)) +* some clusters exists for Moose and Dancer. + +As we can see on the previous charts, the number of created accounts for the Perl developpers is stalling. + +## United States (2646 nodes, 11344 edges) + +([download PDF](http://maps.startgit.net/unitedstates/unitedstates.pdf), [download GDF](http://maps.startgit.net/unitedstates/unitedstates.gdf), [download GEXF](http://maps.startgit.net/unitedstates/unitedstates.gexf)) + +This one is really nice. We can clearly see all the communities. There is something interesting: + +* C and Ruby are on the opposite side (C on the left, Ruby on the right) +* Python and Perl are also opposed (Perl at the bottom and Python at the top) + +I'll let you take some conclusion by yourself on this one ;) + +## France (706 nodes, 1059 edges) + +([download PDF](http://maps.stargit.net/france/france.pdf), [download GDF](http://maps.stargit.net/france/france.gdf), [download GEXF](http://maps.stargit.net/france/france.gexf)) + +We have a lot of small clusters on this one, and some very big authorities. + +## Japan (464 nodes, 1091 edges) + +([download PDF](http://maps.stargit.net/japan/japan.pdf), [download GDF](http://maps.stargit.net/japan/japan.gdf), [download GEXF](http://maps.stargit.net/japan/japan.gexf)) + +There is three dominants clusters on this one: + +* Ruby +* Perl +* C + +The Ruby and Perl one are well connected. There is a lot of japanese hacker on CPAN using both languages. + +## StarGit + +[StarGit](http://stargit.net) is a great tool we built with Alexis to let you explore **your** community on GitHub. You can read more about the application on [Alexis' blog](http://ofnodesandedges.com/2011/06/20/stargit.html). + +It's hosted on [dotcloud](http://dotcloud.com) (I'm still amazed at how easy it was to deploy the code ...), using the Perl [Dancer web framework](http://perldancer.org), MongoDB to store the data, and Redis to do some caching. + +## Credits + +I would like to thanks the whole GitHub team for being interested in the previous poster and to ask another one this year :) + +A **huge** thanks to Alexis for his help on building the awesome StarGit. Another big thanks to Antonin for his work on the poster. diff --git a/content/post/2012-02-17-HTTP_requests_with_python.md b/content/post/2012-02-17-HTTP_requests_with_python.md new file mode 100644 index 0000000..66f64e9 --- /dev/null +++ b/content/post/2012-02-17-HTTP_requests_with_python.md @@ -0,0 +1,131 @@ +--- +date: 2012-02-17T00:00:00Z +summary: In which I express my frustration toward HTTP libraries in Python. +title: The state of HTTP's libraries in Python +--- + +## Hey! I'm alive! + +I've started to write some Python for work, and since I'm new at the game, I've decided to start using it for some personal project too. + +Most of what I do is related to web stuff: writing API, API client, web framweork, etc. At [Say](http://www.saymedia.com/) I'm working on our platform. Nothing fancy, but really interesting (at least to me) and challenging work (and we're recruting, drop me a mail if you want to know more). + +## Writing HTTP requests with Python + +### httplib + +[httplib](http://docs.python.org/library/httplib.html) is part of the standard library. The documentation says: *It is normally not used directly*. And when you look at the API you understand why: it's very low-level. It uses the HTTPMessage library (not documented, and not easily accessible). It will return an HTTPResponse object, but again, no documentation, and poor interface. + +### httplib2 + +[httplib2](http://code.google.com/p/httplib2/) is a very popular library for writing HTTP request with Python. It's the one used by Google for it's [google-api-python-client](http://code.google.com/p/google-api-python-client/) library. There's absolutly nothing in common between httplib's API and this one. + +I dont like it's API: the way the library handles the **Response** object seems wrong to me. You should get one object for the response, not a tuple with the response and the content. The request should also be an object. Also, The status code is considered as a header, and you lose the message that comes with the status. + +There is also an important issue with httplib2 that we discovered at work. In some case, if there is an error, httplib2 will retry the request. That means, in the case of a POST request, it will send twice the payload. There is [a ticket that ask to fix that](http://code.google.com/p/httplib2/issues/detail?id=124), marked as **won't fix**. [Even when there is a perfectly acceptable patch for this issue.](http://codereview.appspot.com/4365054/) (it's a [WAT](https://www.destroyallsoftware.com/talks/wat) moment). I'm really curious to know what was the motiviation behind this, because it doesn'nt makes sense at all. Why would you want your client to retry twice your request if it fails ? + +### urllib + +[urllib](http://docs.python.org/library/urllib.html) is also part of the standard library. I was suprised, because given the name, I was expecting a lib to *manipulate* an URL. And indeed, it also does that! This library mix too many different things. + +### urllib2 + +[urllib2](http://docs.python.org/library/urllib2.html) And because 2 is not enough, also ... + +### urllib3 + +[urllib3](http://code.google.com/p/urllib3/). I thought for a moment that, maybe, the number number was related to the version of Python. I'll spare you the suspense, it's not the case. Now I would have expected them to be related to each other (sharing some common API, the number being just a way to provides a better API than the previous version). Sadly it's not the case, they all implement different API. + +At least, urllib3 has some interesting features: + +* Thread-safe connection pooling and re-using with HTTP/1.1 keep-alive +* HTTP and HTTPS (SSL) support + +### request + +A few persons pointed me to [requests](http://pypi.python.org/pypi/requests). And indeed, this one is the nicest of all. Still, not exactly what *I*'m looking for. This library looks like [LWP::Simple](https://metacpan.org/module/LWP::Simple), a library build on top of various HTTP components to help you for the common case. For most of the developers it will be fine and do the work as intented. + +## What I want + +Since I'm primarly a Perl developer (here is were 99% of the readers are leaving the page), I've been using [LWP](https://metacpan.org/module/LWP) and HTTP::Messages for more than 8 years. LWP is an awesome library. It's 16 years old, and it's still actively developed by it's original author [Gisle Aas](https://metacpan.org/author/GAAS). He deserves a lot of respect for his dedication. + +There is a few other library in Perl to do HTTP request, like: + + * [AnyEvent::HTTP](https://metacpan.org/module/AnyEvent::HTTP): if you need to do asynchronous call + * [Furl](https://metacpan.org/module/Furl): by Tokuhiro and his yakuza gang + +but most of the time, you end up using LWP with HTTP::Messages. + +One of the reason this couple is so popular is because it provides the right abstraction: + +* a user-agent is provided by LWP::UserAgent (that you can easily extends to build some custom useragent) +* a Response class to encapsulates HTTP style responses, provided by HTTP::Message +* a Request class to encapsulates HTTP style request, provided by HTTP::Message + +The response and request objects use HTTP::Headers and HTTP::Cookies. This way, even if your building a web framework and not a HTTP client, you'll endup using HTTP::Headers and HTTP::Cookies since they provide the right API, they're well tested, and you only have to learn one API, wether you're in an HTTP client or a web framework. + +## http + +So now you start seeing where I'm going. And you're saying "ho no, don't tell me you're writing *another* HTTP library". Hell yeah, I am (sorry, Masa). But to be honest, I doubt you'll ever use it. It's doing the job *I* want, the way *I* want. And it's probably not what you're expecting. + +[http](http://git.lumberjaph.net/py-http.git/) is providing an abstraction for the following things: + +* http.headers +* http.request +* http.response +* http.date +* http.url (by my good old friend "bl0b":https://github.com/bl0b) + +I could have named it **httplib3**, but **http** seems a better choice: it's a library that deals with the HTTP protocol and provide abstraction on top of it. + +You can found the [documentation here](http://http.readthedocs.org/en/latest/index.html) and install it from [PyPI](http://pypi.python.org/pypi/http/). + +### examples + +A few examples + +```python +>>> from http import Request +>>> r = Request('GET', 'http://lumberjaph.net') +>>> print r.method +GET +>>> print r.url +http://lumberjaph.net +>>> r.headers.add('Content-Type', 'application/json') +>>> print r.headers +Content-Type: application/json + + +>>> +``` + +```python +>>> from http import Headers +>>> h = Headers() +>>> print h + + +>>> h.add('X-Foo', 'bar') +>>> h.add('X-Bar', 'baz', 'foobarbaz') +>>> print h +X-Foo: bar +X-Bar: baz +X-Bar: foobarbaz + + +>>> for h in h.items(): +... print h +... +('X-Foo', 'bar') +('X-Bar', 'baz') +('X-Bar', 'foobarbaz') +>>> +``` + +### a client + +With this, you can easily build a very simple client combining thoses classes, or a more complex one. Or maybe you want to build a web framework, or a framework to test HTTP stuff, and you need a class to manipulate HTTP headers. Then you can use http.headers. The same if you need to create some HTTP responses: http.response. + +I've started to write [httpclient](http://git.lumberjaph.net/py-httpclient.git/) based on this library that will mimic LWP's API. + +I've started [to document this library](http://httpclient.readthedocs.org/en/latest/index.html) and I hope to put something on PyPI soon. diff --git a/content/post/2012-10-31-virtualenv-and-checkouts.md b/content/post/2012-10-31-virtualenv-and-checkouts.md new file mode 100644 index 0000000..81b11e9 --- /dev/null +++ b/content/post/2012-10-31-virtualenv-and-checkouts.md @@ -0,0 +1,33 @@ +--- +date: 2012-10-31T00:00:00Z +summary: In which I share a trick for virtualenv +title: Virtualenv and checkouts +--- + +I've started to do some Clojure in my spare time. The default tool adopted by the community to manage projects is [leiningen](http://leiningen.org). For those of you who don't know what `lein` is, it's a tool to automate your Clojure project: it will boostrap a new project, install the dependencies, and there's a plugin mechanism to extend the default possibilities of the tool. + +One of the nice feature of the tool is the **checkouts** directory. From the [FAQ](https://github.com/technomancy/leiningen/blob/preview/doc/FAQ.md): + +> If you create a directory named checkouts in your project root and symlink some other project roots into it, Leiningen will allow you to hack on them in parallel. + +For Python projects at [$work](http://www.saymedia.com/careers) I use [virtualenvwrapper](http://virtualenvwrapper.readthedocs.org/en/latest/) to easily work on them without having to deal with conflicting dependencies. When I need to change a library that is used by one of the project, usually I go to the virtualenv directory and create a symlink so it uses the one I'm editing. + +What I really want is a mechanism similar to `lein`, where I can have a **checkouts/** directory inside the main project, where I can clone a library or create a symlink. Since `virtualenvwrapper` provides a hook mechanism, I wrote a small hook inside **~/.virtualenvs/postactivate**: + +```sh +#!/bin/bash + +# move to the directory of the project +proj_name=$(echo $VIRTUAL_ENV|awk -F'/' '{print $NF}') +proj_path=/home/vagrant/src/$proj_name + +cd $proj_path + +if [ -d checkouts ]; then + for ext in $(ls checkouts); do + export PYTHONPATH=proj_path/checkouts/$ext:$PYTHONPATH + done +fi +``` + +Then, when I type `workon $project_name` in my shell, the environment is activated, I'm moved to the right directory, and the library inside the **checkouts/** directory are added to my **PYTHONPATH**. diff --git a/content/post/2012-11-14-two-tech-talks-in-a-day.md b/content/post/2012-11-14-two-tech-talks-in-a-day.md new file mode 100644 index 0000000..1cf2734 --- /dev/null +++ b/content/post/2012-11-14-two-tech-talks-in-a-day.md @@ -0,0 +1,34 @@ +--- +date: 2012-11-14T00:00:00Z +summary: In which I write a summary for two tech talks. +title: Two tech. talks in a day +--- + +Today I assisted to two tech. talks. One of them was our "Reading Group" session at lunch and the second one was the Python meetup tonight. + +## Say's tech talk + +I'm trying to organize at [work](http://saymedia.com), every two weeks during lunch time, a session where engineers can discuss about an article, tool, or paper they find interesting. Today we were a very small group (only 4 peoples), and we talked about two tools that [Masa](http://sekimura.typepad.com/blog/) wanted to explore: [Kage](https://github.com/cookpad/kage) and [HTTP Archive](http://www.igvita.com/2012/08/28/web-performance-power-tool-http-archive-har/). + +### Kage + +We talked about Kage first (kah-geh means "shadow" in Japanese). Masa started to explain what was the goal of the tool and how it works. Basically, it's a proxy that will send your HTTP request to your production and stage environment. The basic [example](https://github.com/cookpad/kage/blob/master/examples/proxy.rb) show how easy it is to write a simple proxy with a few rules to dispatch only your GET request to your two environments, and then compare the payload returned. + +It's using [em-proxy](https://github.com/igrigorik/em-proxy/) which is a DSL to write proxy for EventMachine. We then discussed where we could use it in our architecture and what benefit we could get from it, but for now we don't have any plan to use it. + +### HAR + +Then the second thing we talked about what HAR. You can find more detailed information on [Ilya's blog](http://www.igvita.com/2012/08/28/web-performance-power-tool-http-archive-har/) and on this [gist](https://gist.github.com/3500508). To summarize, HAR is an HTTP archive [format](http://www.softwareishard.com/blog/har-12-spec/). When you use the developers tools in Chrome, you can see the request and their response time, but you can also save this information in the HAR format. + +There's a few tools then to analyse this data, like YSlow, or browse them with the HARviewer. Another interesting thing is that you don't have to use a browser to get them, you can use [Phantomjs](http://phantomjs.org). Masa and Kyle talked about it and if they could use that with Nagios, or with Jenkins to measure the response time. + +## SF Python Meetup + +The last talk of the day was the [SF Python Meetup](http://www.meetup.com/sfpython/) after a 4 or 5 months break, at the Yelp headquarter. We were supposed to have some lightning talks to start, but we only had one, from someone who think that "lightning talk" means "publicity for Google and ho by the way, we're hiring" (yeah like no one in the room knew that ...). + +Then David Schachter presented [How to Speed Up A Python Program 114,000 times](http://www.rtortell.com/SF_Python_Meetup_slides_public.pdf). He showed us how he improved the performance of a script by 114,000 times. I will start by saying the talk was interesting and that David was an entertaining speaker. He went through some optimization he used, like using multiprocessing modules, [cython](http://cython.org), and some more difficult optimization he was not able to get from cython (like the permuted vector code). + +He had a very strong opinion about cluster and I really disagree with him. One of his complains was that cluster are hard, are young, and we don't have any tools, so we should not use them. But we've been using cluster in universities and laboratories for years now. Even if the tools are still not very great, they exist, and they work. And more importantly to me, the fact that he spend 12 weeks on optimizing his program, and doing stuff that I would not be able to do, he proved that optimizing is *also* very hard, and that doesn't seem easier to me that using a cluster. + +All in all, that was a good day, and I learned new things. Now I need to find a subject for our next reading group session. + diff --git a/content/post/2012-11-27-ansible-and-chef.md b/content/post/2012-11-27-ansible-and-chef.md new file mode 100644 index 0000000..de71b9c --- /dev/null +++ b/content/post/2012-11-27-ansible-and-chef.md @@ -0,0 +1,90 @@ +--- +date: 2012-11-27T00:00:00Z +title: Where we talk about ansible and chef +--- + +I've been using [Chef](http://www.opscode.com/chef/) for some time now, but it was always via [Vagrant](http://vagrantup.com), +so a few weeks ago I decided to get more familiar with it. A friend +of mine had set up a Chef server for his own use and was OK to let me use +it for my personal server. There was a four days weekend for Thanksgiving +coming, so it was the perfect occasion to take a better look at it, +and to re-install my Linode server with Chef. And since it was a +really long weekend, I also decided to take a look at [ansible](http://ansible.cc), another +tool to push stuff to your server. + +I'm not going to talk about installation, configuration, set up and +all that kind of stuff, there's enough material available around (blog +posts, articles, books, etc). Instead, I will talk about my +experience and share my (very valuable) opinion (because, clearly, the +world deserve to know what I think). + +## Writing cookbooks for Chef + +For the few of you who don't know, cookbooks, in Chef's world, are a +group of files (templates, static files) and code that are used to +automate your infrastructure. Usually, you'll create a cookbook for +each of your application (one for nginx, one for MySQL, etc). + +I've a few services on my server (git, gitolite, Jenkins, graphite, +collectd, phabricator, ...), and I wanted a coobook for each of them. +I've started by looking for the one already existing (there's a lot of +them on GitHub, on the +[opscode's account](https://github.com/opscode-cookbooks/)), and I tried to use +them without any modification. Usually, a cookbook will let you set +some configuration details in your role or node to override the +defaults it provides (like +the password for MySQL, or the path where to put logs). So what I did +was to set the interesting cookbook as a git submodule in my cookbook +repository. Unfortunately, for almost all of them, I had to give up +and import them in the repo, so I could edit and modify them. + +That's probably my biggest complaint with cookbooks: I doubt code +re-usability is possible. You can use a cookbook as a base for your +own version, but either they are too generic; or sometimes you need a +workaround; or they do way too many things. And as a result, you need to edit the +code to make them behave the way you want. + +In my opinion, developers/ops should just publish [LWRP](http://docs.opscode.com/essentials_cookbook_lwrp.html) (Lightweight +Resources and Providers) and templates, that's the only thing that I +can see as really re-usable (take a look at +[perl-chef](https://github.com/dagolden/perl-chef), I think that this one is a good +example). + +## Using ansible + +ansible was a new tool for me. A few friends mentionned it to me last +October when I was at the [OSDC.fr](http://osdc.fr) and it was also suggested to me by a +colleague at work. + +This tool is definitely less known that Chef, so I'll give a quick +introduction. In ansible world, you write "playbooks", which are the +orchestration language for the tool. That sounds very similar with +Chef, but the main difference is they are not actual code, but a +scenario with actions. + +On the web site of the project, there's a quote saying: + +> You can get started in minutes. + +and for once, that's true. I only had to read the first page of the +documentation, and I was able to write a very simple playbook that I +was able to evolve very quickly to do something actually useful. + +Another difference with Chef is that they don't incite you to share +your playbooks, but instead to share your modules. Modules could be +compared to Chef's LWRP. They are Python code to do something +specific (like the [`pip`](http://ansible.cc/docs/modules.html#pip) module, to install Python package, or the +[`template`](http://ansible.cc/docs/modules.html#template)'s one). + +## Chef vs Ansible + +For now, I've decided to stick to this: use Chef for my supporting +application (nginx, MySQL, etc) and ansible for my own applications. + +So far, I prefer ansible to Chef. There's definitely less available +material about ansible on the net, but the quality is better, and the +main documentation is very (I insist on the *very*) well organized. I've never spend more than +10 minutes looking for something and to implement it. I can't say +the same with Chef: the wiki is confusing; there's way too many +cookbooks available; their quality is very disparate. + diff --git a/content/post/2012-11-28-perl-redis-and-anyevent-at-craiglist.md b/content/post/2012-11-28-perl-redis-and-anyevent-at-craiglist.md new file mode 100644 index 0000000..3daf241 --- /dev/null +++ b/content/post/2012-11-28-perl-redis-and-anyevent-at-craiglist.md @@ -0,0 +1,113 @@ +--- +date: 2012-11-28T00:00:00Z +title: Perl, Redis and AnyEvent at Craiglist +--- + +Last night I went to the +[SF.pm](http://www.meetup.com/San-Francisco-Perl-Mongers/) meetup, +hosted by Craiglist (thanks for the food!), where +[Jeremy Zawodny](https://twitter.com/jzawodn) talked about +[Redis](http://redis.io) and his module +[AnyEvent::Redis::Federated](https://metacpan.org/module/AnyEvent::Redis::Federated). +We were about 30 mongers. + +I was eating at the same table as Craiglist CTO's, and he went through +some details of their infrastructure. I was surprised by the quantity +of place where they use Perl, and the amount of traffic they deal with. + +## Redis + +Jeremey started his talk by explaining what is their current problem: +they have hundred of hosts in multiple data center, and they collect +continuously dozen of metrics. They looked at MySQL to store them, +but it was too slow to support the writes. Another thing +important for them is that mostly only the most recent data matters. They +want to know what's going on *now*, they don't really care about the +past. + +So their goal is simple: they need something fast, *really* fast, and +simple. That's where Redis enter the game. + +They want data replication, but +Redis don't have this feature: there's only a master/slave replication +mechanism (so, one way), and they need a solution with multi master, +where a node becoming master does not drop data. They address this +issue with a "syncer", that I'll describe later. + +Because Redis is single thread, and servers have multiple cores, they +start 8 process on each node to take advantages of them. + +To me, the main benefit of Redis over Memcached is that you can use +it as a data structure server. If you only need something to store +key value, I'll prefer to stick to memcached: the community around is +bigger, there's a lot of well know patterns, and a lot of big +companies are contributing to it (lately, Twitter and FaceBook). + +The structure they use the most are the +[*sorted set*](http://redis.io/commands#sorted_set). The format to store a metric is: + + * key: `$time_period:$host:$metric` (where the $timeperiod is + usually a day) + * score: `$timestamp` + * value: `$timestamp:$value` + +In addition of storing those metrics in the nodes, they also keep a +journal of what has changed. The journal looks like this: + + * score: `$timestamp` of the last time something has changed + * value: `$key` that changed + +The journal is only one big structure, and it's used by their syncer +(more about that in a moment). The benefit of having ZSET is that +they can delete old data easily by using the key (they don't have +enough memory to store more than a couple of days, so they need to be +able to delete by day kickly). + +The journal is use for replication. Each process has a syncer that +track all his peers, pull the data from those nodes and merge them +with the local data. Earlier Jeremy mentioned that they have 8 +instances on each node, so a the syncer from process 1 on node a will +only check for the process 1 on node b. + +He also mentioned a memory optimization done by Redis (you can read +more about that [here](http://redis.io/topics/memory-optimization)). + +## AnyEvent::Redis::Federated + +Now, it's time to see the Perl code. `AnyEventE::Redis::Federated` is +a layer on top of `AnyEvent::Redis` that implements a consistent +hashing. I guess now every body has gave up hope to see someday +[redis cluster](http://redis.io/topics/cluster-spec) (and I'm more and +more convinced that hit should never be implemented, and let the +client implement their own solution for hashing / replication). + +Some of the nice feature of the modules: + + * call chaining + * [you can get singleton object for the connection](https://metacpan.org/module/AnyEvent::Redis::Federated#SHARED-CONNECTIONS) + * you can also use it in blocking mode + * query all node (where you send the same command to all the node, + can be useful to do sanity check on the data) + * the client will write to one node, and let the syncer do the job + +He then showed us some code (with a very gross example: `new +AnyEvent::Redis::Federated`, I know at least +[one person](http://search.cpan.org/perldoc?indirect) who would have +probably said something :). + +## Concerns + +The idea seems fine, but, as one person noted during the Q&A, how will +this scale when you have more than 2 or 4 nodes in your cluster ? +Since each process' syncer need to talk to *all* the other nodes, it +will probably be very expensive for this process to gather information +from all the nodes and write them. Also, by adding more nodes, you're +storing less information into each process, since you replicate +everything. Maybe a good solution is to keep many small cluster of +2 to 4 nodes, and let each of them deal with some specific metrics. + +The module is not yet used in production, but they've tested it +heavily, in a lot of conditions (but I would note that there's no unit +test :). They intent to use it soon with some +home made dashboard to display the metrics. + diff --git a/content/post/2012-12-16-about-devops.md b/content/post/2012-12-16-about-devops.md new file mode 100644 index 0000000..a292689 --- /dev/null +++ b/content/post/2012-12-16-about-devops.md @@ -0,0 +1,47 @@ +--- +date: 2012-12-16T00:00:00Z +title: About DevOps +--- + +There's a lot of talk about what is or what is not DevOps, and I'll +throw my opinion in the mix. + +Until a few weeks ago, at [work](http://saymedia.com), we had mostly +two teams: the engineering team and the ops team. Our workflow was +(to simplify) the following: + + * engineers develop services and applications + * they push their change to Jenkins + * a build pass and is pushed to CI + * a few times a week, engineers ask Ops to push the change to + production + +There's already a lot of articles about the kind of frictions created +by this (who owns what; engineers would blame ops when the push was +failing (or the other way around); it's hard for ops to know what's +wrong when something is broken in production; etc). + +A few weeks ago it was decided to create a new team to improve +engineers efficiency, and the team was named "DevOps". At first I was +not sure it was the right name for this team, but now I don't think it +matters. + +I was not sure the name was appropriate because of this +[article](http://continuousdelivery.com/2012/10/theres-no-such-thing-as-a-devops-team/) +(and a few other to respond), explaining why you don't want a DevOps +team, but instead you want the whole organization to be DevOps. We +need engineers to own their applications, to be able to push when they +want, but also to monitor, know what's wrong or slow, etc. + +The **work** [hachi](https://github.com/hachi) and I will have to do is to +help engineers and ops to *be* the DevOps. Our team responsibility is +to choose, evaluate and integrate tools. We will also provide +libraries, documentation, training and support. *We* are not the +DevOps. Our **goals** are to create this culture, to give more +responsibilities to engineers, and to free Ops from the work of +pushing code. The **success** of this team will be measured by the +adoption of our work by engineers and ops. + +So yes, I agree that you don't want a dedicated DevOps team, but you +still need a team coming from different background (hachi is coming +from Ops and I'm an engineer) to build that culture. diff --git a/content/post/2013-01-10-carbons-manhole.md b/content/post/2013-01-10-carbons-manhole.md new file mode 100644 index 0000000..1bafd97 --- /dev/null +++ b/content/post/2013-01-10-carbons-manhole.md @@ -0,0 +1,43 @@ +--- +date: 2013-01-10T00:00:00Z +title: Carbon's manhole +--- + +We're rolling out Graphite and statsd at [work](http://saymedia.com), and I've spend some time debugging our setup. Most of the time, the only thing I need is ``tcpdump`` to verify that a host is sending correctly the various metrics. + +But today, thanks to a [stupid reason](http://if.andonlyif.net/blog/2013/01/the-case-of-the-disappearing-metrics.html), I've learned about another way to debug [carbon](http://graphite.readthedocs.org/en/latest/carbon-daemons.html): the manhole. The idea of the manhole is to give you a access to a REPL attached to the live process. When my boss told me about it, I was at first surprised to see this in a Python application. I've already been exposed to this kind of debugging thanks to Clojure, where it's not uncommon to connect a REPL to your live application (for example, Heroku [document how to connect to a remote live REPL in your application](https://devcenter.heroku.com/articles/debugging-clojure)). When I first heard of that I was very skeptical (give access to a *live* environment, and let the developer mess with the process ?!). But I've learned to love it and I feel naked when I'm working in an environment where this is not available. So I was happy to jump and take a look at that feature. + +Since it's not very well documented and I had a hard time finding some information, let me share here the basics. + +First you'll need to configure Carbon's to allow the connection: + +```ini +ENABLE_MANHOLE = True # by default it's set to False +MANHOLE_INTERFACE = 127.0.0.1 +MANHOLE_PORT = 7222 +MANHOLE_USER = admin +MANHOLE_PUBLIC_KEY = <your public SSH key, the string, not the path to the key> +``` + +Now you can restart carbon, and connect to the Python shell with ``ssh admin@127.0.0.1 -p7222``. This manhole is useful to get an idea of the data structure your process is handling, or to get an idea of what's going on (is there a lot of keys being held in memory? Is the queue size for one metric huge? etc). + +From here, you can execute Python code to examine the data of the process: + +```python +>>> from carbon.cache import MetricCache +>>> print MetricCache['PROD.apps.xxx.yyy.zzz] +[(1357861603.0, 93800.0), (1357861613.0, 98200.0), (1357861623.0, 91900.0)] +``` + +The [``MetricCache``](https://github.com/graphite-project/carbon/blob/master/lib/carbon/cache.py#L19) class is a Python dictionary where you can access your keys. You can also list all the metrics with the size of their queue with ``MetricCache.counts()``. + +Or even force the daemon to write to disk all the data points: + +```python +>>> from carbon.writer import writeCachedDataPoints +>>> writeCachedDataPoints() +``` + +Before doing any of that, I would recommend to read the code of carbon. It's pretty short and quiet straight forward, especially the code of the [writer](https://github.com/graphite-project/carbon/blob/master/lib/carbon/writer.py). + +Of course, you have to know what you're doing when you're executing code from a REPL in a live environment. diff --git a/content/post/2013-01-14-where-im-excited-about-a-keyboard.md b/content/post/2013-01-14-where-im-excited-about-a-keyboard.md new file mode 100644 index 0000000..ed77103 --- /dev/null +++ b/content/post/2013-01-14-where-im-excited-about-a-keyboard.md @@ -0,0 +1,21 @@ +--- +date: 2013-01-14T00:00:00Z +title: Where I'm excited about a keyboard +--- + +One thing that I always find fascinating is how an object, or a way of thinking, was modeled 25, 50 or 100 years ago, and how we keep going on with the same design without trying to think if it's actually the right one, or to challenge our way of thinking with a different one. Keyboards are one of those things. We all know the [origin](http://en.wikipedia.org/wiki/Qwerty) of the QWERTY layout, why the keyboard has this shape, and so on (the [wikipedia](http://en.wikipedia.org/wiki/Computer_keyboard) page is a good read). Let's take a look at this tweet from [Technomancy](http://technomancy.us) (who, for the the record, also has some [interesting](http://www.flickr.com/photos/technomancy/4397554484/) keyboard ideas :) + +<blockquote class="twitter-tweet tw-align-center"><p>I wonder how long it'll be before we can look back and laugh at the idea of poking a QWERTY soft-keyboard on a touch screen. Hope it's soon.</p>— Phil Hagelberg (@technomancy) <a href="https://twitter.com/technomancy/status/290959908302647296" data-datetime="2013-01-14T23:13:42+00:00">January 14, 2013</a></blockquote> +<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> + +That's exactly what I'm talking about: we keep reproducing the same design without questioning it's benefits (I've no doubt that people at Apple and Google have tried a lot of alternatives, an they probably decided to stick to this one because it's the most common and people are used to it. Still, it doesn't mean we have to stick with it). But even for physical keyboard, we keep the same design: a large space key (why ?!), a capslock key (who need that one, even remapped to "control" I feel like it's a bad place to physically put a key on the keyboard), large enter/shift keys, etc. + +I'm not a keyboard nerd. But sometimes I take a look at some new and strange models, sometimes I try one of them, and I might even buy one, like the [TypeMatrix](http://www.typematrix.com) (that one I really liked). + +So, when [Jesse](https://twitter.com/obra) announced that he was working on a keyboard, I got *really* excited. His layout is very interesting (it seems to be inspired by the kinesis), and the shape is also innovative. I also remember reading that he suffers from RSI, which mean it's something he will be careful bout. Take a look at the mapping: there's only 52 keys, control/shift/alt are repeated, but with this shape it makes sense, since they're dedicated to one half of the keyboard. + +<center><a href="http://www.flickr.com/photos/obra/8361045529/" title="A reduced travel keyboard layout by jesse, on Flickr"><img src="http://farm9.staticflickr.com/8096/8361045529_4694afe187.jpg" width="500" height="281" alt="A reduced travel keyboard layout"></a></center> + +You can [read](http://blog.fsck.com/2012/12/building-a-keyboard-part-1.html) [about](http://blog.fsck.com/2012/12/building-a-keyboard-part-2.html ) [his](http://blog.fsck.com/2013/01/a-pound-of-sculpey.html ) [progress](http://blog.fsck.com/2013/01/pinkies-and-your-brain.html ). He's also publishing [pictures](http://www.flickr.com/photos/obra/ ) on his flickr account. + +I really hope he will be able to build this keyboard and manage to manufacture it (maybe a kickstarter project ?). I'll definitely be in line to get one. diff --git a/content/post/2013-01-28-let-s-talk-about-graphite.md b/content/post/2013-01-28-let-s-talk-about-graphite.md new file mode 100644 index 0000000..629d37e --- /dev/null +++ b/content/post/2013-01-28-let-s-talk-about-graphite.md @@ -0,0 +1,102 @@ +--- +date: 2013-01-28T00:00:00Z +summary: In which I share my experience with Graphite +title: Let's talk about Graphite +--- + +As I've already mentionned in a [previous post](/carbons-manhole/), at [$work](http://saymedia.com) we are currently deploying Graphite and the usual suspects. + +Finding articles on how to install all these tools is easy, there's plenty of them. But what's *really* hard to find, are stories on *how to use them*: what's collected, how, why, how do you organize your metrics, do you rewrite them, etc. + +What I want with this post is to start a discussion about usages, patterns, and good practices. I'm going to share how *we*, at SAY, are using them, and maybe this could be the start of a conversation. + +## Graphite/collectd/statsd + +We're using [statsd](https://github.com/etsy/statsd), [collectd](https://collectd.org) [Graphite](https://github.com/graphite-project) (and soon [Riemann](http://riemann.io) for alerting). + +### Retention + +Our default retention policy is: `10s:1h, 1m:7d, 15m:30d, 1h:2y`. We don't believe that Graphite should be used for alerting: it's a tool for looking at history and trends. + +360 points for the last hour is enough to refer to a graph when an incident occurs. Most of the teams are releasing at least once a week, so 1 minute definition for a week is enough to compare trends between two releases. Then we go to a month (2 sprints) and they 2 years. We thought at first to keep only 15 months (1year + 1 quarter to compare), but since we have enough disk space, we decided to keep two years, however we might decide to change that in the future. + +### System + +We don't do anything fancy here. `collectd` is running on each host, and then write to a central `collectd` server. + +### Services + +For shared services (Memcached, Varnish, Apache, etc), we talk to `statsd`. We have a Perl script named `sigmoid`, with the following usage: + +```sh +Usage: ./sigmoid [<options>] <metricname> <value> + Exactly one of: + --counter (value defaults to 1) + --aggregate + --gauge + --event + --raw + + Other options: + --disable-multiplex (for statsd only) + --appname + --hostname (to log on behalf of another machine) +``` + +This script is used by other scripts who monitor logs, status of apps, etc. This way it's very easy for a Perl, Python, Shell script to just call `sigmoid` via `system`, and then send the metric and the value to `statsd`. + +For some other services we might need something more specific. Let's take a look at Apache. We have another Perl script for the **CustomLog** settings (`CustomLog "|/usr/local/bin/apache-statsd"`). The script is doing the following things: + +* compute the size of the HTTP request in bytes +* compute how long it took to return the response + +Then, it will send the following lines to `statsd` (with $base being the vhost in our case): + +* `$base.all.requests:1|c` increases the total of HTTP requests we're receiving +* `$base.all.bytes:$bytes|ms` send the size, in bytes, of that request +* `$base.all.time:$msec|ms` the time spend to get the response + +Now we will send the same line two more times, with a different prefix: `$base.method.$request_method` *and* `$base.status.$status`. + +### Applications + +Here, developers decide what they want to collect, and send the metric to `statsd`. + +### Events + +And finally we have events. Every time we push an application or a configuration, we create a new event. + +## Proxying statsd + +We want metrics to be well organized, in a clear hierarchy. [Jason Dixon](https://github.com/obfuscurity) wrote in a [blog post](http://obfuscurity.com/2012/05/Organizing-Your-Graphite-Metrics) that *Misaligned paths are ok*. I disagree. We're collecting more than 100k metrics so far. If things are not well organized, it will become quickly very difficult to find what you're looking for. + +So, here's how we organize our metrics. The first level is the environment (PROD, CI, DEV, ...). Then we have *apps* and *hosts*. For the *host* section, we group by cluster type (Hadoop cluster, Web servers for TypePad, etc), and then you have the actual host, with all the metrics collected. For *apps*, we have four main categories: *aggregate*, *counters*, *events* and *gauges* (I'll come back on that later). + +Earlier I said that apps where sending metrics to `statsd`, but that's not exactly true. We (mostly) never write directly to statsd or Graphite. + +On each host, we have a Perl script listening. This proxy will rewrite all the incoming metrics by appending to the name the environment, the cluster and so on. This way when someone want to send a key, he doesn't have to care convention or using the correct prefix. + +Also, it will also multiplex the metric: we want the same key to end-up under *host* and under *app*. Let's take an example here. If you're writing a web service, you may want to send a metric for the total time taken by an endpoint (this will be an aggregate). Our key will be something like: **\<application-name\>.\<endpoint-name\>.\<http-method\>.\<total-time\>**. The proxy, based on the network address, will determine that it's environment is CI, and that it's an application. But it also knows the name of the server, and the cluster. So two keys will be created: + +* **\<CI\>.\<apps\>.\<aggregate\>.\<application-name\>.\<endpoint-name\>.\<http-method>.<total-time\>** +* **\<CI\>.\<hosts\>.\<cluster-name\>.\<host-name\>.\<aggregate\>.\<application-name\>.\<endpoint-name\>.\<http-method\>.\<total-time\>** + +This way we can find the metric aggregated by application, or if we think there's a problem in one machine, we can compare per host the same metric. + +## Other problems with statsd and Graphite + +I don't know if it's a problem with vocabulary, or our maths (I admit that my maths are not good, but I trust Abe and Hachi's maths), but you can't imagine how much time we spend debating around the words gauges, counters and aggregates. What they mean, how they work, when to use them. So here's my questions: are we missing something obvious? do we over think it? or is it also confusing, and people are misusing them? + +Let's take **gauge** as an example. If you read [the documentation for gauges](https://github.com/etsy/statsd/blob/master/README.md#gauges), it seems very simple: you send a value, and it will be recorded. Well, the thing is it will record only the last value send during the 10 seconds interval. This work well when you have a cron job that will look at something every minute and report a metric to `statsd`, not if you're sending that 10 times a second (and yes, we will provide a patch for documentation soon). + +Another one where we lost a good amount of time: if you're smallest retention is different from the interval used by statsd to flush the data, they will be graphed incorrectly (see this [comment](https://github.com/etsy/statsd/issues/32#issuecomment-1830985)). + +The best "documentation" for `statsd`, so far, are the discussions in the [issues](https://github.com/etsy/statsd/issues). + +We have some other complains about Graphite. Even after reading the [rationals](http://graphite.wikidot.com/whisper#toc1) for Whisper, I'm not convinced it was a good idea to replace RRD with it. We also discovered some issues with [Graphite's functions](http://if.andonlyif.net/blog/2013/01/graphites-derivative-function-lies.html). + +## Meetup + +We've a huge basement at work that can be used to host meetup. There's already a few meetup in the San Francisco about "devops" stuff ([Metrics Meetup](http://www.meetup.com/San-Francisco-Metrics-Meetup/events/98875712/), [SF DevOps](http://www.meetup.com/San-Francisco-DevOps/), etc), but maybe there's room for another one with a different format. + +What I would like, is a kind of forum, where a topic is picked, and people share their *experiences* (the bad, the good and the ugly), not how to configure or deploy something. And there's a lot of topics where I've questions: deployment (this will be the topic of my next entry I think), monitoring, alerting, post-mortem, etc. If you're interested, send me an email, or drop a comment on this post. diff --git a/content/post/2013-02-19-should-I-read-the-code.md b/content/post/2013-02-19-should-I-read-the-code.md new file mode 100644 index 0000000..95b4b22 --- /dev/null +++ b/content/post/2013-02-19-should-I-read-the-code.md @@ -0,0 +1,20 @@ +--- +date: 2013-02-19T00:00:00Z +title: Should I read the code ? +--- + +This conversation happened twice in the last few weeks at work, the first time during my 1:1 with my manager, and a second time with the whole team. + +We were investigating [Riemann](http://riemann.io/), and we started to discuss what it would means to adopt this technology in our stack. Riemann is written in Clojure, and no one at work is really familiar with this language (except for me, and I'm don't consider myself efficient with it). + +The question is how do you deal with a new tool when there's only one person in the team that can read the code, and therefore contribute to the project to add features, fix bugs, etc. Are we supposed to be familiar with the code of the things that we use? + +I've never read the code of MySQL, I've read parts of Apache's code, and I've never looked at the source of the Linux' kernel. At the same time, I usually read the code of the Perl and Python libraries I use frequently, I've also read the source of statsd and Graphite (two other tools that we looked at) in order to understand what they do and hunt for issues (in the way we use them) or bugs. + +I see two ways to approach this question so far: as a developer and as a user. As a developer, I consider that I **have to** read and understand the code of the libraries my code depends on (we've found some serious issues in libraries we use daily because of this approach). + +For services we use in the infrastructure, it depends of the size of the tool and it's community. For a new product, or when the documentation is too sparse, or when the community is rather small, it's a good thing to be able to look at the code and explain what it does. For bigger projects (MySQL, Apache, Riak), so far I've relied on the experiences people had with the tools, the community. + +I'll conclude this post with an anecdote. Last Thursday we were trying to understand why the CPU load on the Graphite's box went over the roof when we added about 25% more metrics to it. With Abe and Hachi we said "ok let's dig into this problem". You could have guessed who are the ops while looking at the scene. We were looking for the same things: reads and write. Abe and Hachi started to do that with the help of ``strace``, while I started to walk through the code. I think the two ways are valid, at least you can use one to correlate the other, and they gave you different information (``strace`` will help you to time the operations, while the code would explain what you're writing and reading). + +I'm curious to hear how other approach this problem. diff --git a/content/post/2013-02-24-practical-joke.md b/content/post/2013-02-24-practical-joke.md new file mode 100644 index 0000000..ec60aff --- /dev/null +++ b/content/post/2013-02-24-practical-joke.md @@ -0,0 +1,34 @@ +--- +date: 2013-02-24T00:00:00Z +title: Practical Joke +--- + +I've nothing exciting to write about so I'll share a prank I did a few years ago (because I'm kinda proud of this one). + +To protect the innocent, I'll change all the names. + +I was working at a company named $LF. One of my colleague ($NG), always looks at his keyboard when he types. Some time he can type for a few minutes. Then he looks at his screen, only to realize that it's full of typo, or worse, that he had typed in the wrong window. This used to really bother me, because I know he's been typing for years, but I was not sure if he was looking at his keyboard because he needs to, or just because his head is too heavy to look at the screen. + +So one day, while everybody (except $GM) were away taking a break, I decided to conduct an experiment. + +The french keyboard uses the [AZERTY](http://en.wikipedia.org/wiki/Azerty) layout. $NG uses the [best editor in the world](http://www.gnu.org/software/emacs/), where the *x* key is *really* important (try to do something without 'Meta-x'). He also uses a laptop, connected to an external monitor and with an external keyboard. So I decided to swap the *x* and the *w* letter on the external keyboard, by switching the caps. The actual *x* and *w* were still at the same place, only the label on the key was switched. + +When he came back from the break, he starts typing as usual. Quickly he realized that something was wrong, and he did a lot of things to figure out what. Among them: + +* verify that the content of ``xmodmaprc`` was correct, and then tweak it +* reinitialize the layout with ``setxkbmap`` +* use a tool (maybe ``showkey`` ?) to see what was the actual code send from the keyboard while hitting the key +* probably recompiled a bunch of stuff +* definitely upgraded his Linux distribution + +At this point, he understood that when hitting *x*, *w* was received, and for *w*, *x*. He rebooted his laptop (because when in doubt, you should "[tried turning it off and on again](http://www.youtube.com/watch?v=nn2FB1P_Mn8)"). But the problem was still there. + +If I remember correctly, that's when he decided to look at the laptop's keyboard and tried to use it, only to realize that it was working correctly! So he asked the ops guy, $GM, who was sitting next to me, if there was a spare keyboard somewhere. I slowly turned my head toward $GM, whispering 'noooo' to him, and he replied to $NG "nop, sorry, no spare keyboard, <insert stupid excuse #49>". + +The other developers started to realize that something was wrong with $NG, but no one said or did anything yet. We were about 30 minutes in the experiment now. + +Because he still had work to do, he tried to cop with the problem, and every time he needed a *x* or a *w*, he would type something, stop, look at the screen, sigh/grumble, and use the laptop's keyboard to fix all the *w* and *x*. That was difficult to miss, and soon other people asked him what was wrong, and so that's when I revealed . + +When I explained what I did he was surprised, starred at the two keyboards for a while, and saw that, effectively, the keys were swapped! I'm pretty sure the rest of the team made fun of him but I don't really remember exactly how. + +I consider the experiment to be a success: I proved that he need to look at his keyboard to type, and I'm also confident that I can reproduce it with the same subject multiple time and obtain the same result again. diff --git a/content/post/2013-07-24-patch.pm.md b/content/post/2013-07-24-patch.pm.md new file mode 100644 index 0000000..542ed96 --- /dev/null +++ b/content/post/2013-07-24-patch.pm.md @@ -0,0 +1,10 @@ +--- +date: 2013-07-24T00:00:00Z +title: Patch.pm +--- + +Next week-end (from Friday, July 26 to Sunday, July 29), the French [Mongueurs](http://mongueurs.pm/) are organizing the first [patch.pm](http://patch.pm/p0/), a new hackathon around Perl. + +If you can't attend the event, feel free to hop on *#mongueurs*, on *irc.perl.org*, and join us to work on your favorite module! I'll be working [Dancer2](http://github.com/perldancer/Dancer2) during the week-end. + +A huge thanks to the [French Mongueurs](https://twitter.com/mongueurs_perl) (and especially [Laurent](https://twitter.com/elbeho)) for organizing this one. diff --git a/content/post/2013-07-28-patch.pm-report.md b/content/post/2013-07-28-patch.pm-report.md new file mode 100644 index 0000000..03b968a --- /dev/null +++ b/content/post/2013-07-28-patch.pm-report.md @@ -0,0 +1,25 @@ +--- +date: 2013-07-28T00:00:00Z +summary: In which I summarize my activity during the first patch.pm +title: Patch.pm - report +--- + +This week-end I participated to [patch.pm](http://patch.pm/p0/), the first version of a new hackathon organized by the Mongueurs. Sadly (or maybe not) I was not physically with my fellow mongueurs to work on code, and did it remotely. + +My goal for this two days was to get some work done on [Dancer2](https://github.com/PerlDancer/Dancer2): code review, fixes, submit bugs, and get some work done on the core. + +Saturday started by a discussion with [Alberto](https://github.com/ambs) and [Sawyer](https://github.com/xsawyerx) about what should be our priorities. None of us had really worked on Dancer2 so far, so we had a lot to catch up. We started with a list of things we think should be easy to work on, so we could focus on them in order to be able to deliver something by the end of the weekend. We quickly agreed on it, and then we started to code. + +Sawyer started his chainsaw and pushed a dozen of commits to clean up some code. He also write a few emails to the team with more short and medium terms objectives. + +On my side, I wanted to fix the code related to the hooks. Not all the hooks available in Dancer 1 were present in Dancer 2, and some of them were a little bit buggy. [I pushed some code](https://github.com/PerlDancer/Dancer2/pull/331) to fix that, hopefully we will find the time this week to review it. + +During that time, Alberto merged a few pull requests and fixed the travis build! + +I only managed to get two hours of work done on Sunday. The biggest change was a refactoring of the Response and Request objects, to give them a similar API. Then I took a look at some of the changes made by Sawyer and did some review. + +It was fun. Working again with Alberto and Sawyer was really great, and I realized how much I missed working with this guys. I'm excited for the next comming weeks, more work will be done (there's still some issues that we need to fix, and some documentation clean up) and some releases will be pushed out. + +And a huge shout out to [Yanick](https://github.com/yanick), because while we were having fun on Dancer2, he was pushing fixes and merging pull requests for Dancer 1. You should look closely at the CPAN because a release will appear very soon :) + +And to conclude, I've a question for [Laurent](https://twitter.com/elbeho) (who is probably the most important member of the french community at this point): when do you organize the next one ? :) (also, I'm very disapointed that I didn't get the free croissants and coffee). diff --git a/content/post/2014-01-04-setting-up-cgit-with-ansible.md b/content/post/2014-01-04-setting-up-cgit-with-ansible.md new file mode 100644 index 0000000..4af7ebd --- /dev/null +++ b/content/post/2014-01-04-setting-up-cgit-with-ansible.md @@ -0,0 +1,235 @@ +--- +date: 2014-01-04T00:00:00Z +summary: In which I share how I install cgit with Ansible. +title: Setting up cgit with Ansible +--- + +I've [already write](/ansible-and-chef/) about [Ansible](http://www.ansibleworks.com/). I use it to manage and configure my servers (most of them are VMs running on my laptop, but the idea is the same). One of the server is used to store my personal git repositories, and I wanted to use [cgit](http://git.zx2c4.com/cgit/) as the front end for the public repositories instead of the old and slow [gitweb](https://git.wiki.kernel.org/index.php/Gitweb). + +Since there's no package in Debian for cgit, I need to have an easy procedure to install it. I'll show how I do it with Ansible. This could be useful if you're learning about Ansible are you're looking for a simple use case. + +## Work directory + +The work directory contains a bunch of files: + + * $workdir/hosts - local inventory with all the hosts, grouped by categories + * $workdir/$hostname.yml - play book for a given host (more about this file later) + * $workdir/roles/git - directory containing templates, tasks and handlers for installing cgit + * $workdir/vars/$hostname.yml - contains all the variable needed to install cgit + +> Replace $hostname with the name of the host you want to use for cgit. + +## Handlers + +In my case, cgit is hosted behind Nginx, so first, we need a handler to restart it after changing Nginx's configuration. + +```yaml +# roles/git/handlers/main.yml +- name: restart nginx + service: name=nginx state=restarted +``` + +## Roles + +Now we need to define our role for cgit. The idea is to install the required packages to be able to build cgit, to create the directories where we will store our repositories, and actually build cgit. + +```yaml +# roles/git/tasks/main.yml +- name: Set the directory for public repos + file: path=/srv/git/public + owner=www-data + group=www-data + mode=0770 recurse=yes + state=directory + +- name: Set the directory for private repos + file: path=/srv/git/private + owner=www-data + group=www-data + mode=0770 + recurse=yes + state=directory + +- name: Install necessities for cgit + apt: pkg={{ item }} state=installed + with_items: + - build-essential + - autoconf + - automake + - libtool + - libfcgi-dev + - libssl-dev + - spawn-fcgi + - highlight + - fcgiwrap + +- name: Create cgit web directory + file: path=/srv/www/{{ cgit_subdomain }}.{{ domain }} + recurse=yes + state=directory + owner=www-data + +- name: Download cgit tarbal + get_url: url=http://git.zx2c4.com/cgit/snapshot/cgit-0.9.2.zip + dest=/tmp/cgit-0.9.2.zip + force=no + +- name: Unzip cgit + command: unzip -qo /tmp/cgit-0.9.2.zip -d /tmp + +- name: Configure cgit installation + template: src=cgit.conf.j2 dest=/tmp/cgit-0.9.2/cgit.conf + +- name: Install cgit + shell: make get-git && make && make install chdir=/tmp/cgit-0.9.2 + +- name: Set permissions for cgit + file: path=/srv/www/{{ cgit_subdomain }}.{{ domain }} + owner=www-data + state=directory + recurse=yes + +- name: Configure the nginx HTTP server for cgit + template: src=etc_nginx_sites-available_cgit.j2 + dest=/etc/nginx/sites-available/{{ cgit_subdomain }}.{{ domain }} + group=www-data + owner=www-data + +- name: Configure cgit + template: src=etc_cgitrc.j2 + dest=/etc/cgitrc + group=www-data + owner=www-data + +- name: Enable cgit + file: src=/etc/nginx/sites-available/{{ cgit_subdomain }}.{{ domain }} + dest=/etc/nginx/sites-enabled/{{ cgit_subdomain }}.{{ domain }} + state=link + group=www-data + owner=www-data + notify: restart nginx + +- name: Backup git directory + template: src=etc_cron.hourly_git-backup.j2 + dest=/etc/cron.hourly/git-backup + mode=0755 +``` + +## Templates + +We need a bunch of templates to configure and build our tools. Let's start with **cgit.conf**. + +```sh +# roles/git/templates/cgit.conf.j2 + +CGIT_SCRIPT_PATH = /srv/www/{{ cgit_subdomain }}.{{ domain }} +``` + +This file is used when we build cgit to install it to a specific location. + +The next template is to configure cgit. + +```ini +# roles/git/templates/etc_cgitrc.j2 + +root-desc=Franck Cuny's projects +virtual-root=/ +logo=/cgit.png +css=/cgit.css +scan-path=/srv/git/public +remove-suffix=1 +clone-prefix=http://git.$hostname.net +``` + +This template is to configure nginx. + +```nginx +# roles/git/templates/etc_nginx_sites-available_cgit.j2 + +server { + listen 80; + server_name "{{ cgit_subdomain}}.{{ domain }}"; + root /srv/www/{{ cgit_subdomain }}.{{ domain }}; + + location / { + try_files $uri @cgit; + } + + location @cgit { + index cgit.cgi; + + fastcgi_param SCRIPT_FILENAME $document_root/cgit.cgi; + + fastcgi_pass unix:/run/fcgiwrap.socket; + fastcgi_param HTTP_HOST $server_name; + fastcgi_param PATH_INFO $uri; + fastcgi_param QUERY_INFO $uri; + include "fastcgi_params"; + } + + error_log /var/log/nginx/{{ cgit_subdomain }}.{{ domain }}-error.log; + access_log /var/log/nginx/{{ cgit_subdomain }}.{{ domain }}-access.log; +} +``` + +## Backing up on s3 + +I backup all my git repositories to a bucket on s3. In order to do that, you'll need either a new role or to update the current one by adding the following instructions. + +```yaml +- name: Install s3cmd + apt: pkg=s3cmd + +- name: Configure s3cmd + sudo: false + template: + src="s3cfg.j2" + dest="/root/.s3cfg" + +- name: Backup git directory + template: src=etc_cron.hourly_git-backup.j2 + dest=/etc/cron.hourly/git-backup + mode=0755 +``` + +We need a template to configure our access to s3. + +```ini +[default] +access_key = {{ aws_access_key }} +secret_key = {{ aws_secret_key }} +use_https = True +``` + +And another template for our cron job. + +```sh +#!/bin/sh +s3cmd sync -v /srv/git/ s3://$hostname-backup/git/ > /tmp/s3_backup_git.log 2>&1 +``` + +## Variables + +I have a file named **vars/$hostname.yml** that contains the + +```yaml +domain: $hostname.net +cgit_subdomain: git + +aws_access_key: access-key +aws_secret_key: secret-key +``` + +## Play time + +The content of the playbook + +```yaml +- hosts: $hostname + vars_files: + - vars/$hostname.yml + roles: + - git +``` + +Now I can tell Ansible to run this playbook, and this will install cgit on my server: `ansible-playbook -i hosts lj.yml`. diff --git a/content/post/2014-01-11-ansible-role-for-bittorrent-sync.md b/content/post/2014-01-11-ansible-role-for-bittorrent-sync.md new file mode 100644 index 0000000..a9b69d4 --- /dev/null +++ b/content/post/2014-01-11-ansible-role-for-bittorrent-sync.md @@ -0,0 +1,137 @@ +--- +date: 2014-01-11T00:00:00Z +summary: In which I share how I install bittorrent sync with Ansible. +title: Setting up bittorrent sync with Ansible +--- + +Recently, I've started to look at the services I'm using and I'm trying to find alternatives for some of them. One of them was [Dropbox](https://www.dropbox.com/). I've been using this service for 4 or 5 years at this point, and I was a paid customer for at least 3 years. My plan was about to renew in a few days, and I decided to look for a free (either as "doesn't cost a buck" or OSS) alternative. To make it quick, since it's not the purpose of this post, I've decided to go with Bittorrent Sync. Still, this solution is not perfect, so here's a short list of pros and cons: + +* Pros: + * Free + * The only limit on storage is based on the free space you have + * Decentralized +* Cons + * Harder to set up + * Slower than Dropbox for syncing files + * Decentralized (yes I know): you need to always have at least one node up if you want to get a copy of a file + +I decided to set btsync on one of my server that is always up, that I way I can access my copies whenever I want from any devices (this is similar to Dropbox). To do that I've create a set of tasks for Ansible, following the instructions provided by [Leo Moll](https://github.com/tuxpoldo) (who created the package for Debian) on [this forum](http://forum.bittorrent.com/topic/18974-debian-and-ubuntu-server-packages-for-bittorrent-sync-121-1/). + +I've created a role named 'btsync' that contains a handler, some templates and a bunch of tasks. Let's start with the handler, since this one is really simple. For this role, the only y service we care about is the bittorrent sync daemon: + +```yaml +- name: restart btsync + service: name=btsync state=restarted +``` + +The tasks are pretty straightforward. First we need to fetch Leo's PGP key so we can install the package using `apt`. + +```yaml +- name: Install GPG key for btsync + apt_key: + id=6BF18B15 + state=present + url=http://stinkfoot.org:11371/pks/lookup?op=get&search=0x40FC0CD26BF18B15 +``` + +Now that we have the PGP key, we can add the repository to our sources and install the package: + +```yaml +- name: Add the deb repo for btsync + apt_repository: + repo='deb http://debian.yeasoft.net/btsync wheezy main' + state=present + +- name: Install btsync + apt: pkg=btsync state=installed +``` + +For the remaining tasks, we need some configuration. + +```yaml +btsync_shared: + - dir: /path/to/shared/photos + secret: a-secret + use_relay_server: true + use_dht: false + search_lan: false + use_sync_trash: true + - dir: /path/to/shared/documents + secret: another-secret + use_relay_server: true + search_lan: false + use_sync_trash: true +``` + +The daemon expect all the directories that it will write to, to exist. So let's create them for him, and also set the right permissions on them: + +```yaml +- name: Create the directories where we need to sync + file: path={{ item.dir }} state=directory owner={{ main_user_name }} group={{ main_user_name }} mode=0700 + with_items: btsync_shared +``` + +We need a specific configuration file for our user: + +```yaml +- name: Configure btsync + template: + src=etc_btsync_user.conf.j2 + dest=/etc/btsync/{{ main_user_name }}.conf + group={{ main_user_name }} + owner={{ main_user_name }} + mode=0600 + notify: restart btsync +``` + +and the template that goes with it: + +```json +{ + "device_name": "{{ domain }}", + "storage_path" : "/home/{{ main_user_name }}/.btsync", + "listening_port" : 12589, + "check_for_updates" : false, + "use_upnp" : false, + "download_limit" : 0, + "upload_limit" : 0, + "disk_low_priority" : true, + "lan_encrypt_data" : true, + "lan_use_tcp" : false, + "rate_limit_local_peers" : false, + "folder_rescan_interval" : 600, + "shared_folders" : {{ btsync_shared | to_json }} +} +``` + +To complete the setup, we need to tell the daemon to start with our newly created configuration: + +```sh +# This is the configuration file for /etc/init.d/btsync +# +# Start only these btsync instances automatically via +# init script. + +AUTOSTART="{{ main_user_name }}" + +# Optional arguments to btsync's command line. Be careful! +# You should only add thngs here if you know EXACTLY what +# you are doing! +DAEMON_ARGS="" +``` + +Finally, the task to setup the default configuration for the daemon: + +```yaml +- name: Configure btsync server + template: + src=etc_default_btsync.j2 + dest=/etc/default/btsync + group=root + owner=root + notify: restart btsync +``` + +That's it! + +I'll try to find some time this weekend to upload this to [Galaxy](https://galaxy.ansibleworks.com/). diff --git a/content/post/2014-02-01-provision-an-ec2-instance-with-vagrant-and-ansible.md b/content/post/2014-02-01-provision-an-ec2-instance-with-vagrant-and-ansible.md new file mode 100644 index 0000000..f311d3e --- /dev/null +++ b/content/post/2014-02-01-provision-an-ec2-instance-with-vagrant-and-ansible.md @@ -0,0 +1,55 @@ +--- +date: 2014-02-01T00:00:00Z +summary: In which I provision an EC2 instance with Vagrant and Ansible +title: Provisioning an EC2 instance with Vagrant and Ansible +--- + +I like to use [Ansible](http://www.ansible.com/) to manage my personal servers. It forces me to make the environment reproducible so I don't have to care about a specific box: I can throw them away easily, knowing I can get a new one when I need, with the exact same configuration. + +I also find Ansible easier to reason about than Chef or Puppet. The fact that I have to manage and maintain only a few machines is probably why. + +When I develop on my personal projects, I use a lot of VMs, and that's where [Vagrant](http://www.vagrantup.com/) enters the picture. Being able to start a local VM and get a clean environment quickly is invaluable to me. It makes it easy to test an application or library in different situation, without carrying about local dependencies and conflicts created by having multiple versions of the same library installed on the system. + +But sometimes a local VM is not enough, and you need a more powerful server, so now you need an EC2 instance. + +My goal with this article is to show how easy you can combine Vagrant with Ansible to provision an EC2 instance. + +## The basic + +I have a private repository with all my rules for Ansible. But for this post, all we need is a simple playbook. So let's start by creating a directory named *vagrant*, and put inside a configuration file named *playbook.yml*, with the following content: + +<script src="https://gist.github.com/franckcuny/fae46135ad0f3581ce6b.js"></script> + +What we're describing, is that for all the hosts in our inventory we will use `sudo` to install the program `htop` using `apt-get` (yes, I assume you're using a debian-based system, but you get the idea). + +First, we will try the setup on a local box. If you don't already have a Vagrant box installed, you can grab a new one by running `vagrant box add precise64 http://files.vagrantup.com/precise64.box`. + +Now we can add the configuration file named *Vagrantfile* with this content. + +<script src="https://gist.github.com/franckcuny/aadd788101c08744a22a.js"></script> + +This file says that we will use the box named *precise64*, located at the given URL, and we want to provision it using Ansible, and the path to the playbook. + +By running `vagrant up`, a box gets started and provisioned. An inventory file is generated for us inside the directory, so ansible will know what to do. The output should be similar to this: + +<script src="https://gist.github.com/franckcuny/e3df9a2424e4a4a12f60.js"></script> + +As we can see, everything went well, and the application `htop` was successfully installed. We can now run `vagrant ssh` and once logged inside the VM, run `htop`. + +## AWS + +I've created a key pair for Vagrant in the AWS console. Note the access and secret access keys, and download the SSH private key too. For this article, we will put the key into the same directory as our playbook and Vagrant's configuration. + +We need to install a plugin for that: `vagrant plugin install vagrant-aws`. We also need to modify our *Vagrantfile* to use a different box, and also add the configuration for AWS. + +<script src="https://gist.github.com/franckcuny/ac8cad84af5f51a923f6.js"></script> + +We need to override the user name to *ubuntu* and specify the path to the private key (the one we got from the AWS console when we created our new key pair) to log into the instance. The box also needs to be overridden. + +Running `vagrant up --provider=aws` will provision the box. It will takes a few minutes to start the instance and run the provisioning part. Wait a few minutes, but if it looks like the system is stuck, you can re-run the previous command by exporting `VAGRANT_LOG=debug` in order to get more detailed information. + +> If the provisioning blocks while trying to connect to ssh, it's probably because your security group doesn't allow SSH connections. + +Now `vagrant ssh` will dump you into the VM and you should be able to run `htop`. + +Don't forget to run `vagrant halt` and `vagrant destroy` once you're done! diff --git a/content/post/2014-02-15-cursive-nrepl-cljs-oh-my.md b/content/post/2014-02-15-cursive-nrepl-cljs-oh-my.md new file mode 100644 index 0000000..6d21ce4 --- /dev/null +++ b/content/post/2014-02-15-cursive-nrepl-cljs-oh-my.md @@ -0,0 +1,95 @@ +--- +date: 2014-02-15T00:00:00Z +summary: In which I share my quick experiment with cljs +title: Cursive, nREPL, Clojurescript .. oh my +--- + +> This notes are mostly for me to remember how to get an environment up and running. + +I’ve tried to play with [Clojurescript](https://github.com/clojure/clojurescript) a few times already, but I never managed to get an environment working as I wanted. Granted, I didn’t try very hard, and my requirements are probably not commons, so I’m not blaming the tools. + +This time I’ve decided to dedicate a little bit more times and try to see how far I could go. My requirements are: + +* it needs to work in a linux VM +* I want to have a nREPL connection + * so I can execute code from the REPL and execute them in the browser + * I also want to be able to call functions from the browser REPL +* It needs to work within Emacs and/or Intellij IDEA + +The Intellij IDEA requirement is new. I’ve never used an IDE before and I’ve decided to give it a try. I’ve installed the [Cursive](http://cursiveclojure.com) plugin for Intellij and so far I'm happy with it. + +I’m pleased to say that after a few hours I had a completely working environment meeting all my requirements. I’m pretty sure this notes will become outdated quickly, so I’ll try update this post as I discover more things. + +## The VM + +I like Vagrant and there’s no reason to not use it for this kind of projects. I’ve created a [repository](https://github.com/franckcuny/devbox) on GitHub with my setup for that. It’s a simple VM with VirtualBox and Vagrant. I’m using Ansible to do the provisioning part. + +After cloning the repository, you can run `vagrant up` and a virtual machine, running Ubuntu 13.10, will be started and provisionned (a bunch of tools like tmux are installed, but what really interest us here is the openjdk). Once it's up, a I run `vagrant ssh` then I start a `tmux` session. + +## Create a project + +I create a project using the [cljs-start](https://github.com/magomimmo/cljs-start) template for leiningen. To be honest, all the hard work is done by this template, I'm just putting the pieces together to have the setup I want build around it. With this template, you get: + +* the ability to start a HTTP server for serving your static files +* a browser-repl from nREPL + +This plugin relies on [Austin](https://github.com/cemerick/austin). + +All I needed to do was to run `lein new cljs-start project` (where project is the name of my project). + +## The REPL + +Once you’ve started to work with a REPL it’s hard to go back to a language that don’t have one. When working with Clojure, you get used very quickly to it, especially since nREPL is so nice, and allows you to work on a remote box. + +The only tricky thing when setting up the VM is to be sure to forward a few ports for: + +* nREPL - so you can connect from the host to the REPL running on the guest +* the browser REPL - so your REPL can talk to the browser + +I run the REPL in headless mode on the VM (in my `tmux` session), from my project's directory: + +```sh +# specify a port that I will be forwarded from my host to the guest +AUSTIN_DEFAULT_SERVER_PORT=4343 lein repl :headless :host 0.0.0.0 :port 4242 +``` + +The `AUSTIN_DEFAULT_SERVER_PORT` variable is the port that will be used by your REPL to talk to the browser. That's why you need to forward this port in Vagrant. The other options (**host** and **port**) are here to tell the repl to listen on all the interfaces (so I can connect from the host) on the given port. + +## Editor + +> I’m focusing on Intellij IDEA here, but it works the same with Emacs/[CIDER](https://github.com/clojure-emacs/cider). + +To install the Cursive plugin, you need to go to [this page](http://cursiveclojure.com/userguide/index.html) and follow the instructions. + +I can now open a project in Intellij and start coding. I've configured my project to use a remote REPL. + + + +Now I can connect to the remote REPL and do a quick test to see if it works: + + + +Great! It's time to start the web server to serve our static files and see if I can connect the browser-repl to it too. Running the following code in the REPL should do the trick: + +```clj +(run) ;; will start a server with jetty on port 3000, that I can reach from port 4000 +(browser-repl) ;; that’s the *really* cool part +``` + +If I want to test something, all I have to do is to load the file into the REPL and then call a function. For example: + +```clj +(.log js/console "Hi from Intellij IDEA!") +``` + +and see the output in my browser's console! + + + +When working on the project, I can run evaluate the file or a form and send it to the browser. Again, this would be the same with Emacs, instead of having CIDER to use a local nREPL session, you'll just connect to a remote one. + +## Conclusion + +I realize that it’s not the easiest setup. I’m maintaining the build system we have at work for our sites; we use javascript and nodejs, and I’m really upset by the complexity of our process. If I had to put with all of that to build a site I would be pretty mad. Still, I think this setup can be simplified a lot. But using a VM also makes it easier to give a working environment to a new developer, and it's easy to throw it away, after all, I'm using it mostly to run the REPL and to have it working in an environment similar to what it would be in production. + +I have to admit that so far, I enjoy Cursive, it's stable and it works well. I'm still learning how to use the IDE, but some features are usefull (creating the functions, checking the number of parameters, displaying docstring, etc). We will see how long I stick to it. diff --git a/content/post/2014-03-23-golang-for-sysadmin.md b/content/post/2014-03-23-golang-for-sysadmin.md new file mode 100644 index 0000000..3372de5 --- /dev/null +++ b/content/post/2014-03-23-golang-for-sysadmin.md @@ -0,0 +1,19 @@ +--- +date: 2014-03-23T00:00:00Z +summary: In which I'm looking for a solution on how to deploy golang programs +title: golang for sysadmins +--- + +Recently I've been reading articles about golang and how it can be used to replace scripts that you would usually write in Python Perl or Bash. I can understand why you would do that (you get a binary, so it's faster, it works without the need of an interpreter, etc). + +At work we have a "ops.git" repository with all our rules for cfengines, configurations for services, zones for DNS, and also a tons of scripts. Most of them, today, are in Python/Perl/Bash. I've been looking and toying to replace some of them with golang, but I don't see how we could do that without modifying the existing setup. + +This scripts are copied on a system by `cfengine`, usually in `/usr/local/bin`. I'm trying to figure out how would you do that for programs in go ? I can't imagine a solution where you would have to compile the binary and commit it to the repository (the size of the repository would just explode), and I don't like the idea of having a hook in `cfengine` to compile and put them in place. + +The main solution that I can see here is to have a different repository with all the scripts, and let jenkins build a debian package (something like $company-ops-tools.deb) and then have it deployed/upgraded by cfengine via `apt`. + +Another solution would have to put a Makefile into this repository and let jenkins build some artifacts, put them in a package, and get it deployed. + +Is there another simpler solution ? By simpler I really mean faster than having to go through a build system, since it has to compile, build a package, and then get it deployed. It's not really great when you want to have a quick feedback on a script you're writing, and the current ops are quiet used to this, so having to go through a longer loop would be annoying. + +If you're using go in your company with this kind of setup, I'll be interested in feedback, feel free to contact me by [email](mailto:franck.cuny@gmail.com), on [Twitter](https://twitter.com/franckcuny) or [Google+](https://plus.google.com/+franckcuny). diff --git a/content/post/2014-08-08-google-is-using-https-as-a-ranking-signal.md b/content/post/2014-08-08-google-is-using-https-as-a-ranking-signal.md new file mode 100644 index 0000000..c38edb5 --- /dev/null +++ b/content/post/2014-08-08-google-is-using-https-as-a-ranking-signal.md @@ -0,0 +1,30 @@ +--- +date: 2014-08-08T00:00:00Z +summary: In which I don't understand why people are upset with Google's decision +title: Google is using HTTPS as a ranking signal +--- + +Earlier this week Google has announced that they will use HTTPS as a ranking signal for the [Page Rank](http://en.wikipedia.org/wiki/Page_rank). + +<center><blockquote class="twitter-tweet" lang="en"><p>HTTPS is now a lightweight ranking signal: <a href="http://t.co/hOr1DS9trV">http://t.co/hOr1DS9trV</a>. Secure those sites! <a href="https://twitter.com/hashtag/WebmasterNews?src=hash">#WebmasterNews</a> <a href="http://t.co/qk9v9L8lvR">pic.twitter.com/qk9v9L8lvR</a></p>— Google Webmasters (@googlewmc) <a href="https://twitter.com/googlewmc/statuses/497440606440792064">August 7, 2014</a></blockquote></center> +<script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> + +A lot of people on Hacker News and Twitter were annoyed (even upstet for some) by that decision. I understand some of the concern, but I don't think they are legitimate: + + * Yes, most SSL certs are not free. But hosting a website has also a cost. + + * However, certificates are not always expensive. You can get one for $16 with [Gandi](https://www.gandi.net/ssl). + + * Yes, there is probably additional cost. You'll need the technical knowledge on how to set up the certificate. But it's the same for running a web site. If you don't know how to do it, you'll need someone to do it for you. + + * There's an opportunity for hosting companies to compete on this, and make it easier and cheaper for small business and individual to run a web site with TLS. + + * The Page Rank is already secret, no one know how important having TLS will impact the ranking. + + * Yes, it matters even for a "content only" site. How can you trust the content was not altered otherwise ? + + * This decision impact business more than anyone else. If you're worried that your own personal blog is going to be impacted, I don't want to be mean, but I doubt this will have a huge impact for you. + + * Google won't stop ranking you because you don't have TLS. + +I really believe this is a first small step in the right direction. Plain HTTP should die at this point. diff --git a/content/post/2014-08-18-new-job.md b/content/post/2014-08-18-new-job.md new file mode 100644 index 0000000..7c146a9 --- /dev/null +++ b/content/post/2014-08-18-new-job.md @@ -0,0 +1,9 @@ +--- +date: 2014-08-18T00:00:00Z +summary: In which I get a new job. +title: New Job! +--- + +<center> +<blockquote class="twitter-tweet" lang="en"><p>First day at the new job! ☺ <a href="http://t.co/8ZpyFL2yfr">pic.twitter.com/8ZpyFL2yfr</a></p>— Franck Cuny (@franckcuny) <a href="https://twitter.com/franckcuny/statuses/501393676677816322">August 18, 2014</a></blockquote> <script async src="//platform.twitter.com/widgets.js" charset="utf-8"></script> +</center> diff --git a/content/post/2014-11-20-opening-a-file.md b/content/post/2014-11-20-opening-a-file.md new file mode 100644 index 0000000..d240298 --- /dev/null +++ b/content/post/2014-11-20-opening-a-file.md @@ -0,0 +1,114 @@ +--- +date: 2014-11-20T00:00:00Z +summary: In which I try to understand what happens when you open a file +title: On opening a file +--- + +A very common task for a programmer is to open a file. This seems to be a trivial operation, and we don’t think twice about it. But what is really happening when we’re opening that file ? + +## A simple C program + +For this exercise, I’m going to use this very simple C program: + +<script src="https://gist.github.com/franckcuny/d208c34a0b8397f3e4ca.js"></script> + +The code does the following things: + +* opens a file in read-only mode +* checks that we got a file descriptor +* if we don’t have the file descriptor, we print an error and exit +* we close the file descriptor +* we exit + +This is really simple and not much is going on, right ? Let’s take a better look at it. + +The `fopen()` function that we use is provided by the libc. It's documentation is pretty straight forward (`man 3 fopen`): *"The fopen() function opens the file whose name is the string pointed to by path and associates a stream with it."*. + +## Run the program + +We’re going to compile the source code first, so we can run the program: + +```bash +gcc -o test test.c +``` + +## Overview + +First I want to have an overview of the execution of this program. For this we will use `strace`. + +<script src="https://gist.github.com/franckcuny/7b9b9ab4fdccab364674.js"></script> + +We can ignore most of that output, only the last few lines interest us. We can see two functions related to the code we wrote: + +* a call to open, with **/etc/issue** as the first argument +* a call to close, again, with 3 as the first argument + +The first function is the system call `open()`, and we see that it returns 3, which is our file descriptor. When `close()` is called, it's only argument is again 3, which is the file descriptor returned by `open()`, and then we exit. + +## Deeper + +Now let’s invoke the program with gdb: + +<script src="https://gist.github.com/franckcuny/5ab16ac3a075200aafa1.js"></script> + +We can see the calls (the `callq` instructions) to our three functions: `fopen()`, `perror()` and `fclose()`, but we want to take a look at what exactly is behind this functions. Let's try to dig the `fopen` instruction a little bit more (I've removed all the lines that are not the `callq` instructions): + +<script src="https://gist.github.com/franckcuny/1d7883696306611e9bd3.js"></script> + +OK, so here we can see that we’re calling the function `_IO_new_file_fopen()`. + +## libc + +In our program, we're using functions provided by the libc. We're going to take a look at `_IO_new_file_fopen`, and we can read the source [here](http://fxr.watson.org/fxr/source/libio/fileops.c?v=GLIBC27#L252). + +Most of the function is to set a bunch of flags, and then the next call we’re interested in is [`_IO_file_open`](http://fxr.watson.org/fxr/source/libio/fileops.c?v=GLIBC27#L335). The function is defined [here](http://fxr.watson.org/fxr/source/libio/fileops.c?v=GLIBC27#L217). As you can see, here we end up calling `open()`. + +## system call + +The `open()` function is one of the linux system calls. If we look at [the list of syscalls](http://lxr.free-electrons.com/source/include/linux/syscalls.h), we can see that it is mapped to [`sys_open`](http://lxr.free-electrons.com/source/include/linux/syscalls.h#L512). + +The function is defined in [fs/open.c](http://lxr.free-electrons.com/source/fs/open.c#L992), and do a call to [do_sys_open](http://lxr.free-electrons.com/source/fs/open.c#L964). + +The interesting part of the function starts with the call to `get_unused_fd_flags()`, where we get a file descriptor. Then we do the call to `do_filp_open()`, where we end up (via more functions call): + +* geetting a file struct +* find the inode +* populate the file struct + +To finish, we do a call to `fsnotify()`, which will notify the watchers on this file, and add the file descriptor with the other struct files. + +## inodes + +To open a file, you need to locate it on the disk. A file is associated with an inode, which contains meta data about your file, and they are stored on your disk. When you want to reach a file, the kernel will find the inode and from that the location on the disk. You can read more about inodes on [wikipedia](https://en.wikipedia.org/wiki/Inode), and this [great page about ext4](https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout). + +You can run `man 1 stat` in your shell on the file to see the information we can find. + +<script src="https://gist.github.com/franckcuny/0104bdea0e515f809ad4.js"></script> + +An inode is a data structure to represent an object on the filesystem. If you look at the previous output, you can see information like the size, the number of blocks, how many references exists to this file (links), etc. + +Here, we can see that the inode is 679618. Now let’s take a look with the FS debugger: + +<script src="https://gist.github.com/franckcuny/016e6fc5be47a1fd4b4b.js"></script> + +There's many cools things you can do with inode, like using `man 1 find` to find a file based on it's inode instead of file name. + +## Deeper! + +Valgrind is another amazing tool to do analysis of a program. Let's recompile our binary with the `-g` option, to embed debugging information in our binary: + +```bash +gcc -g -o test test.c +``` + +`valgrind` has an option `--tool` to use specific tool. Let's run valgrind with the **callgrind** tool, followed by `callgrind_annotate` to get a more readable output: + +<script src="https://gist.github.com/franckcuny/313fb41e150dfb28a2f7.js"></script> + +With the `--cache-sim=yes` option, we count all the instructions for read access, cache misses, etc. Another nifty tool is **cachegrind**, which shows the cache misses for different level of caches. + +<script src="https://gist.github.com/franckcuny/71c1ae266b26aa8bf6e1.js"></script> + +## The end + +As you can see, using various tools (and there's more tools available!), you can see that opening a file involves a lot of operations behind the scene. diff --git a/content/post/2015-01-02-on-video-games.md b/content/post/2015-01-02-on-video-games.md new file mode 100644 index 0000000..0a87593 --- /dev/null +++ b/content/post/2015-01-02-on-video-games.md @@ -0,0 +1,32 @@ +--- +date: 2015-01-02T00:00:00Z +summary: In which I try predict the future of video games +title: On video games +--- + +Some times ago I had a very short conversation with [Maddingue](https://twitter.com/maddingue) about what could be the future of video games. Since then I've been thinking a little bit more about it. + +There's a possibility that things will change quickly and a lot for the consoles, and I believe that we will move toward streaming platforms quickly. Microsoft, Google, Facebook, Amazon have the resources to make this possible. This also shows why Facebook spent a lot of money on Oculus. + +Still, streaming has its own challenges (read [dormando’s post](http://www.dormando.me/2014/11/09/3ds_dqx/) on what he had to do to get a smooth game experience with DQX on 3DS) : latency, lag, you need a fast and always on internet connection, etc. + +We already know that Nintendo is already exploring this market with remakes of some of its games on the DS. Sony tried with Gaikai, unsuccessfully so far. They service is running and available (at least in north America) but I don't know anyone using it. + +This generation of console (PS4 and xbox one) is growing faster than any previous generation. Sales for the PS4 are amazing, and even if Microsoft is lagging behind, the xbox one is selling extremely well compared to the xbox 360. + +But it might be hard for MS to catch up. Because there’s already a lot of PS4 outside, some publishers might decide to focus on this platform instead. Developing a game to support two or three platforms is very expensive for them, and if the majority of players are on one platform, why bother. Also it seems that the PS4 is easier to develop on (it was not the case on the PS3, so Sony has learned from this error), and is more powerful. + +Now, MS also have the resources to announce in two years that they have a new and inexpensive console to play games streamed from their platform (Azure). We can even imagine that they offer it for free if you take a 2 or 3 years subscription to the xbox live. In this scenario, Sony would not be able to compete: they don't have the cash, and I really doubt that Gaikai is going to be ready anytime soon. + +Let’s take a look at the four other giants now: Google, Amazon, Facebook and Apple. They all build phones, tablets, and devices that you plug into your TV. They also have huge cloud infrastructure, and know how to distribute content that way. + +The thing that is not clear to me is how publisher would join. Right now, all the companies provide their own SDK and if you want your game to work on different devices, you have to develop different version of the games for each device. It’s in the interest of the constructor/cloud providers, but not the publishers, and this can’t scale. What they might end up doing is to provide a SDK to create a light client for each platform (android, iOS, windows, Oculus), and the publisher can get his game running on the cloud infrastructure of his choice (e.g: Google Compute or AWS). Cloud providers will compete on the infrastructure (fast platform, a lot of [POP](http://en.wikipedia.org/wiki/Point_of_presence), etc). This could also helps issues that we saw this year, where publishers are releasing a game but they don’t have enough servers to accept all the players on day one. + +The [Oculus Rift](https://www.oculus.com/) is going to be a huge game changer, but for now it still requires a PC to play, and a lot of people either don't have one, or don't have one powerful enough to play games. If the game can be rendered in the cloud and just be streamed to the oculus, everybody with an internet connection can play games. + +This also “fix” an old an annoying problem for gamers: retrocompatibility. If we stop developing games for specific hardware, but instead just for PCs, either on Linux or Windows, and all we have is a stream, it should be easier to play a game in 10 years that it is today. + +I might be totally wrong on this. But my gut tells me that’s where we’re heading. We’ve already started to move all the computation in the cloud, video gaming is the next step (and it’s one of the last because of it’s complexity). + +Here’s my prediction: the move will come from the new players (Google/Amazon/FB/Apple) and it will be there in less than 5 years. + diff --git a/content/post/2015-01-31-ubuntu-on-dell-xps13.md b/content/post/2015-01-31-ubuntu-on-dell-xps13.md new file mode 100644 index 0000000..c355313 --- /dev/null +++ b/content/post/2015-01-31-ubuntu-on-dell-xps13.md @@ -0,0 +1,60 @@ +--- +date: 2015-01-31T00:00:00Z +summary: In which I install Ubuntu on a Dell XPS13 +title: On Ubuntu and a Dell XPS13 +--- + +Being fed up with Yosemite, I decided to order the new [Dell XPS13 (2015 +edition)](http://www.dell.com/us/p/xps-13-9343-laptop/pd?ST=dell%20xps13&dgc=ST&cid=79646&lid=2024370&acd=123098073120560) +after reading a few reviews. I've installed Ubuntu 14.10 on it. + +Installing Ubuntu was (as expected) straightforward: + +* you go to the BIOS (F2 when the Dell logo shows up on screen) +* boot from an USB disk with Ubuntu (page to the instructions) +* use an ethernet -> usb adaptor since the wifi doesn’t work out of the box with the broadcom driver +* select what ever you want for the disk setup (I went with dm-crypt / llvm / btrfs) +* once it’s installed and you’re logged in as your user, install the wifi package (apt-get install bcmwl-kernel-source) + +then at this point you have a “functional” laptop. + +Sadly, there's still a few issues so far with Ubuntu 14.10: + +* no sound (I don’t really mind for now, I use my phone/tablet to listen to music / watch youtube and stuff). There’s a [ticket](https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1413446) to track the problem +* as is the keyboard (there's a bug where a key can be repeated a few time) +* the touchpad is ... touchy. I need to hold the button down for a second before it get registered, + and there's some kind of issues where it get lost. But I think this is fixed with newer kernel + (need to test). + +<script src="https://gist.github.com/franckcuny/70b6959eef1892d00197.js"></script> + +But there's also the good stuff! + +* the screen! really awesome definition. The resolution was properly detected, didn’t tweak anything +* the keyboard is nice, however I’m looking for a solution to swap fn and control +* the battery last, I get about ~7 hours of battery life with normal usage + +## Hardware + +The hardware is attractive. It’s really small (smaller than the MBPr 13”, but probably a little bit +bigger than the MBA 11”). It’s light. There’s 2 USB ports, 1 SD card (that’s one of the reason I +went with this model instead of the carbon x1). + +My main complaint is the 8GB limit, I would have prefered 16GB. Not having a broadcom +wireless card would have also been nice (maybe they will change this for the developer edition ?). +But (and that's the main difference with Apple :), there's a +[manual](ftp://ftp.dell.com/Manuals/all-products/esuprt_laptop/esuprt_xps_laptop//xps-13-9343-laptop_Service%20Manual_en-us.pdf) +and instructions on how to replace the wireless card. The good alternaties seems to be the [Intel +7260ngw](http://www.amazon.com/gp/product/B00GUNZUG0/) (I might replace the card). + +For the curious, here’s the output of +[dmesg](https://gist.github.com/franckcuny/02da991e4b4c6bcaabef) and +[lspci](https://gist.github.com/franckcuny/bc9a486dc17e8c9acef1). + +I’ll update this page with more details / information in the next few weeks (I'm currently on a 3.16, I'll try the 3.18 during the weekend). + +*Updates* + +* I've replaced the wireless card with the Intel 7260ngw. Opening the laptop is + easy, but replacing the card was not that easy. Fixing the antenna is kind of + a pain. However, the quality of the signal is way better now. diff --git a/content/post/2015-07-25-dont-remove-white-spaces.md b/content/post/2015-07-25-dont-remove-white-spaces.md new file mode 100644 index 0000000..4b421ea --- /dev/null +++ b/content/post/2015-07-25-dont-remove-white-spaces.md @@ -0,0 +1,62 @@ +--- +date: 2015-07-25T00:00:00Z +summary: In which I explain why removing blindly white spaces is harmful +title: Don't blindly remove trailing white spaces +--- + +I don't like trailing white spaces in my source code. I've configured my editors to highlight them +so I don't add them by accident, and when possible, I remove them. But it doesn't mean that all of +them should be removed blindly. + +In this post, I'm talking about files that are managed by a SCM. When working on such a text file, +editor's hooks that delete them when writing a file can be more annoying than keeping them in place. +A change should only touch lines that are relevant to the fix or feature beeing added. Touching +lines that are not relevant are creating noise in the history. I've made this mistake in the past, +and I've learned my lessons. + +## Pain for the reviewer + +The person who will review the change will have to make an extra effort to understand why the diff +highlight some lines where it looks like there's no change. It's a distraction to his main task, and +it doesn't bring any benefit to the change beeing submitted. + +## Pain for the person browsing the history + +When someone browse the history and try to understand what changed between two versions, the +deletion is just noise. It's already hard to make the mental effort to read a diff, and understand +what and why things have changed. Adding some extra noise is annoying. + +Running a tool like `git blame` shows how useless this is, for both the person reading the history +and the author. + +## Tips + +Configure your editor to highlight them. If you are using Emacs, you can do it with + +``` +(require 'whitespace) +(global-whitespace-mode 1) +(setq whitespace-style '(face trailing tabs tab-mark)) +``` + +With vim, you can add the following: + +``` +set list lcs=trail:·,tab:»· +highlight ExtraWhitespace ctermbg=red guibg=red +``` + +It's also possible to configure `git` to highlight them when running `git add -p`, by running + +``` +git config --global core.whitespace trailing-space,space-before-tab +``` + +`git` will complain if it finds white spaces in your change, so you have time to fix and remove +them. + +If you really don't want any trailing white spaces, you can also configure your SCM with a +post-commit hook to reject commits that contains them. + +If you're using `vimdiff` to read a diff, it's possible to not highlight white spaces with `set +diffopt+=iwhite`. This can makes it a little bit easier to read messy diff. diff --git a/content/post/2015-08-22-deployment-on-friday.md b/content/post/2015-08-22-deployment-on-friday.md new file mode 100644 index 0000000..eba7786 --- /dev/null +++ b/content/post/2015-08-22-deployment-on-friday.md @@ -0,0 +1,36 @@ +--- +date: 2015-08-22T00:00:00Z +summary: In which we deploy our code on a Friday afternoon. +title: Deploying to production on a Friday +--- + +There's a lot of reasons to not deploy code to production on a Friday. If something goes wrong, you +might end up spending your Friday night, or worst, your entire weekend, fixing some issue. But +there's no reasons Friday should be different from any other day. It's not OK to spend a night +during the week fixing an issue in production. + +To be in a place where you can deploy code at any time requires a few things: tools, automation and +culture. + +In my team, every body is on call: SWEs and SREs. We all share the same responsibilities. There's no +rule saying that deploying on Friday is forbidden. We don't do continuous deployment, and we also +don't have a schedule saying which days of the week we deploy. + +Artifacts are build by our build system and pushed to some archives. We don't deploy artifacts that +we build on a developer laptop. + +Before deploying an artifact to production, we usually run it as a a canary on some shards. +Depending of the service, it can be for a few hours up to a couple of weeks. Once we are confident +in this artifact, after reviewing our dashboards, we know it's going to be safe to deploy it. + +We communicate with the team (and outside the team) before doing a deployment. A deploy starts with +a ticket in Jira, saying what we are going to deploy, how long it will take, and what are the steps +to validate, and do a rollback. At least one another person need to review the ticket and give his +consent. If that person has some concerns, the deploy is pushed back, and we figure out what's +wrong. On-call has to ACK the procedure before starting too. + +Our deployments are hooked with our alerts. If something goes wrong, the deployment is paused, or +roll backed. + +Once it becomes easy to build test deploy and rollback, then there's no reasons to not deploy code on +Friday. diff --git a/content/post/2015-09-03-talking-about-technology.md b/content/post/2015-09-03-talking-about-technology.md new file mode 100644 index 0000000..8457ac2 --- /dev/null +++ b/content/post/2015-09-03-talking-about-technology.md @@ -0,0 +1,52 @@ +--- +date: 2015-09-03T00:00:00Z +summary: In which I particularly enjoyed a talk from MesosCon, and hope to see more + like this one +title: Talking about technology at conferences or meet-up. +--- + +I'm more and more annoyed by how the tech community is super enthusiastic about new pieces of +technology, and how hard they try to convince you it's the best next thing in the world. Way too +often, at conferences or meet-ups, the talks tend to glorify a product or a technology, and only +focus on how it will make your life easier. It's too common to have someone do a demo on stage on +how to build, in 5 minutes, a trivial application running with X many instances in a container in +the cloud and be like "see how easy it was !?". + +What will not be mentioned is how your team is going to transition to this technology or +infrastructure; what are the traps hiding; what will not work; what are the real limitations (can it +scale to more than 10 instances ? 100 instances ? 10k instances ?); how do you manage it in your +data-center; in your cloud; how easy is it to debug; what are the current issues that people running +it in production have met; what's the worst case scenario for an incident; how long can it take to +recover; and way too many other things. + +Over the last few days, I binge-watched many of the +[MesosCon](https://www.youtube.com/playlist?list=PLVjgeV_avap2arug3vIz8c6l72rvh9poV)'s' videos. One of the +talk I really enjoyed was by [Joseph Smith](https://twitter.com/Yasumoto). In [his +talk](https://www.youtube.com/watch?v=nNrh-gdu9m4&index=8&list=PLVjgeV_avap2arug3vIz8c6l72rvh9poV), +he shared about various ways Mesos and Aurora failed at Twitter. + +Joseph's talk was the opposite of what I described earlier. He mentioned at length issues and +problems we've encountered running Aurora. Some of the issues he explored were recent (from a couple +of weeks ago); some were pretty old and are fixed by now; and also what would be the worst case +scenario that could happen. This is exactly what I want to hear when someone introduces a piece of +technology. I need to be aware of them. It doesn't mean that I'm going to be scared and will not use +it. + +I believe this is important. The public who come to a talk is, most of the time, here to learn about +a piece of technology. They might have some prior knowledge, but most of them don't. They want to +learn what can be done with it; how to use it; how it's an improvement. But more importantly, we +need to talk about the cost and path to adopt the piece of technology. Going from a simple demo +running on 2 hosts to a something running on production with hundred of thousands of users and on +thousands of instances is a different story. + +And yes, these could be questions asked by the public at the end of the talk. But not everybody +feel comfortable asking them out loud in front of their peers. + +I feel the same way about post-mortems. Companies should share them more frequently. Some companies +are [pretty good about it](https://github.com/danluu/post-mortems). I can understand, if your +product is not a service for developers, that you might not want to share them on your blog to not +scare your users. But we should still share them during conferences. Maybe there's even an +opportunity for a meet-up focused on post-Mort em ? + +Talking about issues and how difficult it might be to adopt something is not doing is disservice to +something you really enjoy working with. diff --git a/content/post/2015-12-28-books-of-2015.md b/content/post/2015-12-28-books-of-2015.md new file mode 100644 index 0000000..701f9e2 --- /dev/null +++ b/content/post/2015-12-28-books-of-2015.md @@ -0,0 +1,60 @@ +--- +date: 2015-12-28T00:00:00Z +summary: In which I quickly review the books I've read in 2015 +title: Books of 2015 +--- + +I was hoping to read more this year, but I've started very slowly, with only a few books the first half of the year. But almost all of them were good, and only one really disappointed me. I'm not covering purely technical books this time, maybe I'll do it next year. + +- [Countdown to zero day](http://www.amazon.com/gp/product/B00KEPLC08): First book of the year, it was mentioned at lunch by a [coworker](https://github.com/rgs1). The book is about the [Stuxnet](https://en.wikipedia.org/wiki/Stuxnet) virus: how it was discovered; the impact; the remaining questions; etc. You'll read it as a spy novel. The style is not great, but the story is so unbelievable, it's hard to stop reading. Also, the author did a great job to not be too technical, you don't need any particular background to understand it. Now would be a good time to read it, while following conversations about [#32c3](https://twitter.com/search?f=tweets&vertical=news&q=%2332c3&src=typd). + +- [Proxima](http://www.amazon.com/gp/product/B00INIJJAI): Your classic space opera. A group is send to a new planet to colonize it, and everything goes wrong. Good read while commuting to work, and there's some good ideas. I will read more in this series next year. + +- [Witches Abroad](http://www.amazon.com/gp/product/B001AW2OYC): First Pratchett in a while! We follow a small group of witches traveling in foreign parts of the discworld. As always, it's hilarious. I really need to get back to the discworld universe. One book a year is not enough! + +- [The prince](http://www.amazon.com/gp/product/B00KWPSND4): My first time reading it, and it was good! I'll admit that the content is different from what I was expecting. It's not as "cold" or "brutal" as I thought it would be if you put the text in its historical context. Machiavelli is very practical in his explanations. + +- [How to stop sucking and be awesome instead](http://www.amazon.com/gp/product/B00BU3KPQU): It's fun to read in commute. There's some good advice, and it's not only for engineers / managers. I think everybody can read it and get something from it, you don't need any technical background to enjoy the essays. + +- [Seveneves](http://www.amazon.com/gp/product/B00LZWV8JO): I love Stephenson. I'm a huge fan of the [Cryptonomicon trilogy](http://www.amazon.com/Cryptonomicon-Neal-Stephenson-ebook/dp/B000FC11A6/) and the [Baroque Cycle](http://www.amazon.com/Baroque-Cycle-Quicksilver-Confusion-System-ebook/dp/B00KVIBWPI). However, [Reamde](http://www.amazon.com/Reamde-Novel-Neal-Stephenson-ebook/dp/B004XVN0WW) was kind of disappointing, and so I had high expectation for this novel. The book is divided in two parts: for the first one, it was like I watching [Gravity](https://en.wikipedia.org/wiki/Gravity_(film)), where everything that will go wrong, will go wrong. The second part is not as entertaining, but is still good. Despite the last part, it's still one of my favorite read for this year. The first part is really captivating, and you just want to know what's going to happen next. + +- [The Pentium chronicles](http://www.amazon.com/gp/product/B001CBCRCA): Great book on project management. When I got the book, I was expecting something similar to [Inside The Machine](http://www.amazon.com/Inside-Machine-Introduction-Microprocessors-Architecture-ebook/dp/B004OEJO0A): a very technical book about the Pentium: how it works, what were the decision taken while designing the processor, the various challenges, etc. Although you'll find some of that in the book, it's more about how to manage a large and risky project in a company like Intel. Different teams want different things at different time. Marketing want to go in different direction; how to respond to change of schedule; how to handle frustrations in the team, etc. I highly recommend this book to anyone in a manager position or who want to become manager. + +- [caliban’s war: book 2](http://www.amazon.com/gp/product/B005SCRR1A): Another typical space opera, ideal for the commute. There’s a TV show now, saw the first episode ([it is/was free on Google Play](https://play.google.com/store/tv/show/The_Expanse?id=qSBdK4fbIYc)). I think I’ll stick with the book for now, and wait to hear what people are saying about the show. I’ll keep reading this serie. + +- [The Three Body Problem](http://www.amazon.com/gp/product/B00IQO403K): Another book mentioned by a co-worker during lunch. It's an interesting SF book. Not the best I’ve read, but entertaining. There's also some insight on Chinese culture (the author is chinese). + +- [The Martian: a novel](http://www.amazon.com/gp/product/B00EMXBDMA): Great SF book. Fun to read, lot’s of details. Engineers will love this book. Interesting to see how he approaches problems and solve them The book was in my "to-read" list for a while, and I wanted to read it before seeing the movie, which I did. + +- [Structures: or why things don’t fall](http://www.amazon.com/gp/product/B009G1PHP2): if, in general, you are curious, definitely take a look at this one. I’ve learned some things about architecture and why some things are the way they are. I’ll definitely look at buildings with a different yey now (especially churches / cathedrals / bridges). Also really good to force you to ask the question ‘why is it that way’ instead of saying ‘this is stupid’. + +- [Nexus](http://www.amazon.com/Nexus-Trilogy-Book-1-ebook/dp/B00TOZI7FM): This is a great SF series. It reminds me of [babylon babies](http://www.amazon.com/Babylon-Babies-Science-Fiction-French/dp/2070417530/), I have the same excitement reading the books. If you are interested in AI, and what it could mean for the future of humanity, read this book: there's a lot of interesting ideas. + +- [The soul of a new machine](http://www.amazon.com/Soul-New-Machine-Tracy-Kidder-ebook/dp/B005HG4W9W): Interesting book about a computer build by Data General in the late 70s. There's some technical information, but it's mostly about the people who worked on the project. The part I really enjoyed is that the author put a great effort to put everything in context. Like for "The Pentium Chronicles", if you are a manager or interested in learning more about working with people, I highly recommend this book. Also, I would not be surprised if the authors of [Halt and catch fire](https://play.google.com/store/tv/show/Halt_and_Catch_Fire?id=0B6GRUwiZH4) got some ideas from this book :) + +- [Console wars](http://www.amazon.com/Console-Wars-Nintendo-Defined-Generation-ebook/dp/B00FJ379XE): The war between Sega and Nintendo. This two companies created amazing hardware and software in the late 80s and early 90s. Like a lot of kids from my generation, I got interested in computers because of video games. And so I had high expectation for this book, but sadly it was a disappointment for me. I was expecting something more technical, but the author almost only talk about the marketing part. It also focus only on the U.S. market, and I was hoping to learn more about the launch of the consoles in Japan. I can see how this would be interesting if you are interested in the pop culture side of video games though, but this is not my case. + +- [Crux](http://www.amazon.com/gp/product/B00TOZI7J8): Second volume in the Nexus trilogy. Still entertaining, and there's some really great quotes. I loved the action. The last volume in the series will probably be my first novel of 2016. + +- [The Dark Forest](http://www.amazon.com/Dark-Forest-Cixin-Liu-ebook/dp/B00R13OYU6): Second volume in "The Three Body Problem" trilogy, and I can't say I enjoyed this volume. There's three parts: the first two are really not that exciting, but the last one is really good. I've spend a lot of time questioning the actions the characters take in the first two parts, and I don't feel like the explanations given in the last part are satisfactory. Also, the style is still not really good: I can't say if it’s due to the translation or not (I remember having a similar problem with some of Gibson’s book in their french version). + + + + + + + + + + + + + + + + + + + + + |
