Jumat, 26 Februari 2010

{Logging, Messaging, Notification, Auditing, ...} frameworks

They're all the same, in many respects.

In my module code, I want to be as flexible as possible. I want to be as detached from implementation details as possible.

I just want to generate a log message (or a notification, or an audit entry). I don't want to care where it ends up, how it is sent, how it is filtered/categorized, who the recipient(s) is/are, what medium(s) is/are used, etc. Let whoever uses the module configure it all.

And I don't want to reinvent the wheel, I want to use an existing framework. A logging framework seems to be a sensible choice. Let's use Log::Any for example. This is a snippet from a module for a file-manager-type web app:

my (@success, @failed);
for (@files) {
unlink $_;
if ($!) { push @failed, $_ } else { push @success, $_ }
$log->info("Done deleting files. Files deleted: %s. Files not deleted: %s", \@success, @failed);

The message can end up:
  • in a system-wide log-file;
  • discarded (if configured level of output is less than INFO);
  • in a per-user log file;
  • in a notification email to user (should the user configure it);
  • in a notification email to sysadmin (should the admin configure it);
  • into audit table in database (if the application is using a database);
  • in the console (if this module is also used in a command-line based app);
  • as a desktop notification (if this module is also used in a desktop app);
  • in an internal web-based forum;

However, some of those outputs usually need some additional metadata. In email and desktop notification we usually need separate subject and body. In the audit table in database we need to fill in who (the logged in user). In web applications we usually need to log the remote IP address (or even User-Agent string). In desktop notification or forum sometimes we would like to be able to update the message instead of creating a new one.

Instead of in the logging output module (the appender and formatter, in Log4perl-speak), sometimes I *do* care and need to specify this in my module.

Logging APIs usually do not allow us to do so. Output/appender modules are only given the message as a string.

So I intend to cheat by embedding a JSON-/YAML-encoded metadata in front of the message, e.g.:
$log->info("\x00{subject: 'Progress of copying $foo to $bar', id: xasd8f7d}\x00Copying $foo to $bar. 0%");
for (1..100) {
$log->info("\x00{id: xasd8f7d}\x00$_%");
sleep 1;

The presence of \x00 signify that until the next \x00 there will be additional metadata in the form of YAML (or JSON). This way, my desktop notification and web-based forum output module knows that the 2nd-101st message is an update and thus can adjust accordingly.

It should, and I don't have to reinvent any framework or wrapper. But it feels so hackish and ugly.

Kamis, 25 Februari 2010

The problems with the older CPAN clients

Thank God for cpanminus. Now that I'm free from having to use them, allow me to rant, no, bitch about them.

1. Bad defaults. Some default values might make sense 10-15 years ago, but not so much nowadays. For example, I'd argue that "follow" should now be default. See #2.

2. Too developer-oriented. For example, I believe "notest" should be on by default. This is compounded by the fact that installing Perl modules is so damn-slow already. See #3.

3. Too slow. Startup takes around 10-30 seconds or more. Installing Moose usually takes minutes (but with cpanminus, it only takes about 1 minute with --notest on my PC). Autocomplete takes one to a couple of seconds.

4. Too interactive, too verbose. The older clients are getting better but not quiet enough, cpanminus is such a breathe of fresh air.

5. Too bloated (which is the reason why cpanminus was developed in the first place).

The older CPAN clients are an embarassment if we compare it to "apt-get", "yum", "urpmi", which are way faster, way quieter, way less interactive. There's no reason why a CPAN client cannot be like those. And fortunately cpanminus proves it.

The shorter path to deployment heaven

Now that there is cpanminus, I've modified the handy little module CPAN::AutoINC to prefer cpanminus over CPAN.

So now when users download and run my programs/scripts for the first time, instead of failing with the dreaded message:
Can't locate Foo/Bar.pm in @INC (@INC contains: [a dozen or more of paths....]).
BEGIN failed--compilation aborted at /some/path line 123

which users or even Perl novices have no clue on how to fix, the program will instead automatically download every necessary CPAN modules into the user's home directory and runs out-of-the-box on the first try!
$ download-bca
Fetching http://search.cpan.org/CPAN/authors/id/A/AD/ADAMK/File-HomeDir-0.89.tar.gz
Building File-HomeDir-0.89 for File::HomeDir...
File::HomeDir installed successfully.
Fetching http://search.cpan.org/CPAN/authors/id/D/DR/DROLSKY/File-Slurp-9999.13.tar.gz
Building File-Slurp-9999.13 for File::Slurp...
File::Slurp installed successfully.
Fetching http://search.cpan.org/CPAN/authors/id/S/SH/SHARYANTO/Finance-Bank-ID-BCA-0.07.tar.gz
==> Found dependencies: Pod::Coverage, DateTime, Test::Pod::Coverage, Log::Any, Any::Moose, Mouse
Fetching http://search.cpan.org/CPAN/authors/id/R/RC/RCLAMP/Pod-Coverage-0.20.tar.gz
==> Found dependencies: Devel::Symdump
Fetching http://search.cpan.org/CPAN/authors/id/A/AN/ANDK/Devel-Symdump-2.08.tar.gz
Building Devel-Symdump-2.08 for Devel::Symdump...
Devel::Symdump installed successfully.
Building Pod-Coverage-0.20 for Pod::Coverage...
Pod::Coverage installed successfully.
Fetching http://search.cpan.org/CPAN/authors/id/S/SP/SPADKINS/App-Options-1.07.tar.gz
Building App-Options-1.07 for App::Options...
App::Options installed successfully.
(the program finally runs)

As the developer, you only need to put "use CPAN::AutoINC" at the top of your main script. How's that for deployment heaven? Zero installation, zero configuration (thanks to cpanminus), zero dependency outside of Perl core modules.

But there's a catch. The user (or the user's sysadmin) still needs to install CPAN::AutoINC and needs to install and bootstrap local::lib first. So it's not really an out-of-the-box experience yet.

miyagawa++ is already planning to automatically bootstrap local::lib from cpanminus, and the logic behind CPAN::AutoINC is just a couple of dozen lines that can be embedded easily into the script (this process can be made automatic with distribution building tools like Dist::Zilla plugin).

Since cpanminus has zero dependencies, we can simply include it too in our application.

And maybe in the future there can be GUI interfaces for cpanm, so it can display a nice dialog to ask confirmation via the desktop.

Be prepared to be able to provide a much nicer experience for your users.

Rabu, 24 Februari 2010


This is a draft/RFC document.

Log::Any is great if you're writing modules. You only need to say:
use Log::Any qw($log);

and then you're off producing logs with:
# etc

But if you're writing scripts/applications (and thus need to "consume" or display the logs as well), it becomes a bit of a hassle. For example, if you want to display logs to the screen with Log::Dispatch, this is the incantation you need:
use Log::Any qw($log);
use Log::Any::Adapter;
use Log::Dispatch;
my $disp = Log::Dispatch->new(outputs => ["Screen", min_level=>"debug", newline=>1]);
Log::Any::Adapter->set("Dispatch", dispatcher=>$disp);

which I'm sure I'll never ever remember and will just copy paste everytime.

The goal is to be able to write only this in your scripts:
use Log::Any::App qw($log);

and Log::Any::App (which currently does not exist) will take care of all the rest:

  • Choose the best available adapter(s);
  • Configure the adapter(s) with the best defaults, e.g. to screen, as well as to file /var/log/SCRIPTNAME.log (if running as root) or ~/SCRIPTNAME.log (if running as user). The defaults can of course be changed via configuration;
  • Pick configuration from various sources, like environment variables (e.g. turning level to debug if DEBUG is set to true), command line options (e.g. log level from --log_level/--log-level/--debug/--verbose/etc, as well as detecting result from Getopt::Long or App::Options so we can avoid parsing by ourselves)

I think the main challenge is arranging a set of defaults that are acceptable and comfortable for a lot of people, and working together nicely with available modules like adapter modules (Log::Any::Adapter::Dispatch, Log::Any::Adapter::Log4perl), command line parsing modules, configuration modules, etc.

And later should you refactor your script into modules, the logging part can be left untouched as they already use the Log::Any framework.

Modules to look at: Log::Dispatchouli.

Rabu, 17 Februari 2010

A small plea to non-English bloggers

I absolutely welcome and cheer the non-English posts in the Iron Man feed (as they are also welcome by the rules). I even try to read one or two whenever I can.

But, at the risk of sounding like a party pooper, guys, could you please not give your post an English title/subject when the content is not in English? I'm sure other people will appreciate it as it saves them time when they can't read your language anyway.

Thank you very very much, and keep blogging!

Rabu, 10 Februari 2010

The (lack of) readability of Perl

Can't believe it's almost 8 years since I wrote this Indonesian article about "8 things that make Perl relatively unreadable".

To summarize, I think the readability issue in Perl largely boils down to the high usage of symbols/non-alphanumeric characters (e.g. regex and special variables). This admittedly won't change for years to come, as Perl 6 also offers us lots of new operators, even allows us to define new ones, using the full range of the Unicode charset! So we will almost certainly be hearing and need to be fighting against this meme for a long long time.

I also believe that most of the complaints about Perl being unreadable come from non-Perl users. When they read some code, they expect it to be able to read/guess it based on the prior knowledge of some other programming language. For example, even before I started learning French or German, I could guess the general meaning of some French and German (or some other European language) phrases based solely on my knowledge of English. This does not happen with, say, Chinese or Math or regular expression, as the symbols are just gibberish to the non-initiated.

Perhaps marketing that Perl is one of the hardest languages to master will attract more people to learning it, as that's what's happening with Cantonese and some westerners.

Setelah 7 tahun...

Tahun 2003 saya menulis artikel ini: 8 Hal Yang Membuat Perl Relatif Unreadable. Wah, 1 tahun lagi sudah 8 tahun deh, secepat itukah waktu berlalu?

Eniwei, setelah lewat 7 tahun, tentu saja ada beberapa hal/poin di artikel tersebut yang sudah tidak berlaku lagi. Bukti bahwa Perl tetap hidup karena tetap berubah dan berevolusi.

1. Moose dan sistem OO di Perl6. Kini pengguna Perl tidak perlu minder lagi, karena Perl kini sudah punya sistem objek yang modern dan "tidak mengerikan lagi", bahkan berani deh beradu superior dengan Ruby, Python, Javascript, dll.

2. Beberapa perbaikan readability dapat kita jumpai di Perl 6, antara lain penggunaan prefiks variabel yang lebih konsisten, Untuk mengakses elemen array dan hash kini menggunakan @array[n] dan %array{s}, tidak lagi $array[n] dan $array{s}, bahkan sebetulnya Anda bisa "hidup" hanya dengan mengenal satu prefiks saja, $, karena array dan hash dan struktur kompleks semuanya dapat ditampung oleh $.

Terdapat pula grammar untuk mengizinkan kita memodularkan pola regex yang kompleks sehingga menjadi jauh lebih readable.

Tapi... bukan Perl namanya kalau tidak mengizinkan kita meringkas-ringkas program menggunakan banyak simbol. Tak kurang dari puluhan operator baru, belum lagi metaoperator, dan juga berbagai idiom baru, hadir di Perl 6. Bahkan seluruh karakter Unicode dapat dipakai!

Apakah ini membuat Perl tetap/makin unreadable? Mengulangi kata-kata di artikel 2003 saya, readability sesuatu yang subjektif. Apakah matematik yang penuh simbol juga unreadable? Tentu saja iya, bagi mereka yang buta/awam matematik, tapi tentu tidak bagi matematikawan.

Rabu, 03 Februari 2010


Over the course of many years, I have written lots of lots of short command-line scripts in Perl (Perl's great for that, you know). Most of these scripts are small utilities, or replacement for shell scripts, or automation tools.

All of these scripts invariably need some ability to take command-line options. Of course, back in the days, I used Getopt::Std and Getopt::Long. They both did their task well, but also invariably I will need to provide -h or --help for usage information. I always hate having to write this usage text manually.

Also, if you use your scripts often enough, you will end up with the same incantation for some, that you will want to put the command line options to a config file to avoid repeating yourself. Passwords are also not appropriate in command-line options due to security issues.

So I now use App::Options and have never looked back. You can look at it as a pretty straightforward replacement for Getopt::Long, but it gives you automatic --help and --version (--program_version actually, but --version also does something else wonderful). And it automatically enables you to read config files. I emphasize "automatically" because you absolutely do not have to do a single thing, as App::Options gives you some nice defaults on where to find the config files.

How great is that? Just write your code as if you're using Getopt::Long, but gain all these extra abilities for free!

There are other solutions for command-line scripts, like App::Cmd, but really App::Options is the easiest way especially for old-timers like me who do not want to change their Getopt::Long-style habit.

App::Options is not perfect though. There are a couple of small annoyances I have about it, but I'm not hung up on them. The first is that "=" in command-line option is required, i.e. you have to write "--opt=args" instead of "--opt args". Second is, --version defaults to displaying Perl modules version instead of your program's version.

For larger apps, I also sometimes need hierarchical/multilevel configuration. It'd also be nice to have YAML support. These are something I hope to accomplish with Config::Tree, but I guess I still need to work on it quite a bit before I can replace my usage of App::Options with it.