Bench: a simpler benchmark module

There was a post in blogs.perl.org or Planet Perl Iron Man (sorry, forgot the exact article) that said something along the line of: "Benchmark is a fine module, but for simplicity I'll use the time command". Which immediately hit home with me, because I too very seldomly use Benchmark. I guess the problem is I almost always have to perldoc it before using it, and there are quite some extra characters to type.

So last weekend I wrote Bench (repo) that's hopefully simpler enough to get used more.

To benchmark your program, just type: perl -MBench yourscript.pl. Sample output:

$ perl -Ilib -MBench -MMoose -e1
0.229s

Bench exports a single function, bench(), by default. To time a single sub, use: perl -MBench -e'bench sub { ... }'. By default it will call your sub at most 100 times or 1 second. Here's a sample output:

100 calls (12120/s), 0.0083s (0.0825ms/call)

To benchmark several subs: perl -MBench -e'bench {a=>sub{...}, b=>sub{...}}' or perl -MBench -e'bench [sub{...}, sub{...}]'. Sample output:

a: 100 calls (12120/s), 0.0083s (0.0825ms/call)
b: 100 calls (5357/s), 0.0187s (0.187ms/call)

Bench will automatically use Dumbbench if it's already loaded, e.g.: perl -MDumbbench -MBench -e'...'. Or you can force Bench to use Dumbbench: perl -MBench -e'bench sub { ... }, {dumbbench=>1}'.

That's about it currently.

Org::Parser

If you're like me, over the years you'll have had your todo lists scattered over multiple programs and places. First a simple text file with homebrewn format, then various Windows programs, then various Linux GUI programs, then back to Notepad and joe/gedit/kate, then various apps on cellphones, then pencil & paper (due to cellphones keep getting lost/stolen), then some cloud apps, then todo.txt, then finally org-mode. And if you're anything like me or many others, you'll find that org-mode is *it*.

I'm now in the (long, boring) process of consolidating everything in Org. For todo lists, contact lists, and even long documents and all journals/diaries. I've written a preliminary version of Org::Parser to help automate stuffs via the command line. It only supports the basic stuffs at the moment but has been able to parse all my *.org files.

The code is available on GitHub.

Backup data dengan dengan Perl, rsync, dan git

Saat ini, saya menyimpan data pribadi di 2 direktori utama: ~/repos dan ~/media. Semua file-file teks (termasuk source code, website, catatan/tulisan, konfigurasi, agenda .org Emacs) ditaruh di bawah ~/repos di dalam repo-repo git, per proyek (Contoh: ada ~/repos/settings, ~/repos/writings, ~/repos/perl-Git-Bunch, dsb). Semua file lain yang berupa file media besar-besar ditaruh di ~/media.

Untuk membackup data di ~/media, saya menggunakan File::RsyBak, yang menyediakan skrip command-line rsybak. Skrip ini pada dasarnya hanyalah wrapper untuk perintah rsync dan membuat snapshot-snapshot backup sesuai jangka waktu histori yang diinginkan (defaultnya: 7 harian + 4 mingguan + 3 bulanan). Skrip ini dijalankan tiap hari lewat cron dan backupnya disimpan di harddisk terpisah /backup.

Untuk membackup data di ~/repos, saya menggunakan Git::Bunch, yang menyediakan skrip command-line gitbunch. Pada dasarnya, gitbunch membackup menggunakan rsync juga, tapi tanpa histori (karena git sudah menyimpan sejarah perubahan). Selain itu, yang dibackup juga hanya subdirektori .git/ dari tiap repo. Ini mengirit ruang disk, karena ~/repos masih sering saya kopi ke flashdisk yang kapasitasnya terbatas. Untuk merestore dari backup, kita tinggal melakukan "git checkout" dari hasil backup .git/ tiap repo ini.

Skrip gitbunch juga dapat melakukan sinkronisasi dari satu direktori ~/repos ke direktori ~/repos lainnya. Pada intinya, "gitbunch sync" hanyalah wrapper untuk "git pull". Dengan cara ini, saya bisa mensinkronkan pekerjaan PC ke netbook atau sebaliknya dengan mudah.

Artikel yang lebih mendetil, pernah ditulis untuk majalah InfoLINUX: Manajemen data pribadi dengan git.

Bagaimana strategi backup Anda?

The coming bloated Perl apps?

A few weeks ago, I got annoyed by the fact that one of our command line applications was getting slower and slower to start up (the delay was getting more and more noticable), so I thought I'd do some refactoring, e.g. split large files into smaller ones and delay loading modules until necessary.

Sure enough, one of the main causes of the slow start up was preloading too many modules. Over the years I had been blindly sticking 'use' statements into our kitchen sink $Proj::Utils module, which was used by almost all scripts in the project. Loading $Proj::Utils alone pulled in over 60k lines from around 150 files!

After I split things up, it became clearer which modules are particularly heavy. This one stood out:

% time perl -MFile::ChangeNotify -e1
real 0m0.972s

% perl -MDevel::EndStats -e1
# Total number of module files loaded: 129
# Total number of modules lines loaded: 46385

So almost 130 files and a total of 45k+ lines just from loading File::ChangeNotify alone. 130 files just for a filesystem monitoring routine! Who would've thought that a filesystem monitor needs so many lines of program? Compare with, say, a recent HTTP client:

% perl -MHTTP::Tiny -e1
# Total number of module files loaded: 18
# Total number of modules lines loaded: 6089

I quickly switched to Linux::Inotify2 and things are much better now (but I might have to revisit this since we want to give the new Debian/kFreeBSD a Squeeze).

As I suspected (since the module is written by Dave Rolsky also), File::ChangeNotify utilizes Moose, which is not particularly lightweight either:

% time perl -MMoose -e1
real 0m0.712s

% perl -MDevel::EndStats -MMoose -e1
# Total number of module files loaded: 100
# Total number of modules lines loaded: 35760

Compare with:

% time perl -MMouse -e1
real 0m0.089s

% perl -MDevel::EndStats -MMouse -e1
# Total number of module files loaded: 20
# Total number of modules lines loaded: 6675

Come to think of it, running Dist::Zilla is also quite painfully slow these days. Just running "dzil foo" pulled in around 60k lines and took 1.7s! Of course, dzil is Moose-based.

While it is a good thing that Moose is getting more popular, it's a bit shameful to see that Ruby and Python scripts "get OO for free" while Moose scripts have to endure a 0.7s startup penalty. Mouse, Moo, Role::Basic come to the rescue but I wonder what would Ruby/Python programmers think (you have how many object systems?? Why do you people can never agree on one thing and TIMTOWTDI everything?)

Disclaimer: Number of lines includes all blanks/comment/POD/DATA/etc from all files loaded in %INC, actual SLOC is probably significantly less. Timing is done on a puny HP Mininote netbook (Atom N450 1.66GHz) which I'm currently stuck with in the past few weeks. With all due respects to all authors of modules mentioned. They all write fantastic, working code.

Outputting pretty data structure on console programs

Our application has a command-line API interface for convenient access via shell/console. It used to output API result data in YAML:

# /c/sbin/spanel api --yaml File list --account steven --volume data --dir /public
--- 
dir: 
  atime: '1270675429'
  ctime: '1285916065'
  gid: 1023
  group: steven
  is_dir: 1
  mtime: '1285916065'
  perms: 493
  uid: 1012
  url: ~
  user: steven
entries: 
  - 
    atime: '1284665000'
    ctime: '1289609859'
    dir: /public
    gid: 1023
    group: steven
    icon_file: folder.gif
    inode: 1984908
    is_dir: 1
    is_link: 0
    mtime: '1289609859'
    name: git
    perms: 493
    reldir: ''
    size: 4096
    uid: 1012
    url: ~
    user: steven
  - 
    atime: '1270675424'
    ctime: '1285130140'
    dir: /public
    gid: 1023
    group: steven
    icon_file: text.gif
    inode: 1976727
    is_dir: 0
    is_link: 0
    mtime: '1155368486'
    name: .htaccess
    perms: 436
    reldir: ''
    size: 48
    uid: 1012
    url: ~
    user: steven
  - 
    atime: '1270675424'
    ctime: '1285130140'
    dir: /public
    gid: 1023
    group: steven
    icon_file: unknown.gif
    inode: 1976725
    is_dir: 0
    is_link: 0
    mtime: '1155368397'
    name: .htaccess~
    perms: 436
    reldir: ''
    size: 63
    uid: 1012
    url: ~
    user: steven
total_num_entries: 3
url: ~

YAML is relatively readable if you compare to JSON or (shudder) XML, but I soon grew tired of reading YAML for data that should be tabulated and better formatted for human consumption.

Thus, Data::Format::Pretty::Console. The idea is for me, a lazy programmer, to throw it data structure of various kind and it will display it nicely suitable for console viewing. The command line API interface now by default shows nicely formatted text for API result data (but still provides --yaml and --json option). Please bear with this blog post's misformatting and assume it's all pretty:

dir:
.---------------------.
| key    | value      |
+--------+------------+
| atime  | 1270675429 |
| ctime  | 1285916065 |
| gid    |       1023 |
| group  | steven     |
| is_dir |          1 |
| mtime  | 1285916065 |
| perms  |        493 |
| uid    |       1012 |
| url    |            |
| user   | steven     |
'--------+------------'

entries:
.----------------------------------------------------------------------------------------------------------------------------------------------------------------------.
| atime      | ctime      | dir     | gid  | group  | icon_file   | inode   | is_dir | is_link | mtime      | name       | perms | reldir | size | uid  | url | user   |
+------------+------------+---------+------+--------+-------------+---------+--------+---------+------------+------------+-------+--------+------+------+-----+--------+
| 1284665000 | 1289609859 | /public | 1023 | steven | folder.gif  | 1984908 |      1 |       0 | 1289609859 | git        |   493 |        | 4096 | 1012 |     | steven |
| 1270675424 | 1285130140 | /public | 1023 | steven | text.gif    | 1976727 |      0 |       0 | 1155368486 | .htaccess  |   436 |        |   48 | 1012 |     | steven |
| 1270675424 | 1285130140 | /public | 1023 | steven | unknown.gif | 1976725 |      0 |       0 | 1155368397 | .htaccess~ |   436 |        |   63 | 1012 |     | steven |
'------------+------------+---------+------+--------+-------------+---------+--------+---------+------------+------------+-------+--------+------+------+-----+--------'

total_num_entries:
3

url:

Comparison of INI-format modules on CPAN

I'm not terribly happy with the state of Perl/CPAN support for the INI file format.

I have this requirement of modifying php.ini files programmatically from Perl like: set register_globals to On/Off, add/remove some extension (via the extension=foo lines), adding/removing some functions from the disabled_functions list, etc. So I would like to find a CPAN module that can just set/unset a parameter and leave formatting/comments alone as much as possible.

Turns out that among a dozen or so of INI modules on CPAN, a few of them do not do writes at all (e.g. Config::INI::Access or Config::Format::INI). And a few that do, write a la dump. That is, they just rewrite the whole INI file with the in-memory structure. All comments and formatting (even ordering, in some cases) are lost. Example: Config::INI::Writer and Tie::Cfg. And, last time I tried, I couldn't even install Config::IniHash from the CPAN client. Not good.

So I ended up with Config::IniFiles. And I needed to patch two features in before I could even read php.ini and write to it properly. This is an old module, which although still maintained, probably needs a rewrite or modernization. One reviewer in CPAN Ratings also wrote that this module fails in edge cases and the test suite is incomplete.

But, at least this module gets the work done. It tries to maintain comments, and even has a host of other features like delta, multiline, default section, etc. Most features seem to be rather exotic to me personally, but then none of the other INI modules on CPAN has the basic features that I needed.

(Short, grossly incomplete) comparison of Perl logging frameworks

After doing this post on comparison of Perl serialization modules, I intended to continue with other comparisons, and even thought on setting up a wiki or creating/maintaining a section on the Official Perl 5 Wiki, which already has a Recommended Modules section, although there is not much comparison being written for each recommendation. (Btw, I just noticed a change of domain for the Wiki, from perlfoundation.org to socialtext.net).

But of course other tasks soon took precedence, so until the Wiki idea is realized, I thought I'll just continue posting on the blog as usual.

There are **seriously** too many Perl logging frameworks out there. As the Log4perl FAQ mentions, "Writing a logging module is like a rite of passage for every Perl programmer, just like writing your own templating system."

So I'm not going to pretend like I've evaluated even half of logging modules that are on CPAN. Instead I'm just going to include a few worth mentioning.

Log::Dispatch and Log::Log4perl. Two of the arguably most popular Perl logging frameworks are Log::Dispatch and Log::Log4perl. They are like the Moose of logging frameworks: mature, feature rich, flexible, has a lot of support/extra/3rd party modules, but... "slow". I quote the slow part because first of all, speed is not an issue for the majority of applications. And second of all, they are not relatively slow at all compared to other modules until they actually log stuff to output. For example, doing debug() on a warn level is around 1,5mils/sec with with Log4perl, and 3mils/sec with Log::Fast. But for actual logging, Log::Fast can be 10-45 times faster than these two.

Log::Any. For most people, Log::Dispatch and Log4perl should suffice. I personally haven't been unable to produce a case where I can't customize Log4perl they way I want. This shows the flexibility of the module. So the only thing left for flexibility is a thin wrapper where you might want to switch logging framework (kind of like Any::Moose for logging). There are a few of these on CPAN, but I prefer Log::Any (and I've also made a thin wrapper for *that*, Log::Any::App). RJBS also made one: Log::Dispatchoulli. You might be interested in using it if you are interested in using String::Flogger.

Performance-wise, as with Moose, there are other alternatives: Log::Fast, for one. There are also a few other minimalistic frameworks, but I do not recommend using them as many of them are not flexible at all. Unless your application is really performance-critical.

I've most probably left out a lot of possibly interesting alternatives.