Perl Indonesia

Menginstal Finance::Bank::ID::Mandiri untuk mengunduh transaksi Mandiri

2011-10-19T00:08:00.001-07:00

Atas permintaan user, berikut ini panduan cara menggunakan modul Perl Finance::Bank::ID::Mandiri untuk mengunduh transaksi rekening bank Mandiri Anda. Panduan ini berasumsi Anda menggunakan Linux (Debian atau Ubuntu) dengan Perl 5.10 ke atas. Jika Anda menggunakan Windows, atau Linux dengan Perl di bawah 5.10 (mis: CentOS 5.x) harap menyesuaikan sendiri (atau, jika ada yang mau membuatkan tutorialnya, silakan hubungi saya).

Modul serupa untuk BCA Finance::Bank::ID::BCA juga tersedia, cara menggunakannya mirip.

Prasyarat:

Komputer dengan koneksi Internet bersistem operasi Linux (Debian/Ubuntu) dan Perl 5.10 ke atas
Program curl (untuk mendownload cpanminus)
Akses root (aplikasi bisa juga diinstal tanpa akses root, tapi agar mudahnya kita pakai root)
Rekening bank Mandiri dengan akses internet banking aktif (ada username dan password).

Langkah-langkah:

Instal modul-modul Perl yang dibutuhkan. Agar mudahnya, kita menggunakan cpanminus untuk menginstal modul-modul Perl. Jika Anda belum menginstal cpanminus, silakan instal dulu sbb:

Buka konsol, lalu ketik:
```
$ curl -L http://cpanmin.us | perl - --sudo App::cpanminus
```
Setelah itu, kita menginstal modul Mandiri dengan cpanminus:
```
$ sudo cpanm -n Finance::Bank::ID::Mandiri
```
Mengkonfigurasi program. Setelah selesai, Anda akan mendapatkan perintah download-mandiri. Konfigurasi perintah ini dengan membuat file konfigurasi:
```
$ mkdir ~/.app
$ (buat/edit file download-mandiri.conf)
```
Isi file konfigurasi adalah sbb:
```
[ALL]
username = (username akun Mandiri Anda)
password = (password akun Mandiri Anda)
```
Setelah itu tinggal jalankan perintah download-mandiri dari konsol. Defaultnya program akan mengunduh transaksi dalam format YAML selama sebulan terakhir. Bisa juga dioutput format JSON dan kustomisasi tanggal. Tambahkan opsi --debug jika ingin melihat pesan debugging. Anda juga bisa menjalankan skrip ini lewat cron agar berjalan otomatis secara periodik (mis: seminggu sekali atau sehari sekali).

Jika mengalami masalah, silakan reply posting blog ini.

App::UniqFiles (a case for building app with Dist::Zilla and Sub::Spec)

2011-07-27T20:37:00.001-07:00

When watching videos at Tudou or Youku, both Chinese YouTube-like video sites, you'll often get one/two 15- or 30-second video ads at the beginning. Since I download lots of videos recently, my Opera browser cache contains a bunch of these video ads files, each usually ranging from around 500k to a little over 1MB. But there are also duplicates.

I thought I'd collect these ads, for learning Chinese, but I don't want the duplicates, only one file per different ad. The result: App::UniqFiles, which contains a command-line script called uniq-files. Now all I need to do is just type mkdir .save; mv `uniq-files *` .save/ and delete the duplicate videos, which are files not moved to .save/.

With the help of Dist::Zilla, Sub::Spec::CmdLine, Pod::Weaver::Plugin::SubSpec, and Log::Any::App, I managed to finish App::UniqFiles, from scribbling down the concept to uploading the first release to CPAN and github, in just about under an hour (00:54 to be exact). Not super-speedy for a small script (I can probably write a one-off script version in 15-30 minutes), but for an extra 30 minutes, I get:

a proper Perl distribution, with tests and POD and all;
all the core functionality contained in subroutines (which is much more reusable than a script);
a POD API documentation for the subroutines;
a command-line application with --help message, argument parsing, configurable log levels, even bash completion with just 3 lines of code.

I think developing with Dist::Zilla and Sub::Spec is great, mainly because they realize the DRY ("Don't Repeat Yourself") principle and free you from mundane tasks. Having to repeat the same stuffs or do mindless/tedious tasks is indeed a significant source of frustation for programmers. It deflects us from the real, important task: writing the code to actually solve our problems.

Dist::Zilla allows you to generate dist's README from the main module's POD instead of you having to create this file manually. It inserts LICENSE, AUTHORS, VERSION sections to your POD instead of you having to insert and update them manually. It frees you from the mundane tasks like creating dist tarballs, checking ChangeLog, incrementing version numbers, uploading to CPAN, etc. Really, I wouldn't want to build dists manually ever again without tools like Dist::Zilla.

Sub::Spec allows you to specify rich metadata for your sub in one place, from which you can generate Getopt::Long options, POD documentation, command-line --help message, etc from it, instead of you having to maintain each of them manually. Module like Sub::Spec::CmdLine also frees you from many mundane UI issues (which, coincidentally, I hate) like parsing arguments and formatting output data to screen.

Undocumented Getopt::Long::Configure feature

2011-07-25T21:43:00.000-07:00

Getopt::Long has a Configure() function to let you customize its parsing behaviour, e.g. whether or not to be case-sensitive, whether or not unknown options are passed unmodified or generate an error, etc. However, this customization is global: it affects every piece of code using Getopt::Long.

Since I use Getopt::Long in a utility module, which might conflict with the module user using Getopt::Long along with my module, I need to localize my Configure() effect. I was about to submit an RT wishlist ticket pertaining to this, but some quick checking revealed that Configure() already has this feature.

Configure() returns an arrayref containing all the current options. If you pass this arrayref to it, it will set all the options. This way, you can save and restore options.

Thanks to the Getopt::Long author, Johan Vromans, who apparently has maintained this module since 1990!

Using Org format to document code

2011-06-16T20:54:00.000-07:00

My most recent hacktivity includes preparing Org::Export::Pod and Org::Export::Text (both not yet ready) following Org::Export::HTML. I am planning to document source code (currently just for functions) using Org as the master format instead of POD. From Org, I'll be exporting to various target formats, including POD itself, inserted to modules' source code in the build process using a simple Dist::Zilla plugin.

Now why Org? First and foremost, obviously because I use Emacs, and the last few months I've migrated practically all of my notes/todolists/addressbooks to this format. Also, it's visually nicer to look at than POD when it comes to things like headings and lists. Org also supports tables (I understand that there's an extension to POD that supports tables too, but I imagine it will not be as easy to write?). BTW, among other lightweight markup languages, Markdown Extra also supports tables with an equally nice syntax.

A couple of concerns for Org. First, writing literal examples is a bit more cumbersome. Where in Pod or Markdown or most Wiki format you only need to indent to go verbatim, in Org you need to enclose with #+BEGIN_SRC ... #+END_SRC or prepend each line with ": ". But I've come to accept it.

Second is parser support in other languages. Since I envision ultimately my function specs is to be processed by other languages too, it would be nice if there are support for the document parser in these languages, including for Javascript and PHP. In this regard, Markdown seems to be a win.

But hey, Org is still readable as-is, and currently nothing beats Org-mode for writing notes. So Org FTW!

Bench: a simpler benchmark module

2011-03-30T00:29:00.000-07:00

There was a post in blogs.perl.org or Planet Perl Iron Man (sorry, forgot the exact article) that said something along the line of: "Benchmark is a fine module, but for simplicity I'll use the time command". Which immediately hit home with me, because I too very seldomly use Benchmark. I guess the problem is I almost always have to perldoc it before using it, and there are quite some extra characters to type.

So last weekend I wrote Bench (repo) that's hopefully simpler enough to get used more.

To benchmark your program, just type: perl -MBench yourscript.pl. Sample output:

$ perl -Ilib -MBench -MMoose -e1
0.229s

Bench exports a single function, bench(), by default. To time a single sub, use: perl -MBench -e'bench sub { ... }'. By default it will call your sub at most 100 times or 1 second. Here's a sample output:

100 calls (12120/s), 0.0083s (0.0825ms/call)

To benchmark several subs: perl -MBench -e'bench {a=>sub{...}, b=>sub{...}}' or perl -MBench -e'bench [sub{...}, sub{...}]'. Sample output:

a: 100 calls (12120/s), 0.0083s (0.0825ms/call)
b: 100 calls (5357/s), 0.0187s (0.187ms/call)

Bench will automatically use Dumbbench if it's already loaded, e.g.: perl -MDumbbench -MBench -e'...'. Or you can force Bench to use Dumbbench: perl -MBench -e'bench sub { ... }, {dumbbench=>1}'.

That's about it currently.

Org::Parser

2011-03-17T07:02:00.001-07:00

If you're like me, over the years you'll have had your todo lists scattered over multiple programs and places. First a simple text file with homebrewn format, then various Windows programs, then various Linux GUI programs, then back to Notepad and joe/gedit/kate, then various apps on cellphones, then pencil & paper (due to cellphones keep getting lost/stolen), then some cloud apps, then todo.txt, then finally org-mode. And if you're anything like me or many others, you'll find that org-mode is *it*.

I'm now in the (long, boring) process of consolidating everything in Org. For todo lists, contact lists, and even long documents and all journals/diaries. I've written a preliminary version of Org::Parser to help automate stuffs via the command line. It only supports the basic stuffs at the moment but has been able to parse all my *.org files.

The code is available on GitHub.

Backup data dengan dengan Perl, rsync, dan git

2011-02-14T23:27:00.000-08:00

Saat ini, saya menyimpan data pribadi di 2 direktori utama: ~/repos dan ~/media. Semua file-file teks (termasuk source code, website, catatan/tulisan, konfigurasi, agenda .org Emacs) ditaruh di bawah ~/repos di dalam repo-repo git, per proyek (Contoh: ada ~/repos/settings, ~/repos/writings, ~/repos/perl-Git-Bunch, dsb). Semua file lain yang berupa file media besar-besar ditaruh di ~/media.

Untuk membackup data di ~/media, saya menggunakan File::RsyBak, yang menyediakan skrip command-line rsybak. Skrip ini pada dasarnya hanyalah wrapper untuk perintah rsync dan membuat snapshot-snapshot backup sesuai jangka waktu histori yang diinginkan (defaultnya: 7 harian + 4 mingguan + 3 bulanan). Skrip ini dijalankan tiap hari lewat cron dan backupnya disimpan di harddisk terpisah /backup.

Untuk membackup data di ~/repos, saya menggunakan Git::Bunch, yang menyediakan skrip command-line gitbunch. Pada dasarnya, gitbunch membackup menggunakan rsync juga, tapi tanpa histori (karena git sudah menyimpan sejarah perubahan). Selain itu, yang dibackup juga hanya subdirektori .git/ dari tiap repo. Ini mengirit ruang disk, karena ~/repos masih sering saya kopi ke flashdisk yang kapasitasnya terbatas. Untuk merestore dari backup, kita tinggal melakukan "git checkout" dari hasil backup .git/ tiap repo ini.

Skrip gitbunch juga dapat melakukan sinkronisasi dari satu direktori ~/repos ke direktori ~/repos lainnya. Pada intinya, "gitbunch sync" hanyalah wrapper untuk "git pull". Dengan cara ini, saya bisa mensinkronkan pekerjaan PC ke netbook atau sebaliknya dengan mudah.

Artikel yang lebih mendetil, pernah ditulis untuk majalah InfoLINUX: Manajemen data pribadi dengan git.

Bagaimana strategi backup Anda?

The coming bloated Perl apps?

2011-02-07T08:06:00.001-08:00

A few weeks ago, I got annoyed by the fact that one of our command line applications was getting slower and slower to start up (the delay was getting more and more noticable), so I thought I'd do some refactoring, e.g. split large files into smaller ones and delay loading modules until necessary.

Sure enough, one of the main causes of the slow start up was preloading too many modules. Over the years I had been blindly sticking 'use' statements into our kitchen sink $Proj::Utils module, which was used by almost all scripts in the project. Loading $Proj::Utils alone pulled in over 60k lines from around 150 files!

After I split things up, it became clearer which modules are particularly heavy. This one stood out:

% time perl -MFile::ChangeNotify -e1
real   0m0.972s

% perl -MDevel::EndStats -e1
# Total number of module files loaded: 129
# Total number of modules lines loaded: 46385

So almost 130 files and a total of 45k+ lines just from loading File::ChangeNotify alone. 130 files just for a filesystem monitoring routine! Who would've thought that a filesystem monitor needs so many lines of program? Compare with, say, a recent HTTP client:

% perl -MHTTP::Tiny -e1
# Total number of module files loaded: 18
# Total number of modules lines loaded: 6089

I quickly switched to Linux::Inotify2 and things are much better now (but I might have to revisit this since we want to give the new Debian/kFreeBSD a Squeeze).

As I suspected (since the module is written by Dave Rolsky also), File::ChangeNotify utilizes Moose, which is not particularly lightweight either:

% time perl -MMoose -e1
real    0m0.712s

% perl -MDevel::EndStats -MMoose -e1
# Total number of module files loaded: 100
# Total number of modules lines loaded: 35760

Compare with:

% time perl -MMouse -e1
real    0m0.089s

% perl -MDevel::EndStats -MMouse -e1
# Total number of module files loaded: 20
# Total number of modules lines loaded: 6675

Come to think of it, running Dist::Zilla is also quite painfully slow these days. Just running "dzil foo" pulled in around 60k lines and took 1.7s! Of course, dzil is Moose-based.

While it is a good thing that Moose is getting more popular, it's a bit shameful to see that Ruby and Python scripts "get OO for free" while Moose scripts have to endure a 0.7s startup penalty. Mouse, Moo, Role::Basic come to the rescue but I wonder what would Ruby/Python programmers think (you have how many object systems?? Why do you people can never agree on one thing and TIMTOWTDI everything?)

Disclaimer: Number of lines includes all blanks/comment/POD/DATA/etc from all files loaded in %INC, actual SLOC is probably significantly less. Timing is done on a puny HP Mininote netbook (Atom N450 1.66GHz) which I'm currently stuck with in the past few weeks. With all due respects to all authors of modules mentioned. They all write fantastic, working code.

Outputting pretty data structure on console programs

2010-11-19T04:07:00.000-08:00

Our application has a command-line API interface for convenient access via shell/console. It used to output API result data in YAML:

# /c/sbin/spanel api --yaml File list --account steven --volume data --dir /public
--- 
dir: 
  atime: '1270675429'
  ctime: '1285916065'
  gid: 1023
  group: steven
  is_dir: 1
  mtime: '1285916065'
  perms: 493
  uid: 1012
  url: ~
  user: steven
entries: 
  - 
    atime: '1284665000'
    ctime: '1289609859'
    dir: /public
    gid: 1023
    group: steven
    icon_file: folder.gif
    inode: 1984908
    is_dir: 1
    is_link: 0
    mtime: '1289609859'
    name: git
    perms: 493
    reldir: ''
    size: 4096
    uid: 1012
    url: ~
    user: steven
  - 
    atime: '1270675424'
    ctime: '1285130140'
    dir: /public
    gid: 1023
    group: steven
    icon_file: text.gif
    inode: 1976727
    is_dir: 0
    is_link: 0
    mtime: '1155368486'
    name: .htaccess
    perms: 436
    reldir: ''
    size: 48
    uid: 1012
    url: ~
    user: steven
  - 
    atime: '1270675424'
    ctime: '1285130140'
    dir: /public
    gid: 1023
    group: steven
    icon_file: unknown.gif
    inode: 1976725
    is_dir: 0
    is_link: 0
    mtime: '1155368397'
    name: .htaccess~
    perms: 436
    reldir: ''
    size: 63
    uid: 1012
    url: ~
    user: steven
total_num_entries: 3
url: ~

YAML is relatively readable if you compare to JSON or (shudder) XML, but I soon grew tired of reading YAML for data that should be tabulated and better formatted for human consumption.

Thus, Data::Format::Pretty::Console. The idea is for me, a lazy programmer, to throw it data structure of various kind and it will display it nicely suitable for console viewing. The command line API interface now by default shows nicely formatted text for API result data (but still provides --yaml and --json option). Please bear with this blog post's misformatting and assume it's all pretty:

dir:
.---------------------.
| key    | value      |
+--------+------------+
| atime  | 1270675429 |
| ctime  | 1285916065 |
| gid    |       1023 |
| group  | steven     |
| is_dir |          1 |
| mtime  | 1285916065 |
| perms  |        493 |
| uid    |       1012 |
| url    |            |
| user   | steven     |
'--------+------------'

entries:
.----------------------------------------------------------------------------------------------------------------------------------------------------------------------.
| atime      | ctime      | dir     | gid  | group  | icon_file   | inode   | is_dir | is_link | mtime      | name       | perms | reldir | size | uid  | url | user   |
+------------+------------+---------+------+--------+-------------+---------+--------+---------+------------+------------+-------+--------+------+------+-----+--------+
| 1284665000 | 1289609859 | /public | 1023 | steven | folder.gif  | 1984908 |      1 |       0 | 1289609859 | git        |   493 |        | 4096 | 1012 |     | steven |
| 1270675424 | 1285130140 | /public | 1023 | steven | text.gif    | 1976727 |      0 |       0 | 1155368486 | .htaccess  |   436 |        |   48 | 1012 |     | steven |
| 1270675424 | 1285130140 | /public | 1023 | steven | unknown.gif | 1976725 |      0 |       0 | 1155368397 | .htaccess~ |   436 |        |   63 | 1012 |     | steven |
'------------+------------+---------+------+--------+-------------+---------+--------+---------+------------+------------+-------+--------+------+------+-----+--------'

total_num_entries:
3

url:

Comparison of INI-format modules on CPAN

2010-11-17T20:53:00.003-08:00

I'm not terribly happy with the state of Perl/CPAN support for the INI file format.

I have this requirement of modifying php.ini files programmatically from Perl like: set register_globals to On/Off, add/remove some extension (via the extension=foo lines), adding/removing some functions from the disabled_functions list, etc. So I would like to find a CPAN module that can just set/unset a parameter and leave formatting/comments alone as much as possible.

Turns out that among a dozen or so of INI modules on CPAN, a few of them do not do writes at all (e.g. Config::INI::Access or Config::Format::INI). And a few that do, write a la dump. That is, they just rewrite the whole INI file with the in-memory structure. All comments and formatting (even ordering, in some cases) are lost. Example: Config::INI::Writer and Tie::Cfg. And, last time I tried, I couldn't even install Config::IniHash from the CPAN client. Not good.

So I ended up with Config::IniFiles. And I needed to patch two features in before I could even read php.ini and write to it properly. This is an old module, which although still maintained, probably needs a rewrite or modernization. One reviewer in CPAN Ratings also wrote that this module fails in edge cases and the test suite is incomplete.

But, at least this module gets the work done. It tries to maintain comments, and even has a host of other features like delta, multiline, default section, etc. Most features seem to be rather exotic to me personally, but then none of the other INI modules on CPAN has the basic features that I needed.

(Short, grossly incomplete) comparison of Perl logging frameworks

2010-11-17T20:53:00.001-08:00

After doing this post on comparison of Perl serialization modules, I intended to continue with other comparisons, and even thought on setting up a wiki or creating/maintaining a section on the Official Perl 5 Wiki, which already has a Recommended Modules section, although there is not much comparison being written for each recommendation. (Btw, I just noticed a change of domain for the Wiki, from perlfoundation.org to socialtext.net).

But of course other tasks soon took precedence, so until the Wiki idea is realized, I thought I'll just continue posting on the blog as usual.

There are **seriously** too many Perl logging frameworks out there. As the Log4perl FAQ mentions, "Writing a logging module is like a rite of passage for every Perl programmer, just like writing your own templating system."

So I'm not going to pretend like I've evaluated even half of logging modules that are on CPAN. Instead I'm just going to include a few worth mentioning.

Log::Dispatch and Log::Log4perl. Two of the arguably most popular Perl logging frameworks are Log::Dispatch and Log::Log4perl. They are like the Moose of logging frameworks: mature, feature rich, flexible, has a lot of support/extra/3rd party modules, but... "slow". I quote the slow part because first of all, speed is not an issue for the majority of applications. And second of all, they are not relatively slow at all compared to other modules until they actually log stuff to output. For example, doing debug() on a warn level is around 1,5mils/sec with with Log4perl, and 3mils/sec with Log::Fast. But for actual logging, Log::Fast can be 10-45 times faster than these two.

Log::Any. For most people, Log::Dispatch and Log4perl should suffice. I personally haven't been unable to produce a case where I can't customize Log4perl they way I want. This shows the flexibility of the module. So the only thing left for flexibility is a thin wrapper where you might want to switch logging framework (kind of like Any::Moose for logging). There are a few of these on CPAN, but I prefer Log::Any (and I've also made a thin wrapper for *that*, Log::Any::App). RJBS also made one: Log::Dispatchoulli. You might be interested in using it if you are interested in using String::Flogger.

Performance-wise, as with Moose, there are other alternatives: Log::Fast, for one. There are also a few other minimalistic frameworks, but I do not recommend using them as many of them are not flexible at all. Unless your application is really performance-critical.

I've most probably left out a lot of possibly interesting alternatives.

Sometimes you don't want circular checking

2010-10-01T08:39:00.001-07:00

I use the nifty Data::Rmap to "flatten" DateTime objects into strings so they can be exported to JSON and handled outside Perl. But due to circular checking in Data::Rmap, this:

$ perl -MData::Rmap=:all -MData::Dump \
-e'$d = DateTime->now; $doc = [$d, $d]; 
rmap_ref { $_ = $_->ymd if UNIVERSAL::isa($_, "DateTime") } $doc; 
dd $doc'

produces something like this:

["2010-10-01", ...unconverted DateTime object...]

For now I work around this by defeating Data::Rmap's circular checking, though I wonder if there's a better way.

$ perl -MData::Rmap=:all -MData::Dump \ 
-e'$d = DateTime->now; $doc = [$d, $d]; 
rmap_ref { $_[0]{seen} = {}; $_ = $_->ymd if UNIVERSAL::isa($_, "DateTime") } $doc; 
dd $doc'

will correctly produce:

["2010-10-01", "2010-10-01"]

Yet another stupid mistake #1

2010-09-30T17:56:00.000-07:00

During a refactor of a data from array @foo to hash %foo, I used 'each' to iterate over the hash, but forgot to change the 'for' statement with 'while'. So I ended up with something like:

$ perl -MData::Dump -E'%a=(a=>1, b=>2);
for (my ($k, $v) = each %a) { $_ = "$k x"; dd {k=>$k, v=>$v, "\$_"=>$_} }'

And this is nasty because for(@ary) aliases $_ to each element in @ary, and in this case it modifies $k (quiz #1: and $v too, do you know why?) right under your nose! Thus the result are really messed up:

{ "\$_" => "a x", "k" => "a x", "v" => 1 }
{ "\$_" => "a x x", "k" => "a x", "v" => "a x x" }

Not to mention the loop stops after processing two items (quiz #2: do you know why?) But you might not realize that after you add some pairs to %a and wondering why they don't get processed.

The error message Perl gives is not really helpful, to say the least :)

Comparison of Perl serialization modules

2010-09-23T03:28:00.002-07:00

A while ago I needed a Perl data serializer with some requirements (supports circular references and Regexp objects out of the box, consistent/canonical output due output will be hashed). Here's my rundown of currently available data serialization Perl modules. A few notes: the labels fast/slow is relative to each other and are not the result of extensive benchmarking.

Data::Dumper. The grand-daddy of Perl serialization module. Produces Perl code with adjustable indentation level (default is lots of indentation, so output is verbose). Slow. Available in core since the early days of Perl 5 (5.005 to be exact). To unserialize, we need to do eval(), which might not be good for security. Usually the first choice for many Perl programmers when it comes to serialization and arguably the most popular module for that purpose.

Storable. Fast. Produces compact, portable binary output. Also available in core distribution. Does not support Regexp objects out of the box (though adding support for that requires only a few lines). Binary format used to change several times in the past without backward compatibility in the newer version of the module, giving people major PITA. Supposedly stabilized now.

YAML::XS. Fast. Verbose YAML output (currently doesn't seem to have option to output inline YAML). My personal experience in the past is sometimes this module behaved weirdly and died with a cryptic error, but I guess currently it's pretty stable.

There are other YAML implementations like YAML::Syck (also pretty speedy) and the old Pure-Perl YAML.pm and partial implementation YAML::Tiny. The last two might not be a good choice for general serialization needs.

Data::Dump. Very slow. Produces nicely indented Perl output. The strength of this module is in pretty output and flexibility in customizing the formatting process. Based on Data::Dump I've hacked two other specialized modules: Data::Dump::PHP for producing PHP code, and Data::Dump::Partial to produce compact and partial Perl output for logging.

XML::Dumper. Produces *very* verbose (as is the case with all XML) XML output. Slow. Aside from the XML format, I don't think there's a reason why you should choose this over the others.

JSON::XS. Fast, outputs pretty compact but still readable code, but does not support circular references or Regexp objects.

JSYNC. Slow, outputs JSON and in addition supports circular references but not yet Regexp objects.

FreezeThaw. Slow, produces compact output but not as compact as Storable. Does not support Regexp objects out of the box.

Apart from these there are many other choices too, but I personally don't think any of them is interesting enough to be a favorite. For example, last time I checked PHP::Serialization (and all the other PHP-related modules) does not support circular references. There's also, for example, Data::Pond: cute concept but of little practical use as it is even more limited than JSON format.

There are also numerous alternatives to Data::Dumper/Data::Dump, producing Perl or Perl-like code or indented formatted output, but they are either: not unserializable back to data structures (so, they are more of a formatting module instead of serialization module) or focus on pretty printing instead of speed. In general I think most Data::Dumper-like modules are slow when it comes to serializing data.

In conclusion, choice is good but I have not found my perfect general serialization module yet. My two favorites are Storable and YAML::XS. If JSYNC is faster and supports Regexp, or if YAML::XS or YAML::Syck can output inline/compact YAML, that would be as near to perfect as I would like it.

Hope this comparison is useful. Corrections and additions welcome.

Perl vs PHP (a bit of credit to PHP)

2010-09-23T03:28:00.001-07:00

Just read this blog post. Comments are disabled, so I thought I'd add a blog post.

There are endless ways we can sneer at PHP's deficiencies, but since 5.3 PHP already supports anonymous subroutines, via the function (args) { ... } syntax. So:

$longestLine = max(
    array_map(
        create_function('$a', 'return strlen($a);'), 
        explode("\n", $str)
    )
);

can be rewritten as:

$longestLine = max(
    array_map(
        function($a) { return strlen($a); }, 
        explode("\n", $str)
    )
);

though the example is not a good one since it might as well be:

$longestLine = max(
    array_map(
        'strlen',
        explode("\n", $str)
    )
);

Book review: Catalyst 5.8 The Perl MVC Framework

2010-09-01T21:03:00.000-07:00

Book information
Title: Catalyst 5.8 The Perl MVC Framework.
Subtitle: Build Scalable and extendable web applications using the Agile MVC framework.
Author: Antano Solar John.
Publisher: Packt Publishing.
Country: UK/India.
Year: 2010.

This book is a follow up to the 2007 Catalyst book by Jonathan Rockway (member of Catalyst core developer team). I have no idea how much of the content is changed between the two.

About the review(er)
This is a review on the electronic (PDF) edition of the book. I am a Perl developer and a CPAN author, but have not used Catalyst (or most other recent web frameworks, for that matter) before.

About Catalyst
So far I've managed to avoid learning about web frameworks and continue to create web applications the old way (CGI/CGI::Fast, direct DBI/SQL, a homemade simple templating language, and recently lots of jQuery and CSS play). Part of this is due to laziness, and part due to lack of need. I've never needed to create complex web applications in Perl. And the apparently heavy learning curve and complexities of Catalyst, Mojo, Dancer, etc just make me say don't bother.

But, thanks to this book, I find out that Catalyst project is not unlike a Perl CPAN module, with files/subdirectories like Makefile.PL, Changes, README, lib/, t/, etc. You can now even manage your project with Dist::Zilla (not mentioned in the book though as the plugin for this is new).

The good
This book is only about 200 (instead of 500+) pages long, which I appreciate. The preface is concise, and the explanation in the chapters are straightforward enough. The author uses clear and simple English sentences instead of long complex ones. The organization of topics into chapters is quite appropriate.

Missing topics
I didn't find any mention of Strawberry Perl, only ActivePerl. The examples are all using SQLite and no other databases. I wish AJAX and integration with one/more JavaScript frameworks like jQuery (and thus, CSS) is discussed more, as this is now very popular and common. But that will add significantly to the length of the book.

The first chapter on MVC also deserves some more extension.

There is no comparison whatsoever with any other Perl web frameworks or other non-Perl frameworks like Django and Rails.

I would've liked a chapter/subchapter on performance tuning and benchmarking (there is a 'Performance considerations' section in the Deployment chapter but that only covers the choice of webserver).

Plack/PSGI is not yet covered on this edition, which is a pity.

The rather bad
The author gives CPAN links to pages of specific release versions, e.g. http://search.cpan.org/~ash/DBIx-Class-0.08013/lib/DBIx/Class/Schema/
Versioned.pm which tends to break as new releases added and old releases removed from CPAN. ~~But this is understandable because currently CPAN only provides http://search.cpan.org/dist/DBIx-Class/ and not something like http://search.cpan.org/dist/DBIx-Class/current/pod/Foo/Bar.pm.~~ search.cpan.org does provide a more stable URL: http://search.cpan.org/dist/DBIx-Class/lib/DBIx/Class/Manual/FAQ.pod

The author also uses 2-space indent instead of 4, which I suspect is because he also uses Ruby/Rails.

The really ugly
The general editing of the book, and especially the code/output formatting, is the deal breaker here. I have not found another book that fares equally poorly in this regard.

The first paragraph of the preface already contains two very off-putting typos: "Frednic Brooks" (of Mythical Man-Month fame) and "MOOSE". Boxes drawn with ASCII characters which should align become wrapped and misaligned. When the long lines of code/output are wrapped, it is not clear which lines are wrapped and which are just new lines (some visual indicator should've added like a + or \ sign, line number, or striped background/lines).

There is a plain error in YAML syntax in p67, plain wrong MySQL configuration in p69.

Code formatting/editing is atrocious, with __PACKAGE__ sometimes becomes PACKAGE, or __Package__. Blank lines (which are significant for POD) are removed. And there is some garbage/random characters added in a few places. Totally unacceptable.

Verdict
Unfortunately I cannot recommend this book due to the utterly poor code formatting. I have no major problem with the content though.

Coding Style As A Failure Of Language Design?

2010-09-01T19:18:00.000-07:00

Read this older blog post the other day. Hilarious at best, creepy at worst.

Arbitrary limitations should not be added to a general-purpose programming language unless for a really good reason. Do you really want to code in a language that forces you to indent with 2 spaces, never cross 80-column line, or require/forbid whitespace here and there? And besides, is there any language (no matter how strict the syntax of which is) which do not have some sort of coding style?

Wishlist for a service framework and/or manager

2010-08-27T18:23:00.001-07:00

I maintain code for a few daemons/services written in Perl (most of them serve requests by forking/preforking). Reading post on Ubic, I started to feel that I am reinventing a lot of wheels. Currently I am doing these by writing my own code, as much as which I hope can be offloaded to CPAN in the future:

Autorestart when process size is becoming too big. We need to do this gracefully, meaning wait until there is no more clients being serviced, unless process size really gets too big in which case we need to restart immediately. Checking period can be configured.
Autorestart if script or modules change on disk. Also needs to be done gracefully. This is usually being recommended to be used only on development environment, but I use this too in production, for ease of deployment. But we need to check first (e.g. via "perl -c" or "eval + require" whether the new code from disk is okay.
Avoid duplicate instance (currently always using Proc::PID::File, but I'm open to better mechanism).
Limit clients concurrency. Sometimes this is simple (a single limit for all clients) and sometimes not so much (different limits for different IP/IP blocks/authenticated users/groups/etc).
Reap dead child processes and maintain a count of child processes.
Handle timed out clients. This is rather cumbersome with blocking I/O.
Write init script. This is the part I dislike the most, since there are tons of different OS flavors out there, and with more recent efforts like upstart, launchd, systemd, sooner or later I will certainly have to write different init scripts. I wish there is something equivalent to PSGI/Plack for general services, which can plug my code to whatever service manager might be out there.

Random Perl wishlists #1: uncapture modifier, require+import, backtick function

2010-08-25T11:58:00.001-07:00

Uncapture modifier. The new /r regexp substitution modifier in Perl 5.13.2 indicates that there might be hope for even more modifiers in the future. A possible modifier might be one that ignores all capture groups, which can solve this guy's problem.

require that can also import. I wonder why "require" doesn't also support importing like "use" does: use MODULE_NAME LIST... since it already support "require MODULE_NAME". This way, whenever I want to defer loading some modules, I can just replace "use" with "require" and put the statement under some "if" statement.

the backtick function.. Do you find yourself often having to do use a temporary variable like this, $cmd = "some longish and ".quote("complexly formed")."command"; before doing backtick `$cmd`? This is because unlike in PHP, we have system() but no backtick(). Most probably remnants from shell. There is Capture::Tiny for more general solution but of course it's a bit more cumbersome.

Startup overhead still matters

2010-07-28T17:19:00.000-07:00

We all love Moose, and the subject of this question could have been rephrased better, but why do I get the feeling that not many people write pure CGI or command-line scripts in Perl (that got executed many times) anymore? After all, didn't Perl begin as a tool for sysadmin and only in the mid 1990's got picked up as the darling of CGI/web programming?

There are still many cases where/reasons why Perl scripts need to be run many times (instead of persistently long running).

It's much more stable (I've often need to kill or ulimit or periodically restart a Perl process because after days it grows to 500+ MB).
Sometimes CGI is all you get (especially in shared hosting environment, which is related to point 1).
Sometimes you need to run the scripts for many users, and it's not feasible (e.g. memory-wise) to let them all run persistently.
Many old scripts are designed that way.
Some environments require them that way (e.g. scripts run in .qmail are run for every incoming mail, scripts run by tcpserver are started for every incoming connection, etc).

There used to be projects like PersistentPerl or SpeedyPerl to let us easily make a Perl script persistent by just changing the shebang line (e.g. from #!perl to #!pperl), but these projects are currently not actively developed, probably due to lack of demand (?), or becase this kind of deployment tends to cause subtle bugs (I did get bitten by this a couple of times in the past). You can't just convert a script that is designed/written to be a one-off run into a long-running one without expecting some bugs, anyway.

And the Perl compiler (B::*, *.pmc) is also now deprecated, probably because it does not give that many startup cost saving after all (the fact that Perl has phasers like BEGIN/CHECK blocks means it has to execute code as it compiles them anyway).

And thus we're stuck with having to accept the startup cost of parsing & compiling for every script run. That's why startup cost matters. On our servers awstats runs many thousand of times everyday (2000-5000 sites x 10+ HTML pages), and since it's a giant script (10k-ish line) it has a startup overhead of almost 1s. I really would like to shave this startup overhead as it is a significant part of server load.

Until today many of my scripts/programs are still deployed as one-off command line scripts. And that's why instead of Moose I use Mouse (or Any::Moose, to be exact) whenever I can. And so far I can.

Spot the error

2010-07-05T21:24:00.000-07:00

use Data::Rmap qw(:all);
use JSON;
use Data::Dump;
use Clone;
use boolean;

my $arg = from_json(q{{"1":true,"2":false}});
# convert JSON booleans to boolean's booleans
rmap_all { bless $_,"boolean" if ref($_) =~ /^JSON::(XS|PP)::Boolean$/ }, $arg;
dd $arg;

Hint: it's one character long.

In fact, this piece of code is full of Perl's traps (from Perl's lack of booleans obviously, to less obviously having to clone and rmap not working), it disgusts me.

Reduce in Perl

2010-06-29T20:34:00.001-07:00

Perl has grep/map/sort since probably forever (actually, sort() starts appearing since Perl 2.0). But even now, reduce is still not a builtin in Perl 5 (though available via List::Util), so doing reduce is probably not something that comes as naturally to Perl programmers. Meanwhile Ruby, JavaScript, and even PHP have their reduce operation builtin.

But then, reduce is "not really that useful" (you can just replace it with a simple for loop). So much that Python 3.0 now removes the function from the global namespace and reduces it (pun intended) to a mere member of functools. I guess reduce is really handy only if you are in a heavily functional language that lacks procedural basics.

This can be thought of as a testament to the level of language design skill that Larry has.

The rather funny thing is, in Perl 6, in addition to the reduce() List method there is also the reduce ([]) metaoperator as well.

Perl vs JavaScript

2010-06-23T03:39:00.001-07:00

Here are some notes I made while hacking on Language::Expr::Compiler::JS. Of course, there are a million differences between the two, but these focus mostly on operators and types. Hope it can be useful.

Double vs single quotes. There are practically no functional differences between double-quoted string and single-quoted one in JavaScript. In Perl, single quotes do not interpret escape sequences other than \\ and \', but in JavaScript both single- and double-quoted strings interpret the same set of escape sequences.
String escape sequences. Perl does not support JavaScript's \v (vertical tab), while JavaScript does not support \N{NAME} (named Unicode character), \e and \c[ (escape/control), \a (alarm bell).
Two undefs. JavaScript has two special nothingness/undefinedness: null and undefined. Strangely, null == undefined and they are equal to themselves, but they are not equal to any other values (including 0, '', false). The difference between the two is just this: undefined is not a keyword but a global variable (you can assign to it, but of course you shouldn't). If you want less confusion, just use null.
Behaviour of "+". Since JavaScript only has "+" (while Perl has "+" and "."), you should be aware that "+" in JavaScript coerces to strings when one of the operands is a string (e.g. 1 + "2" becomes "12"). In Perl, "+" coerces to numbers. So be careful when mixing numbers and strings.
You need to explicitly "return" value from a function. But there's a cute shortcut for one statement functions introduced in JavaScript 1.8: 'function (x) x*3' which is equivalent to 'function (x) { return x*3 }' so in this case you don't need the "return".
Boolean. JavaScript has real boolean. In Perl you can use 'boolean' from CPAN which gives you practically the same stuff.
JavaScript lacks a lot of Perl convenience operators, including <=> cmp, low-precedenced and/or/not, //, =~, ~~, qq(), qx(), qw(), m//, s///, **, etc.

All in all, I think JavaScript is quite nice and simple language with familiar syntax (at least compared to PHP). It also has lexical variables, anonymous functions, OO, etc.

JSYNC is brilliant!

2010-06-22T09:00:00.000-07:00

It's a brilliant idea: bank on JSON's popularity and more widespread implementations, add some of the important YAML features not present in JSON on top of it. The result is JSYNC, along with its preliminary CPAN module. (I'd probably picked a different name and choose something like "\" for prefix instead of ".", but hey, it's not my project :-)

A few months ago I was really desperate with the YAML situation in Perl. We have the largest number of YAML implementations, but none of them are good enough compared to Ruby's libsyck. I even contemplated converting all my YAML documents to JSON, but of course that plan was cancelled because JSON doesn't even support references nor objects.

Here's to hoping JSYNC will rocket to popularity soon enough. Ingy++.

Custom dumping in Data::Dump

2010-06-09T05:10:00.000-07:00

After blogging about my small patch to Data::Dump, I contacted Gisle Aas. He is quite responsive and finally comes up with a new release (1.16) of Data::Dump containing the cool new filter feature. My previous example after converted to use the new feature becomes:

$ perl -MData::Dump=dumpf -MDateTime -e'dumpf(DateTime->now, sub { my ($ctx, $oref) = @_; return unless $ctx->class eq "DateTime"; {dump=>qq([$oref])} })'
[2010-06-09T12:22:58]

This filter mechanism is quite generic and allows you to do some other tricks like switching classes, adding comments, and ignore/hide hash keys. The interface is also pleasant to work with, although starting with this release the "no OO interface" motto should perhaps be changed to "just a little bit of OO interface" :-)

Aren't we glad that stable and established modules like this are still actively maintained and getting new features.

Thanks, Gisle!

Menunggu statistik-statistik menarik dari sensus 2010

2010-06-02T09:12:00.000-07:00

Koran PR beberapa minggu lalu menulis, Sensus Penduduk 2010 selain bertujuan mencacah jiwa, juga ingin "mencari fakta-fakta unik." Salah satu (dan satu-satunya) contoh: mencari orang tertua. Dan benar memang, selama beberapa minggu ini sudah ada beberapa artikel yang meliput tentang nenek di kampung anu yang berumur 115th, lalu rekornya dipecahkan oleh yang berumur 120th, lalu 125th, dst. Terakhir kalau tidak salah ada yang lebih dari 140th (walaupun semua klaim usia ini berdasarkan ucapan semata, bahkan tidak ada akte lahir atau bukti tertulis lainnya).

Tentu saja, selain hanya mencari orang tertua, masih ada banyak sekali hal menarik yang bisa diekstrak dari data sensus ini. Misalnya, saya harapkan, BPS dapat menerbitkan adalah daftar nama depan dan nama belakang yang paling lazim, seperti yang dilakukan oleh badan sensus Amrik.

Saat menyusun versi awal modul Perl Locale::ID::GuessGender::FromFirstName, saya kesulitan mencari basis data nama yang bisa dipakai, karenanya saya mengambil 1000 nama depan terlazim dari database pelanggan kantor. Tentu saja, andaikan ada data yang lebih representatif, seperti dari sensus penduduk, tentu jauh lebih baik.

Seandainya diberi kesempatan, saya bersedia mengolah data mentahnya ;-)

Optimizing for $money?

2010-05-27T06:28:00.000-07:00

While some sites optimize for speed and bandwidth usage, others do the opposite. About a decade ago when Internet connection at the office (then at pathetic speed of 128-256kbps) was experiencing serious slowdown, I noticed that images and even flag icons from the Summer Olympic website are deliberately made totally uncacheable, by setting Expires header value to a past date. Apparently IBM is still doing the same trick for the Grand Slams sites.

$ cctrl() { perl -MLWP::UserAgent -E'$ua=LWP::UserAgent->new; $res=$ua->get($ARGV[0]); say $res->header("cache-control")' $1; }

RSS icon, only cacheable for several hours:

$ cctrl http://www.rolandgarros.com/images/nav/rgr_nv_00000g3.gif
max-age=15000

In fact all content photos are also cacheable for several hours only, despite already having unique URLs.

Yellow button, which surely won't change a lot (and has a fairly unique URL anyway), cacheable only to a little over 11 minutes!

$ cctrl http://www.rolandgarros.com/images/misc/rgr_ms_00000g2.gif
max-age=700

Compare to:

$ cctrl http://www.facebook.com/images/app_icons/newsfeed.gif
max-age=2592000

or even:

$ cctrl http://www.ibm.com/i/v16/t/ibm-logo.gif
max-age=2592000

Custom class dumping for Data::Dump

2010-05-21T01:59:00.000-07:00

I'm using DateTime objects a lot these days: anytime I get some date/time data from outside of Perl, the first thing I do is convert them to DateTime object, to avoid calculation/formatting hassle ahead.

However, the dumps are not pretty.

% perl -MDateTime -MData::Dump -e'dd [DateTime->now]'

[

bless({

formatter => undef,

local_c => {

day => 21,

day_of_quarter => 51,

day_of_week => 5,

day_of_year => 141,

hour => 8,

minute => 55,

month => 5,

quarter => 2,

second => 36,

year => 2010,

},

local_rd_days => 733913,

local_rd_secs => 32136,

locale => bless({

"default_date_format_length" => "medium",

"default_time_format_length" => "medium",

en_complete_name => "English United States",

en_language => "English",

en_territory => "United States",

id => "en_US",

native_complete_name => "English United States",

native_language => "English",

native_territory => "United States",

}, "DateTime::Locale::en_US"),

offset_modifier => 0,

rd_nanosecs => 0,

tz => bless({ name => "UTC" }, "DateTime::TimeZone::UTC"),

utc_rd_days => 733913,

utc_rd_secs => 32136,

utc_year => 2011,

}, "DateTime"),

]

It gets worse when you have some records each with DateTime object in it.

That's why I added a couple of mechanisms to allow us to custom a class' dump.

$ perl -Ilib -MDateTime -MData::Dump -e'$Data::Dump::CUSTOM_CLASS_DUMPERS{"DateTime"} = sub { "$_[0]" }; dd [DateTime->now]'

[2010-05-21T08:57:45]

or:

$ perl -Ilib -MDateTime -MData::Dump -e'package DateTime; sub dump { "$_[0]" }; package main; dd [DateTime->now]'

[2010-05-21T08:58:09]

I know some other dumper in CPAN probably has this ability, but I like Data::Dump's output.

If you want to take a look at a couple of small patches to Data::Dump: http://github.com/sharyanto/data-dump

I've also contacted Gisle Aas to ask what he thinks of it.

On RJBS's automatic version numbering scheme

2010-05-13T03:49:00.001-07:00

Everytime I browse through CPAN recent uploads, and see versions of modules with RJBS's automatic numbering scheme, like 2.100920 or 1.091200 I tend to read it as 2.(noise) and 1.(more noise).

The problem is that it doesn't look like a date at all (is there any country or region using day of year in their date??). I've never been bothered enough with this though, as I don't use this scheme myself, but have always had a suspicion that this obfuscation is deliberate for some deep reason.

Turns out that it's just a matter of space saving and floating point issue. I'm not convinced though, is x.100513n (YYMMDD, 6 digits + 1 digit serial = 7 digits) really that much longer than x.10133n (YYDDD, 5 digits + 1 digit serial = 6 digits)? Is there a modern platform where Perl's numbers are represented with 32-bit single precision floating point (only 7 decimal digit precision) where it will present a problem when n becomes 2 digit using YYMMDD scheme?

Based on past experiences, since it is unlikely that I will do more than 20 releases in one month (usually even only once or twice a month or less frequently), if I were to adopt a date-based automatic versioning policy, perhaps I'll pick x.YYMMn where n is omitted for the first release, and then 1..9, and then 91..99 (and then 991..999 and so on). This way, most releases have the shortest number of digits. I don't "incur cost" for the first few releases (which anyway will be all there is, most of the time).

1.1005
1.10051
1.10052
...
1.10059
1.100591
1.100592
...
1.100599
1.1005991
...

In fact, I bet most modules have only a few releases per year. So how about this scheme, x.YYn:

1.10 - first release of the year
1.101 - second
1.102 - third
...
1.109 - tenth
1.1091 - eleventh
1.1092 - 12th
...
1.1099 - 19th
1.10991 - 20th

Or how about x.Dn (releases per decade) or even x.Cn (releases per century)? :-)

My brain prefers that I don't use long version numbers. Except when the version number is long because of some date (e.g. to indicate freshness of release). But why torture ourselves with a date that we need several seconds to parse in our head?

So I'll stick with 0.01, 0.02, 0.03, ... for now.

Menebak gender orang Indonesia berdasarkan nama depan

2010-05-13T02:57:00.000-07:00

Sesuai janji di posting blog beberapa bulan lalu, hari ini saya merilis Locale-ID-GuessGender-FromFirstName. Nama modulnya jadi panjang ya? :-p

Sebab ke depannya, seiring dengan modul pelengkap yang direncanakan, Locale-ID-ParseName-Person, kita juga bisa menebak gender seseorang dari atribut nama lainnya, misalnya dari sapaan (Bapak/Ibu/Bung/Mbak), dari gelar keagamaan (H/Hj), dari pola nama kedaerahan (mis: I Ketut/Ni Ayu), dll.

Rilis pertama ini akurasi dan kelengkapannya belum bisa diandalkan, tapi sudah bisa dicoba-coba. Saya sudah menambahkan sekitar 1000 nama-nama umum dari database klien kantor (soalnya kesulitan mencari database yang lebih bagus, tidak seperti di Amrik yang bisa mengambil data dari biro sensus di sana). Algoritma heuristik (sangat) sederhana juga sudah ditambahkan, beserta dengan algoritma untuk mencari dari Google.

Ada yang punya waktu luang membuat skrip CGI sederhana, atau aplikasi Facebook, untuk interface web modul ini? Sekalian mengumpulkan lebih banyak data dan koreksi. Saya sih pengen aja, cuma males :p

perlmv: Renaming files with Perl code

2010-05-12T11:18:00.000-07:00

perlmv is a script which I have personally been using all the time for years, but has only been uploaded to CPAN today. The concept is very simple, to rename files by manipulating $_ in specified Perl code. For example, to rename all .avi files to lowercase,

$ perlmv -de '$_=lc' *.avi

The -d option is for dry-run, so that we can test our code before actually renaming the files. If you are sure that the code is correct, remove the -d (or replace it with -v, for verbose).

perlmv can also save your code into scriptlets (files in ~/.perlmv/scriptlets/), so if you do:

$ perlmv -e 's/\.(jpe?g|jpe)$/.jpg/i' -W normalize-jpeg

You can later do this:

$ perlmv -v normalize-jpeg *.JPG *.jpeg

In fact, perlmv comes with several scriptlets you can use (more useful scriptlets will be added in the future):

$ perlmv -L
lc
pinyin
remove-common-prefix
remove-common-suffix
uc
with-numbers

Let me know if you have tried out the script.

So is wantarray() bad or not?

2010-05-05T13:00:00.000-07:00

The style of returning different things in list vs scalar context has been debated for a long time (for a particular example, this thread in Perlmonks).

A few months ago I made a decision that all API functions in one of my projects should return this:

return wantarray ? ($status, $errmsg, $result) : $result;

That is, we can skip error checking when we don't want to do it.

Now, in the spirit of Fatal and autodie, I am changing the above to:

return wantarray ? ($status, $errmsg, $result) : 
    do { die "$status - $errmsg" unless $status == SUCCESS; $result };

But somehow I can still see myself and others tripping over this in the future, as I have, several times so far. It's bad enough that for each API function one already has to remember the arguments and their types, and one kind of return and its type.

Maybe I should just bite the bullet and admit the misadventure into wantarray(), and that context-sensitive return should be left to @foo, localtime(), and a few other classical Perl 5 builtins that have been ingrained in every Perl programmer's mind.

Module Wishlist: magical loading of module

2010-04-28T10:43:00.000-07:00

This Module Wishlist series is meant to surprise me with the power of CPAN. I wish about or dream up some module without first checking on CPAN, and hopefully can be delighted when what I want is already there.

Don't you hate it when you have to do:

$ perl -MSome::Really::Long::Module -e'print Some::Really::Long::Module->foo'

The goal is to be able to say something very close to:

$ perl -e'print Some::Really::Long::Module->foo'

and my module is loaded automaticaly.

To die or to croak, that is the question

2010-04-28T10:21:00.000-07:00

Lately I've been tempted to use croak() instead of die(). Somehow it seems more considerate to users. But finally in the end I'm sticking with die(). In fact, I think the Carp module should be, well, croaked.

The reasons:

1. Even though Carp has been included in Perl 5 since forever (Module::CoreList tells me: "5"), carp(), croak(), cluck(), and confess() are still not builtins, which means I still need an extra "use Carp".

2. Too many keywords! Most other languages only have "throw" or "raise".

3. Names are too weird! I understand the difficulty of coming up with a concise set of names that are similar but slightly different. But requiring these weird names might also indicate that there is something fishy about the concept itself.

4. The choice of showing a stack trace or not should not be in the individual functions. That burdens the programmer with too much thinking.

5. Even with Carp qw(verbose), what's to be done with codes that still die() and warn()? (But luckily there's Carp::Always.)

6. Showing stack trace should not be this difficult. I still think there should be a command-line switch for Carp::Always (or alias it to 'oan' :-)

7. Programmers (module writers) make mistake. They should not skip a call frame.

In short, I think Carp makes things a little bit too complicated. But what's Perl without complication? :-)

You know you're a Perl programmer when...

2010-04-28T09:26:00.000-07:00

You know you're a Perl programmer (or a CPAN author) when...

When you're thinking of packaging every piece of code as a CPAN module.

A couple of days ago I need a subroutine that takes a nested data structure (e.g. {vol1 => {a=>{b=>{c=>10}}}, vol2 => {a2=>{b2=>{c2=>20}}}}), a Unix-like path string (e.g. "vol1:/a/b/c") and return the branch/leaf node of the data structure according to the specified path (in this example, 10).

After browsing CPAN and a few minutes of reading the POD of some modules and not finding exactly what I wanted [*], that subroutine idea quickly transformed into an idea of a full-fledged CPAN module. The next day I uploaded Data::Filesystem to CPAN, which is actually yet another Data::Walker- / Data::Path- / Data::DPath-like module.

Turns out that I really don't need that module (yet, maybe someday). What I needed is just a simple Perl subroutine, and nothing more, because I will need to create a Javascript and PHP equivalent for it. Porting a whole module is not something I even want to do.

I wonder just how many CPAN authors that (do not) start their modules this way: overengineering of a small problem after not finding exactly what they want in CPAN.

[*] Btw, not finding what you want in one of the millions of CPAN modules has got to be one of the saddest thing in the universe. :-)

Yada Yada, Buat apa ?

2010-04-18T09:08:00.000-07:00

Perl 5.12.0 baru saja keluar dan salah satu yang baru yaitu Yada Yada Operator.

Saya hanya bingung kapan atau situasi seperti apa yang membuat yada yada operator bisa (dan bagus) untuk digunakan ?,
Selintas saya teringat seperti pending nya RSpec.

Tip sprintf()

2010-04-15T19:28:00.000-07:00

Salah satu fitur sprintf() (dan printf()) yang agak jarang diketahui/dipakai orang adalah bahwa sprintf() mendukung spesifikasi posisi argumen di dalam string formatnya, menggunakan prefiks NOMOR + "$":

% perl -E'say sprintf(q[%d %d %d], 1, 2, 3)'
1 2 3

% perl -E'say sprintf(q[%2$d %3$d %1$d], 1, 2, 3)'
2 3 1

Sayangnya, sprintf() tidak mendukung binding berdasarkan nama, seperti di Python:

print 'This {food} is {adjective}.'.format(adjective='absolutely horrible', food='spam')
This spam is absolutely horrible.

Kadang-kadang binding berdasarkan nama lebih nyaman, karena jika terjadi penambahan/pengurangan argumen, kita tidak harus menggeser-geser posisi lagi. Beberapa aplikasi tertentu seperti translasi juga kadang bisa dibuat lebih enak interfacenya seandainya menggunakan binding berbasis nama.

Jadi, bagaimana solusinya di Perl? Bisa dengan modul seperti String::Formatter, atau membuat sendiri :-) (seperti yang saya lakukan baru-baru ini di Data::Schema):

# $extra = {mverb => "harus"}; # mverb juga bisa 'sebaiknya'
# $args = [1, 10];
print stringf("Data %(mverb)s di antara %(0)d sampai %(1)d", $args, $extra);
Data harus lebih kecil di antara 1 sampai 10.

Fungsi stringf() mencoba mencari nilai binding di argumen kedua dst. Argumen dapat berupa hashref maupun arrayref, jadi saya bisa menggunakan binding berdasarkan posisi maupun nama. Nyaman :-)

Data::Dumper (Unfortunately, Part 2)

2010-04-14T11:26:00.000-07:00

One of the first things a Perl programmer will notice when learning about Data::Dumper is: how weird and "inside out" the OO interface is. This is, I think, another unfortunate accident in the Perl history, as Data::Dumper, being the first of such modules, gets into the core in early Perl 5 and remains popular up until this day. But the interface and default settings apparently annoy a lot of people so much that alternatives and wrappers like Data::Dump, Data::Dumper::Again, Data::Dumper::Concise, among others, sprung up to life.

A loose analogy would be CVS which was popular for (too long) a time, and following it the explosion of alternative version control systems. Eventually after this phase a winner will emerge or dominate. In the version control system case it appears to be git. And in the Perl case I think it will be a builtin perl() method/function, like in Perl 6. Probably in 5.14? 5.16? 5.18? Don't you think it's about time Perl can "natively" dump its own structures in Perl, just like Python, Ruby, PHP, etc have been able to for a long time?

(Btw, lest anyone thinks otherwise: I do love DD. It has lots of options and has served its purpose well over the years.)

List::Util, List::MoreUtils, Util::Any (Unfortunately, Part 1)

2010-04-14T10:52:00.000-07:00

The dichotomy of List::Util and List::MoreUtils is one of the unfortunate annoyances in Perl. One is without s, one is with s. Which function belongs to which? And no, you can't simply say, "f*ck it, just import everything!" as List::Util doesn't provide the usual ":all" import tag (RT).

Some thoughts (from someone who is largely ignorant on the history of both modules), all IMO:

1. Since List::Util is basically a convenient library, convenience should've been its main design goal. It should've been inclusive enough. The decision to deny the inclusion of any(), all(), none() just because they are too "trivial" to implement in one line of Perl was a bit strange, since max(), min(), etc are also trivial to implement in Perl.

2. List::MoreUtils should've included all the functionalities of List::Util, so one can use it *instead of* List::Util.

But hey, what happened happened.

Btw, we also have Perl 6's junction taking the "all", "any", "none" keyword.

And we'll see whether solutions like Util::Any will catch on, as it's another syntax to learn, another module to download and install. As with many annoyances, they are actually not that big of a deal. One can just spend a few seconds looking up the documentation to find the functions he/she wants, and after about tens of uses should remember which ones are in which.

Just when I'm warming to 5.10, comes 5.12!

2010-04-14T09:27:00.000-07:00

Perl is far from dead/dying nowadays, with 5.12 being released recently, and the yearly timed-based release plan and all. In fact, just after I start to be comfortable using some of the 5.10 niceties, here comes a whole new version with even more niceties waiting to be explored!

Features in 5.10 I'm using regularly.

Defined-or (if there's only one feature I can have in 5.10, I pick this one).

State variables (love it!).

Features in 5.10 I'm starting to use.

-E switch (but my reflex still says -e all the time).

Recursive pattern in regex (e.g., via Regexp::Grammars).

say() (maybe if I say it often enough I'll start to say say more).

Features in 5.10 I rarely/ever touch.

Smart match (I know it's a godsend, but strangely I never feel the need for it so far).

given/when (I'm sticking with if/elsif/else, especially since given/when cannot be used as an expression yet).

Named capture in regex (yeah, old habits die hard).

5.10 and 5.12. IMO, 5.10 contains more "significant" visible new features for end users (i.e. Perl programmers), especially in the area of new syntax addition. It's 5 years in the making and delivers many features borrowed from Perl 6. But that is not meant to belittle 5.12 which also packs some major goodies, especially pluggable keywords. This one promises to usher us into a world of new syntaxes and mini languages, though it also confirms Perl as being a language that is "impossible to parse" and it surely will pose a challenge/headache for PPI and syntax highlight/Intellisense writers. I look forward to something like a better embedded SQL and templates (using pluggable keywords instead of treating everything as strings all the time).

Data::Dump::PHP

2010-04-07T03:50:00.000-07:00

I actually don't believe there isn't something like this in CPAN yet. Well, actually there is PHP::Var, but it has bugs, doesn't handle scalars, and doesn't do recursive structure. But then I am equally surprised to be able to hack Data::Dump::PHP in just a couple of hours, by blatanly copying from Gisle Aas' Data::Dump and just modifying only what's necessary.

And another note, PHP's var_export() currently can't dump recursive structures, which Data::Dump::PHP can.

Bahasa yang buruk vs programer yang buruk

2010-04-01T12:01:00.000-07:00

Apakah kita seharusnya menangisi kenyataan bahwa Perl tidak lagi menjadi bahasa primadona untuk Web? Rasanya sulit bisa mengejar popularitas PHP, atau Ruby dan Python saat ini di domain pemrograman Web. Dulu pertengahan 1990-an Perl dipilih karena belum banyak bahasa lain yang tersedia default di server-server Unix. Alternatifnya saat itu hanyalah C, shell, atau Tcl. Kini persaingan amat ketat/banyak. Perl termasuk salah satu yang lebih sulit/lama dipelajari dan selain itu memiliki imej "tua" (padahal umurnya gak beda jauh dengan rekan2nya, hanya sekitar 1-2 tahun dengan Python dan hanya 5 tahunan lebih dengan Ruby; semua bahasa2x ini sudah hampir atau lebih tua dari 20 tahun).

Di satu sisi kehilangan pamor/momentum/posisi jawara/apalah tentu gak mengenakkan. Tapi di sisi lain, ada manfaatnya. Para "programer" web yang cenderung lebih banyak menghasilkan kode-kode yang buruk jadi meninggalkan Perl. Saya ingat dulu saat Perl popular, betapa komunitas Perl dianggap elitist, eksklusivist, sombong, angkuh, tidak ramah terhadap pemula. Dan bahasa-bahasa lain mulai mendapat tempat di hati khalayak ramai karena menawarkan komunitas yang lebih ramah pemula. (Belakangan, komunitas Perl pun mulai melunak dan menginisiasi effort2x untuk lebih merangkul pemula, seperti membuat milis beginners@, dsb. Tapi mungkin sudah terlambat).

Salah satu alasan mengapa komunitas Perl "benci" pemula adalah: karena begitu banyak niubi yang jadi programer karbitan/jadi2xan berbondong2x mempelajari Perl, kadang setengah2x (atau seperempat2x!), dan selalu mencampurkan konsep Perl dan CGI. Selalu menulis Perl dengan PERL. Selalu mendecode parameter CGI sendiri (karena mengikuti instruksi buku2x Perl tak bermutu), padahal di Perl 4 pun sudah ada cgi-lib.pl. Selalu menanyakan persoalan sepele yang sudah sejak lama ada di FAQ. Selalu mengkopi paste kode dan menulis skrip yang begitu hancur2xan jeleknya.

Sekarang rupanya mayoritas dari mereka sudah berpindah ke PHP. Sebagai pengurus server hosting Linux, sudah sering saya harus mengecek aplikasi PHP milik klien hosting yang bermasalah. Dan tiap kali saya mengintip kode sumbernya, kadang saya tersenyum pahit, kadang mengelus dada, kadang geleng2x kepala. Program2x jelek dan berantakan ternyata tidak pernah punah. Dulu di Perl, sekarang di PHP. Kalau dulu Matt's Script Archive jadi biang hole, kini ada phpBB, WordPress, Joomla sebagai penerusnya.

Apakah Rails atau Django akan kebal dari para programer buruk? Don't underestimate the power of stupid people :)

Maaf, saya tidak bermaksud berarogan ria di sini. Ada berbagai macam alasan mengapa seseorang bisa disebut programer buruk, seringkali itu bukan karena dia bodoh. Deadline yang terlalu singkat menyebabkan harus kopi paste kode. Pengetahuan yang minim karena pengalaman kurang menyebabkan desain yang naif. Peran bahasa untuk melakukan "nudging" dan manajemen insentif untuk memperbaiki kebiasaan-kebiasaan yang salah memang berpengaruh, tapi selalu ada ruang untuk beginner's mistakes. Dan selalu harus ada refactoring. Programer yang buruk tidak pernah melakukan refactoring.

Jadi, bersyukurlah karena nanti 10 tahun lagi para generasi programer baru tidak lagi banyak mengutuk Perl karena harus memaintain kode lama CGI yang sudah membusuk. Melainkan mengutuk PHP karena diwarisi segunung kode spageti busuk bercampur HTML. Atau mengutuk Ruby karena peninggalan kode-kode busuk Rails dengan desain objek yang terbalik-balik dan pattern-pattern salah kaprah.

Programer yang buruk selalu ada sepanjang masa. Bahasa yang saat itu banyak dipakai yang akan jadi kambing hitamnya. :-)

Kompetisi, kompetisi

2010-03-31T10:47:00.000-07:00

(Berhubung lagi gak mut dan ada kerjaan2x lain, kelewat seminggu deh posting Iron Man. ~~Menurut aturan, seharusnya status ane jadi kertas lagi :-(~~ Ternyata tidak, karena ada posting dari zak. Thanks zak :) )

Tadi barusan baca postingan Sawyer X tentang postingan Adam Kennedy yang berencana mengadu Dancer dan Mojolicious. Exciting banget!

IMO, dari dulu seharusnya komunitas Perl menggelar inisiatif2x kompetisi seperti ini.

Biasanya yang sering terjadi di CPAN adalah kelahiran modul demi modul baru yang merupakan eksperimen atau proyek pribadi milik orang2x berbeda. Kurang suka dengan sebuah modul? Buat saja tandingannya, alternatifnya, versi Anda sendiri. Lihat ada berapa belas framework web di Perl sejak Catalyst, ada berapa puluh modul untuk validasi data (ane juga nyumbang satu nih hehe), dan mungkin seratusan modul konfigurasi.

Tapi yang membuat pengguna bingung, bagaimana memilih modul terbaik? Terpopular? Tercepat? Dsb. Kurang ada perbandingan2x atau benchmark2x atau kontes. Komunitas Perl seolah gak suka dengan kompetisi. Bahkan counter download aja dari dulu gak jadi dibuat2x.

Kita perlu ingat bahwa kompetisi mendorong inovasi. Kurang inovasi berarti mati.

Ayo kita berkompetisi, ayo kita saling beradu secara positif.

-- sh (Yang sedang menggodok versi Data::Schema berikutnya untuk menjadi DFV-killer :-)

Autovivification

2010-03-26T20:12:00.000-07:00

Berdasarkan pertanyaan dari beberapa orang yang agak kebingungan dengan istilah ~~autovivication~~autovivification, mungkin secara singkat saya akan langsung memberikan contoh tentang apa itu yang disebut dengan ~~autovivication~~.autovivification
Ini dia ~~autovivication~~ autovivification:

my $hash;
$hash->{aaa}{bbb}{ccc} = 'sebuah nilai';

Ketika dideklarasikan, $hash tidak mempunyai key dan value, tetapi kemudian kita memberikan suatu value pada key yang 'undefined'.

Ya sesingkat itulah definisi dari ~~autovivication~~.autovivification

Modules AND applications

2010-03-17T08:00:00.001-07:00

"Why choose?" -- Fatima Dinssa

In Modules vs Applications, Sawyer X noted that one of the "issues" (emphasis mine) of Perl people is the tendency to write modules instead of applications. That CPAN is great but due to the lack of end user programs there is no WOW factor. He suggested that we write programs/applications that everyone can use to attract more people to Perl.

While I agree with the last suggestion, I don't agree with the preference to modularize everything as an issue. As someone who wrote a program years ago that is comprised of many separate scripts and duplicated code (even in different languages, just for the fun of it) and still have to maintain it today, I'd say that not putting as much code as possible into reusable modules is a mistake.

I'd instead suggest that we still write modules (which is what made CPAN great anyway), but try to *also* accompany each distribution with a demo app (preferably in the App:: namespace).

I myself will try to do that from now on.

Log::Any::App (2)

2010-03-17T07:15:00.000-07:00

Following up on my previous post, I've just uploaded Log::Any::App to CPAN.

Do you write or use modules that use Log::Any? Or, do you want to use Log::Any conveniently in an application? Now you can just do this:

% perl -MLog::Any::App -MOtherModuleThatUsesLogAny -e ...

and the logs will be displayed to screen. The default level is WARN, but if you want to debug things:

% DEBUG=1 perl -MLog::Any::App ...

or if you want to quiet things down:

% LOGLEVEL=error perl -MLog::Any::App ...

If you put 'use Log::Any::App' in your script, when run it will by default log to file too (~/prog.log or /var/log/prog.log). It can even automatically log to syslog.

Zero-conf. No more long incantation.

Please tell me what you think.

Choosing test names

2010-03-11T23:02:00.000-08:00

Which test names do you prefer?

"tong() method can connect to database"
"tong() method can disconnect from database"
"sha() method can delete an existing file"
"sha() method fails when deleting a non-existing file"

or:

"tong 1"
"tong 2"
"sha 1"
"sha 2"

They are both rather extreme, but if I had to choose, I would still rather go with the shorter ones. I tend to treat test names more like unique IDs, and when things go wrong I just look up the actual test code.

I wouldn't mind verbose test names though if they can somehow be automatically generated (a future Google Translate project, perhaps?) from code, because they are just repeating what the code says.

To repeat myself, it's the DRY principle.

On reading code

2010-03-03T04:26:00.000-08:00

Last week I started using github, forked a project, and read some of miyagawa's beautiful code. Later on the weekend, I imitated a particular style I found from his code to improve my own code.

And then I realized: during the course of many years as a programmer, I really really seldom read other people's code, especially real-world code. Sure, I do whenever I have to patch something. But other than that, practically never.

Other than code, I do read an awful lot: books, magazines, mailing lists, forums, blogs, web pages. I have never doubted the benefits of reading for improving knowledge and understanding, so why haven't I read more code? A couple of reasons I can think of:

1. We programmers are not paid to read code. We are paid to write programs, to churn out lines upon lines of code. Heck, we're not even paid to write good code, we're paid to get the job done.

2. Reading code is hard. It can really drain your brain. Imagine a recipe book where the flow of instructions is not linear (jump to line X, jump to page Y and then return here), where a misplaced character can destroy the whole recipe, and where everything is so interdependent and interlinked that you need to read everything twice first before you can begin to understand it.

3. Reading code is boring. Can you really curl up with a good program the way you can with an exciting novel?

4. Reading code is not necessary. A good reusable piece of code need not be read anyway, all you need is its API documentation. Programmers are hired regardless of their ability to read real-world code, there are no interview tests in reading code other than a few or at most a dozen lines of it. Heck, programmers are even hired despite their inability to write any code at all.

After experiencing that one session of code reading can do some wonders, I'll make it a program to read more (good) other people's code.

{Logging, Messaging, Notification, Auditing, ...} frameworks

2010-02-26T19:56:00.000-08:00

They're all the same, in many respects.

In my module code, I want to be as flexible as possible. I want to be as detached from implementation details as possible.

I just want to generate a log message (or a notification, or an audit entry). I don't want to care where it ends up, how it is sent, how it is filtered/categorized, who the recipient(s) is/are, what medium(s) is/are used, etc. Let whoever uses the module configure it all.

And I don't want to reinvent the wheel, I want to use an existing framework. A logging framework seems to be a sensible choice. Let's use Log::Any for example. This is a snippet from a module for a file-manager-type web app:

my (@success, @failed);
for (@files) {
    unlink $_;
    if ($!) { push @failed, $_ } else { push @success, $_ }
}
$log->info("Done deleting files. Files deleted: %s. Files not deleted: %s", \@success, @failed);

The message can end up:

in a system-wide log-file;
discarded (if configured level of output is less than INFO);
in a per-user log file;
in a notification email to user (should the user configure it);
in a notification email to sysadmin (should the admin configure it);
into audit table in database (if the application is using a database);
in the console (if this module is also used in a command-line based app);
as a desktop notification (if this module is also used in a desktop app);
in an internal web-based forum;

However, some of those outputs usually need some additional metadata. In email and desktop notification we usually need separate subject and body. In the audit table in database we need to fill in who (the logged in user). In web applications we usually need to log the remote IP address (or even User-Agent string). In desktop notification or forum sometimes we would like to be able to update the message instead of creating a new one.

Instead of in the logging output module (the appender and formatter, in Log4perl-speak), sometimes I *do* care and need to specify this in my module.

Logging APIs usually do not allow us to do so. Output/appender modules are only given the message as a string.

So I intend to cheat by embedding a JSON-/YAML-encoded metadata in front of the message, e.g.:

$log->info("\x00{subject: 'Progress of copying $foo to $bar', id: xasd8f7d}\x00Copying $foo to $bar. 0%");
for (1..100) {
    $log->info("\x00{id: xasd8f7d}\x00$_%");
    sleep 1;
}

The presence of \x00 signify that until the next \x00 there will be additional metadata in the form of YAML (or JSON). This way, my desktop notification and web-based forum output module knows that the 2nd-101st message is an update and thus can adjust accordingly.

It should, and I don't have to reinvent any framework or wrapper. But it feels so hackish and ugly.

The problems with the older CPAN clients

2010-02-25T23:13:00.000-08:00

Thank God for cpanminus. Now that I'm free from having to use them, allow me to rant, no, bitch about them.

1. Bad defaults. Some default values might make sense 10-15 years ago, but not so much nowadays. For example, I'd argue that "follow" should now be default. See #2.

2. Too developer-oriented. For example, I believe "notest" should be on by default. This is compounded by the fact that installing Perl modules is so damn-slow already. See #3.

3. Too slow. Startup takes around 10-30 seconds or more. Installing Moose usually takes minutes (but with cpanminus, it only takes about 1 minute with --notest on my PC). Autocomplete takes one to a couple of seconds.

4. Too interactive, too verbose. The older clients are getting better but not quiet enough, cpanminus is such a breathe of fresh air.

5. Too bloated (which is the reason why cpanminus was developed in the first place).

The older CPAN clients are an embarassment if we compare it to "apt-get", "yum", "urpmi", which are way faster, way quieter, way less interactive. There's no reason why a CPAN client cannot be like those. And fortunately cpanminus proves it.

The shorter path to deployment heaven

2010-02-25T21:14:00.000-08:00

Now that there is cpanminus, I've modified the handy little module CPAN::AutoINC to prefer cpanminus over CPAN.

So now when users download and run my programs/scripts for the first time, instead of failing with the dreaded message:

Can't locate Foo/Bar.pm in @INC (@INC contains: [a dozen or more of paths....]).
BEGIN failed--compilation aborted at /some/path line 123

which users or even Perl novices have no clue on how to fix, the program will instead automatically download every necessary CPAN modules into the user's home directory and runs out-of-the-box on the first try!

$ download-bca
Fetching http://search.cpan.org/CPAN/authors/id/A/AD/ADAMK/File-HomeDir-0.89.tar.gz
Building File-HomeDir-0.89 for File::HomeDir...                                    
File::HomeDir installed successfully.                                              
Fetching http://search.cpan.org/CPAN/authors/id/D/DR/DROLSKY/File-Slurp-9999.13.tar.gz
Building File-Slurp-9999.13 for File::Slurp...                                        
File::Slurp installed successfully.                                                   
Fetching http://search.cpan.org/CPAN/authors/id/S/SH/SHARYANTO/Finance-Bank-ID-BCA-0.07.tar.gz
==> Found dependencies: Pod::Coverage, DateTime, Test::Pod::Coverage, Log::Any, Any::Moose, Mouse
Fetching http://search.cpan.org/CPAN/authors/id/R/RC/RCLAMP/Pod-Coverage-0.20.tar.gz             
==> Found dependencies: Devel::Symdump                                                           
Fetching http://search.cpan.org/CPAN/authors/id/A/AN/ANDK/Devel-Symdump-2.08.tar.gz              
Building Devel-Symdump-2.08 for Devel::Symdump...                                                
Devel::Symdump installed successfully.                                                           
Building Pod-Coverage-0.20 for Pod::Coverage...                                                  
Pod::Coverage installed successfully.                                                            
...
...
...
Fetching http://search.cpan.org/CPAN/authors/id/S/SP/SPADKINS/App-Options-1.07.tar.gz
Building App-Options-1.07 for App::Options...
App::Options installed successfully.
...
(the program finally runs)

As the developer, you only need to put "use CPAN::AutoINC" at the top of your main script. How's that for deployment heaven? Zero installation, zero configuration (thanks to cpanminus), zero dependency outside of Perl core modules.

But there's a catch. The user (or the user's sysadmin) still needs to install CPAN::AutoINC and needs to install and bootstrap local::lib first. So it's not really an out-of-the-box experience yet.

miyagawa++ is already planning to automatically bootstrap local::lib from cpanminus, and the logic behind CPAN::AutoINC is just a couple of dozen lines that can be embedded easily into the script (this process can be made automatic with distribution building tools like Dist::Zilla plugin).

Since cpanminus has zero dependencies, we can simply include it too in our application.

And maybe in the future there can be GUI interfaces for cpanm, so it can display a nice dialog to ask confirmation via the desktop.

Be prepared to be able to provide a much nicer experience for your users.

Log::Any::App

2010-02-24T06:33:00.000-08:00

This is a draft/RFC document.

Log::Any is great if you're writing modules. You only need to say:

use Log::Any qw($log);

and then you're off producing logs with:

$log->debug(...);
$log->warn(...);
# etc

But if you're writing scripts/applications (and thus need to "consume" or display the logs as well), it becomes a bit of a hassle. For example, if you want to display logs to the screen with Log::Dispatch, this is the incantation you need:

use Log::Any qw($log);
use Log::Any::Adapter;
use Log::Dispatch;
Log::Any::Adapter->set('Dispatch');
my $disp = Log::Dispatch->new(outputs => ["Screen", min_level=>"debug", newline=>1]);
Log::Any::Adapter->set("Dispatch", dispatcher=>$disp);
...
$log->warn(...);

which I'm sure I'll never ever remember and will just copy paste everytime.

The goal is to be able to write only this in your scripts:

use Log::Any::App qw($log);

and Log::Any::App (which currently does not exist) will take care of all the rest:

Choose the best available adapter(s);
Configure the adapter(s) with the best defaults, e.g. to screen, as well as to file /var/log/SCRIPTNAME.log (if running as root) or ~/SCRIPTNAME.log (if running as user). The defaults can of course be changed via configuration;
Pick configuration from various sources, like environment variables (e.g. turning level to debug if DEBUG is set to true), command line options (e.g. log level from --log_level/--log-level/--debug/--verbose/etc, as well as detecting result from Getopt::Long or App::Options so we can avoid parsing by ourselves)

I think the main challenge is arranging a set of defaults that are acceptable and comfortable for a lot of people, and working together nicely with available modules like adapter modules (Log::Any::Adapter::Dispatch, Log::Any::Adapter::Log4perl), command line parsing modules, configuration modules, etc.

And later should you refactor your script into modules, the logging part can be left untouched as they already use the Log::Any framework.

Modules to look at: Log::Dispatchouli.

A small plea to non-English bloggers

2010-02-17T01:59:00.000-08:00

I absolutely welcome and cheer the non-English posts in the Iron Man feed (as they are also welcome by the rules). I even try to read one or two whenever I can.

But, at the risk of sounding like a party pooper, guys, could you please not give your post an English title/subject when the content is not in English? I'm sure other people will appreciate it as it saves them time when they can't read your language anyway.

Thank you very very much, and keep blogging!

The (lack of) readability of Perl

2010-02-10T18:10:00.000-08:00

Can't believe it's almost 8 years since I wrote this Indonesian article about "8 things that make Perl relatively unreadable".

To summarize, I think the readability issue in Perl largely boils down to the high usage of symbols/non-alphanumeric characters (e.g. regex and special variables). This admittedly won't change for years to come, as Perl 6 also offers us lots of new operators, even allows us to define new ones, using the full range of the Unicode charset! So we will almost certainly be hearing and need to be fighting against this meme for a long long time.

I also believe that most of the complaints about Perl being unreadable come from non-Perl users. When they read some code, they expect it to be able to read/guess it based on the prior knowledge of some other programming language. For example, even before I started learning French or German, I could guess the general meaning of some French and German (or some other European language) phrases based solely on my knowledge of English. This does not happen with, say, Chinese or Math or regular expression, as the symbols are just gibberish to the non-initiated.

Perhaps marketing that Perl is one of the hardest languages to master will attract more people to learning it, as that's what's happening with Cantonese and some westerners.

Setelah 7 tahun...

2010-02-10T17:49:00.000-08:00

Tahun 2003 saya menulis artikel ini: 8 Hal Yang Membuat Perl Relatif Unreadable. Wah, 1 tahun lagi sudah 8 tahun deh, secepat itukah waktu berlalu?

Eniwei, setelah lewat 7 tahun, tentu saja ada beberapa hal/poin di artikel tersebut yang sudah tidak berlaku lagi. Bukti bahwa Perl tetap hidup karena tetap berubah dan berevolusi.

1. Moose dan sistem OO di Perl6. Kini pengguna Perl tidak perlu minder lagi, karena Perl kini sudah punya sistem objek yang modern dan "tidak mengerikan lagi", bahkan berani deh beradu superior dengan Ruby, Python, Javascript, dll.

2. Beberapa perbaikan readability dapat kita jumpai di Perl 6, antara lain penggunaan prefiks variabel yang lebih konsisten, Untuk mengakses elemen array dan hash kini menggunakan @array[n] dan %array{s}, tidak lagi $array[n] dan $array{s}, bahkan sebetulnya Anda bisa "hidup" hanya dengan mengenal satu prefiks saja, $, karena array dan hash dan struktur kompleks semuanya dapat ditampung oleh $.

Terdapat pula grammar untuk mengizinkan kita memodularkan pola regex yang kompleks sehingga menjadi jauh lebih readable.

Tapi... bukan Perl namanya kalau tidak mengizinkan kita meringkas-ringkas program menggunakan banyak simbol. Tak kurang dari puluhan operator baru, belum lagi metaoperator, dan juga berbagai idiom baru, hadir di Perl 6. Bahkan seluruh karakter Unicode dapat dipakai!

Apakah ini membuat Perl tetap/makin unreadable? Mengulangi kata-kata di artikel 2003 saya, readability sesuatu yang subjektif. Apakah matematik yang penuh simbol juga unreadable? Tentu saja iya, bagi mereka yang buta/awam matematik, tapi tentu tidak bagi matematikawan.

App::Options

2010-02-03T16:07:00.000-08:00

Over the course of many years, I have written lots of lots of short command-line scripts in Perl (Perl's great for that, you know). Most of these scripts are small utilities, or replacement for shell scripts, or automation tools.

All of these scripts invariably need some ability to take command-line options. Of course, back in the days, I used Getopt::Std and Getopt::Long. They both did their task well, but also invariably I will need to provide -h or --help for usage information. I always hate having to write this usage text manually.

Also, if you use your scripts often enough, you will end up with the same incantation for some, that you will want to put the command line options to a config file to avoid repeating yourself. Passwords are also not appropriate in command-line options due to security issues.

So I now use App::Options and have never looked back. You can look at it as a pretty straightforward replacement for Getopt::Long, but it gives you automatic --help and --version (--program_version actually, but --version also does something else wonderful). And it automatically enables you to read config files. I emphasize "automatically" because you absolutely do not have to do a single thing, as App::Options gives you some nice defaults on where to find the config files.

How great is that? Just write your code as if you're using Getopt::Long, but gain all these extra abilities for free!

There are other solutions for command-line scripts, like App::Cmd, but really App::Options is the easiest way especially for old-timers like me who do not want to change their Getopt::Long-style habit.

App::Options is not perfect though. There are a couple of small annoyances I have about it, but I'm not hung up on them. The first is that "=" in command-line option is required, i.e. you have to write "--opt=args" instead of "--opt args". Second is, --version defaults to displaying Perl modules version instead of your program's version.

For larger apps, I also sometimes need hierarchical/multilevel configuration. It'd also be nice to have YAML support. These are something I hope to accomplish with Config::Tree, but I guess I still need to work on it quite a bit before I can replace my usage of App::Options with it.

Four months after...

2010-01-27T15:42:00.000-08:00

Four months ago I joined the Iron Man challenge. So what have I done and what did I get out of this?

What I have done: wrote several new Perl modules. I didn't feel like I have enough things to say (that's why I set up the blog as "Perl Indonesia" and invited others to join instead of "Steven Haryanto's Perl Adventure"), and I still don't, really. So I write more code instead, and post about them.

The modules are typically small, ones which I can finish in one or two sittings. The ideas for these modules usually have been floating in my head for some time, but since they are trivial enough I have never gotten around to write them. The blogging challenge changed that.

Some of the other modules actually come from code I have written, in one form or another, but have not been properly packaged. Dist::Zilla changed that, as it lowers the cost of maintaining and releasing Perl distributions. Thanks, RJBS!

What I get: joy and satisfaction that I am actively doing something for Perl.

All in all, Ironman is a great idea and a good cause. I encourage everybody to keep blogging and writing.

Lingua::ZH::PinyinConvert::ID

2010-01-20T04:00:00.000-08:00

Today I released Lingua::ZH::PinyinConvert::ID. This release is dedicated to the recently deceased former President of Indonesia, Abdurrahman Wahid, which in 2001 declared Chinese new year as national holiday as well as re-allowed the use of Chinese characters and the expression of Chinese culture in daily life, thus ending the 30 years of anti-Chinese laws during the Suharto regime.

Sadly, a generation of Indonesian Chinese people grew up while being cut off from their heritage, not learning Chinese languages nor a lot of the Chinese culture. This will probably be fixed in the next couple of generations as many elementary schools now start to include Mandarin in their curriculum.

Log::Any

2010-01-12T18:21:00.000-08:00

A few weeks ago I found about Log::Any, and have since migrated many of modules to it (usually from Log::Log4perl).

The two main advantages of Log::Any for me:

your users don't need to configure anything if they don't want logging. When my module, say Foo, uses Log4perl to produce logs, then when you use Foo, Log4perl will emit a warning: Log4perl: Seems like no initialization happened. Forgot to call init()?. To remove this warning you need to initialize Log4perl, e.g. using use Log::Log4perl qw(:easy); Log::Log4perl->easy_init($FATAL); This is annoying if you happen to not care about logging.

With Log::Any, the default is null logging.
printf-style logging. For example, $log->debugf("data = %s", $data); It can also handle references/nested data structures, so you don't have to resort to something like $log->debug(sub { "data = " . Dumper($big_data) });. In fact, 99% of the time I use sub{} is precisely because I want to avoid the cost of dumping.

Of course, there are some features in Log4perl that is missing in Log::Any, like logdie() and the TRACE level. But it is a small price to pay. You gain other benefits, the most important of which is compatibility with other logging frameworks. I'm sure some people prefer something else to Log4perl. Log::Any currently supports Log4perl as well as Log::Dispatch, and possibly others too in the future.

Thanks Jonathan for Log::Any!

The Help MySQL petition

2010-01-05T16:09:00.000-08:00

As much as I love(d) MySQL and am still using it a lot (mostly for PHP web applications that are married to it), there seems to be too much politics surrounding it these days.

From helpmysql.org (emphasis mine):

If those IPRs fall into the hands of MySQL's primary competitor, then MySQL immediately ceases to be an alternative to Oracle's own high-priced products. So far, customers had the choice to use MySQL in new projects instead of Oracle's products. Some large companies even migrated (switched) from Oracle to MySQL for existing software solutions. And every one could credibly threaten Oracle's salespeople with using MySQL unless a major discount was granted. If Oracle owns MySQL, it will only laugh when customers try this. Getting rid of this problem is easily worth one billion dollars a year to Oracle, if not more.

Is Oracle really MySQL's primary competitor? I thought they represent two very distinct segments?

Also, if Oracle owns MySQL, why can't I still threaten Oracle sales reps to use MySQL to get a deep discount on Oracle DB? Because Oracle will threaten to kill MySQL, or sue every other company that provides paid support for MySQL, or deliberately delay fixing critical bugs in MySQL? I'm really not convinced with this argument. In any case MySQL would still be a cheaper substitute for Oracle. And I were an Oracle client wanting to get a discount (and not laughs), I think I would rather threaten to switch to SQL Server or Postgres instead.

Also, online petition immediately conjures up the image of a teenager trying to save his/her favorite cartoon TV show that is being cancelled.

Also, let's never forget the bigger "politics" before this, regarding the position of MySQL AB on the usefulness of things like transactions or foreign key constraints, depending on whether its product has support for them.

Let's take one particular example with foreign key constraints. It shows that you really can't trust the opinion of people with ulterior motives. Here's a snippet from the old MySQL 3.23.x manual, when MySQL has no support for foreign key checking (emphasis mine):

5.4.5.1 Reasons NOT to Use Foreign Keys constraints

There are so many problems with foreign key constraints that we don't know where to start:
Foreign key constraints make life very complicated, because the foreign key definitions must be stored in a database and implementing them would destroy the whole ``nice approach'' of using files that can be moved, copied, and removed.
The speed impact is terrible for INSERT and UPDATE statements, and in this case almost all FOREIGN KEY constraint checks are useless because you usually insert records in the right tables in the right order, anyway.
There is also a need to hold locks on many more tables when updating one table, because the side effects can cascade through the entire database. It's MUCH faster to delete records from one table first and subsequently delete them from the other tables.
You can no longer restore a table by doing a full delete from the table and then restoring all records (from a new source or from a backup).
If you use foreign key constraints you can't dump and restore tables unless you do so in a very specific order.
It's very easy to do ``allowed'' circular definitions that make the tables impossible to re-create each table with a single create statement, even if the definition works and is usable.
It's very easy to overlook FOREIGN KEY ... ON DELETE rules when one codes an application. It's not unusual that one loses a lot of important information just because a wrong or misused ON DELETE rule.

The only nice aspect of FOREIGN KEY is that it gives ODBC and some other client programs the ability to see how a table is connected and to use this to show connection diagrams and to help in building applicatons.

And here's a snippet from MySQL 5.1 manual, when, through InnoDB, MySQL now has support for foreign keys. Notice the complete change of heart (emphasis mine):

1.8.5.4. Foreign Keys

The InnoDB storage engine supports checking of foreign key constraints, including CASCADE, ON DELETE, and ON UPDATE. See Section 13.6.4.4, “FOREIGN KEY Constraints”.

[...]

Foreign key enforcement offers several benefits to database developers:
Assuming proper design of the relationships, foreign key constraints make it more difficult for a programmer to introduce an inconsistency into the database.
Centralized checking of constraints by the database server makes it unnecessary to perform these checks on the application side. This eliminates the possibility that different applications may not all check the constraints in the same way.
Using cascading updates and deletes can simplify the application code.
Properly designed foreign key rules aid in documenting relationships between tables.

Do keep in mind that these benefits come at the cost of additional overhead for the database server to perform the necessary checks. Additional checking by the server affects performance, which for some applications may be sufficiently undesirable as to be avoided if possible. (Some major commercial applications have coded the foreign key logic at the application level for this reason.)

[...]

Be aware that the use of foreign keys can sometimes lead to problems:

[...]

This either means MySQL AB deliberately added misleading opinion about foreign key constraints, or MySQL AB grew up and saw the benefits of foreign key constraints during the later days of 3.23/4.x. Either way doesn't bode well on MySQL.

But anyway, I guess all of these so-called "politics" exist in any product advocacy. No product can support all possible features, so unsupported features sometimes get downplayed, either deliberately or innocently. Since Perl subscribes to the multiparadigm and TIMTOWTDI thinking, we suffer less from these. But haven't we all heard more than a handful of otherwise brilliant Perl programmers casting aside things like block indentation or even OOP as overrated, just because other languages support these features (better) than us?

Parsing with Perl

2010-01-05T08:56:00.000-08:00

When it comes to text processing and manipulation, I suspect that the majority of Perl programmers depend on their regex skill to do the job. So much that it becomes their hammer. At least it is mine, because I whip up regexes for almost anything, from web scraping to converting dictionaries.

Which is a pity because there are other useful techniques, like parsing.

Admittedly, parsing is a bit harder (to do correctly), and libraries for parsing most formats out there, from XML to YAML, from CSV to PPI, already exist anyway. So ironically it's even harder to find useful, practical applications for parsing that are easy to implement. Most parser tools seem to invariably give the tired, if not overused, calculator example (i.e., parsing simple mathematical expression like '3 * (2 + 4)').

In the recent years, I've also have only encountered one instance where I want to do some parsing. Two weeks ago I thought it would be nice for my Data::Schema module to provide some shortcuts.

'int*' would be translated to [int => {set=>1}]

'int[]' to [array => {of => "int"}],

'(int*)[]' to [array => {of => [int => {set=>1}]}]

'(int[])*' to [array => {set => 1, of => "int"}]

'(int|int[])[]' to [array => {of => {either => {of => ["int", [array => {of => "int"}]]}}}]

and so on. (These shortcuts, along with some other goodies, will be in the next release of Data::Schema.)

I ended up using Regexp::Grammars for this, since I'm already on Perl 5.10. Regexp::Grammars is basically just a syntactic sugar and helper on top of Perl 5.10's regex which is already capable of doing recursive matching. Parse::RecDescent is also equally easy to use, but I prefer the simpler interface of Regexp::Grammars. I quickly gave up on Parse::Yapp and Parse::Eyapp though. Not that they're no good, it's just that they are too much trouble for my particularly simple need.

Anyway, parsing is fun, as long as--like anything else in life--it doesn't get too complicated :-) I just need to find other small use cases where I can do some more parsing...

CPAN CPAN on the disk, ...

2009-12-30T16:26:00.000-08:00

Interestingly, there are some ID's up there that I've never ever heard of, ever. Either I'm the hobbit, or this list is not representative at all, or both. Maybe we should rank on total number of prereq'ed modules...

$ cd /minicpan && ( for dir in */*/*; do 
echo -e `ls $dir/*.tar.gz 2>/dev/null | perl -ne's/-[^A-Za-z_].+//; 
$m{$_}++; END{print 0+keys %m}'`\\t${dir#*/*/}; done ) | sort -rn | head -10
213     ADAMK
212     ZOFFIX
206     RJBS
169     MIYAGAWA
124     SMUELLER
122     NUFFIN
117     BINGOS
111     GUGOD
109     MARCEL
100     TOKUHIROM

Keywords: CPAN authors with the most modules, most distributions

Given/when expression

2009-12-28T06:30:00.000-08:00

Given/when in Perl 5.10 is great (and will be even greater in Perl 6, for example we can remove many of the now-required parentheses). But it is very statement-oriented. Sometimes I miss CASE expressions a la SQL:

SELECT store_name, CASE store_name
  WHEN 'Los Angeles' THEN sales*2
  WHEN 'San Diego' THEN sales*1.5
  ELSE sales
  END AS new_sales
FROM store;

So, how do you do given/when expressions in Perl? I can think of a couple alternatives, but none is very appealing. Do you a have better one?

my @store = (
    {store_name=>'Los Angeles', sales=>100_000},
    {store_name=>'San Diego', sales=>100_000},
    {store_name=>'Anaheim', sales=>100_000}
);

1. do {} + if/elsif (con: needs do{}, needs explicit assignment to $_, no implicit smart matching)

for my $s (@store) {
    printf "%s %d %d\n", $s->{store_name}, $s->{sales}, do {
        local $_ = $s->{store_name};
        if (/Angel/) { $s->{sales}*2 }
        elsif ($_ eq 'San Diego') { $s->{sales}*1.5 }
        else { $s->{sales} }
    };
}

2. do {} + given/when (con: needs do{}, needs $tmp assignments)

for my $s (@store) {
    printf "%s %d %d\n", $s->{store_name}, $s->{sales}, do {
        my $tmp;
        given($s->{store_name}) {
            when (/Angel/) { $tmp = $s->{sales}*2 }
            when ('San Diego') { $tmp = $s->{sales}*1.5 }
            default { $tmp = $s->{sales} }
        }
        $tmp
    };
}

3. sub {} + given/when (con: needs sub{}, slower perhaps?, needs explicit return's)

for my $s (@store) {
    printf "%s %d %d\n", $s->{store_name}, $s->{sales}, sub {
        given ($s->{store_name}) {
            when (/Angel/) { return $s->{sales}*2 }
            when ('San Diego') { return $s->{sales}*1.5 }
            default { return $s->{sales} }
        }
    }->();
}

Kalender adven Perl 6, RSS di HP

2009-12-20T21:27:00.000-08:00

Blog apa yang paling menarik dibaca tahun 2009 ini? Tak diragukan lagi, Perl 6 Advent Calendar, http://perl6advent.wordpress.com/ . Selama sebulan menjelang Natal, Anda akan disuguhi artikel-artikel menarik bergaya bahasa santai yang masing-masing mengulas satu fitur baru/keren dari Perl 6. Saat ini sudah ada 20-an artikel yang diterbitkan. Masih ada beberapa hari lagi tersisa, jangan lewatkan!

Betewe, sejak taun ini ikut-ikutan beli hape kuerti yang bisa internetan, walau gak ikut-ikutan fesbukan, saya menemukan kegiatan yang mengasyikkan: membaca RSS di hape dengan Google Reader. Terasa sekali, perusahaan yang paling gets it, paling memperhatikan usability mobile browsing tak lain tak bukan adalah Google. Dan juga Facebook mungkin (tapi saya jarang pakai). Karena itu gak sabar rasanya menanti ponsel dan netbuk besutan Google tahun depan.

Storable, Regexp, bugs, bugs, bugs

2009-12-16T08:34:00.000-08:00

A few hours spent yesterday trying to find out why some of my tests keep failing under certain conditions. Turns out I did a Storable::dclone() on an object, and that object contains a regular
expression. And Storable Don't Do No Regex. Worse is, Storable doesn't complain but will just freeze/thaw the regexes into garbage.

$ perl -MStorable=dclone -MData::Dumper -e'
    print "Storable version = $Storable::VERSION\n";
    $re = qr/abc/;
    print Dumper $re;
    print Dumper dclone $re;'
Storable version = 2.21
$VAR1 = qr/(?-xism:abc)/;
$VAR1 = bless( do{\(my $o = undef)}, 'Regexp' );

This means:

$ perl -MStorable=freeze -E'say "yikes" if freeze(qr/a/) eq freeze(qr/b/)'
yikes

Since regexes are so common in Perl, maybe some warnings in Storable documentation should be in order? I'm sure many more bums are in line waiting to be bitten by this. (Bug report filed).

There is Regexp::Copy which contains Regexp::Storable, which is supposed to add regexp (de)serialization to Storable, but turns out that it still has bugs. Even this very simple case will yield a wrong answer:

$ perl -MData::Dumper -MStorable=dclone -MRegexp::Copy -E'say "Regexp::Copy version = $Regexp::Copy::VERSION"; say Dumper dclone([qr/a/, qr/b/])'
Regexp::Copy version = 0.06
$VAR1 = [
          qr/(?-xism:b)/,
          qr/(?-xism:b)/
        ];

(Btw, if you "use Regexp::Copy" before/without "use Storable", it will also result in an error. So that's couple of bug reports filed in).

Sadly it's unclear whether these will be fixed soon. The bug queues for Storable and Regexp::Copy contains unresolved entries several years old.

I first tried to switch to Data::Compare (as actually I was just comparing data structure freeze()'s for comparison, as well as some cloning). But turns out that Data::Compare doesn't deal with recursive/circular structure yet (bug filed).

Finally I resorted to using the good ol' Data::Dumper for serializing/comparison part, and Clone for the cloning part.

Is it just me (I hope it's just me) or do other people find serializing/deserializing modules in CPAN tend to be more buggy than, say, Ruby ones? I've never *once* encountered a problem with Ruby's yaml module, yet aside from Storable and Regexp case above, I have also been bitten several times by bugs in YAML.pm, YAML::Syck, and YAML::XS. The last one is this.

And with a couple of bugs in my own code unrelated to all the above, that makes the most number of bugs found/reported between yesterday and today. Not bad after all, but still I'm worried.

Arti sebuah nama

2009-12-16T03:31:00.000-08:00

“What’s in a name?” Apalah arti sebuah nama, begitu tulis Shakespeare. “That which we call a rose. By any other name would smell as sweet.” Atau diterjemahkan, terasi dibilang mawar pun tetap bau.

Tentu saja, dalam kenyataan, sebuah nama biasanya mengandung banyak arti, karena kita memberi nama tidak secara acak melainkan disertai asosiasi, maksud, ekspresi, atau harapan tertentu. Saya ingat saat sebuah serial drama Jepang 1980-an beken di Indonesia, dari anak kenalan sampai anjing tetangga diberi nama Oshin. Jiwa setiap zaman dan budaya terjejak dalam nama-nama yang diberikan pada era/kultur tersebut. Masa revolusi dan perjuangan China dulu banyak bayi lelaki diberi nama Jian Guo (bangun negara) atau Guo Qing (hari kemerdekaan). Di Internet, muncul situs-situs bernama aneh seperti digg, reddit, twitter, semuanya karena ingin nama yang pendek di tengah kelangkaan domain .com.

Baru-baru ini saya menulis 2 buah modul kecil dalam bahasa pemrograman Perl, yang satu untuk menebak gender nama orang berdasarkan nama depan (menurut statistik dan sejumlah aturan heuristik), dan yang satu lagi untuk mengurai sebuah nama menjadi komponen-komponennya. Pada waktu Anda membaca tulisan ini, kemungkinan kedua modul tersebut sudah bertengger di situs repositori Perl CPAN.

Berbeda dengan beberapa modul serupa yang sudah ditulis untuk bahasa lain seperti Inggris yang hanya berkutat soal penebakan gender, modul pengurai nama Indonesia ini saya lengkapi dengan rutin untuk mengekstrak segala macam aspek yang memang terindikasi dalam nama. Termasuk agama (dari keberadaan titel seperti Haji/Hj, nama depan seperti Muhammad/Muh, atau singkatan nama baptis seperti FX), suku/etnik (dari pola penamaan tertentu misalnya di Bali dengan nama-nama seperti I Gusti Agung atau Ni Made, di Jawa dengan Raden, atau dari nama depan/marga yang amat khas seperti Liem untuk etnik China, Siregar untuk Batak, dll), hingga profesi/tingkat pendidikan (dari titel akademik). Sudah sangat “SARA” bukan? Tapi apa yang sebetulnya dimaksud dengan isu SARA?

Modul penebak gender biasanya digunakan untuk memberi kata sapaan yang cocok (bisa Bapak atau Ibu) saat menulis surat/email, karena ada studi yang mengatakan bahwa penyebutan kata sapaan yang salah dapat mengurangi efektivitas/tingkat respon/dsb (selain tentunya menyinggung perasaan!). Namun modul pengurai nama saya ini, termasuk alat bantu lain seperti perangkat lunak pendeteksi ras dalam foto wajah, dapat membantu proses diskriminasi lebih lanjut. Bayangkan proses penyaringan mahasiswa/karyawan/pejabat yang kini dapat lebih praktis dalam membuang calon tak diinginkan dari ras, suku, agama/keyakinan tertentu.

Saya sempat ragu sesaat untuk tidak jadi merilis modul ini, namun berdasarkan pertimbangan-pertimbangan di bawah, akhirnya saya berkeputusan untuk tetap merilisnya.

Pertama, perangkat lunak semacam ini tidak membuat jadi mungkin diskriminasi yang sebelumnya tidak dimungkinkan. Maksudnya adalah, semua informasi untuk diskriminasi seperti agama, suku, gender, dsb tersebut sebetulnya sudah terkandung di dalam nama itu sendiri. Perangkat lunak hanya merupakan enkoding informasi ini dalam bentuk instruksi komputer. Entah karena kebanggaan, kebiasaan, atau untuk meneruskan garis keturunan, orang tetap mencantumkan berbagai elemen indikator ke dalam nama mereka, walaupun konsekuensinya mempermudah dirinya didiskriminasi berdasarkan nama.

Kedua, argumen “pedang bermata dua”. Sama seperti senjata pisau atau senapan yang bisa digunakan untuk membunuh maupun menyelamatkan, memulai atau menghentikan perang, demikian juga perangkat lunak dapat dipakai oleh polisi untuk melakukan racial profiling ataupun bagi para organisasi untuk melaksanakan affirmative action. Keberadaan perangkat lunak itu sendiri tidak mengubah kecenderungan ke arah salah satu.

Ketiga, dalam kaitannya dengan SARA, UU ITE di Bab tentang perbuatan yang dilarang menyebutkan bahwa hanya informasi yang ditujukan untuk menimbulkan kebencian atau permusuhan antarindividu/antargolonganlah yang termasuk dilarang disebarkan. Perangkat lunak pengurai (parser) sama sekali tidak dibibiti informasi kebencian/permusuhan.

Bagaimana menurut pandangan para pembaca? Saya menanti masukan dari Anda semua. Apakah merilis perangkat lunak untuk menguraikan (parsing) nama orang dengan tujuan mengetahui gender, agama, suku, golongan, termasuk ke dalam tindakan terlarang? (PCMedia Edisi Jan 2010)

Variabel state di Perl 5.10

2009-12-07T21:10:00.000-08:00

Bahasa Perl termasuk memberi banyak pilihan skop variabel bagi programer. Pertama, ada variabel global (tepatnya variabel package, karena sebetulnya tidak ada variabel global di Perl; ck ck ck di awal artikel sudah berbohong? :-)

Kedua, yang mungkin paling sering kita pakai, variabel leksikal alias variabel privat yang dideklarasikan dengan my() yang hanya bisa dilihat/diakses oleh blok atau file atau eval tempat si variabel dideklarasi.

Ketiga, variabel lokal untuk dynamic scoping, menggunakan kata kunci local(), biasanya kita pakai untuk menyimpan/mem-backup sebuah variabel, lalu mengubahnya di dalam sebuah blok, dan nanti otomatis saat kita keluar dari blok tersebut nilai si variabel akan terpulihkan kembali. Kita bisa melokalkan variabel package, filehandle, glob, dsb. Bahkan kita bisa melakukan hal seperti ini:

$config = { foo=>'ujan', bar=>'kemarau' };
...
{
    local $config->{foo} = 'blah';
    sini();
    sana();
}
print $config->{foo}; # 'ujan' lagi

Pada contoh di atas, kita melokalkan satu pair dari hash saja. Saat masuk ke sini() dan sana(), nilai lokal $config->{foo} akan terus dipertahankan. Inilah yang dimaksud dynamic scoping, jadi tidak berbasis pada source code melainkan pada alur running program. Setelah blok selesai, barulah nilai lama $config->{foo} pulih. Asyik kan?

Keempat, ada lagi yang namanya our(), mulai diperkenalkan sejak Perl 5.6. Ini pada dasarnya adalah pengganti untuk "use vars qw($foo)". our() membuat alias leksikal untuk sebuah variabel global, eh, variabel package. Tentu saja variabel tersebut nanti bisa diakses dari package lain.

Kelima, yang menjadi topik posting blog ini, yaitu variabel state. Variabel ini diperkenalkan sejak Perl 5.9.sekian dan resmi menjadi bagian fitur baru dari 5.10. Untuk menggunakannya, di awal skrip kita harus melakukan:

use feature 'state';

atau:

use feature ':5.10'; # jangan lupa kutipnya ya...

Kegunaan variabel state adalah untuk membuat variabel leksikal yang persisten. Sebelumnya hal ini memang bisa dilakukan menggunakan closure, tapi terus terang saya malas menghafalnya. Setelah ada state(), barulah jadi lebih termotivasi untuk menggunakan leksikal persisten.

Akhir-akhir ini saya sering membuat metode yang mengembalikan nilai konstan/statik, contohnya:

sub config_vars {
    [qw/
     recurse_hash
     recurse_array
     parse_prefix
     ...
    /]
}

Kenapa tidak pakai variabel biasa saja (mis menggunakan our())? Tujuannya sih agar bisa memanfaatkan inheritance.

Tapi, tahukah Anda, bahwa setiap kali dipanggil, si metode tersebut akan membuat arrayref baru? Buktinya:

$ perl -le'sub f { [1,2,3,4,5,6,7,8,9,10] } print f for 1..5'
ARRAY(0x9d1c40)
ARRAY(0xa083c8)
ARRAY(0xa083b0)
ARRAY(0xa08398)
ARRAY(0xa08380)

Pemborosan bukan? Nah, solusinya kita bisa menggunakan variabel state:

sub config_vars {
    state $a = [qw/
     recurse_hash
     recurse_array
     parse_prefix
     ...
    /];
}

Variabel $a ini akan hidup terus walaupun sudah keluar dari skop (tentu saja, pada ujungnya nanti akan di-garbage collect kalau memang tidak ada yang memakai lagi).

Kini:

$ perl -lE'sub f { state $f = [1,2,3,4,5,6,7,8,9,10] } print f for 1..5'
ARRAY(0x12ffc40)
ARRAY(0x12ffc40)
ARRAY(0x12ffc40)
ARRAY(0x12ffc40)
ARRAY(0x12ffc40)

Oya, -E sama seperti -e tapi menghidupkan semua fitur Perl terbaru (dalam kasus ini, ekivalen dengan "use feature ':5.10'").

Selamat bermain dengan variabel di Perl!

Requiring 5.10

2009-12-02T19:44:00.000-08:00

So I decided to add "perl = 5.010000" in some of my dist.ini's. This is just because of one particular habit I recently acquired: writing "$a //= 1" and "$b = $a // 2". (Well actually I've rejoiced since defined-or is announced for 5.10, but for some reason have only begun to really use it in the past weeks.)

So much nicer than writing "$a = 1 unless defined($a)" and "$b = defined($a) ? $a : $2". It's the having to repeat myself aspect which I find disgusting, especially if $a is an expression. Which is one of the reason I always hate coding in PHP because in PHP you can't even say "$b = $a || 2".

I'm not switching, state-ing, say-ing, doing recursive patterns, or any of the other cool stuffs in 5.10. So am I so selfish for forcing 5.10 down the throats of other people just for such a minor convenience? The word externality comes to mind (having completed reading Superfreakonomics 2 days ago).

But it's a positive externality, really. ;-)

minicpan

2009-12-01T04:15:00.000-08:00

Last weekend I got a new toy: Asus EEE PC S101. This is actually my second netbook (and fourth laptop overall). I sort of dumped my first netbook only after a month of use because I now totally hate hard drives on netbooks: they're hot, they're loud, they're a power drain.

The EEE has a SSD drive, but at only 32GB, putting the whole CPAN (currently at 6.9GB) on it is a bit taxing. With the help of minicpan, I got it down to only 1.2GB. So thanks again Ricardo!

Then I immediately wondered how big BackPAN is, guessing it might be between 30-100GB (with 14 years of CPAN's history and all). But then:

$ perl -MParse::BACKPAN::Packages -e'$p = Parse::BACKPAN::Packages->new(); printf "BACKPAN is %.1fGB\n", $p->size/1024/1024/1024'
BACKPAN is 12.4GB

Much smaller than I had thought. It's just about twice the current size of CPAN. Which probably means that a lot of CPAN authors still leave much of their stuffs around and not delete them.

More flexible configuration merging with Data::PrefixMerge

2009-11-26T23:07:00.000-08:00

(I'm planning a refactoring for Data::PrefixMerge and will be renaming it to Data::ModeMerge. Thought I'd post something on the blog.)

In a typical Unix program, there are three levels of configuration: system-wide config file (/etc/myapp.conf), per-user config file (~/.myapprc), and command-line options. It's convenient programatically to load each of those in a hash and then merge (e.g. using Data::Merger or Hash::Merge) system-wide hash with the per-user hash, and then merge again the result with the command-line hash to get the a single hash as the final configuration. Your program can from there on deal with this just one hash instead of three.

In a typical merging process between two hashes (left-side and right-side), when there is a conflicting key, then the right-side key will override the left-side. This is usually the desired behaviour in our said program as the system-wide config is there to provide defaults, and the per-user config (and the command-line arguments) allow a user to override those defaults.

But suppose that the user wants to unset a certain configuration setting that is defined by the system-wide config? She can't do that unless she edits the system-wide config (in which she might need admin rights), or the program allows the user to disregard the system-wide config. The latter is usually what's implemented by many Unix programs, e.g. the -noconfig command-line option in mplayer. But this has two drawbacks:

a slightly added complexity in the program. The program needs to provide a special, extra comand-line option.
the user loses all the default settings in the system-wide config. What she needed in the first place was to just unset a single setting (a single key-value pair of the hash).

Here's where Data::PrefixMerge comes in. It provides a so-called DELETE mode.

prefix_merge({foo=>1, bar=>2}, {"!foo"=>undef, bar=>3, baz=>1});

will result ini:

{bar=>3, baz=>1}

The ! prefix tells Data::ModeMerge to do a DELETE mode merging. So the final result will lack the foo key.

On the other hand, what if the system admin wants to protect a certain configuration setting from being overriden by the user or the command-line? This is useful in a hosting or other retrictive environment where we want to limit users' freedom to some degree. This is possible via the KEEP mode (prefix ^):

prefix_merge({"^bar"=>2, "^baz"=>1}, {bar=>3, "!baz"=>0, quux=>7});

will result in:

{bar=>2, baz=>1, quux=>7}

effectively protecting bar and baz from being overriden/deleted/etc.

Aside from the two mentioned modes, there are also a few others available by default: ADD (prefix +), CONCAT (prefix .), SUBTRACT (prefix -), as well as the plain ol' NORMAL/override (optional prefix *).

You can add other modes by writing a mode handler module. (planned in upcoming Data::ModeMerge release)

You can change the default prefixes for each mode if you want. You can disable each mode individually. (planned in upcoming Data::ModeMerge release)

You can default to always using a certain mode, like the NORMAL mode, and ignore all the prefixes, in which case Data::ModeMerge will behave like most other merge modules.

You can change default mode, prefixes, disabling/enabling mode, etc on a per-hash basis using the so-called options key.

WEHT Emperl?

2009-11-24T07:39:00.001-08:00

Speaking of templating systems in the previous post, I also got reminded of Embperl, and wondered why it didn't get more popular. I remember back in 1998-1999 enjoying working with Embperl before moving on to Mason. It has a nice syntax, and one very nice feature: (HTML- and/or URL-)escaping output by default. Say you have in $foo "<script>evil()</script>" then this template:

[+ $foo +]

will output "<script>evil()</script>". You are protected from XSS by default. And if you want to turn off this escaping, you can set EMBPERL_ESCMODE to 0, or, do this:

[+ do { local $escmode = 0; $foo } +]

But then maybe this is akin to what earlier versions of PHP attempted to do with default magic_quotes_gpc and magic_quotes_runtime set to on. These two default configuration have helped spread the backslashitis/toothpick syndrome all over the web and are currently deprecated (and will be removed in PHP 6.0). A majority of PHP programmers apparently never understood the need of these escaping, and got confused/mad by the insistence of PHP to add those pestering backslashes. And most would turn off the configuration, or add a routine to reverse the escaping at the beginning of their programs.

So is the moral of the story: do not overprotect programmers (especially ignorant ones)? Or don't try to fix the problem the wrong way? Or both?

Perl program that generates... Perl programs

2009-11-24T06:12:00.000-08:00

Data::Schema is a module that I wrote to do data structure validation (using another data structure acting as a schema). I had not been terribly happy with the validation speed, ranging in about 100 validations/sec for schemas that are only barely complex. This is not bad, actually, because Data::FormValidator also performs more or less the same. But that was below expectation.

Last week I suddenly realized, writing data validator or a templating system is essentially writing a [mini] language, with our schema/template as the miniprogram and our [Perl] program as the interpreter. To make our miniprogram faster, we can compile it into Perl.

FastTemplate from PHP quickly comes to mind. I remember that it was advertised as being fast because it converts the template into PHP code (e.g. into a bunch of echo's, if/else's, and for loops). And I think there must be at least half a dozen Perl-based template languages on CPAN doing the same. Although I'm not sure about how common data validators compile schemas into Perl code.

So anyway, starting from 0.12, Data::Schema can convert schema into Perl subroutines and use it to validate data structures. Validation speed jumps about one order of magnitude faster, and I'm a happy guy again.

Who's using your module?

2009-11-17T18:16:00.000-08:00

To see who on CPAN are using your module (distribution), go to this page: http://cpants.perl.org/dist/used_by/Your-DistName

So I casually entered my distributions, being happily surprised that there are people using them.

If you're like me, and I'm sure thousands of other CPAN authors, you're putting stuffs on CPAN either for fun, or for your own use. Knowing other people are using your module gives you some perspective. I was happily breaking/removing/refactoring stuffs as I please, but now I'll need to think twice everytime.

Merge key support in YAML

2009-11-17T02:59:00.000-08:00

None of the many Perl YAML modules support merge key:

$ cat merge.yaml
foo: 1
bar: 2        
<<:
  baz: 3      

$ perl -MYAML::Tiny -MFile::Slurp -e'print Dump Load(scalar read_file "merge.yaml")'
---                                                                                               
<<:             
  baz: 3      
bar: 2        
foo: 1

$ perl -MYAML -MFile::Slurp -e'print Dump Load(scalar read_file "merge.yaml")'
---
<<:
  baz: 3
bar: 2
foo: 1

$ perl -MYAML::Syck -MFile::Slurp -e'print Dump Load(scalar read_file "merge.yaml")'
---
<<:
  baz: 3
bar: 2
foo: 1

$ perl -MYAML::XS -MFile::Slurp -e'print Dump Load(scalar read_file "merge.yaml")'
---
<<:
  baz: 3
bar: 2
foo: 1

However, good ol' Ruby handles it just fine:

$ ruby -ryaml -e'print YAML::load(File.open("merge.yaml").read).to_yaml'
---
baz: 3
foo: 1
bar: 2

Kinda strange since Ruby's yaml and YAML::Syck both use why's syck library, which has not been updated for quite a while.

Dist::Zilla for Module::Starter Users: A 2-Minute Guide

2009-11-10T19:07:00.000-08:00

So you're a module author. You typically do this when starting a new distribution:

$ module-starter --module=Foo::Bar --author="Your Name" --email="you@example.com"

and then hack away under the resulting Foo-Bar directory. Easy enough right?

Problems are:

too much generated boilerplate code and text;
lack of automation for the building and release process.

I'm sure you have experienced one or more of these:

Having to search+replace copyright year in every file;
Forgetting to "make clean" or remove backup files before creating tarball;
Forgetting to update MANIFEST;
Feeling tired and bored of all the tedious and laborious tasks;
Wondering if there is a better way.

You need a distribution builder like Dist::Zilla. It helps:

eliminate a lot of duplicate text;
automate updating MANIFEST;
automate generating README;
build tarball;
automate a lot of other stuffs;
upload to CPAN;
and more.

Dist::Zilla is flexible and has a lot of plugin/plugin bundles, but it can be less straightforward to use it. Here's a simple step-by-step guide you can follow.

Install these modules from CPAN:
- Dist::Zilla
- Dist::Zilla::Plugin::PodWeaver
- Dist::Zilla::Plugin::ReadmeFromPod
Create ~/.dzil/config.ini, containing:

[!new] author = Your Name copyright_holder = Your Name initial_version = 0.01 [!release] user = YOUR-PAUSE-ID password = YOUR-PAUSE-PASSWORD
Now, instead of using module-starter, you run dzil new to start your new distribution:

$ dzil new Foo-Bar

Instead of a bunch of files under Foo-Bar/, there's now just one file, dist.ini:

name = Foo-Bar version = 0.01 author = youruser license = Perl_5 copyright_holder = youruser [@Classic]

There's currently a small bug in Dist::Zilla not supplying the correct author/copyright_holder, so edit dist.ini, as well as for adding some lines:

name = Foo-Bar version = 0.01 author = Your Name <you@example.com> license = Perl_5 copyright_holder = Your Name [@Classic] [PodWeaver] [ReadmeFromPod]
Create the most basic distribution structure:

$ mkdir -p lib/Foo t

Put some test files into t/. Default tests generated from module-starter like 00-load.t might be a good start.

Create Changes file. You can copy paste from the one generated by module-starter:
```
0.01    2009-11-11
        First version, released on an unsuspecting world.
```
As for lib/Foo/Bar.pm, here's what I use for template. You can just fill out the [[...]] parts:

package Foo::Bar; # ABSTRACT: [[Abstract of module]] use strict; use warnings; [[YOUR CODE]] 1; __END__ =head1 SYNOPSIS [[YOUR SYNOPSIS]] =head1 DESCRIPTION [[YOUR DESCRIPTION]] =cut

It's much simpler and shorter than what module-starter generates. Some POD sections like VERSION, NAME, AUTHOR, LICENSE AND COPYRIGHT are deliberately omitted. They will be generated by Dist::Zilla later when building the distro.
Hack away. Write your code in lib/Foo/Bar.pm. Add some tests in t/. Add other files when needed.
To test the distribution, run dzil test.
To build the distribution, run dzil build. This will create Foo-Bar-0.01.tar.gz which contains all the necessary goodies of a standard classic distribution, like README, LICENSE, MANIFEST, META.yml, etc.
To release the distribution, run dzil release. This will upload your module to CPAN. Sweet!
To release a new distribution, just update version number in dist.ini and $VERSION in your main module file. Don't forget to add an entry to Changes. Repeat dzil test, build, release.

In the future "dzil new" might allow creating a more complete skeleton.

There are lots of other nice things Dist::Zilla can do for you, like checking Changes file, do automatic version numbering, etc. Welcome to the nice world of Dist::Zilla!

Comments/corrections are welcome.

Regex Editor in Padre 0.50

2009-11-10T03:31:00.000-08:00

Thanks to Gábor Szabó, we now have a regex editor in Padre. Yay! Sure it's really basic at the moment, but it's a start.

The Padre Features page says this about the regex editor: "A tool that allows the less experienced users as well to build and debug a regular expressions".

Well, if your inspiration is a tool to *teach* regex, then maybe. But in general I don't quite agree on the "less experienced users" emphasis. Regex is a minilanguage, and a featureful IDE/editor and debugger helps beginners as well as experienced users a lot. Would you say Padre is a tool for the less experienced Perl programmers? Up to a certain point, it's becoming really hard/cumbersome to debug rather long and complex regular expressions.

Personally I've only used Rx Toolkit in Komodo. Its interface is simple and not as colorful as some Windows regex editors on the market, but it does the job well. If I can cite a wishlist for Rx Toolkit, it would just be to add position indicator (line number + column + offset) in each of regex/string/matches subwindow. Being able to step regex is nice too I guess, but not essential. Usually my regexes aren't *that* long/complex.

Of course the future for Padre's regex editor is to handle Perl 6 grammars, which will be a full-fledged language in itself, so there *should* be step over/trace into/watch/etc debugger features for it.

Creating shortcuts for long Perl module names

2009-11-03T09:10:00.000-08:00

My main interface with the computer is the shell, the terminal, the keyboard. Thus I love shortcuts. My .bash_profile is littered with one- and two-letter shortcuts like:

alias m='mplayer -fixed-vo -osdlevel 3'
alias mn='mplayer -fixed-vo -osdlevel 3 -nosound'
alias m11='m -speed 1.1'
alias m12='m -speed 1.2'
alias m20='m -xy 2'
alias m21='m20 -speed 1.1'
...

And when aliases are not enough, because they do not work outside the shell (like in KDE's mini commander), I create scripts. My ~/bin is also littered with alias commands like:

# k
#!/bin/bash
exec konsole "$@"

As well as my homedir, project directories, and the whole filesystem ornamented with short symlinks to here and there.


$ ln -s public p
$ ln -s proj/perl pp
$ ln -s sites/steven.builder.localdomain/www .
...

When I have long Perl module names like Spanel::API::Account::Shared, Spanel::API::Account::XenVPS, I would also like to create shortcuts for these.

There are several modules to do this. I ended up using Package::Alias because I like the interface. Other ones I tried include aliased and namespace::alias.

Suppose I want to create a shortcut of SAAS for Spanel::API::Account::Shared (which can export foo). All I need to do is:

# in SAAS.pm
use Spanel::API::Account::Shared;
use Package::Alias 'SAAS'=>'Spanel::API::Account::Shared';
1;

Now aside from these:

$ perl -MSpanel::API::Account::Shared -e'Spanel::API::Account::Shared::foo()'
$ perl -MSpanel::API::Account::Shared=foo -e'foo()'

All of below work too:

$ perl -MSAAS -e'Spanel::API::Account::Shared::foo()'
$ perl -MSAAS -e'SAAS::foo()'
$ perl -MSAAS=foo -e'foo()'

My fingers thank Joshua Keros, the author of Package::Alias.

The while(1) {...last...} construct

2009-11-03T08:53:00.000-08:00

In the past year or two I've been comfortably using this construct in Perl as well as in PHP (and probably others too):

while (1) {
    do_stuff;
    do_some_checks;
    do { warn error 1; last } if not OK;

    do_more_stuff;
    do_some_checks;
    do { warn error 2; last } if not OK;

    do_even_more_stuff;
    do_some_checks;
    do { warn error 3; last } if not OK;

    ...

    #finally
    last;
}

I prefer it over the one below:

do_stuff;
do_some_checks;
if (!OK) {
    warn error 1;
} else {
    do_more_stuff;
    do_some_checks;
    if (!OK) {
        warn error 2;
    } else {
        do_even_more_stuff;
        do_some_checks;
        if (!OK) {
             ...
        }
        # finally
        success!
    }
}

The while (1) { ... last ... } style is clearer and avoids extraneous indentation. The only thing biting me in the past when using this construct is that sometimes I forgot to add the final last, resulting in an infinite loop. But this series-of-checks-and-early-bail pattern happens so often in my code that the construct quickly became second nature.

Anyone got the same habit, or perhaps using some alternative (like the new given-when)?

Using mod_perl aside from deploying web apps?

2009-10-27T09:30:00.000-07:00

History has it that mod_perl was (is?) marketed as a high performance alternative to Perl CGI, and that virtually all success stories mentioned on the website are about using it to deploy web apps.

It is unfortunate because: first of all, mod_perl is much more than just about running web apps, it is actually a tool to customize and extend Apache using Perl instead of C. There's so much you can do with it.

Second, I believe mod_perl is suboptimal for running web apps: it makes your Apache processes fat (and thus you'll often need a front-end proxy), it's tied to Apache (so you can't experiment with other webservers), it's relatively more complex to configure and tune compared to, say, FastCGI (and thus potentially more insecure), and it's just too damn powerful if you just want a fast CGI.

mod_perl is also theoretically more insecure because it bundles webserver and the application engine together instead of cleanly separates it. Some of your Perl code might run as root too. All of these are unnecessary if you just want to run web apps.

Here at work we have been using mod_perl for years, not for deploying web apps but for creating custom Apache handlers in Perl (basically it was because my C sucks). The handlers do the following:

connect to CGI daemon (because we also write our own CGI daemon, which is much more paranoid in some respects but also more flexible in others like running PHP scripts under different configurations);
filter URLs using own rules (this can be done using a series of regexps with mod_rewrite, but much more readable and comfortable if done with full-fledged Perl code);
authenticate hosting users;
do per-vhost aliases (mod_alias can also do this, but we are using mass/dynamic virtual hosts);
etc.

The servers on which we run mod_perl are shared hosting servers, and we allow users to install their own .htaccess, so we have to patch mod_perl to restrict the Perl* directives from the users. This is because mod_perl does not have something like mod_ruby's RubyRestrictDirectives. This kind of functionality is available as a build-time configuration.

So there you have it, an unusual/unpopular application of mod_perl: customizing Apache in a shared hosting environment.

Anyone else using mod_perl not to deploy web apps?

Trying out Padre 0.47

2009-10-27T02:20:00.000-07:00

This is written after a few days of trying Padre. Normally I use emacs, joe, Komodo Edit/IDE, kate, and recently geany, so this post is basically about comparing Padre's editing features with these other editors.

Note: Yup, 0.48 was already out a couple of weeks ago, but I haven't been able to install it due to segfaults in libwx*. Incidentally my box was recently upgraded from Jaunty to Karmic so that might have caused the problem. Anyway, I did read the 0.48's ChangeLog just in case.

First and foremost, I'm blown away! Between my first try a few months back and 0.47, Padre has tranformed into a very usable editor/IDE, and it's pretty fast too, faster than Komodo IDE/Edit which is notoriously sluggish. Kudos to the hardworking Padre team, you guys rock!

Here are the features which I find still missing:

a button (or keyboard shortcut) to quickly toggle on/off the directory tree.
next-tab and prev-tab functionality. There is next-file and prev-file, but after you rearrange the tabs, those two become less useful. Maybe assign Ctrl-PgUp/PgDn or Alt-Left/Right for this?
justify/reflow paragraph/manual word wrap. A la emacs' M-q or joe's C-kj.
autodelete trailing whitespaces when saving (and autoadd newline at the end if missing).
remember folding state when reopening files.
(Bug?) the "autofold POD when folding is enabled" feature is very nice, but right now is behaving rather strangely. It only works after I open Preferences and hit Save. Opening or saving files does not automatically fold PODs as advertised.
maybe include something like Rx Toolkit in Komodo? Because I do write a *lot* of regexes when coding in Perl.
(Suggestion) soft characters for automatic bracket completion. I find this feature in Komodo very very nice as I seem to have the habit of typing { or (, and then after thinking a while cancel it.

I'd gladly submit these into Padre's Trac if someone would give me an account. Last time I visited #padre no one had the admin rights to do this. Signup from the website is temporarily disabled to overwhelming amount of spam.

Anyway, keep up the wonderful work, guys! Looking forward to using Padre more and more often in the future. Envisioning writing Padre config files in Perl, just like in emacs using Lisp... Life will be good...

HTTP-style sub return

2009-10-20T02:48:00.000-07:00

In a subroutine, we often want to return the status of operation (success/failure/error code) as well as the result of the operation.

When a function does not need to return any result, it can just return the status, usually as an integer scalar. In C and Unix the convention is to return 0 for success and non-zero for the error code. In Perl and many other languages, it is the other way around: zero/false/undef for failure and true values for success. You can also return the result as well right there and then as long as the result evaluates to true. This can be a problem if there is a possibility that the result be zero/false/undef.

Alternatively, you can return the result instead. How do we return the error code then? Usually via some global variable like $? and $! in Perl. This has drawbacks of its own, like in multithreaded/reentrant code.

It is safer thus, to return both the status and the result separately and explicitly, e.g. using a 2-element array:

($status, $result) = foo(...);

For roughly a year now, I have been adopting something like the above, with what I call HTTP-style return convention. Instead of 2, I return a 3-element array in my subs:

return ($status, $extra_info, $result);

$status is a 3-digit integer values, with values taken as much as possible from the HTTP spec: 200 means success, 4xx means generic "client side" (i.e. caller side) error like missing or invalid arguments, 404 means not found, 403 means forbidden, 5xx means error in the "server side" (our side, the sub side), 501 means unimplemented, and so on.

$extra_info is a hashref which contains, well... extra info, like error string, debugging messages, or intermediate results. This is the equivalent of HTTP response headers. But it can be an undef too if the sub does not offer extra information. So it will avoid creating unnecessary hashref.

$result is the actual result.

An example code:

my @resp = process_stuff(@stuffs);
if ($resp[0] != 200) {
    die "Failed ($resp[1]{errstr})";
} else {
    print "Number of stuffs input: ",     scalar(@stuffs);
    print "Number of stuffs processed: ", $resp[2];
}

Another example:

my @resp = search_stuff($stuff);
if ($resp[0] == 404) {
    die "Stuff $stuff not found";
} elsif ($resp[0] != 200) {
    die "Failed";
} else {
    print "Stuff $stuff found in $resp[2]";
}

It is a bit confusing for readers not familiar with this style, but I find this clear as it is (i.e., without declaring/using constants for 200, 404, etc).

The advantages of this return style:

You can return the status as well as the result as well as extra info.
The convention for the status codes is already familiar to many. HTTP has been so popular for most of the lifetime of Perl, that most Perl programmers who have dabbed in CGI or Internet programming should be familiar with it. Even many nonprogrammers know what 404 or 500 mean since they often see this while browsing. The 2xx, 4xx, 5xx convention is also used in other protocols like SMTP.

If you are not, you can always use a comment or a constant to explain the meaning of the numbers.
You can easily wrap your sub later in a REST API or web service. Just pass $status as HTTP status code, (selected) pairs in $extra_info into HTTP response headers, and $result (possibly encoded in JSON/YAML/whatever).

A bonus, because Perl has contexts, you can also do this:

if (wantarray) {
    return ($status, $extra_info, $result);
} else {
    return $result;
}

so that you can fallback to a very simple style when all the other stuffs are not needed:

my $result = foo(...); # don't care about status, assume always success

I find this return style somewhat relevant in the light of PSGI's deservedly speedy upshoot to popularity.

Dumping content to files using Log::Dispatch::Dir

2009-10-14T21:07:00.000-07:00

Logging frameworks like Log::Log4perl and Log::Dispatch are great. They relieve you of the burden of reinventing your own (which, admit it, will probably suck more), and they decouple your code from logging details (which can be changed later independently). You just say I want to log "something something" in the code, and you can later configure whether those messages are actually written to the logs, where those logs are, how the messages are formatted, etc without changing your log-using code.

Logging can be used, to some extent, to replace debugging or to aid it. Each log message is usually only a single line, like "Starting foo ...", "Ended foo ...", "The value of foo is $val", although we can also log large data structure dumps or file contents.

When writing web robots like HTML scrapers or interfaces to online banking sites, which are particularly fragile, it is often convenient to save each server's full response into a separate file so you can easily check each step by opening the saved file in a browser.

If you want to use Log::Log4perl or Log::Dispatch for this, you can too, using Log::Dispatch::Dir. This module will write each log message to a separate file in a specified log directory.

An example, in Finance::Bank::ID::Base I have code like this:


$self->logger_dump->trace(
    "<!-- result of mech request #$i ($method ".Dump($args)."):\n".
    $mech->response->status_line."\n".
    $mech->response->headers->as_string."\n".
    "-->\n".
    $mech->content
);

where $mech is a WWW::Mechanize object. I use $self->logger for "normal" log messages and $self->logger_dump specifically for dumping contents. Both the logger and logger_dump attributes can be supplied by the module user, e.g.:


my $ibank = Finance::Bank::ID::BCA->new(
    ...
    logger      => Log::Log4perl->get_logger("Messages"),
    logger_dump => Log::Log4perl->get_logger("Dumps"),
);

and the Log::Log4perl configuration is something like this:


log4perl.logger.Messages=TRACE, SCREEN, LOGFILE
log4perl.logger.Dumps=TRACE, LOGDIR

log4perl.appender.SCREEN=Log::Log4perl::Appender::ScreenColoredLevels
log4perl.appender.SCREEN.layout=PatternLayout
log4perl.appender.SCREEN.layout.ConversionPattern=[\%r] %m%n

log4perl.appender.LOGFILE=Log::Log4perl::Appender::File
log4perl.appender.LOGFILE.filename=/path/to/logs/main.log
log4perl.appender.LOGFILE.layout=PatternLayout
log4perl.appender.LOGFILE.layout.ConversionPattern=[\%d] %m%n

log4perl.appender.LOGDIR=Log::Dispatch::Dir
log4perl.appender.LOGDIR.dirname=/path/to/logs/dumps
log4perl.appender.LOGDIR.layout=PatternLayout
log4perl.appender.LOGDIR.layout.ConversionPattern=%m

This is convenient enough for me, but in the future I want to do some MIME checking to the log messages, so Log::Dispatch::Dir can automatically add a suitable file extension e.g. .html, .txt, .jpg, etc.

Mengecek rekening BCA dan Mandiri dengan Perl

2009-10-14T20:44:00.000-07:00

Akhirnya, kesampean juga merilis modul Finance::Bank::ID::BCA dan Finance::Bank::ID::Mandiri. Kini Anda bisa mengecek rekening BCA dan Mandiri dengan Perl!

Bertahun-tahun lalu saya pernah membuat skrip serupa untuk KlikBCA tapi pake kombinasi curl/wget. Sejak terjadi perubahan layout/program di KlikBCA dari ASP ke Java, gak pernah lagi ngupdate skrip ini sampe beberapa bulan yang lalu. Dan akhirnya minggu lalu dan minggu ini menyempatkan memodulkan kodenya.

Oya, sebetulnya sebagian kodenya pertama-tama ditulis dalam PHP ;-p. Cuma, kode PHP-nya untuk kantor dan gak dirilis (dan mungkin gak akan pernah dirilis, karena saya males memaintain kode PHP utk publik).

Planet CPAN

2009-10-12T23:42:00.000-07:00

Recently I have been enjoying Iron Man blog posts that talk about some particular CPAN module like HTML::FormHandler, Finance::Quote, Term::ProgressBar, local::lib, Log::Log4perl, Dist::Zilla, even the good ol' Digest::MD5.

I'm hoping though that even more people (authors and users alike) would blog more about CPAN modules, because although we arguably have one of the richest sets of interfaces to our wonderful software library, with more than 16000 modules it's near impossible to browse them all. Feature blog posts certainly help people stumble on interesting stuffs even if they don't follow Recent CPAN Uploads.

Since ~ 95% of all interesting things in the Perl world are happening inside CPAN (not to belittle the huge efforts of the p5p team or the Parrot & Perl 6 designers/implementors, of course) shouldn't we be blogging more about it?

Filtering RSS

On a somewhat unrelated note, lately I've also been tired of all the Microsoft/Windows/Vista/7/8/9 that are filling up from the Slashdot RSS feed. Most are irrelevant to me as I use Linux, besides, those news are really minor/unimportant/of marketing type, like rereading old Vista reviews, intentionally ambiguous 128-bit version of future Windows, or repeated news items telling me that products are being delayed yet again. Who cares?

Nowadays I'm using Google Reader on a cellphone to read most feeds, so less junk would be nice. Google Reader doesn't have filtering yet, so I ended up using Yahoo! Pipes. Filtering and doing other stuffs to feeds (and other kinds of data like CSV) are surprisingly quick and easy using this web-based visual editor. A user-friendly Perl- and Unix-killer? :-) Just go to pipes.yahoo.com, click on My Pipes, create a new pipe, and do some drag-and-drops and text field filling, publish your pipe, and get RSS. Perfect (here are two examples of pipes I've created: slashdot-noms and slashdot-nofb). Looking forward to a more Microsoft-free news reading ahead.

CPAN download counter? +1!

2009-10-12T22:40:00.000-07:00

From time to time people (including me once, a few years back) would ask questions like: what are the 'top' (or 'most popular' or 'widely used' or most downloaded) dists/modules on CPAN? Is there a download counter for each dist/module? Like this one from prz, a budding module author.

The answer is there isn't one, because CPAN is just a bunch of static files. The upside of this, CPAN is very easily mirrored (e.g. via FTP or rsync or offline via CDs) and served (e.g. via FTP, HTTP, or local filesystem). The downside, there isn't a place for much intelligence/logic on the serving side.

To implement this feature, we can put some stats gathering code on the client side, like what Debian has been doing for a while; in fact you can already see the list of most widely installed Perl modules from the data. Or we can add some stats to search.cpan.org like most viewed/clicked/downloaded dists and modules, and maybe top search keywords. Not representative of all mirrors, sure, but it's better than nothing.

Download counter, or at least Popular/Top Downloads, is a common feature on download/catalog/shopping/news sites, from freshmeat and Download.com, to Amazon and iTunes Store. So common that many users expect it to be there as a standard feature.

It's not hard to imagine why people like to know what's popular, what everybody else is using/doing, what's in, what's hot. It's a social side of human nature. And it's beneficial to know which modules are getting downloaded and used more, to direct development efforts to the more important stuffs. Volunteers can surely take the top modules list as one consideration when picking which project to spend their valuable time on.

What I'm not very clear on though is why, aside from PHP, many programming languages' communities don't like this particular feature? Do we hate competition, do we hate popularity contest, or are we just plain lazy?

Anyway, effort like CPANHQ might soon make the Top/most $foo modules, and more, possible. Yay!

Moving from bzr to git

2009-10-06T19:18:00.000-07:00

As with a lot of coders out there, I've moved on from no source control, to CVS, to Subversion, and recently to one of the distributed ones. In fact, I tried both Bazaar and git roughly at the same time a couple of years back and have been quite comfortably using both to manage my projects.

Last week though, I've migrated all but one of my projects to git. The only bzr repository left is the one for $work where I want to avoid retraining people to use git since they are not coders.

I had no real complaints with bzr. Sure, it's generally slower than git, the repository size is slightly larger than git's, and branching is a bit more cumbersome than in git, but all of those have never bothered me enough to part with bzr.

The clincher, however, came when I needed to write some post-commit hook to post my commit messages to an internal web-based bulletin board. In bzr you need to write a plugin written in Python. Not that there's anything wrong with it, but I don't think I even want a Perl-based SCM where I need to write a Perl plugin for everything when a two-line shell script will do. Add to that fact that bzr doesn't provide a template/skeleton for the plugin and I have to spend a few minutes to google around the plugin API (in addition to refreshing my memory on Python syntax), etc.

I think I'm much more comfortable with git nowadays.

Some Indonesian-specific modules to check SIM, NPWP, NIK

2009-10-06T10:28:00.000-07:00

Today for some reason I finally got motivated enough to write and release Business-ID-{NIK,SIM,NPWP} to CPAN. Been wanting to for years, but haven't got around to it due to lack of challenge and, more importantly, reference. For example, so far I haven't found a publicly available and authoritative document explaining the numbering scheme and the valid area codes for the SIM. I had to do some deducing (read: guessing) from some thousands of actual user-submitted SIM, NPWP, and NIK/KTP numbers (some of them most certainly bogus). Isn't it sad?

If you encounter some valid numbers being rejected by the modules, or vice versa, feel free to report them as bugs.

Perlsatu kita teguh, Perlcerai kita runtuh

2009-09-28T22:16:00.000-07:00

Yup, garing abis judulnya, tapi yang kepikir dalam 10 detik cuma itu sih.

Blog ini pertama dibuat untuk ikutan Perl Iron Man Blogging Challenge. Karena gw gak yakin bisa keep up ngeblog sendirian seminggu sekali (tepatnya, 4 post dalam 32 hari) maka gw buat nama blog ini mewakili komunitas Perl Indonesia.

Bila Anda cinta/suka Perl, ingin berbagi tentang Perl, atau sedang belajar Perl, saya harapkan mau ikut bergabung ngeblog di blog ini. Posting boleh dalam bahasa Indonesia maupun English. Gak harus murni tentang Perl, yang kira-kira cuma 10-20% berhubungan dengan Perl juga gpp. Dan sekali-kali OOT juga no problem. Bahasa gak harus resmi, yang penting enak dibaca dan komunikatif. Level post gak harus advanced, semua level diterima.

Berminat? Silakan hubungi Steven di: steven2 setrip id setrip perl et masterweb titik net. Nanti 'tak tambahkan akun Google Anda ke dalam daftar anggota tim blog ini supaya bisa ikutan ngepost. Yuk?

Buat para pembaca, enjoy dan doakan kita2x!