Startup overhead still matters

We all love Moose, and the subject of this question could have been rephrased better, but why do I get the feeling that not many people write pure CGI or command-line scripts in Perl (that got executed many times) anymore? After all, didn't Perl begin as a tool for sysadmin and only in the mid 1990's got picked up as the darling of CGI/web programming?

There are still many cases where/reasons why Perl scripts need to be run many times (instead of persistently long running).

• It's much more stable (I've often need to kill or ulimit or periodically restart a Perl process because after days it grows to 500+ MB).
  
  
• Sometimes CGI is all you get (especially in shared hosting environment, which is related to point 1).
  
  
• Sometimes you need to run the scripts for many users, and it's not feasible (e.g. memory-wise) to let them all run persistently.
  
  
• Many old scripts are designed that way.
  
  
• Some environments require them that way (e.g. scripts run in .qmail are run for every incoming mail, scripts run by tcpserver are started for every incoming connection, etc).

There used to be projects like PersistentPerl or SpeedyPerl to let us easily make a Perl script persistent by just changing the shebang line (e.g. from #!perl to #!pperl), but these projects are currently not actively developed, probably due to lack of demand (?), or becase this kind of deployment tends to cause subtle bugs (I did get bitten by this a couple of times in the past). You can't just convert a script that is designed/written to be a one-off run into a long-running one without expecting some bugs, anyway.

And the Perl compiler (B::*, *.pmc) is also now deprecated, probably because it does not give that many startup cost saving after all (the fact that Perl has phasers like BEGIN/CHECK blocks means it has to execute code as it compiles them anyway).

And thus we're stuck with having to accept the startup cost of parsing & compiling for every script run. That's why startup cost matters. On our servers awstats runs many thousand of times everyday (2000-5000 sites x 10+ HTML pages), and since it's a giant script (10k-ish line) it has a startup overhead of almost 1s. I really would like to shave this startup overhead as it is a significant part of server load.

Until today many of my scripts/programs are still deployed as one-off command line scripts. And that's why instead of Moose I use Mouse (or Any::Moose, to be exact) whenever I can. And so far I can.

Spot the error

use Data::Rmap qw(:all);
use JSON;
use Data::Dump;
use Clone;
use boolean;

my $arg = from_json(q{{"1":true,"2":false}});
# convert JSON booleans to boolean's booleans
rmap_all { bless $_,"boolean" if ref($_) =~ /^JSON::(XS|PP)::Boolean$/ }, $arg;
dd $arg;

Hint: it's one character long.

In fact, this piece of code is full of Perl's traps (from Perl's lack of booleans obviously, to less obviously having to clone and rmap not working), it disgusts me.

Reduce in Perl

Perl has grep/map/sort since probably forever (actually, sort() starts appearing since Perl 2.0). But even now, reduce is still not a builtin in Perl 5 (though available via List::Util), so doing reduce is probably not something that comes as naturally to Perl programmers. Meanwhile Ruby, JavaScript, and even PHP have their reduce operation builtin.

But then, reduce is "not really that useful" (you can just replace it with a simple for loop). So much that Python 3.0 now removes the function from the global namespace and reduces it (pun intended) to a mere member of functools. I guess reduce is really handy only if you are in a heavily functional language that lacks procedural basics.

This can be thought of as a testament to the level of language design skill that Larry has.

The rather funny thing is, in Perl 6, in addition to the reduce() List method there is also the reduce ([]) metaoperator as well.

Perl vs JavaScript

Here are some notes I made while hacking on Language::Expr::Compiler::JS. Of course, there are a million differences between the two, but these focus mostly on operators and types. Hope it can be useful.

• Double vs single quotes. There are practically no functional differences between double-quoted string and single-quoted one in JavaScript. In Perl, single quotes do not interpret escape sequences other than \\ and \', but in JavaScript both single- and double-quoted strings interpret the same set of escape sequences.
  
  
• String escape sequences. Perl does not support JavaScript's \v (vertical tab), while JavaScript does not support \N{NAME} (named Unicode character), \e and \c[ (escape/control), \a (alarm bell).
  
  
• Two undefs. JavaScript has two special nothingness/undefinedness: null and undefined. Strangely, null == undefined and they are equal to themselves, but they are not equal to any other values (including 0, '', false). The difference between the two is just this: undefined is not a keyword but a global variable (you can assign to it, but of course you shouldn't). If you want less confusion, just use null.
  
  
• Behaviour of "+". Since JavaScript only has "+" (while Perl has "+" and "."), you should be aware that "+" in JavaScript coerces to strings when one of the operands is a string (e.g. 1 + "2" becomes "12"). In Perl, "+" coerces to numbers. So be careful when mixing numbers and strings.
  
  
• You need to explicitly "return" value from a function. But there's a cute shortcut for one statement functions introduced in JavaScript 1.8: 'function (x) x*3' which is equivalent to 'function (x) { return x*3 }' so in this case you don't need the "return".
  
  
• Boolean. JavaScript has real boolean. In Perl you can use 'boolean' from CPAN which gives you practically the same stuff.
  
  
• JavaScript lacks a lot of Perl convenience operators, including <=> cmp, low-precedenced and/or/not, //, =~, ~~, qq(), qx(), qw(), m//, s///, **, etc.
  
  

All in all, I think JavaScript is quite nice and simple language with familiar syntax (at least compared to PHP). It also has lexical variables, anonymous functions, OO, etc.

JSYNC is brilliant!

It's a brilliant idea: bank on JSON's popularity and more widespread implementations, add some of the important YAML features not present in JSON on top of it. The result is JSYNC, along with its preliminary CPAN module. (I'd probably picked a different name and choose something like "\" for prefix instead of ".", but hey, it's not my project :-)

A few months ago I was really desperate with the YAML situation in Perl. We have the largest number of YAML implementations, but none of them are good enough compared to Ruby's libsyck. I even contemplated converting all my YAML documents to JSON, but of course that plan was cancelled because JSON doesn't even support references nor objects.

Here's to hoping JSYNC will rocket to popularity soon enough. Ingy++.

Custom dumping *in* Data::Dump

After blogging about my small patch to Data::Dump, I contacted Gisle Aas. He is quite responsive and finally comes up with a new release (1.16) of Data::Dump containing the cool new filter feature. My previous example after converted to use the new feature becomes:

$ perl -MData::Dump=dumpf -MDateTime -e'dumpf(DateTime->now, sub { my ($ctx, $oref) = @_; return unless $ctx->class eq "DateTime"; {dump=>qq([$oref])} })'
[2010-06-09T12:22:58]

This filter mechanism is quite generic and allows you to do some other tricks like switching classes, adding comments, and ignore/hide hash keys. The interface is also pleasant to work with, although starting with this release the "no OO interface" motto should perhaps be changed to "just a little bit of OO interface" :-)

Aren't we glad that stable and established modules like this are still actively maintained and getting new features.

Thanks, Gisle!

Menunggu statistik-statistik menarik dari sensus 2010

Koran PR beberapa minggu lalu menulis, Sensus Penduduk 2010 selain bertujuan mencacah jiwa, juga ingin "mencari fakta-fakta unik." Salah satu (dan satu-satunya) contoh: mencari orang tertua. Dan benar memang, selama beberapa minggu ini sudah ada beberapa artikel yang meliput tentang nenek di kampung anu yang berumur 115th, lalu rekornya dipecahkan oleh yang berumur 120th, lalu 125th, dst. Terakhir kalau tidak salah ada yang lebih dari 140th (walaupun semua klaim usia ini berdasarkan ucapan semata, bahkan tidak ada akte lahir atau bukti tertulis lainnya).

Tentu saja, selain hanya mencari orang tertua, masih ada banyak sekali hal menarik yang bisa diekstrak dari data sensus ini. Misalnya, saya harapkan, BPS dapat menerbitkan adalah daftar nama depan dan nama belakang yang paling lazim, seperti yang dilakukan oleh badan sensus Amrik.

Saat menyusun versi awal modul Perl Locale::ID::GuessGender::FromFirstName, saya kesulitan mencari basis data nama yang bisa dipakai, karenanya saya mengambil 1000 nama depan terlazim dari database pelanggan kantor. Tentu saja, andaikan ada data yang lebih representatif, seperti dari sensus penduduk, tentu jauh lebih baik.

Seandainya diberi kesempatan, saya bersedia mengolah data mentahnya ;-)