Wishlist for a service framework and/or manager

I maintain code for a few daemons/services written in Perl (most of them serve requests by forking/preforking). Reading post on Ubic, I started to feel that I am reinventing a lot of wheels. Currently I am doing these by writing my own code, as much as which I hope can be offloaded to CPAN in the future:

• Autorestart when process size is becoming too big. We need to do this gracefully, meaning wait until there is no more clients being serviced, unless process size really gets too big in which case we need to restart immediately. Checking period can be configured.
  
  
• Autorestart if script or modules change on disk. Also needs to be done gracefully. This is usually being recommended to be used only on development environment, but I use this too in production, for ease of deployment. But we need to check first (e.g. via "perl -c" or "eval + require" whether the new code from disk is okay.
  
  
• Avoid duplicate instance (currently always using Proc::PID::File, but I'm open to better mechanism).
  
  
• Limit clients concurrency. Sometimes this is simple (a single limit for all clients) and sometimes not so much (different limits for different IP/IP blocks/authenticated users/groups/etc).
  
  
• Reap dead child processes and maintain a count of child processes.
  
  
• Handle timed out clients. This is rather cumbersome with blocking I/O.
  
  
• Write init script. This is the part I dislike the most, since there are tons of different OS flavors out there, and with more recent efforts like upstart, launchd, systemd, sooner or later I will certainly have to write different init scripts. I wish there is something equivalent to PSGI/Plack for general services, which can plug my code to whatever service manager might be out there.
  
  

Random Perl wishlists #1: uncapture modifier, require+import, backtick function

Uncapture modifier. The new /r regexp substitution modifier in Perl 5.13.2 indicates that there might be hope for even more modifiers in the future. A possible modifier might be one that ignores all capture groups, which can solve this guy's problem.

require that can also import. I wonder why "require" doesn't also support importing like "use" does: use MODULE_NAME LIST... since it already support "require MODULE_NAME". This way, whenever I want to defer loading some modules, I can just replace "use" with "require" and put the statement under some "if" statement.

the backtick function.. Do you find yourself often having to do use a temporary variable like this, $cmd = "some longish and ".quote("complexly formed")."command"; before doing backtick `$cmd`? This is because unlike in PHP, we have system() but no backtick(). Most probably remnants from shell. There is Capture::Tiny for more general solution but of course it's a bit more cumbersome.

Startup overhead still matters

We all love Moose, and the subject of this question could have been rephrased better, but why do I get the feeling that not many people write pure CGI or command-line scripts in Perl (that got executed many times) anymore? After all, didn't Perl begin as a tool for sysadmin and only in the mid 1990's got picked up as the darling of CGI/web programming?

There are still many cases where/reasons why Perl scripts need to be run many times (instead of persistently long running).

• It's much more stable (I've often need to kill or ulimit or periodically restart a Perl process because after days it grows to 500+ MB).
  
  
• Sometimes CGI is all you get (especially in shared hosting environment, which is related to point 1).
  
  
• Sometimes you need to run the scripts for many users, and it's not feasible (e.g. memory-wise) to let them all run persistently.
  
  
• Many old scripts are designed that way.
  
  
• Some environments require them that way (e.g. scripts run in .qmail are run for every incoming mail, scripts run by tcpserver are started for every incoming connection, etc).

There used to be projects like PersistentPerl or SpeedyPerl to let us easily make a Perl script persistent by just changing the shebang line (e.g. from #!perl to #!pperl), but these projects are currently not actively developed, probably due to lack of demand (?), or becase this kind of deployment tends to cause subtle bugs (I did get bitten by this a couple of times in the past). You can't just convert a script that is designed/written to be a one-off run into a long-running one without expecting some bugs, anyway.

And the Perl compiler (B::*, *.pmc) is also now deprecated, probably because it does not give that many startup cost saving after all (the fact that Perl has phasers like BEGIN/CHECK blocks means it has to execute code as it compiles them anyway).

And thus we're stuck with having to accept the startup cost of parsing & compiling for every script run. That's why startup cost matters. On our servers awstats runs many thousand of times everyday (2000-5000 sites x 10+ HTML pages), and since it's a giant script (10k-ish line) it has a startup overhead of almost 1s. I really would like to shave this startup overhead as it is a significant part of server load.

Until today many of my scripts/programs are still deployed as one-off command line scripts. And that's why instead of Moose I use Mouse (or Any::Moose, to be exact) whenever I can. And so far I can.

Spot the error

use Data::Rmap qw(:all);
use JSON;
use Data::Dump;
use Clone;
use boolean;

my $arg = from_json(q{{"1":true,"2":false}});
# convert JSON booleans to boolean's booleans
rmap_all { bless $_,"boolean" if ref($_) =~ /^JSON::(XS|PP)::Boolean$/ }, $arg;
dd $arg;

Hint: it's one character long.

In fact, this piece of code is full of Perl's traps (from Perl's lack of booleans obviously, to less obviously having to clone and rmap not working), it disgusts me.

Reduce in Perl

Perl has grep/map/sort since probably forever (actually, sort() starts appearing since Perl 2.0). But even now, reduce is still not a builtin in Perl 5 (though available via List::Util), so doing reduce is probably not something that comes as naturally to Perl programmers. Meanwhile Ruby, JavaScript, and even PHP have their reduce operation builtin.

But then, reduce is "not really that useful" (you can just replace it with a simple for loop). So much that Python 3.0 now removes the function from the global namespace and reduces it (pun intended) to a mere member of functools. I guess reduce is really handy only if you are in a heavily functional language that lacks procedural basics.

This can be thought of as a testament to the level of language design skill that Larry has.

The rather funny thing is, in Perl 6, in addition to the reduce() List method there is also the reduce ([]) metaoperator as well.

Perl vs JavaScript

Here are some notes I made while hacking on Language::Expr::Compiler::JS. Of course, there are a million differences between the two, but these focus mostly on operators and types. Hope it can be useful.

• Double vs single quotes. There are practically no functional differences between double-quoted string and single-quoted one in JavaScript. In Perl, single quotes do not interpret escape sequences other than \\ and \', but in JavaScript both single- and double-quoted strings interpret the same set of escape sequences.
  
  
• String escape sequences. Perl does not support JavaScript's \v (vertical tab), while JavaScript does not support \N{NAME} (named Unicode character), \e and \c[ (escape/control), \a (alarm bell).
  
  
• Two undefs. JavaScript has two special nothingness/undefinedness: null and undefined. Strangely, null == undefined and they are equal to themselves, but they are not equal to any other values (including 0, '', false). The difference between the two is just this: undefined is not a keyword but a global variable (you can assign to it, but of course you shouldn't). If you want less confusion, just use null.
  
  
• Behaviour of "+". Since JavaScript only has "+" (while Perl has "+" and "."), you should be aware that "+" in JavaScript coerces to strings when one of the operands is a string (e.g. 1 + "2" becomes "12"). In Perl, "+" coerces to numbers. So be careful when mixing numbers and strings.
  
  
• You need to explicitly "return" value from a function. But there's a cute shortcut for one statement functions introduced in JavaScript 1.8: 'function (x) x*3' which is equivalent to 'function (x) { return x*3 }' so in this case you don't need the "return".
  
  
• Boolean. JavaScript has real boolean. In Perl you can use 'boolean' from CPAN which gives you practically the same stuff.
  
  
• JavaScript lacks a lot of Perl convenience operators, including <=> cmp, low-precedenced and/or/not, //, =~, ~~, qq(), qx(), qw(), m//, s///, **, etc.
  
  

All in all, I think JavaScript is quite nice and simple language with familiar syntax (at least compared to PHP). It also has lexical variables, anonymous functions, OO, etc.

JSYNC is brilliant!

It's a brilliant idea: bank on JSON's popularity and more widespread implementations, add some of the important YAML features not present in JSON on top of it. The result is JSYNC, along with its preliminary CPAN module. (I'd probably picked a different name and choose something like "\" for prefix instead of ".", but hey, it's not my project :-)

A few months ago I was really desperate with the YAML situation in Perl. We have the largest number of YAML implementations, but none of them are good enough compared to Ruby's libsyck. I even contemplated converting all my YAML documents to JSON, but of course that plan was cancelled because JSON doesn't even support references nor objects.

Here's to hoping JSYNC will rocket to popularity soon enough. Ingy++.