Outputting pretty data structure on console programs

Our application has a command-line API interface for convenient access via shell/console. It used to output API result data in YAML:

# /c/sbin/spanel api --yaml File list --account steven --volume data --dir /public
--- 
dir: 
  atime: '1270675429'
  ctime: '1285916065'
  gid: 1023
  group: steven
  is_dir: 1
  mtime: '1285916065'
  perms: 493
  uid: 1012
  url: ~
  user: steven
entries: 
  - 
    atime: '1284665000'
    ctime: '1289609859'
    dir: /public
    gid: 1023
    group: steven
    icon_file: folder.gif
    inode: 1984908
    is_dir: 1
    is_link: 0
    mtime: '1289609859'
    name: git
    perms: 493
    reldir: ''
    size: 4096
    uid: 1012
    url: ~
    user: steven
  - 
    atime: '1270675424'
    ctime: '1285130140'
    dir: /public
    gid: 1023
    group: steven
    icon_file: text.gif
    inode: 1976727
    is_dir: 0
    is_link: 0
    mtime: '1155368486'
    name: .htaccess
    perms: 436
    reldir: ''
    size: 48
    uid: 1012
    url: ~
    user: steven
  - 
    atime: '1270675424'
    ctime: '1285130140'
    dir: /public
    gid: 1023
    group: steven
    icon_file: unknown.gif
    inode: 1976725
    is_dir: 0
    is_link: 0
    mtime: '1155368397'
    name: .htaccess~
    perms: 436
    reldir: ''
    size: 63
    uid: 1012
    url: ~
    user: steven
total_num_entries: 3
url: ~

YAML is relatively readable if you compare to JSON or (shudder) XML, but I soon grew tired of reading YAML for data that should be tabulated and better formatted for human consumption.

Thus, Data::Format::Pretty::Console. The idea is for me, a lazy programmer, to throw it data structure of various kind and it will display it nicely suitable for console viewing. The command line API interface now by default shows nicely formatted text for API result data (but still provides --yaml and --json option). Please bear with this blog post's misformatting and assume it's all pretty:

dir:
.---------------------.
| key    | value      |
+--------+------------+
| atime  | 1270675429 |
| ctime  | 1285916065 |
| gid    |       1023 |
| group  | steven     |
| is_dir |          1 |
| mtime  | 1285916065 |
| perms  |        493 |
| uid    |       1012 |
| url    |            |
| user   | steven     |
'--------+------------'

entries:
.----------------------------------------------------------------------------------------------------------------------------------------------------------------------.
| atime      | ctime      | dir     | gid  | group  | icon_file   | inode   | is_dir | is_link | mtime      | name       | perms | reldir | size | uid  | url | user   |
+------------+------------+---------+------+--------+-------------+---------+--------+---------+------------+------------+-------+--------+------+------+-----+--------+
| 1284665000 | 1289609859 | /public | 1023 | steven | folder.gif  | 1984908 |      1 |       0 | 1289609859 | git        |   493 |        | 4096 | 1012 |     | steven |
| 1270675424 | 1285130140 | /public | 1023 | steven | text.gif    | 1976727 |      0 |       0 | 1155368486 | .htaccess  |   436 |        |   48 | 1012 |     | steven |
| 1270675424 | 1285130140 | /public | 1023 | steven | unknown.gif | 1976725 |      0 |       0 | 1155368397 | .htaccess~ |   436 |        |   63 | 1012 |     | steven |
'------------+------------+---------+------+--------+-------------+---------+--------+---------+------------+------------+-------+--------+------+------+-----+--------'

total_num_entries:
3

url:

Comparison of INI-format modules on CPAN

I'm not terribly happy with the state of Perl/CPAN support for the INI file format.

I have this requirement of modifying php.ini files programmatically from Perl like: set register_globals to On/Off, add/remove some extension (via the extension=foo lines), adding/removing some functions from the disabled_functions list, etc. So I would like to find a CPAN module that can just set/unset a parameter and leave formatting/comments alone as much as possible.

Turns out that among a dozen or so of INI modules on CPAN, a few of them do not do writes at all (e.g. Config::INI::Access or Config::Format::INI). And a few that do, write a la dump. That is, they just rewrite the whole INI file with the in-memory structure. All comments and formatting (even ordering, in some cases) are lost. Example: Config::INI::Writer and Tie::Cfg. And, last time I tried, I couldn't even install Config::IniHash from the CPAN client. Not good.

So I ended up with Config::IniFiles. And I needed to patch two features in before I could even read php.ini and write to it properly. This is an old module, which although still maintained, probably needs a rewrite or modernization. One reviewer in CPAN Ratings also wrote that this module fails in edge cases and the test suite is incomplete.

But, at least this module gets the work done. It tries to maintain comments, and even has a host of other features like delta, multiline, default section, etc. Most features seem to be rather exotic to me personally, but then none of the other INI modules on CPAN has the basic features that I needed.

(Short, grossly incomplete) comparison of Perl logging frameworks

After doing this post on comparison of Perl serialization modules, I intended to continue with other comparisons, and even thought on setting up a wiki or creating/maintaining a section on the Official Perl 5 Wiki, which already has a Recommended Modules section, although there is not much comparison being written for each recommendation. (Btw, I just noticed a change of domain for the Wiki, from perlfoundation.org to socialtext.net).

But of course other tasks soon took precedence, so until the Wiki idea is realized, I thought I'll just continue posting on the blog as usual.

There are **seriously** too many Perl logging frameworks out there. As the Log4perl FAQ mentions, "Writing a logging module is like a rite of passage for every Perl programmer, just like writing your own templating system."

So I'm not going to pretend like I've evaluated even half of logging modules that are on CPAN. Instead I'm just going to include a few worth mentioning.

Log::Dispatch and Log::Log4perl. Two of the arguably most popular Perl logging frameworks are Log::Dispatch and Log::Log4perl. They are like the Moose of logging frameworks: mature, feature rich, flexible, has a lot of support/extra/3rd party modules, but... "slow". I quote the slow part because first of all, speed is not an issue for the majority of applications. And second of all, they are not relatively slow at all compared to other modules until they actually log stuff to output. For example, doing debug() on a warn level is around 1,5mils/sec with with Log4perl, and 3mils/sec with Log::Fast. But for actual logging, Log::Fast can be 10-45 times faster than these two.

Log::Any. For most people, Log::Dispatch and Log4perl should suffice. I personally haven't been unable to produce a case where I can't customize Log4perl they way I want. This shows the flexibility of the module. So the only thing left for flexibility is a thin wrapper where you might want to switch logging framework (kind of like Any::Moose for logging). There are a few of these on CPAN, but I prefer Log::Any (and I've also made a thin wrapper for *that*, Log::Any::App). RJBS also made one: Log::Dispatchoulli. You might be interested in using it if you are interested in using String::Flogger.

Performance-wise, as with Moose, there are other alternatives: Log::Fast, for one. There are also a few other minimalistic frameworks, but I do not recommend using them as many of them are not flexible at all. Unless your application is really performance-critical.

I've most probably left out a lot of possibly interesting alternatives.

Sometimes you *don't* want circular checking

I use the nifty Data::Rmap to "flatten" DateTime objects into strings so they can be exported to JSON and handled outside Perl. But due to circular checking in Data::Rmap, this:

$ perl -MData::Rmap=:all -MData::Dump \
-e'$d = DateTime->now; $doc = [$d, $d];
rmap_ref { $_ = $_->ymd if UNIVERSAL::isa($_, "DateTime") } $doc;
dd $doc'

produces something like this:

["2010-10-01", ...unconverted DateTime object...]

For now I work around this by defeating Data::Rmap's circular checking, though I wonder if there's a better way.

$ perl -MData::Rmap=:all -MData::Dump \
-e'$d = DateTime->now; $doc = [$d, $d];
rmap_ref { $_[0]{seen} = {}; $_ = $_->ymd if UNIVERSAL::isa($_, "DateTime") } $doc;
dd $doc'

will correctly produce:

["2010-10-01", "2010-10-01"]

Yet another stupid mistake #1

During a refactor of a data from array @foo to hash %foo, I used 'each' to iterate over the hash, but forgot to change the 'for' statement with 'while'. So I ended up with something like:

$ perl -MData::Dump -E'%a=(a=>1, b=>2);
for (my ($k, $v) = each %a) { $_ = "$k x"; dd {k=>$k, v=>$v, "\$_"=>$_} }'

And this is nasty because for(@ary) aliases $_ to each element in @ary, and in this case it modifies $k (quiz #1: and $v too, do you know why?) right under your nose! Thus the result are really messed up:

{ "\$_" => "a x", "k" => "a x", "v" => 1 }
{ "\$_" => "a x x", "k" => "a x", "v" => "a x x" }

Not to mention the loop stops after processing two items (quiz #2: do you know why?) But you might not realize that after you add some pairs to %a and wondering why they don't get processed.

The error message Perl gives is not really helpful, to say the least :)

Comparison of Perl serialization modules

A while ago I needed a Perl data serializer with some requirements (supports circular references and Regexp objects out of the box, consistent/canonical output due output will be hashed). Here's my rundown of currently available data serialization Perl modules. A few notes: the labels fast/slow is relative to each other and are not the result of extensive benchmarking.

Data::Dumper. The grand-daddy of Perl serialization module. Produces Perl code with adjustable indentation level (default is lots of indentation, so output is verbose). Slow. Available in core since the early days of Perl 5 (5.005 to be exact). To unserialize, we need to do eval(), which might not be good for security. Usually the first choice for many Perl programmers when it comes to serialization and arguably the most popular module for that purpose.

Storable. Fast. Produces compact, portable binary output. Also available in core distribution. Does not support Regexp objects out of the box (though adding support for that requires only a few lines). Binary format used to change several times in the past without backward compatibility in the newer version of the module, giving people major PITA. Supposedly stabilized now.

YAML::XS. Fast. Verbose YAML output (currently doesn't seem to have option to output inline YAML). My personal experience in the past is sometimes this module behaved weirdly and died with a cryptic error, but I guess currently it's pretty stable.

There are other YAML implementations like YAML::Syck (also pretty speedy) and the old Pure-Perl YAML.pm and partial implementation YAML::Tiny. The last two might not be a good choice for general serialization needs.

Data::Dump. Very slow. Produces nicely indented Perl output. The strength of this module is in pretty output and flexibility in customizing the formatting process. Based on Data::Dump I've hacked two other specialized modules: Data::Dump::PHP for producing PHP code, and Data::Dump::Partial to produce compact and partial Perl output for logging.

XML::Dumper. Produces *very* verbose (as is the case with all XML) XML output. Slow. Aside from the XML format, I don't think there's a reason why you should choose this over the others.

JSON::XS. Fast, outputs pretty compact but still readable code, but does not support circular references or Regexp objects.

JSYNC. Slow, outputs JSON and in addition supports circular references but not yet Regexp objects.

FreezeThaw. Slow, produces compact output but not as compact as Storable. Does not support Regexp objects out of the box.

Apart from these there are many other choices too, but I personally don't think any of them is interesting enough to be a favorite. For example, last time I checked PHP::Serialization (and all the other PHP-related modules) does not support circular references. There's also, for example, Data::Pond: cute concept but of little practical use as it is even more limited than JSON format.

There are also numerous alternatives to Data::Dumper/Data::Dump, producing Perl or Perl-like code or indented formatted output, but they are either: not unserializable back to data structures (so, they are more of a formatting module instead of serialization module) or focus on pretty printing instead of speed. In general I think most Data::Dumper-like modules are slow when it comes to serializing data.

In conclusion, choice is good but I have not found my perfect general serialization module yet. My two favorites are Storable and YAML::XS. If JSYNC is faster and supports Regexp, or if YAML::XS or YAML::Syck can output inline/compact YAML, that would be as near to perfect as I would like it.

Hope this comparison is useful. Corrections and additions welcome.

Perl vs PHP (a bit of credit to PHP)

Just read this blog post. Comments are disabled, so I thought I'd add a blog post.

There are endless ways we can sneer at PHP's deficiencies, but since 5.3 PHP already supports anonymous subroutines, via the function (args) { ... } syntax. So:

$longestLine = max(
    array_map(
        create_function('$a', 'return strlen($a);'), 
        explode("\n", $str)
    )
);

can be rewritten as:

$longestLine = max(
    array_map(
        function($a) { return strlen($a); }, 
        explode("\n", $str)
    )
);

though the example is not a good one since it might as well be:

$longestLine = max(
    array_map(
        'strlen',
        explode("\n", $str)
    )
);