How To Build A Perl Package Database

[demerphq]

I can agree with all of that. Actually, starting a discussion about
this was on my todo-list for the last QA hackathon but I didn't get
around to it. Ideally, it should replace not only packlists but also
perllocal

I was thinking about what you said about packlists, and I wonder how much
information one could scrape out of them. Would it be enough to reconstruct
at least that a group of files belongs to a release? That would be enough to
be able to fully uninstall a requested module. For example, if the user asks
to uninstall ExtUtils::MakeMaker the database could have seen that
ExtUtils/MakeMaker.pm was in a packlist together with ExtUtils/MM_Unix.pm and
so on and uninstall them. Probably given their original purpose was to
provide an uninstaller.
Also what's with this .meta directory I see popping up? I missed a memo.

This is all totally doable, and efficient enough, with a small pile of DBM
files and Storable. Where to put the database is a bit more complicated, see
the list of open problems below.

Given that Storable's format isn't forward-compatible, something more
stable such as JSON would be more appropriate.

That's a good point about Storable. JSON requires a dependency.
ExtUtils::Install could bundle JSON::PP, but it would be simpler to use
Data::Dumper. It makes de/serialization faster and simpler. The main
disadvantage is its only readable by Perl, but that's ok since this pile of
DBM files will be opaque to everything but the Perl API. Too much of a mess
to contemplate otherwise.

IMO the question is whether you want the data human readable or not.

If you dont care then use Sereal as a replacement for Storable. Same
feature set pretty much except it is faster and produces smaller
output.

Yves

[Leon Timmermans]

A separate install database for each install location seems like the only
workable approach.

One minor complication of that is the strictest sense an "install
location" isn't all that well defined. Or perhaps I should say every
dist has 8 install locations with unspecified relationship on the
filesystem (lib, arch, bin, script, libdoc, bindoc, libhtml, binhtml).
In practice you may want to use lib *and* arch (they are never used
simultaneously, but stuff in lib may be shared over different perl
installations, contrary to arch which should be for one specific
install).

Leon

[Leon Timmermans]

I wonder if it isn't time to deprecate all the complex install combinations
that may have made sense when hard disks was rather limited.
In ActivePerl we enforce a pretty simplified install layout to be able to
no sharing of directories between different versions
every install area has its own bin, etc, lib, and html subdirectories
the etc subdirectory records packages installed in that location

I think that ship already sailed, if only because different
distributions have very different layouts.

Anyways, I just wanted to say that without putting some restrictions on how
modules (and corresponding scripts) can be installed, creating a package
manager would seem to get even more complex than ExtUtils::Makemaker... :)

What kind of restrictions are you thinking about exactly?

Leon

[Jan Dubois]

Post by Leon Timmermans

Anyways, I just wanted to say that without putting some restrictions on

how

modules (and corresponding scripts) can be installed, creating a package
manager would seem to get even more complex than ExtUtils::Makemaker...

:)
What kind of restrictions are you thinking about exactly?

Mostly I would prohibit sharing of directories between Perl installations,
and even within a single installation, the sharing of directories between
install locations.

E.g. the default configuration right now has $Config{installbin} and
$Config{installsitebin} pointing to the the same directory. This means that
if you install ExtUtils::ParseXS from CPAN, you end up with the new version
of the module in $Config{installsitelib}, but the xsubpp script installed
into $Config{installsitebin} will overwrite the core version already in
$Config{installbin} because they are the same directory.

This means it is now impossible to remove the ExtUtils::ParseXS module from
the "site" install location and reverting to the core version.

Even if you don't care about "delete" functionality in your package
manager, you may still want to preserve the integrity of core install.
Otherwise it is possible that the package manager updates a package it
relies upon itself that breaks the package manager. Then it is impossible
to fix this situation for a regular user without doing a complete reinstall
of Perl itself.

For this reason the ActivePerl package manager explicitly removes the
"site" directories from @INC and only uses the modules originally included
in the distribution.

Cheers,
-Jan

[Leon Timmermans]

Post by Jan Dubois
Mostly I would prohibit sharing of directories between Perl installations,
and even within a single installation, the sharing of directories between
install locations.
E.g. the default configuration right now has $Config{installbin} and
$Config{installsitebin} pointing to the the same directory. This means that
if you install ExtUtils::ParseXS from CPAN, you end up with the new version
of the module in $Config{installsitelib}, but the xsubpp script installed
into $Config{installsitebin} will overwrite the core version already in
$Config{installbin} because they are the same directory.
This means it is now impossible to remove the ExtUtils::ParseXS module from
the "site" install location and reverting to the core version.
Even if you don't care about "delete" functionality in your package manager,
you may still want to preserve the integrity of core install. Otherwise it
is possible that the package manager updates a package it relies upon itself
that breaks the package manager. Then it is impossible to fix this situation
for a regular user without doing a complete reinstall of Perl itself.
For this reason the ActivePerl package manager explicitly removes the "site"
distribution.

I think that would clash with most vendor distributed perls (or at
least it does with both Debian and Red Hat). It would be nice if this
system was instead able to integrate with them instead of them nuking
it to prevent users from doing something stupid.

Leon

[demerphq]

Post by Leon Timmermans

Post by Jan Dubois
Mostly I would prohibit sharing of directories between Perl installations,
and even within a single installation, the sharing of directories between
install locations.
E.g. the default configuration right now has $Config{installbin} and
$Config{installsitebin} pointing to the the same directory. This means that
if you install ExtUtils::ParseXS from CPAN, you end up with the new version
of the module in $Config{installsitelib}, but the xsubpp script installed
into $Config{installsitebin} will overwrite the core version already in
$Config{installbin} because they are the same directory.
This means it is now impossible to remove the ExtUtils::ParseXS module from
the "site" install location and reverting to the core version.
Even if you don't care about "delete" functionality in your package manager,
you may still want to preserve the integrity of core install. Otherwise it
is possible that the package manager updates a package it relies upon itself
that breaks the package manager. Then it is impossible to fix this situation
for a regular user without doing a complete reinstall of Perl itself.
For this reason the ActivePerl package manager explicitly removes the "site"
distribution.

I think that would clash with most vendor distributed perls (or at
least it does with both Debian and Red Hat). It would be nice if this
system was instead able to integrate with them instead of them nuking
it to prevent users from doing something stupid.

FWIW I think that Perl should use one install format and the distros
should not fight that.

IMO a lot of MakeMakers problems come from trying to please too many
people and ending up pleasing no one.

Yves

[Johan Vromans]

A separate install database for each install location seems like the
only workable approach.

Store the complete distribution in a git repository?

-- Johan

[(Johan Vromans)]

[Quoting Philippe Bruhat (BooK), on January 2 2013, 01:23, in "Re: How To Build A P"]

One issue I had when trying to store distributions with
Git::CPAN::Hook is that if a file does not change between two
versions of a distribution, then git won't "detect" it.

This doesn't look like a problem if the approach is to checkout a
particular tag corresponding to a specific install.

-- Johan
http://johan.vromans.org/seasons_greetings.html