The aim is to be able to replicate Perl environments as a provision for a robust Development - Test - Release process, where versions of modules through this process are kept consistent.
Ingredients:
- Perlbrew
- Subversion repository
- minicpan_webserver
- dpan (for legacy environments)
Recipe (fresh environment)
- Install new Perl via Perlbrew
- Setup the repository to store local environments (scripts/new_mirror.sh)
- Install the modules for development (configuring CPAN to download into correct directory)
- Once happy: check in the environment
- Create a bundle file (check in)
- Start CPAN webserver - to distribute modules
- Check out bundle file on client
- Point client CPANPLUS to local mirror install bundle!
Recipe (Convert existing environment)
- Find all tar.gz modules files that have been installed
- Create a new environment to store mirror in (scripts/new_mirror.sh file)
- Copy all the tars associated to the old environment into the cpan-mirrors/local/envs/ENVNAME/trunk/repos/new directory
- Use dpan to create CPAN mirror (may need to manipulate authors file to add DPAN user and where dpan fails may need to fake success file)
Perlbrew
The idea is to run the Perl applications in their own area. Each time we wish to create a new environment we use a new Perl installation (Q1 - how to run multiple perlbrews on a single machine).
Perlbrew provides a useful way of managing perl installations
Download and run:
curl -kL http://install.perlbrew.pl | bash
This will install perlbrew in $HOME/perl5, to install a new version of perl under$HOME/perl5 list the versions available:
$HOME/perl5/perlbrew/bin/perlbrew available
then install the one you wish to use (even tenths are stable). Once installed switch to this version in teh current shell by:
$HOME/perl5/perlbrew/bin/perlbrew switch
You can set your shell up to use this version by default each time you log in by adding the following line to your ~/.bashrc file:
source ~/perl/perlbrew/etc/bashrc
Subversion repository (or other revision system)
I use
subversion as it is the standard where I work at the moment, we are a small team and haven't seen enough reason to switch allegiances to another system as yet.
The environments will be stored in a revision system to provide a mechanism to track use and make changes. The following outlines the repository used to store the environment versions:
Create a root repository 'cpan-mirrors' $SOMEWHERE on your system under which there will be two directories:
cd $SOMEWHERE
mkdir cpan-mirrors/local (to store the local enivronments)
mkdir cpan-mirrors/sync (to store the upto date mirror)
The repository 'may' contain a full cpan mirror. This full mirror is synchronised using minicpan whether checked in or not, create a directory under cpam-mirrors/sync to store the full mirror (i'll choose cpan/trunk/repos as there may be a need for different versions, other files associated to it etc), and a directory for some scripts (I choose cpan/trunk/scripts)
cd $SOMEWHERE
mkdir cpan-mirrors/sync/cpan/trunk/repos
mkdir cpan-mirrors/sync/cpan/trunk/scripts
Synchronise the upto date mirror and optionally check the mirror in (if you check in this may take some time!)
create a minicpan_cpan_repos file in sync/cpan/trunk/scripts, this will configure how the full cpan is synchronised
minicpan_cpan_repos:
remote: <your favourite mirror goes here>
local: <somehwere directory here>/cpan-mirrors/sync/cpan/trunk/repos
skip_perl: 1
exact_mirror: 1
ignore_source_control: 1
It's handy to a have a script that can run the synchronisation, create such a script (sync_cpan.sh):
sync_cpan.sh
#!/bin/bash
export PERLBREW_ROOT=${HOME}/perl5/perlbrew
export PERLBREW_HOME=/tmp/.perlbrew
source {PERLBREW_ROOT}/etc/bashrc
SOMEWHERE=<Your value for SOMEWHERE>
#update thi sline with your installed perl version
INSTALLED_PERL_VERSION=5.14.2
CPAN_MIRRORS=${SOMEWHERE}/sync/cpan
cd ${PERLBREW_ROOT}/bin
./perlbrew use ${INSTALLED_PERL_VERSION}
cd ${PERLBREW_ROOT}/scripts
./minicpan -C ${CPAN_MIRRORS}/trunk/scripts/minicpan_cpna_repos
cd ${CPAN_MIRRORS}/trunk
svn commit -m"commited latest version" repos
Create the boiler plate for the environment
This will be used to create a new environment tree via a script
In your repository tree under cpan-mirrors/local create a directory called TEMPLATE_ENV then underneath this directory create a number of subdirectories:
TEMPLATE_ENV
TEMPLATE_ENV/branches
TEMPLATE_ENV/trunk
TEMPLATE_ENV/trunk/config
TEMPLATE_ENV/trunk/config/scripts
TEMPLATE_ENV/trunk/config/cpan_home
TEMPLATE_ENV/trunk/repos
TEMPLATE_ENV/trunk/repos/new
In the TEMPLATE_ENV/trunk/config directory create some configuration files:
1. a CPAN configuration file used when downloading new CPAN modules whilst working on new features in development (TEMPLATE_ENV_CPAN_update.pm)
TEMPLATE_ENV_CPAN_update.pm
my $envname="TEMPLATE_ENV";
my $home_dir = '
';
my $ROOT_REPOS="$homedir/perl5/mirrors/cpan-mirrors"; # or where your SOMEWHERE is located
my $cpan_home_dir="$ROOT_REPOS/local/envs/$envname/trunk/config/cpan_home";
$CPAN::Config = {
"cpan_home" => $cpan_home_dir,
"version_timeout" => "15",
"show_unparsable_versions" => "0",
"makepl_arg" => "install_base=",
"histfile" => "/$home_dir/.cpan/histfile",
"unzip" => "/usr/bin/unzip",
"show_upload_date" => "0",
"mbuild_install_build_command" => "./Build",
"yaml_load_code" => "0",
"urllist" => [
"http://host.of.your.full.cpanmirror.org:2963"
],
"trust_test_report_history" => "0",
"gzip" => "/usr/bin/gzip",
"keep_source_where" => "$ROOT_REPOS/local/envs/$envname/trunk/repos/new",
"yaml_module" => "YAML",
"prefer_installer" => "MB",
"build_requires_install_policy" => "yes",
"connect_to_internet_ok" => "0",
"getcwd" => "cwd",
"prefer_external_tar" => "1",
"make_install_make_command" => "/usr/bin/make",
"no_proxy" => "",
"build_cache" => "100",
"make_arg" => "",
"wget" => "/usr/bin/wget",
"auto_commit" => "0",
"patch" => "/usr/bin/patch",
"ftp_proxy" => "",
"ftp_passive" => "1",
"tar" => "/bin/tar",
"inactivity_timeout" => "0",
"use_sqlite" => "0",
"scan_cache" => "atstart",
"mbuildpl_arg" => "--install_base=",
"halt_on_failure" => "1",
"cache_metadata" => "0",
"show_zero_versions" => "0",
"term_ornaments" => "1",
"prefs_dir" => "$home_dir/.cpan/prefs",
"build_dir_reuse" => "0",
"shell" => "/bin/bash",
"prerequisites_policy" => "follow",
"perl5lib_verbosity" => "none",
"make" => "/usr/bin/make",
"gpg" => "/usr/bin/gpg",
"mbuild_arg" => "",
"applypatch" => "",
"inhibit_startup_message" => "0",
"load_module_verbosity" => "none",
"mbuild_install_arg" => "",
"commandnumber_in_prompt" => "1",
"check_sigs" => "0",
"build_dir" => "$home_dir/.cpan/build",
"index_expire" => "1",
"bzip2" => "/usr/bin/bzip2",
"test_report" => "0",
"tar_verbosity" => "none",
"pager" => "less",
"term_is_latin" => "1",
"make_install_arg" => "",
"histsize" => "100",
"http_proxy" => ""
};
1;
__END__
As some of the paths to the various applications (wget gzip etc) may not be the same on your system. You can generate this file from your installed Perl by the command:
cpan -J
You can then edit this file changing the values for:
- cpan_home
- keep_source_where
- urllist(you may wish to change the port that the full mirror is served on)
if sharing between users you may wish to substitute the home directory location with a perl variable. Once generated save in TEMPLATE_ENV/trunk/config/scripts as TEMPLATE_ENV_CPAN_update.pm
2, A CPANPLUS configuration fileused when to release the changes to the environment (TEMPLATE_ENV_release.pm)
my $home_dir = "
"; #insert your home drectory
my $envname = "TEMPLATE_ENV";
$CPAN::Config = {
"cpan_home" => "$home_dir/.cpan/$envname",
"version_timeout" => "15",
"show_unparsable_versions" => "0",
"makepl_arg" => "install_base=",
"histfile" => "$home_dir/.cpan/$envname/histfile",
"unzip" => "/usr/bin/unzip",
"show_upload_date" => "0",
"mbuild_install_build_command" => "./Build",
"yaml_load_code" => "0",
"urllist" => [
"http://minicpan.host.somewhere.org:MYPORT"
],
"trust_test_report_history" => "0",
"gzip" => "/usr/bin/gzip",
"keep_source_where" => "$home_dir/.cpan/$envname/sources",
"yaml_module" => "YAML",
"prefer_installer" => "MB",
"build_requires_install_policy" => "yes",
"connect_to_internet_ok" => "0",
"getcwd" => "cwd",
"prefer_external_tar" => "1",
"make_install_make_command" => "/usr/bin/make",
"no_proxy" => "",
"build_cache" => "100",
"make_arg" => "",
"wget" => "/usr/bin/wget",
"auto_commit" => "0",
"patch" => "/usr/bin/patch",
"ftp_proxy" => "",
"ftp_passive" => "1",
"tar" => "/bin/tar",
"inactivity_timeout" => "0",
"use_sqlite" => "0",
"scan_cache" => "atstart",
"mbuildpl_arg" => "--install_base=",
"halt_on_failure" => "1",
"cache_metadata" => "0",
"show_zero_versions" => "0",
"term_ornaments" => "1",
"prefs_dir" => "$home_dir/.cpan/$envname/prefs",
"build_dir_reuse" => "0",
"shell" => "/bin/bash",
"prerequisites_policy" => "follow",
"perl5lib_verbosity" => "none",
"make" => "/usr/bin/make",
"gpg" => "/usr/bin/gpg",
"mbuild_arg" => "",
"applypatch" => "",
"inhibit_startup_message" => "0",
"load_module_verbosity" => "none",
"mbuild_install_arg" => "",
"commandnumber_in_prompt" => "1",
"check_sigs" => "0",
"build_dir" => "$home_dir/.cpan/$envname/build",
"index_expire" => "1",
"bzip2" => "/usr/bin/bzip2",
"test_report" => "0",
"tar_verbosity" => "none",
"pager" => "less",
"term_is_latin" => "1",
"make_install_arg" => "",
"histsize" => "100",
"http_proxy" => ""
};
1;
__END__
Again this file can be created by:
cpan -J
and edited to provide the correct:
urllist (the server where all the minicpans will sit)
a unique port needs to be given for each environment so you may wish to modify this value (still to do configuring the minicpan_webserver)
3, The last config file to create is the one used by minicpan_webserver to make it think that it is providing a cpan mirror (minicpan_TEMPLATE_ENV):
miniccpan_TEMPLATE_ENV:
#This file is a fake to handle the minicpan_webserver usage (hopefully just needs local: set
remote: http://ignore_mirror.ox.ac.uk/sites/www.cpan.org/
local:
/cpan-mirrors/local/envs/TEMPLATE_ENV/trunk/repos/new
skip_perl: 1
exact_mirror: 1
ignore_source_control: 1
4, Create a script (cpan_mirrors.sh) in the config/scripts director that
will start the minicpan_webervers on the correct port (the port you
selected when creating the environment from the boiler plate)
cpan_mirrors.sh:
#!/bin/bash
#Script to start/stop the cpan mirror webservers:
#At the moment I think that we can only run one webservera at a time unless we figure out how to change the cache directory
#see note on cpan 'cache_dir:/tmp/your/cache/dir' for minicpan_webserver (edit/temporarily manipulate a .minicpanrc file)
MYDIR=`dirname "$0"`
cd ${MYDIR}
CPAN_MINI_CONFIG=${MYDIR}/../minicpan_TEMPLATE_ENV minicpan_webserver -p MYPORT
Scripts to create a new environment from the template (boiler plate)
Now that we have a boiler plate we now create a script that will use it to setup a new environment. essentially it will take a copy of the TEMPLATE_ENV tree and rename under the cpan-mirrors/local/envs directory.
In the cpan-mirrors/local/scripts directory create a place holder script to start the minicpan_webservers, called
webservers_start.sh with a single line:
webservers_start.sh:
#!/bin/bash
In the cpan-mirrors/local/scripts directory create a file called:
new_mirror.sh:
#!/bin/bash
#script will create a mirror for a new environment
if [ "$#" != 2 ]
then
echo "usage: $0
"
exit
fi
MYDIR1=`dirname "$0"`
CURR_DIR=`pwd`
MYDIR=${CURR_DIR}/${MYDIR1}
echo Home:" "${HOME}
echo MYDIR: ${MYDIR}
#exit;
cp -r ${MYDIR}/../envs/TEMPLATE_ENV/ ${MYDIR}/../envs/$1
find ${MYDIR}/../envs/$1 -name '*TEMPLATE_ENV*' -exec rename TEMPLATE_ENV $1 {} \;
find ${MYDIR}/../envs/$1 -type f -exec sed -i "s/TEMPLATE_ENV/$1/g" {} \;
find ${MYDIR}/../envs/$1 -type f -exec sed -i "s#MIRRORS_DIR#${MYDIR}/\.\./\.\./#g" {} \;
find ${MYDIR}/../envs/$1 -type f -exec sed -i "s/MYPORT/$2/g" {} \;
chmod 755 ${MYDIR}/../envs/$1/trunk/config/scripts/webserver_start.sh
chmod 755 ${MYDIR}/webservers_start.sh
echo ${MYDIR}/../envs/$1/trunk/config/scripts/webserver_start.sh >>
echo "your new envirnment can be found at: ${MYDIR}/../../$1" ${MYDIR}/webservers_start.sh
echo ""
cat ${MYDIR1}/README.txt
Also in this directory crate the README.txt file:
README.txt:
To use make sure that the cpan branch/trunk has been checked out locally on the development machine.
Populate your environment with Perl modules by installing them using:
cpan -j < path to file env_name_update.pm > Packages
(where the cpan is th ecpan of the environment you are updating)
To install these modules
1, check the new versions / packages in
2, create the environments snap shot (on dev machine) cpan -a -j < path to branch | trunk conf scripts env_name_update.pm >
3, check snap shot in and distribute (move to TEMPLATE_ENV/trunk/config/ and check in)
4, Check out updated environment on the cpan mirror host and restart the minicpan_webserver
The on the client machine to upgrade:
1, download Snapshot
2, setup the cpanplus configuration:
perl -MCPANPLUS -e'my $conf=CPANPLUS::Configure->new(); $conf->set_conf(prereqs=>1,hosts=>[{"schema"=>"http","path"=>"/","host"=>"host.for.environments.minicpan:<correctport>"}]); $conf->save();'
3, install the downloaded snap shot:
cpanp -i <snapshot - absolute file location>
4, it may also be worth setting prereqs=1 in the cpanplus config!
Providing access to your minicpan mirrors
These are the versions of the Perl modules associated to each environment.
This is done using minicpan_webserver. On the host that will be serving all the mirrors checkout the entire cpan-mirrors repository (note that this must be checked out into the original root directory ($SOMEWHERE) so that all the hard coded paths in our scripts are correct.
On the machine you wish to serve the cpan mirrors install minicpan_webserver. Note that it is probably best not to use one of the Perl installations you are using for a development environment. As this isn't a development environment you can install minicpan_webserver from a live mirror.
Setting up and using a new environment
Assuming that you have Perl installed and repository tree setup checkout your local repository tree into the $SOMEWHERE directory, so that paths are consistent where it was originally created.
cd to the scripts directory in local/scripts and run ./new_mirror.sh
This will create a boiler plate environment.
Before you install any Perl packages you must configure your CPAN so that the downloaded files are placed in your new repository to do this
cpan -j path to conf/scripts/env_name_update.pm Packages_list_to_install
By doing this the required packages will be downloaded into your development environment cpan. Once you are happy with your changes and ready to release you can svn diff the cpan-mirrors/local/envs/YOURENV tree to find the new modules you installed. Create a new bundle file to distribute. Check these in, and then on the host for the minicpan_webserver update the cpan environments and restart the webservers ready for the live checkout.
to checkout into the live location configure your CPANPLUS to point to the correct host and port, set the prereqs (its' also an idea to move the ~/.cpanplus file to force a new download of the authors, packages and modules file from this mirror) and install the bundle.
Convert existing environment
- Create a new environment using new_mirror.sh script.
- Find all the original perl tar.gz copy these into your new environments trunk/repos/new directory
- Install from CPAN dpan
- cd to the trunk/repos/new directory and run dpan
Notes: There may be some packages that are not indexed what dpan does is to unpack the source and attempt to find from the file content the package names that are provided. If you know the package names that are provided then you can 'spoof' a sucess file and on running dpan again they will be added to the packages.tar.gz file. For the mirror to also work you may need to add the users you specified in your dpan_conf file to the authors.tar.gz file (by default DPAN).
You should now hopefully be able to treat this mirror like any other
