[Neurodebian-devel] [RFC] Should we convert to use 'Modules' (http://modules.sourceforge.net/)?

[Alex Waite]

Hey Yarik,

As a result of reading this RFC, I was inspired to convert our internal 
Freesurfer (5.1 and 5.3) packages to environment modules (envmod). I'm 
still new to the framework, but here are my (early) thoughts.

On 12/12/2013 05:30 PM, Yaroslav Halchenko wrote:
 > [snip]
 > [Modules] is being used on many systems, especially in HPC setups

It looks like everyone with a serious HPC is using this. The 
SetUpFreeSurfer and FreeSurferEnv scripts are such a tangled mess that I 
ended up contacting two different HPC IT groups for their Freesurfer 
modfiles -- as a starting point. Both responded within a day with the 
files. From this experience alone, I perceive there to be good potential 
support out there.

 > [snip]
> I was thinking that we should start use 'modules' in place of our ad-hoc
> shell (primarily bash) scripts.

The sooner these scripts die, the better IMO.

> [snip]
 > I have already
> built backports for all releases (only not for ubuntu 10.04
 > [snip]

Was it problematic to build on 10.04 or was it because it's EOLed? In my 
experience, 10.04 was the last sane release by the Ubuntu folks and I 
wouldn't be surprised if it become the Windows XP of the Ubuntu world. 
Do we have stats on what percentage of Neurodebian users are on 10.04?

> a. within a package for tool X ship following files under
>     /usr/share/modules/modulefiles/ directory:
>
>    1.
>     X/UVERSION

This is sane and seems to be the standard.

>    2.
>     X/.version
>
>     file specifying the default version (similar to our current
>     /etc/fsl/fsl.sh -> 5.0/fsl.sh)

 From what I've read, the X/default file is looked for first, then 
X/.version. The default file can merely be a symlink to the default 
version modfile while the .version file is a two line file setting the 
default version number.

In my opinion, it's slightly easier, more intuitive, and less error 
prone to go the default symlink route rather than the .version file --- 
especially if we ever have a need to modify it via a postinst script. 
However, both are so dead easy that I don't have a strong preference.

>    3.
>     For packages for which we maintain multiple major releases (e.g. FSL 4.1 and
>     5.0), provide symlinks to the supported release line (similar to dynamic
>     libraries setup). e.g. if for ants we decided to support both old 1.9
>     and new 2.0: 2.0 -> 2.0.0,   1.9 -> 1.9.8
>
>     this way users could explicitly load a specific 'release line', e.g.
>
>      module load ants/1.9 (or module load fsl/5.0)

Isn't this what point 1 is already doing? The only thing I see different 
is having a link from fsl/5.1 to fsl/5.1.1. But I don't see the point of 
doing that unless we plan to have both fsl 5.1.0 and fsl 5.1.1 installed 
at the same time (which I don't think you're suggesting). I think we can 
just keep it as fsl/5.1 for all bugfix releases of fsl 5.1.

> b. administrators would be welcome to add custom modulefiles
>     pointing to local installations, happen they have them, and place them under
>
>      /etc/environment-modules/modules
>
>     By doing so they might 'overload' definition of the default
>     version specified in /usr/share/modules/modulefiles/X/.version thus it
>     would be up to them either symlink that .version file or just have
>     their possibly newer version being default

This makes sense. An additional option is to allow the admin to select 
the default version via a dialog at install.

> c. we then could replace custom /etc/X/X.sh scripts with simple 'module load X'
>     (versioned or not), to avoid any duplication of the setup while staying compatible
>     with existing deployments.

Yes, this should work. Do other versions have to be explicitly unloaded 
first, or is that taken care of automatically by envmod?

>     The only catch here which I do not know how to resolve yet
>     without interaction with users is:
>
>     $> tail -n 3 /etc/fsl/fsl.sh /etc/afni/afni.sh
>      ==> /etc/fsl/fsl.sh <==
>      if [ -f "${HOME}/.fslconf/fsl.sh" ] ; then
>        . "${HOME}/.fslconf/fsl.sh" ;
>      fi
>
>      ==> /etc/afni/afni.sh <==
>      if [ -f "${HOME}/.config/AFNI/afni.sh" ] ; then
>        . "${HOME}/.config/AFNI/afni.sh" ;
>      fi
>
>     Since these are shell-specific they can't be just "sourced" anyhow within
>     modulefile.  It should be possible to check and warn users though if such a
>     file was found present, and advise to convert it to modulefiles but I am not
>     yet sure on how easy that would be todo/enable.

The envmod website does mention the env2 project 
(http://sourceforge.net/projects/env2/) which helps convert environment 
variables across shell languages. It could be a start.

>     There can be ~/.modulerc file providing generic tune up, e.g.
>
>      $> cat ~/.modulerc
>      #%Module1.0#######################################################################
>      module-version ants/1.9 default
>
>     would specify that I want to use 1.9 regardless of the system-wide
>     defaults.  Thus question:
>
>     Q: How we could enable per user per module specific settings?
>     If someone had experience with Modules -- I would be great full for
>     your help.

I think this is what `module load use.own` is for. Though I am not sure 
if it as as simple as appending/overwriting a single variable. I think 
the entire modfile needs to be duplicated, though I am not certain.

Thanks for finding envmod. It looks to be far superior to the current 
method.

---Alex