Environment Modules Updates

Symbolic version to designate module loaded version

2021-01-05T05:59:00.001Z

When the Advanced module version specifiers is enabled, the loaded symbolic version may be used to designate the currently loaded version of specified module.

$ ml display foo@loaded
-------------------------------------------------------------------
/path/to/modulefiles/foo/1.0:
module-whatis foo/1.0
-------------------------------------------------------------------

If no version of specified module can be found loaded, an error is returned.

$ ml display foo@loaded
ERROR: No loaded version found for 'foo' module

Determining module implementation and version

2020-12-29T09:37:00.001Z

New Modules variables are introduced to determine during the evaluation of a modulefile or a modulerc what module implementation is currently in use. The ModuleTool variable corresponds to the name of the module implementation and is set to Modules for this project. The ModuleToolVersion variable corresponds to the version number of the implementation (e.g. 4.7.0).

With these new variables it is possible to precisely know what module command is in use then adapt modulefile code to handle a specific behavior or leverage a new feature.

The modulefile command versioncmp is also introduced to provide a simple way to compare two version strings and return if first version string is less than, equal to or greater than second one.

if {[info exists ModuleTool] && $ModuleTool eq {Modules}
&& [versioncmp $ModuleToolVersion 4.7] >= 0} {
# here some code specific for Modules 4.7 and later versions
}

The ModuleTool and ModuleToolVersion variables and the versioncmp modulefile command are supported by the Lmod project starting version 8.4.8.

Tracing module execution

2020-09-15T04:21:00.001Z

The trace verbosity is introduced between the verbose and debug levels to report details on module searches, resolutions, selections and evaluations. Trace mode can be enabled by setting the verbosity config to the trace value or by using the -T/--trace command-line switches.

To specifically render trace messages, the tr key is added to the color palette with a default value of 2 (decreased intensity).

$ ml -T foo
Evaluate modulerc: '/path/to/modulefiles/.modulerc'
Get modules: {foo} matching 'foo' in '/path/to/modulefiles'
Resolve: 'foo' into 'bar'
Get modules: {bar bar/1.0} matching 'bar' in '/path/to/modulefiles'
Select module: 'bar/1.0' (/path/to/modulefiles/bar/1.0) matching 'bar/1.0'
Loading bar/1.0
Evaluate modulefile: '/path/to/modulefiles/bar/1.0' as 'bar/1.0'

Forbidding use of modules

2020-09-08T19:04:00.001Z

The module-forbid modulefile command is added to dynamically forbid the evaluation of modulefiles it specifies. When forbidden, a module cannot be loaded and an access error is returned when an attempt is made to evaluate it.

$ cat /path/to/modulefiles/foo/.modulerc
#%Module4.6
module-forbid foo@1:
$ ml foo/1.0
ERROR: Access to module 'foo/1.0' is denied
$ ml
No Modulefiles Currently Loaded.

module-forbid statements can be coupled with :mfcmd:module-hide statements to hide modules in addition to forbid their use. module-forbid supports the --not-user, --not-group, --before and --after options to still allow some users or forbid modules before or after a given date time.

An additional error message can be defined with the --message option to guide for instance users when they try to evaluate a forbidden module:

$ cat /path/to/modulefiles/bar/.modulerc
#%Module4.6
module-forbid --message {Software bar/1.0 is decommissioned, please now use\
bar/2.0} --after 2020-09-01 bar/1.0
$ ml bar/1.0
ERROR: Access to module 'bar/1.0' is denied
Software bar/1.0 is decommissioned, please now use bar/2.0

When an evaluated module will soon be forbidden, a message is returned to the user to warn him/her of the near limit. An additional warning message can also be defined here with the --nearly-message option to guide users.

$ cat /path/to/modulefiles/qux/.modulerc
#%Module4.6
module-forbid --nearly-message {Version 1.0 will soon expire, please now use\
version 2.0} --after 2020-09-15 qux/1.0
$ date
Tue 08 Sep 2020 06:49:43 AM CEST
$ ml qux/1.0
Loading qux/1.0
WARNING: Access to module will be denied starting '2020-09-15'
Version 1.0 will soon expire, please now use version 2.0

The range of time the nearly forbidden warning appears can be controlled with the nearly_forbidden_days configuration option, whose value equals to the number of days prior the module starts to be forbidden. This configuration is set to 14 (days) by default and this value can be controlled at configure time with --with-nearly-forbidden-days option. When the nearly_forbidden_days configuration is set through the config sub-command, the MODULES_NEARLY_FORBIDDEN_DAYS environment variable is set.

Hiding modules

2020-09-03T06:17:00.001Z

The newly introduced module-hide modulefile command enables to dynamically hide modulefiles, module aliases or symbolic versions specified to it:

$ cat /path/to/modulefiles/bar/.modulerc
#%Module4.6
module-version bar/1.0 old
# hide 'old' symbolic version
module-hide bar/old
# hide all version 2 and above
module-hide bar@2:
$ cat /path/to/modulefiles/.modulerc
#%Module4.6
# hide all versions of foo module
module-hide foo

module-hide commands should be placed in module rc files and can leverage the Advanced module version specifiers syntax as shown in the above example.

Hidden modules are excluded from available module search or module selection unless query refers to hidden module by its exact name:

$ ml av
--------------- /path/to/modulefiles ---------------
bar/1.0 bar/2.0
$ module load -v foo
ERROR: Unable to locate a modulefile for 'foo'
$ module load -v foo/1.0
Loading foo/1.0
$ module avail bar/old
--------------- /path/to/modulefiles ---------------
bar/1.0(old)

module-hide command accepts a --soft option to apply a lighter of hiding to modules:

$ cat /path/to/modulefiles/qux/.modulerc
#%Module4.6
# softly hide all qux modules
module-hide --soft qux

The soft hiding mode enables to hide modules from full availability listing yet keeping the ability to select such module for load without having to use module exact name:

$ ml av
--------------- /path/to/modulefiles ---------------
bar/1.0 bar/2.0
$ ml av qux
--------------- /path/to/modulefiles ---------------
qux/1.0 qux/2.0
$ module load -v qux
Loading qux/2.0

Alternatively, a --hard option can be set on module-hide command to ensure designated modules do not unveil even if referred by their exact name:

$ cat /path/to/modulefiles/qux/.modulerc
#%Module4.6
# softly hide all qux modules
module-hide --soft qux
# set highest version of qux hard hidden
module-hide --hard qux/3.0
$ ml av qux/3.0
$ ml qux/3.0
ERROR: Unable to locate a modulefile for 'qux/3.0'

Some users or groups can be set unaffected by hiding mechanism with the --not-user or --not-group options:

$ cat /path/to/modulefiles/quuz/.modulerc
#%Module4.6
# hiding does not apply to grp1 and grp2 groups
module-hide --not-group {grp1 grp2} quuz
$ id --groups --name
grp1 grp7
$ ml av quuz
--------------- /path/to/modulefiles ---------------
quuz/1.0 quuz/2.0
$ ml -v quuz
Loading quuz/2.0

Hiding mechanism can also be set effective only before or after a given date time with the --before and --after options. Accepted date time format is YYYY-MM-DD[THH:MM].

$ cat /path/to/modulefiles/fum/.modulerc
#%Module4.6
# hide only before a given date
module-hide --hard --before 2020-09-01T12:00 fum/1.0
# hide only after a given date
module-hide --hard --after 2020-09-01 fum/2.0
$ date
Fri 04 Sep 2020 06:21:48 AM CEST
$ ml av fum
--------------- /path/to/modulefiles ---------------
fum/1.0

Hidden modules can be included in available module searches if option --all/-a is set on avail, aliases, whatis or search sub-commands. Hard hidden modules are unaffected by this option and stay hidden.

$ ml av -a
--------------- /path/to/modulefiles ---------------
bar/1.0(old) foo/1.0 fum/1.0 quuz/2.0 qux/2.0
bar/2.0 foo/2.0 quuz/1.0 qux/1.0

Querying user's name and groups membership

2020-07-24T12:52:00.001Z

Two new sub-commands are introduced for the module-info modulefile command: username and usergroups. They respectively fetch the name of the user currently running modulecmd.tcl or the name of all the groups this user is member of.

These two new modulefile commands can help to adapt code to specific users or groups. Like for instance to instantiate a modulefile for each group the user is member of:

$ cat /path/to/modulefiles/foo/.modulerc
#%Module
foreach grp [module-info usergroups] {
module-virtual foo/$grp .common
}
$ id -G -n
grp1 grp2 grp3
$ module avail
--------------- /path/to/modulefiles ---------------
foo/grp1 foo/grp2 foo/grp3

username and usergroups sub-commands of module-info modulefile command are only supported on Unix platform.

Embed shell scripts in modulefiles

2020-07-16T15:09:00.001Z

The source-sh modulefile command is introduced to source environment changes done by the evaluation of a shell script passed as argument. With newly introduced sh-to-mod sub-command resulting environment changes done by script are output as modulefile commands. source-sh applies those modulefile commands as if they were directly written in loading modulefile.

source-sh is useful for software providing a shell script for their enablement. If you want to enable such software with module yet using shell script provided by software for this task, just write a modulefile using source-sh command to call the shell script.

Keeping the same example used to describe sh-to-mod sub-command: foo software provides a foo-setup.sh script for its activation. Create a modulefile foo/1.2 that calls this script:

$ cat /path/to/modulefiles/foo/1.2
#%Module
source-sh sh /path/to/foo-1.2/foo-setup.sh arg1

Displaying this modulefile indicates the environment changes done by script:

$ module display foo/1.2
-------------------------------------------------------------------
/path/to/modulefiles/foo/1.2:
prepend-path PATH /path/to/foo-1.2/bin
set-alias foo {foobin -q -l}
setenv FOOENV arg1
-------------------------------------------------------------------

Loading the modulefile applies the environment changes seen above:

$ module load -v foo/1.2
Loading foo/1.2
$ echo $FOOENV
arg1
$ alias foo
alias foo='foobin -q -l'

Track of these changes is kept in user environment to be able to undo them when modulefile is unloaded:

$ module unload -v foo/1.2
Unloading foo/1.2
$ echo $FOOENV
$ alias foo
bash: alias: foo: not found

Changes on environment variables, shell aliases, shell functions and current working directory are tracked. The following shells are supported: sh, dash, csh, tcsh, bash, ksh, ksh93, zsh and fish.

Translating shell scripts into modulefiles

2020-07-07T13:17:00.001Z

The sh-to-mod sub-command is added to output as a modulefile content the environment changes done by the evaluation of a shell script passed as argument. sh-to-mod is especially useful for software providing a shell script for their enablement in shell session: it can convert these scripts into modulefiles.

Say for instance, a foo software has been installed and it provides a foo-setup.sh script to activate foo software in user environment:

$ cat /path/to/foo-1.2/foo-setup.sh
#!/bin/sh
export FOOENV="$1"
export PATH=/path/to/foo-1.2/bin:$PATH
alias foo='foobin -q -l'

Calling module sh-to-mod on this shell script outputs the environment changes it performs as a modulefile content:

$ module sh-to-mod sh /path/to/foo-1.2/foo-setup.sh arg1
#%Module
prepend-path PATH /path/to/foo-1.2/bin
set-alias foo {foobin -q -l}
setenv FOOENV arg1

sh-to-mod acts as a full replacement for the standalone createmodule.sh and createmodule.py scripts. However those two scripts are currently still provided for compatibility purpose.

This new feature is available in git repository and will be included in next release (v4.6).

Automatic default and latest symbolic versions

2020-04-07T07:39:00.001Z

When the implicit default mechanism and the advanced module version specifiers are both enabled, a default and a latest symbolic versions are automatically defined for each module name.

This new feature gives the ability to select the highest version available for a module, without knowing beforehand this version name:

$ module load -v foo@latest
Loading foo/1.10

The symbolic versions are automatically defined unless a symbolic version, an alias or a regular module version already exists for these default or latest version names.

Error stack trace

2020-03-31T13:20:00.001Z

Error messages will now embed a stack trace for unknown errors to help localize the root cause of issues. This change applies to modulefile evaluation:

Loading foo/1.2
Module ERROR: add-path cannot handle path equals to separator string
while executing
"append-path PATH :"
(file "/path/to/modulefiles/foo/1.2" line 24)
Please contact <root@localhost>

A stack trace is also returned when an unknown error occurs in modulecmd.tcl script, which facilitates issue report and analysis:

$ module load bar
ERROR: invalid command name "badcommand"
while executing
"badcommand"
(procedure "module" line 14)
invoked from within
"module load bar"
("eval" body line 1)
invoked from within
"eval $execcmdlist"
Please report this issue at https://github.com/cea-hpc/modules/issues

This new feature is available in git repository and will be included into the upcoming v4.5 release.