Environment Modules Updates

Modules 4.6.1 released

2020-11-14T20:30:00.001Z

Modules 4.6.1 is out. This release brings a significant performance improvement for the dynamic hiding/forbidding feature and several other fixes.

The changes introduced in this bugfix release are:

Lib: implement initStateClockSeconds as a Tcl command in libtclenvmodules to provide an optimized way to retrieve current Epoch time.
Lib: implement parseDateTimeArg as a Tcl command in libtclenvmodules to provide an optimized way to convert a datetime string into an Epoch time.
When full module specification is equal to @, raise an error as no module name is provided. (fix issue #362)
Optimize internal recording of hidden module and tag specification when parsing modulerc files in order to reduce the time taken to test if a given module is hidden or if a given tag applies to it.
Script: add the ability to select the benchmark test to perform on mb utility.
Doc: add Use new features without breaking old module command cookbook recipe
Doc: rework option description for module-hide and module-forbid commands in modulefile(4) document.
Doc: describe in diff_v3_v4 document that shell special characters like backticks are escaped when used in values starting Modules 4.0. (fix issue #365)
Doc: make the ENVIRONMENT section from modulefile(4) man page point to the ENVIRONMENT section of module(1) man page.
Fix clear sub-command to unset the MODULES_LMSOURCESH environment variable. (fix issue #367)
Correctly return on avail sub-command a symbolic version defined in a global RC file when specifically searched. (fix issue #368)
Fix module hiding resolution for symbolic versions defined in a global RC file when module-hide statements are set in the modulepath where the modulefiles targeted by these symbols are located. (fix issue #369)
When a module fails to unload during a purge sub-command, preserve loaded the modules it requires to keep environment consistent. (fix issue #370)
Doc: add Hide and forbid modules cookbook recipe.

The list of all changes made in this new version are described in the NEWS document. Details on the new features introduced in the 4.6 series are provided in the MIGRATING document.

http://modules.readthedocs.io/en/stable/MIGRATING.html
http://modules.readthedocs.io/en/stable/NEWS.html

The tarball of this new version can be downloaded at:

http://downloads.sourceforge.net/modules/modules-4.6.1.tar.gz

The zipball to install this new version on Windows platform can be downloaded at:

http://downloads.sourceforge.net/modules/modules-4.6.1-win.zip

If you encounter any issue, please let us know by creating a ticket on the project bug tracker at:

https://github.com/cea-hpc/modules/issues

Special thanks to to all the people who reported issues.

Modules 4.6.0 released

2020-09-16T05:04:00.001Z

Modules 4.6.0 is here with many new stuff to handle the life cycle of your modulefiles. It is as always backward-compatible with previous version 4 releases and primarily adds features and fixes bugs. Major new features introduced in v4.6.0 since v4.5 are:

Details on these features included in this new release are provided in the MIGRATING document. The list of all changes made in this new version are described in the NEWS document.

The tarball of this new version can be downloaded at:

http://downloads.sourceforge.net/modules/modules-4.6.0.tar.gz

The zipball to install this new version on Windows platform can be downloaded at:

http://downloads.sourceforge.net/modules/modules-4.6.0-win.zip

If you encounter any issue, please let us know by creating a ticket on the project bug tracker at:

https://github.com/cea-hpc/modules/issues

Many thanks also to all of you who improve Modules by reporting bugs or sharing new ideas.

Tracing module execution

2020-09-15T04:21:00.001Z

The trace verbosity is introduced between the verbose and debug levels to report details on module searches, resolutions, selections and evaluations. Trace mode can be enabled by setting the verbosity config to the trace value or by using the -T/--trace command-line switches.

To specifically render trace messages, the tr key is added to the color palette with a default value of 2 (decreased intensity).

$ ml -T foo
Evaluate modulerc: '/path/to/modulefiles/.modulerc'
Get modules: {foo} matching 'foo' in '/path/to/modulefiles'
Resolve: 'foo' into 'bar'
Get modules: {bar bar/1.0} matching 'bar' in '/path/to/modulefiles'
Select module: 'bar/1.0' (/path/to/modulefiles/bar/1.0) matching 'bar/1.0'
Loading bar/1.0
Evaluate modulefile: '/path/to/modulefiles/bar/1.0' as 'bar/1.0'

Forbidding use of modules

2020-09-08T19:04:00.001Z

The module-forbid modulefile command is added to dynamically forbid the evaluation of modulefiles it specifies. When forbidden, a module cannot be loaded and an access error is returned when an attempt is made to evaluate it.

$ cat /path/to/modulefiles/foo/.modulerc
#%Module4.6
module-forbid foo@1:
$ ml foo/1.0
ERROR: Access to module 'foo/1.0' is denied
$ ml
No Modulefiles Currently Loaded.

module-forbid statements can be coupled with :mfcmd:module-hide statements to hide modules in addition to forbid their use. module-forbid supports the --not-user, --not-group, --before and --after options to still allow some users or forbid modules before or after a given date time.

An additional error message can be defined with the --message option to guide for instance users when they try to evaluate a forbidden module:

$ cat /path/to/modulefiles/bar/.modulerc
#%Module4.6
module-forbid --message {Software bar/1.0 is decommissioned, please now use\
bar/2.0} --after 2020-09-01 bar/1.0
$ ml bar/1.0
ERROR: Access to module 'bar/1.0' is denied
Software bar/1.0 is decommissioned, please now use bar/2.0

When an evaluated module will soon be forbidden, a message is returned to the user to warn him/her of the near limit. An additional warning message can also be defined here with the --nearly-message option to guide users.

$ cat /path/to/modulefiles/qux/.modulerc
#%Module4.6
module-forbid --nearly-message {Version 1.0 will soon expire, please now use\
version 2.0} --after 2020-09-15 qux/1.0
$ date
Tue 08 Sep 2020 06:49:43 AM CEST
$ ml qux/1.0
Loading qux/1.0
WARNING: Access to module will be denied starting '2020-09-15'
Version 1.0 will soon expire, please now use version 2.0

The range of time the nearly forbidden warning appears can be controlled with the nearly_forbidden_days configuration option, whose value equals to the number of days prior the module starts to be forbidden. This configuration is set to 14 (days) by default and this value can be controlled at configure time with --with-nearly-forbidden-days option. When the nearly_forbidden_days configuration is set through the config sub-command, the MODULES_NEARLY_FORBIDDEN_DAYS environment variable is set.

Hiding modules

2020-09-03T06:17:00.001Z

The newly introduced module-hide modulefile command enables to dynamically hide modulefiles, module aliases or symbolic versions specified to it:

$ cat /path/to/modulefiles/bar/.modulerc
#%Module4.6
module-version bar/1.0 old
# hide 'old' symbolic version
module-hide bar/old
# hide all version 2 and above
module-hide bar@2:
$ cat /path/to/modulefiles/.modulerc
#%Module4.6
# hide all versions of foo module
module-hide foo

module-hide commands should be placed in module rc files and can leverage the Advanced module version specifiers syntax as shown in the above example.

Hidden modules are excluded from available module search or module selection unless query refers to hidden module by its exact name:

$ ml av
--------------- /path/to/modulefiles ---------------
bar/1.0 bar/2.0
$ module load -v foo
ERROR: Unable to locate a modulefile for 'foo'
$ module load -v foo/1.0
Loading foo/1.0
$ module avail bar/old
--------------- /path/to/modulefiles ---------------
bar/1.0(old)

module-hide command accepts a --soft option to apply a lighter of hiding to modules:

$ cat /path/to/modulefiles/qux/.modulerc
#%Module4.6
# softly hide all qux modules
module-hide --soft qux

The soft hiding mode enables to hide modules from full availability listing yet keeping the ability to select such module for load without having to use module exact name:

$ ml av
--------------- /path/to/modulefiles ---------------
bar/1.0 bar/2.0
$ ml av qux
--------------- /path/to/modulefiles ---------------
qux/1.0 qux/2.0
$ module load -v qux
Loading qux/2.0

Alternatively, a --hard option can be set on module-hide command to ensure designated modules do not unveil even if referred by their exact name:

$ cat /path/to/modulefiles/qux/.modulerc
#%Module4.6
# softly hide all qux modules
module-hide --soft qux
# set highest version of qux hard hidden
module-hide --hard qux/3.0
$ ml av qux/3.0
$ ml qux/3.0
ERROR: Unable to locate a modulefile for 'qux/3.0'

Some users or groups can be set unaffected by hiding mechanism with the --not-user or --not-group options:

$ cat /path/to/modulefiles/quuz/.modulerc
#%Module4.6
# hiding does not apply to grp1 and grp2 groups
module-hide --not-group {grp1 grp2} quuz
$ id --groups --name
grp1 grp7
$ ml av quuz
--------------- /path/to/modulefiles ---------------
quuz/1.0 quuz/2.0
$ ml -v quuz
Loading quuz/2.0

Hiding mechanism can also be set effective only before or after a given date time with the --before and --after options. Accepted date time format is YYYY-MM-DD[THH:MM].

$ cat /path/to/modulefiles/fum/.modulerc
#%Module4.6
# hide only before a given date
module-hide --hard --before 2020-09-01T12:00 fum/1.0
# hide only after a given date
module-hide --hard --after 2020-09-01 fum/2.0
$ date
Fri 04 Sep 2020 06:21:48 AM CEST
$ ml av fum
--------------- /path/to/modulefiles ---------------
fum/1.0

Hidden modules can be included in available module searches if option --all/-a is set on avail, aliases, whatis or search sub-commands. Hard hidden modules are unaffected by this option and stay hidden.

$ ml av -a
--------------- /path/to/modulefiles ---------------
bar/1.0(old) foo/1.0 fum/1.0 quuz/2.0 qux/2.0
bar/2.0 foo/2.0 quuz/1.0 qux/1.0

Querying user's name and groups membership

2020-07-24T12:52:00.001Z

Two new sub-commands are introduced for the module-info modulefile command: username and usergroups. They respectively fetch the name of the user currently running modulecmd.tcl or the name of all the groups this user is member of.

These two new modulefile commands can help to adapt code to specific users or groups. Like for instance to instantiate a modulefile for each group the user is member of:

$ cat /path/to/modulefiles/foo/.modulerc
#%Module
foreach grp [module-info usergroups] {
module-virtual foo/$grp .common
}
$ id -G -n
grp1 grp2 grp3
$ module avail
--------------- /path/to/modulefiles ---------------
foo/grp1 foo/grp2 foo/grp3

username and usergroups sub-commands of module-info modulefile command are only supported on Unix platform.

Embed shell scripts in modulefiles

2020-07-16T15:09:00.001Z

The source-sh modulefile command is introduced to source environment changes done by the evaluation of a shell script passed as argument. With newly introduced sh-to-mod sub-command resulting environment changes done by script are output as modulefile commands. source-sh applies those modulefile commands as if they were directly written in loading modulefile.

source-sh is useful for software providing a shell script for their enablement. If you want to enable such software with module yet using shell script provided by software for this task, just write a modulefile using source-sh command to call the shell script.

Keeping the same example used to describe sh-to-mod sub-command: foo software provides a foo-setup.sh script for its activation. Create a modulefile foo/1.2 that calls this script:

$ cat /path/to/modulefiles/foo/1.2
#%Module
source-sh sh /path/to/foo-1.2/foo-setup.sh arg1

Displaying this modulefile indicates the environment changes done by script:

$ module display foo/1.2
-------------------------------------------------------------------
/path/to/modulefiles/foo/1.2:
prepend-path PATH /path/to/foo-1.2/bin
set-alias foo {foobin -q -l}
setenv FOOENV arg1
-------------------------------------------------------------------

Loading the modulefile applies the environment changes seen above:

$ module load -v foo/1.2
Loading foo/1.2
$ echo $FOOENV
arg1
$ alias foo
alias foo='foobin -q -l'

Track of these changes is kept in user environment to be able to undo them when modulefile is unloaded:

$ module unload -v foo/1.2
Unloading foo/1.2
$ echo $FOOENV
$ alias foo
bash: alias: foo: not found

Changes on environment variables, shell aliases, shell functions and current working directory are tracked. The following shells are supported: sh, dash, csh, tcsh, bash, ksh, ksh93, zsh and fish.

Translating shell scripts into modulefiles

2020-07-07T13:17:00.001Z

The sh-to-mod sub-command is added to output as a modulefile content the environment changes done by the evaluation of a shell script passed as argument. sh-to-mod is especially useful for software providing a shell script for their enablement in shell session: it can convert these scripts into modulefiles.

Say for instance, a foo software has been installed and it provides a foo-setup.sh script to activate foo software in user environment:

$ cat /path/to/foo-1.2/foo-setup.sh
#!/bin/sh
export FOOENV="$1"
export PATH=/path/to/foo-1.2/bin:$PATH
alias foo='foobin -q -l'

Calling module sh-to-mod on this shell script outputs the environment changes it performs as a modulefile content:

$ module sh-to-mod sh /path/to/foo-1.2/foo-setup.sh arg1
#%Module
prepend-path PATH /path/to/foo-1.2/bin
set-alias foo {foobin -q -l}
setenv FOOENV arg1

sh-to-mod acts as a full replacement for the standalone createmodule.sh and createmodule.py scripts. However those two scripts are currently still provided for compatibility purpose.

This new feature is available in git repository and will be included in next release (v4.6).