Environment Modules Updates

Modules 4.2.5 released

2019-07-08T18:15:00.001Z

Modules version 4.2.5 has just been released. This new release provides fixes for the issues found recently on the 4.2 serie.

The changes introduced in this bugfix release are:

Correctly escape ? character in shell alias. (fix issue #275)
When resolving the enabled list of modulepaths, ensure resolved path entries are unique. (fix issue #274)
Right trim ‘#’ characters from the fetched modulefile magic cookie string to ensure a correct compatibility version comparison. Useful when modulefile first line is equal to #%Module4.2##############.
Fix argument parsing for the append-path, prepend-path and remove-path modulefile commands to consider every arguments found after the variable name as variable values and not command option even if argument starts with - character. (fix issue #278)
Fix automatic loading of modulefiles when multiple module names are set on a single module load modulefile command. When auto_handling mode was disabled, the load of not loaded modules was not achieved as soon as some modules on this list were already loaded. (fix issue #281)

The list of all changes made in this new version are described in the NEWS document. Details on the new features introduced in the 4.2 series are provided in the MIGRATING document.

http://modules.readthedocs.io/en/stable/NEWS.html
http://modules.readthedocs.io/en/stable/MIGRATING.html

The tarball of this new version can be downloaded at:

http://downloads.sourceforge.net/modules/modules-4.2.5.tar.gz

If you encounter any issue, please let us know by creating a ticket on
the project bug tracker at:

https://github.com/cea-hpc/modules/issues

End of 4.2 strengthening work, road to 4.3

2019-04-28T12:48:00.001Z

Version 4.2 of Modules was released in October last year with new capabilities to automatically handle dependencies between modulefiles. With these new automated mechanisms, corner cases were found which led to a more substantial bugfix work than initially expected.

After 4 bugfix releases, with a specific focus on error management (v4.2.3) and on warning/error/informational message report (v4.2.4), I consider the work on this v4.2 branch over.

Of course there are still things to improve, but I consider the remaining stuff to work on as new features like the introduction of additional automated mechanisms (Conflict Unload, etc).

If you encounter strange situation with v4.2.4, feel free to report them here or directly on the bug tracker.

On my side, I am happy to re-start the work on v4.3. I will progressively disclose here the new features that will be part of it. Hope to roll out this release at the beginning of summer. Stay tuned.

Modules 4.2.4 released

2019-04-26T12:01:00.001Z

Modules version 4.2.4 has just been released. This new release in the 4.2 series has a specific focus on improving the readability of the warning, error and informational messages reported when multiple load or unload modulefile evaluations are performed in a row.

The changes introduced in this bugfix release are:

Better track each module evaluation and the context associated to it in order to report a more accurate information on the additional modules loaded or unloaded when proceeding the main evaluation request. (fix issue #244, #245, #246, #247 and #248)
Doc: preserve quotes and dashes when making HTML docs. (fix issue #250 with contribution from Riccardo Coccioli)
Fix hanging list sub-command when terminal width is equal to the single column text width to be printed. (contribution from Jesper Dahlberg)
During an additional evaluation triggered by an automated module handling mechanism, ensure warning and error messages are reported under the message block of the main evaluation. (fix issue #252)
During the unload of a module when the automated module handling mode is disabled, report a warning message for each unload of a useless requirement that fails as done when the automated module handling mode is enabled. (fix issue #253)
When multiple modules are listed on a prereq command, drop the output of those modules that fails to load (by the Requirement Load automated mechanism) to only keep the output of the module whose load succeed. (fix issue #254)
Fix switch sub-command when the switched-off module cannot be unloaded when other loaded modules depend on it. Whole switch process is failed and no load of the switched-on module is attempted. (fix issue #251)
When switching modules, report failure of switched-off module unload or switched-on module load under the message block of the switch action. A failed switched-off module unload is reported as an error, as it aborts the switch evaluation, whereas a failed switched-on module load is reported as a warning. (fix issue #255)
When a module requirement is seen missing but the load of this module was attempted, report a more specific error or warning message to let user understand that the load of the requirement was attempted but failed. (fix issue #257)
When loading a module, report any missing requirement on the message reporting block corresponding to this module load. This warning or error message comes in addition to the eventual Requirement Load message reported under the message block of the main evaluation. (fix issue #258)
When unloading a module which has some dependent module still loaded, produce a more specific error or warning message if an evaluation of these dependent modules has been realized or if the unload of the required module is forced. (fix issue #259)
When a conflicting module is seen loaded but the unload of this module was attempted, report a Conflict Unload error or warning message toward the main evaluation message block. (fix issue #261)
When loading a module, report any loaded conflict on the message reporting block corresponding to this module load. This warning or error message comes in addition to the eventual Conflict Unload message reported under the message block of the main evaluation. (fix issue #261)
Correctly report loading state of conflicting module. (fix issue #262)
Adapt warning, error and info messages relative to the Dependent Reload mechanism to distinguish the unload phase from the load (reload) phase of this mechanism. In the automated module handling summary report, unloaded modules via this mechanism are reported in the Unloading dependent list and modules reloaded afterward are reported against the Reloading dependent list. (fix issue #263)
When the automated module handling mode is disabled, do not attempt to load a requirement expressed in a modulefile with a module load command, if this requirement is already loaded or loading.
Skip load or unload evaluation of a module whose respectively load or unload was already attempted but failed. If this second evaluation attempt occurs within the same main evaluation frame. (fix issue #264)
When reloading modules through the Dependent Reload automated mechanism, prevent modules to automatically load of other modules with the module load modulefile command, as it is done for the prereq command. (fix issue #265)
Raise an error when an invalid option is set on append-path, prepend-path or remove-path modulefile command. (fix issue #249)
Zsh initializes by default the MANPATH environment variable to an empty value when it starts. To preserve manpath system configuration even after addition to this variable by modulefiles, set MANPATH variable to : if found empty. (improve fix for issue #224)
Doc: provide a short installation guideline in README file. (fix issue #230)

The list of all changes made in this new version are described in the NEWS document. Details on the new features introduced in the 4.2 series are provided in the MIGRATING document.

http://modules.readthedocs.io/en/stable/NEWS.html
http://modules.readthedocs.io/en/stable/MIGRATING.html

The tarball of this new version can be downloaded at:

http://downloads.sourceforge.net/modules/modules-4.2.4.tar.gz

If you encounter any issue, please let us know by creating a ticket on the project bug tracker at:

https://github.com/cea-hpc/modules/issues

Modules 4.2.3 released

2019-03-23T13:11:00.001Z

Modules version 4.2.3 has just been released. This new release in the 4.2 series has a specific focus on error management. When the evaluation of a modulefile fails during an automated module handling mechanisms everything is now done to get or to keep at the end of all evaluations a consistent environment.

The changes introduced in this bugfix release are:

Add all the module dependency-related internal information to those saved prior a modulefile evaluation in order to correctly restore internal state in case modulefile evaluation fails.
Init: in shell initialization scripts, initialize MANPATH if not set with a value that preserves manpath system configuration even after addition of paths to this variable by modulefiles. (fix issue#224)
Enable to define an entire path entry to the MODULEPATH variable which corresponds to a variable reference only. (fix issue#223)
Cookbook: add the modulefiles-in-git recipe. (contribution from Scott Johnson)
When module switch commands are found in modulefiles, track switched-off modulefile as a conflict and switched-to modulefile as a requirement to apply same behaviors than for module load and module unload commands in modulefiles. If module switch has only one argument, do not define a conflict toward switched-off modulefile. CAUTION: it is not recommended to use module switch command in modulefiles. (fix issue#229)
When unloading a module, revert module switch commands found in modulefile: switched-on module is converted to a module unload, like for module load command. Nothing is done for switched-off module, like for module unload command. (fix issue#226)
For default element in a modulefile directory which is a module alias that points to a modulefile, when this modulefile is loaded, it receives as alternative names the eventual module aliases set on the distant directory holding the alias pointing to it. (fix issue#231)
When unloading a module that contains module load or module switch commands in its modulefile, select for unload the automatically loaded requirement module which has been loaded prior its dependent. (fix issue#232)
Doc: describe Emacs settings useful for adhering to coding conventions in CONTRIBUTING guide. (fix issue#233 with contribution from Ben Bowers)
When looking for a loaded or loading dependency requirement, select among the eventual multiple candidates the closest match to the dependent module.
During the unload of a module, if the unload of one of its dependent (by the Dependent Unload mechanism) fails, abort the whole unload process. Exception made if the force mode is enabled. In this case failing module stays loaded and the Dependent Unload mechanism continues with next module to unload.
During the unload of a module, if the unload of one of its useless requirements (by the Useless Requirement Unload mechanism) fails, keep the requirements of this failing module loaded. Such error is reported as a warning and it does not stop the whole unload process. (fix issue#240)
During the load or the unload of a module, if the unload of one of its dependent (by the Dependent Reload mechanism) fails, abort the whole unload or load process. Exception made if the force mode is enabled. In this case failing module stays loaded and Dependent Reload mechanism continues with next module to unload. This failing module is removed from the Dependent Reload list, so it will not take part of the load phrase of the mechanism. (fix issue#239)
During the load or the unload of a module, if the load of one of its dependent (by the Dependent Reload mechanism) fails, abort the whole unload or load process. Exception made if the force mode is enabled. In this case failing module stays loaded and Dependent Reload mechanism continues with next module to load. When the mechanism is applied during a switch command, force mode is enabled by default on the load phase. (fix issue#241)
When reloading all loaded modules with the reload sub-command, if one reloading module fails to unload or load, abort the whole reload process to preserve environment sanity. (fix issue#237)
During the unload of a module when the automated module handling mode is disabled and this module declares its requirements with the module load modulefile command. If the unload of one of its useless requirements (by the Useless Requirement Unload mechanism) fails, whole unload process is not aborted and continue with next module to unload. (fix issue#238)
Contrib: add mtreview utility script that analyzes test suite log file to compare actual and expected output of failed test. mt does not output the full test suite logs anymore but only the information produced by mtreview on failed tests.
Install: exclude Continuous Integration configurations from dist tarballs.

The list of all changes made in this new version are described in the NEWS document. Details on the new features introduced in the 4.2 series are provided in the MIGRATING document.

http://modules.readthedocs.io/en/stable/NEWS.html
http://modules.readthedocs.io/en/stable/MIGRATING.html

The tarball of this new version can be downloaded at:

http://downloads.sourceforge.net/modules/modules-4.2.3.tar.gz

If you encounter any issue, please let us know by creating a ticket on
the project bug tracker at:

https://github.com/cea-hpc/modules/issues

Modules 4.2.2 released

2019-02-18T04:25:00.001Z

Modules version 4.2.2 has just been released. This is a bugfix release in the 4.2 series. Fixes are:

Correct the Dependent Unload mechanism when it triggers the unload of 2 modules making together a requirement from another module. This module is now also added to the dependent modules to unload.
Doc: add a cookbook section in the documentation and port there the 3 pre-existing recipes: inhibit-report-info, top-priority-values and unload-firstly-loaded.
Doc: add a CONTRIBUTING guide.
Doc: fix a typo on the Python initialization example in module man page.
Doc: add a FAQ entry to describe the use of module from Makefile. (with contribution from Robert McLay)
Trim any white-space, newline or ; characters at the beginning or end of the function body passed to set-function modulefile command.
Init: add recognition of the --auto, --no-auto and --force command-line switches in fish shell completion script.
Init: add recognition of the --auto, --no-auto, --force, --paginate and --no-pager command-line switches in zsh shell completion script.
When the load of a modulefile is asked but a conflict is registered against this modulefile by an already loaded module, the load evaluation is now performed and the conflict is checked after this evaluation. If the conflict is still there, this evaluation (and the evaluation of its requirements) is rolled back. This gives the opportunity to solve this conflict by using module unload modulefile command. (fix issue#216)
Init: fix _module_not_yet_loaded alias in tcsh completion script to handle situation when noclobber variable is set. Also ensure actual rm command is called and not an alias. (fix issue#219)
Fix warning message when the load of a modulefile is forced over a reflexive conflict (message was reported twice).
When looking at the dependency of a loaded module, only consider requirement loaded before dependent module (holding a prior position in the loaded module list) as valid. Those loaded after dependent module are considered as an unmet dependency thus they are not taking part in the Dependent Unload, the Useless Requirement Unload and the Dependent Reload mechanisms.

The list of all changes made in this new version are described in the NEWS document. Details on the new features introduced in the 4.2 series are provided in the MIGRATING document.

http://modules.readthedocs.io/en/stable/NEWS.html
http://modules.readthedocs.io/en/stable/MIGRATING.html

The tarball of this new version can be downloaded at:

http://downloads.sourceforge.net/modules/modules-4.2.2.tar.gz

If you encounter any issue, please let us know by creating a ticket on the project bug tracker at:

https://github.com/cea-hpc/modules/issues

Modules 4.2.1 released

2018-11-12T00:08:00.001Z

Modules version 4.2.1 has just been released. This is a bugfix release in the 4.2 series. Fixes are:

Cookbook: add the inhibit-report-info recipe.
Cookbook: port unload-firstly-loaded and top-priority-values recipes to v4.2.
Init: fix listing of loaded modules for fish and tcsh shell completions.
Init: fix saved collection listing when no collection found for bash, zsh, tcsh and fish shell completions.
Adapt system modulefile Tcl command to execute the command passed as argument through shell, like it is performed on compatibility version. (fix issue#205)
Correctly filter modulefile search memory cache entries when using a full search result to search later on a specific modulefile.
Prefix debug messages by information on the current modulefile or modulerc interpreter if any.
Init: fix listing of loaded modules on unload and switch sub-commands for bash shell completion.
Refrain module unload modulefile command from unloading a module required by another loading module.
Enable is-loaded modulefile Tcl command in modulerc interpretation context, like done on compatibility version. (fix issue#207)
Check a required module is not already loading before attempting to load it. Helps to handle cyclic dependencies.
Compute loaded modules requirement dependency relations without cycle and consider the module closing the cycle in a constraint violation state to avoid reloading loops on the Dependent Reload mechanism.
Safely unset dependency reference when computing dependency relations as some dependencies expressed may target same module.
Ensure a loaded module matching multiple entries of a same or prereq will just be considered as one module matching this requirement.
Init: quote prompt in csh and tcsh script with :q rather double-quotes to accommodate prompts with embedded newlines. (fix issue#209 with contribution from Satya Mishra)
Init: skip shell environment alteration if autoinit command fails. (fix issue#208)
Reword path-like variable element counter reference handling to simply ignore the counter values not coherent with the content of related path-like variable. (fix issue#206)

The list of all changes made in this new version are described in the NEWS document. Details on the new features introduced in the 4.2 series are provided in the MIGRATING document.

http://modules.readthedocs.io/en/stable/NEWS.html
http://modules.readthedocs.io/en/stable/MIGRATING.html

The tarball of this new version can be downloaded at:

http://downloads.sourceforge.net/modules/modules-4.2.1.tar.gz

If you encounter any issue, please let us know by creating a ticket on the project bug tracker at:

https://github.com/cea-hpc/modules/issues

Modules 4.2.0 released

2018-10-18T05:34:00.001Z

After 6 months of development, it is a my great pleasure to announce the release of Modules version 4.2.0.

This new version brings a specific focus on the dependency management between modulefiles with a push forward consistency and automation. It is backward-compatible with previous version 4 releases and primarily adds a bunch of new features and fixes some bugs. Major new features introduced in v4.2.0 since v4.1.4 are:

Modulefile conflict constraints consistency
Modulefile prereq constraints consistency
By-passing module defined constraints
Automated module handling mode
Consistency of module load/unload commands in modulefile
Modulefile alias and symbolic modulefile name consistency
Environment variable change through modulefile evaluation context
Express Modules compatibility of modulefile with versioned magic
cookie
Improved module message report
New modulefile commands
set-function, unset-function

Details on these new features are provided in the MIGRATING document. The list of all changes made in this new version are described in the NEWS document.

http://modules.readthedocs.io/en/stable/MIGRATING.html
http://modules.readthedocs.io/en/stable/NEWS.html

The tarball of this new version can be downloaded at:

http://downloads.sourceforge.net/modules/modules-4.2.0.tar.gz

If you encounter any issue, please let us know by creating a ticket on the project bug tracker at:

https://github.com/cea-hpc/modules/issues

Improved module message report

2018-10-12T06:03:00.001Z

Module sub-commands like load, unload or switch, may perform multiple load or unload modulefile evaluations in a row. Also these kind of evaluation modes may sometimes trigger additional load or unload evaluations, when for instance a modulefile contains a module load command.

To improve the readability of the module messages produced relatively to a load or an unload evaluation, these messages are now stacked under a Loading or an Unloading message block that gathers all the messages produced for a given modulefile evaluation:

$ module load --no-auto foo
Loading foo/1.2
ERROR: foo/1.2 cannot be loaded due to missing prereq.
HINT: the following module must be loaded first: bar/4.5

In addition, foreground load, unload, switch and restore actions (ie. asked on the command-line) now report a summary of the additional load and unload evaluations that were eventually triggered in the process:

$ module load --auto foo
Loading foo/1.2
Loading requirement: bar/4.5

This change is available in git repository and was the last addition to the imminent v4.2 release.

Consistency of module load/unload commands in modulefile

2018-09-23T20:21:00.001Z

With the module load modulefile command, a given modulefile can automatically load a modulefile it pre-requires. Similarly with the module unload modulefile command, a given modulefile can automatically unload a modulefile it conflicts with.

Both commands imply additional actions on the loaded environment (loading or unloading extra modulefiles) that should cope with the constraints defined by the loaded environment.

Additionally module load and module unload modulefile commands express themselves constraints on loaded environment that should stay satisfied to ensure consistency.

To ensure the consistency of module load modulefile command once the modulefile defining it has been loaded, this command is assimilated to a prereq command. Thus the defined constraint is recorded in the MODULES_LMPREREQ environment variable. Same approach is used for module unload modulefile command which is assimilated to a conflict command. Thus the defined constraint is recorded in the MODULES_LMCONFLICT environment variable.

To ensure the consistency of the loaded environment, the additional actions of the module load and module unload modulefile commands have been adapted in particular situations:

When unloading modulefile, module load command will unload the modulefile it targets only if no other loaded modulefile requires it and if this target has not been explicitly loaded by user.
When unloading modulefile, module unload command does nothing as the relative conflict registered at load time ensure environment consistency and will forbid conflicting modulefile load.

Please note that loading and unloading results may differ than from previous Modules version now that consistency is checked:

Modulefile targeted by a module load modulefile command may not be able to load due to a registered conflict in the currently loaded environment. Which in turn will break the load of the modulefile declaring the module load command.
Modulefile targeted by a module unload modulefile command may not be able to unload due to a registered prereq in the loaded environment. Which in turn will break the load of the modulefile declaring the module unload command.
If automated module handling mode is enabled, module load modulefile command is interpreted when unloading modulefile as part of the Useless Requirement Unload (UReqUn) mechanism not through modulefile evaluation. As a consequence, an error occurring when unloading the modulefile targeted by the module load command does not break the unload of the modulefile declaring this command. Moreover unload of the module load targets is done in the reverse loaded order, not in the module load command definition order.

This change is available in git repository and will be included into the upcoming v4.2 (only one topic remaining to address prior the release of the new version).

More small changes for 4.2

2018-09-08T19:23:00.001Z

I would like to let you known about 4 small improvements made following to discussions that occurred few months ago on the mailing-list or on bug trackers.

New --with-bin-search-path configure option

New ./configure option added to get in control of the path list used to search the tools required to build and configure Modules (like ps and basename). Default value is /usr/bin:/bin:/usr/local/bin.

New --enable-silent-shell-debug-support configure option

New ./configure option that adds the ability to control whether or not code to support silent shell debug should be added to the module function and sh-kind initialization scripts. Enabled by default.

Useful for those wanting a small and neat module shell function definition (by use of the --disable-silent-shell-debug-support option).

New --enable-quarantine-support configure option

Following same target than the previous new option, this one adds the ability to control whether or not code to support quarantine mechanism should be added to the module function and initialization scripts. Enabled by default.

Express Modules compatibility of modulefile with versioned magic cookie

Any modulefile should start with the #%Module magic cookie and sometimes a version number may be placed right after this string. Until now this version number corresponded to a modulefile format version but it was never checked.

Now this version number reflects the minimum version of Modules required to interpret the modulefile. If the version number is set along the magic cookie string it is now checked and the modulefile is interpreted only if Modules version is greater or equal to this version number. For instance, if a modulefile begins with the #%Module4.3 string, it can only be evaluated by Modules version 4.3 and above. Elsewhere the modulefile is ignored like files without the #%Module magic cookie set.