On July 8th, the conda environment for modules, python/2.7.0 and python/3.6.0, was affected after an incomplete install for seaborn and pandas was aborted. Subsequent steps to fallback to a sane state and install those packages failed. Although the previous conda environments and Python base was still intact, new package installations using conda were failing altogether. Since Intel released their 2019 Parallel Studio XE (compilers and Python) and the 2018 installed version was still functional, effort is being made to migrate to the 2019 versions and eventually abandon the 2018 Python versions.

The 2019 Intel Parallel Studio XE toolkit, including Python versions 2.7.16 and 3.6.7, have just been installed on Memex. To use the 2019 Intel compilers, use module “intel/2019” (includes icc, ifort, etc.), which is an upgrade to the “intel/2018” module (lowercase ‘i’ matters). I am currently working on modules for Intel’s Python 2 and 3, located under /share/apps/intel/2019 as intelpython2/ and intelpython3/.

If you are only interested in the 2019 Intel Compilers and don’t use Intel’s Python versions, you can stop reading here. If you don’t know which python version you’re using, type “python --version” from the command line. It will indicate Intel or GNU. For those who are interested in the 2019 Python installation on Memex, please continue reading…

The 2019 Intel Python installation does not inherit the packages and conda environments from the previous 2018 Intel Python installations, which are modules “python/2.7.0” and “python/3.6.0” on Memex. This means you can continue to use those 2018 Intel installations (including compilers, python, and python conda envs), but updates to those 2018 packages will be abandoned by September 1st, 2019. To save and recreate your current conda environment for the 2019 installation, see Sharing an environment and then Creating an environment from an environment.yml file. This file can be saved and used in other Python/Conda setups (on other machines as well).

I am working to install a few general Python packages for the new 2019 installation, so please feel free to send requests for package installations to [email protected]. The new 2019 Intel Python modules will be “python/2.7.16” and "python/3.6.7". These modules are available now but I am still working to install packages this week and establish a conda "base". These packages include:

numpy
matplotlib
seaborn
tensorflow
pandas
keras
sklearn
r
rstudio
jupyter notebook
r-rgdal
and more… (some packages are easier installed than others!)

Again, these packages will establish the “base” for each Python module and their downstream conda environments, so if you have a general package you’d like me to install, let me know by this week. This is important because If the base for either Python version changes, any conda environment created on top of it will be affected, so please send your requests this week. This work is ongoing…

If request for package installations involve pulling from Github or other third-party sources, then a conda environment might become necessary. Not all packages, and/or package versions, are compatible. For this reason, wait until after this week to create your own conda environments. Personal conda environments can be setup without having Memex admin privileges (recommended if you want full control of your environment). For example (instructions taken from here),

conda create -n myenv
conda activate myenv

will create a conda environment in /home/username/.conda/envs/myenv and then prepend your command prompt with "(myenv)", indicating you’re now in your newly created conda environment. This conda environment allows you to install specific versions of packages as well, but your initial conda environment depends on the module you start with (i.e. “python/2.7.16” or “python/3.6.7”). You can specify what version of a package to install using following command (package here is "python", version is “3.6.8”):

conda install -n myenv python=3.6.8

Conda will try to accommodate this request by downgrading, removing, upgrading, superseding, or installing packages for “python=3.6.8” dependencies, and then ask if you want to proceed (y/n?). I can tell you from experience, Python 3 is easier for Conda to work than Python 2, but most issues can be worked through in conda environments.

For issues, please email [email protected] to create a ticket.

Other useful conda commands/instructions:

conda deactivate #exit conda environment
conda env list #list available conda environments
conda list scipy #list package version in current environment
sharing an environment
deleting an environment

Updates will follow…