changelog Updates

Memex Login Hung on 11/17/21

2021-11-17T21:32:57.411Z

Initial incident and probable cause:

Broken pipe on the login caused by a file update on the login. The master node and login were both rebooted and “wwsh file sync” commands were automated memex_routecheck.sh (crontab and cron.hourly). After reboot, the routing table and maintenance motd were both updated sooner than before. Also added a ping check in order to determine whether the networks needs restarting (after the routing table is updated).

SLURM's Default Memory Per CPU Increased (1GB --> 2GB)

2020-05-07T18:12:00.001Z

Hi All,

Based on previous usage, the default allocation of 1GB of memory per cpu is too low. I have now increased this default to 2GB of memory per cpu.

If you are not setting memory requirements in your job(s), this will change will affect them.

If you are already setting memory requirements in your job(s), this change will not affect you.

Best practice is to specify memory in all jobs. If you haven’t been doing so and are now running into issues (waiting in partitions for longer than usual), you’ll now need to specify memory per cpu in jobs.

--mem-per-cpu=1G # when passed via command line,
or
#SBATCH --mem-per-cpu=1G # when added to a submission script.

Adding either option should set your memory per cpu back to 1GB.

** Again, if you are already using the --mem= or --mem-per-*= options, no changes are required for your job(s).

Thank You

SLURM Priority Adjustment

2020-05-07T15:15:00.001Z

Since priorities were not working for those users who use Memex less frequently and in smaller batches of submitted jobs, these parameters were adjusted:

PriorityWeightFairShare=20000
PriorityWeightTRES=CPU=1000,Mem=2000,GRES/gpu=3000

As a result gres/gpu was added to:

AccountingStorageTRES=cpu,mem,energy,node,billing,fs/disk,vmem,pages,gres/gpu

This functionality changes in SLURM 19+, but our current version is 18.08 which is the latest version packaged with OpenHPC 1.3.

Update: AcctGatherFilesystemType was also enabled for lustre.

Did You Know ... Slack Edition

2020-02-13T20:19:00.001Z

Did you know we have a Slack channel for HPC/Research Computing?

Signup to our Carnegie Institution for Science workspace (click here) and then join the #hpc channel. Please use your Google login, "@carnegiescience.edu", email address.

This channel is useful for a couple of reasons (as well as other Carnegie Science channels!):
- Anyone can share photos, Googe Drive documents, articles, issues and best practices.
- Anyone can send me an informal message to talk about HPC/Research Computing/Memex issues, software questions, usage statistics, and much more. If I can’t answer or address your concerns right away, a formal ticket can be created for a faster response ***.
- Messages can be directed to a particular user or posted to the channel for everyone to see. General issues can be shared to #hpc and searched by other users with similar questions or issues.

To join, click here!!

Other things you can do are create a new private channel, have a group call (type "/call @user1 @user2"), set reminders (type "/remind help" for options), connect with a Github (type "/github help" for options), and much more (formatting messages, built-in commands)!

*** Slack does not replace our ticketing system. It is a way for us to discuss research computing without the formality of a ticketing system (by emailing [email protected]" target="_blank" rel="noopener">[email protected] or typing "/freshservice-ticket" from Slack’s direct messaging system).

Did You Know ... Python Edition

2020-01-23T16:56:00.001Z

Did you know official support for Python 2 is over?

That said, we have Python 3 available on Memex by loading the module, "python/3.6.7". This Python version includes conda, R, Jupyter, IntelMPI, and many other packages. Most Python packages can be installed by sending your request to [email protected]" target="_blank" rel="noopener">[email protected].
Did you know you can install your own Python packages using conda?

For example, if you’d like the latest python version available, 3.8.1, first search for it here first (mainly conda-forge channel) or directly from the command line with:
```
 module purge \
#only needed to ensure a clean environment
module load python/3.6.7
conda search python \
#latest should be 3.8.1 as of last week
```
and use it to set up your own conda environment (both anaconda.org and the command line shows the proper channel, "-c", to use below).
```
 conda create -p /lustre/scratch/$USER/.envs/py38 \
-c conda-forge python=3.8.1
```
The above command assumes /lustre/scratch/$USER/.envs is where you want the conda environment, py38, to reside. It has to be in a directory you own.

To activate any conda environment (using example, py38, above):
```
 source activate py38
```
To deactivate any conda environment:
```
 source deactivate
```
To remove an entire conda environment:
```
 conda-env remove -n py38
```
Did you know you can list existing conda environments already on Memex?

To list the available conda environments (still using the module, python/3.6.7), simply type:
```
 conda env list
```
if you can see them, you can activate them but if you don’t have write permission, you cannot install, remove, or modify them. You can however, clone them (make the new name unique–> "foo", not the clone name --> “foo2”):

conda create -p /lustre/scratch/$USER/.envs/foo --clone foo2
Did you know you can list any installed Python package, in the base conda environment (activated by module load python/3.6.7) or a custom conda environment (activated by source activate foo)?

conda list packagename #replace packagename with real package name

where “packagename” is a placeholder for a real package (like python, or r-base). Of course,
```
 conda list
```
lists all packages installed in your current environment.
Did you know there are other useful commands (#replace packagename and X with real values):

conda search packagename=X.X.X --info #where X.X.X=version_number of packagename
conda clean #removes downloaded packages and caches
conda remove packagename #to remove package
conda list --revisions #to identify a revision to rollback to
conda install --revision X #rollback to revision number, X

Type conda -h and/or conda-env -h for more options.

Note: To manage where your cache directory is located (includes tar balls and downloaded packages), you can set the following in your bashrc:

echo CONDA_PKGS_DIRS=/lustre/scratch/$USER/.envs/pkgs >> \
$HOME/.bashrc
source $HOME/.bashrc

Or in your current environment only with:

export CONDA_PKGS_DIRS=/lustre/scratch/$USER/.envs/pkgs

Did You Know ... Storage Edition

2020-01-15T15:57:00.001Z

Did you know there’s a 256GB quota for all /home directories?

This policy will be fully enforced in the coming weeks. This requirement is needed to manage space and load for /home. Home directories should primarily be used to set up scripts, small software applications, and environment variables across the cluster. To that end, we will be enforcing a hard quota of 256GB. A consolidation of all /home directories to one storage device will take place this year. The exact date will be advertised but please contact me or [email protected]" target="_blank" rel="noopener">[email protected] with any other questions.

Note: If you are currently over the limit, you will have time to move data but start moving (or removing) data now.

Did you know we have about 950TB of storage right now?

This includes the Lustre filesystem (/lustre, 698TB) and the MemexNFS filesystem (/work, /scratch, /share/apps, and /home ~ 252TB). Check out our login banner (see "Mountpoint Information") to see how each filesystem should be used.

Did you know we now have a Memex Globus endpoint, "cisuser#carnegiescience"?

Globus transfers are typically much faster than our command line option. Instructions on how to setup and use Globus on Memex is in our FreshService ticketing system (Solutions --> Computation).
Did you know you can check disk usage in the following ways?

For most users, the new command
```
 $ zquota
```
will show your /home usage and your group’s usage for /work/DEPT. Shared directories such as /share/apps, /scratch, and /lustre usage can seen by the ‘df’ command. For example:
```
 $ df -h /lustre /share/apps /scratch
```
shows usage for all three filesystems. Of course, other commands like $ du -sh /home/username/directory/ for directories or $ ls -Shl filename for files can be used as well.

Did You Know ... SLURM Edition

2020-01-13T21:58:00.001Z

Did you know there’s a gui to view SLURM jobs? Inside a VNC or “ssh -XY …” session, type sview -a from the command line (check out the menu selection as well):

Did you know you can view the maximum resources of each node with:

sinfo -e -o "%20N %10c %10m %25f %10G"
Did you know you can view the maximum memory used for running jobs with (see “/bin/sacct -e” for format options):

/bin/sacct --format="JobID,CPUTime,MaxRSS" -j JOBID

better yet, try sstat --format="jobid,maxrss,avecpu" -j JOBID (“sstat -e” for format options)?
Did you know you can email yourself job status changes?

#SBATCH [email protected]
#SBATCH --mail-type=FAIL,BEGIN,END,SUSPEND

Here are a few more tips and you can contact me directly with any questions.

User reported that rsync/cp/scp too slow on /memexnfs/apps,

2019-12-19T17:35:00.001Z

Temporary Resolution 01/02/20:
Using rclone instead of rsync/cp/scp is 10x faster for large directory and possibly large file syncs to /memexnfs/ mountpoints. Although all reads from /memexnfs mounts are performing performing as expected, all disk to disk writes to /memexnfs are not. As load increases, presumably from cluster jobs and transfers (syncs to /home), write performance suffers. The workaround is to use rclone and this email was sent out to all users:

Please use rclone for large local or remote transfers while using /memenfs/* filesystems, or /share/apps/dept, /home/username, /work/DEPT, or /scratch/username. There seems to be an issue with the common linux commands, rsync and cp, when transferring large directories (size and number of files).

The solution is to use rclone instead of rsync or cp or scp for large directories (size and number of files),

rclone sync /home/username/directory/ /scratch/username/directory/ -LP

This syncing issue currently affects write speeds but not read speeds for large directories to /memexnfs/*. This solution has been tested and should also work fine for small directories and files.

Of course, rclone is used to sync files to/from GDrive as well.

Rclone Tutorial:
https://carnegiescience.freshservice.com/support/solutions/articles/3000040389

Issue 12/19/19:
A user reported rsyncs were tool slow on the /share/apps mount of /memexnfs/apps. Since all /memexnfs/* mounts share the same disks/setup, it was determined the issue was not isolated to /share/apps, /work, /scratch, and /home (all /memexnfs mountpoints across the cluster).

Login Node Slowness (module command hanging on memex.carnegiescience.edu)

2019-12-02T19:57:00.001Z

Resolved 12/10/19:

After tuning the NFS server and clients, the slowness was resolved. Although there were several adjustments, the RPCNFSDCOUNT variable in /etc/sysconfig/nfs was the change that made the biggest improvement (100x bandwidth). This tuning of the system was partially documented in our ticket system for future reference.

Update 12/04/19:

To see improvements to the module command, please log out and log back in. This step is required for you to take advantage of Lmod’s caching feature which improves its responsiveness. In the meantime, I am still investigating ways to improve filesystem performance for /memexnfs/* mountpoints. If you are running SLURM jobs which writes or reads large files, I suggest using /lustre/scratch/$USER as a working directory. If you are running a parallel or multiprocessing job, also use /lustre/scratch/$USER as a working directory. For instance,

mkdir /lustre/scratch/$USER
rsync -aWz /home/$USER/workdir/ /lustre/scratch/$USER/workdir/
cd /lustre/scratch/$USER/workdir/

then submit your job as normal.

After the job finishes, you can rsync the directory back to /home/$USER/workdir for safe keeping. Please keep in mind, you’ll need to add "--delete" to the rsync command for an exact copy of the /lustre/scratch/$USER/workdir (which deletes any files/dirs in /home/$USER/workdir that are not in /lustre/scratch/$USER/workdir).

Use cautiously,

rsync -aWz --delete /lustre/scratch/$USER/workdir/ /home/$USER/workdir/

Please note, the directories under /lustre/scratch are not backed up and are not currently scrubbed.

The use of /lustre/scratch/$USER is recommended because the read/write performance of /memexnfs/* is being hindered by multiple I/O streams, including transfers (by root and users), SLURM jobs, and any other login node activity by users (VNC, shell/interpreter scripts, etc.).

Update 12/03/19:

Lmod’s cache was enabled to improve the performance of the module commands on Memex. However, I/O performance is still a bit sluggish, so more investigation is required to improve performance on the mounted filesystems.

Issue (started week of 11/25/2019):

After logging onto Memex (password/DUO), the user login hangs while trying to load modules. This is an ongoing issue which seems to be caused by remote and/or local mounted filesystems,

while Lmod is traversing one or more of the module paths (usually in “$MODULEPATH”). We are investigating the issue, but in the meantime enter “Ctrl+c” if the bash command prompt doesn’t appear swiftly after the following banner:

System Update & Failed disk in SureStoreHD, memexnfs ZFS pool degraded

2019-10-14T15:40:00.001Z

System Update 11/6/19:

System was updated on November 8th, which includes updates to SLURM, ZFS (0.6 to 0.8, which improves time to rebuild failed disk), and CentOS (7.5 to 7.7).

Resolved 11/2/19:

Issue resolved. Replacement disk has finished resilvering.

Update 10/22/19:

The new drive in our primary filesystem is still rebuilding and will be done in about 8 days. The type of filesystem is ZFS (version 0.6) in RAIDZ1 configuration which means one failed drive puts the filesystem in a degraded state. This degraded state will continue until the drive is "resilvered", or data is copied to the healthy disk and it comes online. This process, which takes entirely too much time, was flagged as a ZFS bug back in November, 2017.

The current version of ZFS, 0.8.0, was released in May of this year and addresses the bug. The improvement to the resilvering process is said to be 5-6x better that the performance we’re currently seeing. The command line slowness you are experiencing on Memex’s login is far worse (up to 5x worse) that the I/O performance on Memex’s compute nodes, but all I/O for /home, /scratch, /work/ and /share/apps will be affected. This means you can still submit jobs from the login but all other activities will be slow in /home, /scratch, and /work.

A way around this is to use your own Lustre scratch directory, /lustre/scratch/username (if it doesn’t exist, you can create it, “mkdir -p /lustre/scratch/username”), to edit files, run local commands, etc. Cleanup for /lustre/scratch/username is turned off for now and you can even submit jobs from here.

**ANNOUNCEMENT**
We are planning a software update for SLURM and ZFS when the disk replacement is completely done. I will be sending out a notice for a planned reboot, which is necessary in order to ensure the ZFS filesystem is truly updated. Please keep this in mind as you are submitting jobs. A job intended to run for a month or so, will be killed prior to these updates in a couple of weeks.

Update 10/14/19:

Drive still resilvering - 24% done and going -

status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Wed Oct 9 12:55:27 2019
18.8T scanned out of 77.8T at 47.3M/s, 363h15m to go
566G resilvered, 24.15% done

Update 10/9/19:
Failed drive replaced and ZFS resilvering with new drive started.

Update 10/7/19:
Increased quota for /memexnfs/home, decreased quota for /scratch as well.

Update 10/7/19:
Memexnfs has become more responsive due to ZFS resilvering after the drive failure. This has resulted in I/O improvements for /home, /work, /scratch, and /share/apps.

Update 10/5/19:
Waiting for new HDD in order to replace the failed disk.

The main issue is a failed drive for all /memexnfs/* mounts. I’ll let you know when the drive has been replaced. Until then Memex’s directories, /memexnfs/scratch (/scratch), /memexnfs/home/ (or /home), /memexnfs/work (or /work), and /memexnfs/apps (or /share/apps) will all be operating in a degraded state.