changelog Updates

Memex Login Hung on 11/17/21

2021-11-17T21:32:57.411Z

Initial incident and probable cause:

Broken pipe on the login caused by a file update on the login. The master node and login were both rebooted and “wwsh file sync” commands were automated memex_routecheck.sh (crontab and cron.hourly). After reboot, the routing table and maintenance motd were both updated sooner than before. Also added a ping check in order to determine whether the networks needs restarting (after the routing table is updated).

Login hangs after kernel message...

2020-06-25T20:09:00.001Z

Update (6/29/20):
The login hang was caused by high I/O load which is returning after a weekend hiatus. Unfortunately, limiting the I/O on the login is not yet feasible due to the design of the system. The issue is not due to a lack of memory, CPU, or bandwidth on the login but the ability of the login to process lots of I/O by users (including transfers and local login processes). The issue is not present when performing the same work on compute nodes (transfers, normal commands, writing files, etc.). Short of an entire system redesign, we are planning a way forward with the way storage is used and configured on Memex. The earliest possible change(s) will be made during the July 8th shutdown. Details will be sent once those plans are set.

Incident (06/24/20 @1540):
Login stalled after the global message below

 kernel:NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [$IP-m:69361]
Message from syslogd@memex at Jun 24 12:23:02 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [$IP-m:69361]
Message from syslogd@memex at Jun 24 12:23:34 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [$IP-m:69361]
Message from syslogd@memex at Jun 24 12:24:02 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#10 stuck for 23s! [$IP-m:69361]
Message from syslogd@memex at Jun 24 12:24:30 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [$IP-m:69361]
Message from syslogd@memex at Jun 24 12:24:58 ...
kernel:NMI watchdog: BUG: soft lockup - CPU#10 stuck for 22s! [$IP-m:69361]```
-------------

Master Node Rebooted

2020-05-22T02:02:00.001Z

**Incident (5/21/20): **

While fixing issues with the GPU nodes, the master node became unstable because of several mount points that were damaged. All of the mount were runtime filesystems so a reboot was requested and fulfilled by SRCF personnel the next morning. No SLURM jobs were reported as affected but new logins were denied until the master node was rebooted. The outage lasted for about 8hrs.

SLURM's Default Memory Per CPU Increased (1GB --> 2GB)

2020-05-07T18:12:00.001Z

Hi All,

Based on previous usage, the default allocation of 1GB of memory per cpu is too low. I have now increased this default to 2GB of memory per cpu.

If you are not setting memory requirements in your job(s), this will change will affect them.

If you are already setting memory requirements in your job(s), this change will not affect you.

Best practice is to specify memory in all jobs. If you haven’t been doing so and are now running into issues (waiting in partitions for longer than usual), you’ll now need to specify memory per cpu in jobs.

--mem-per-cpu=1G # when passed via command line,
or
#SBATCH --mem-per-cpu=1G # when added to a submission script.

Adding either option should set your memory per cpu back to 1GB.

** Again, if you are already using the --mem= or --mem-per-*= options, no changes are required for your job(s).

Thank You

SLURM Priority Adjustment

2020-05-07T15:15:00.001Z

Since priorities were not working for those users who use Memex less frequently and in smaller batches of submitted jobs, these parameters were adjusted:

PriorityWeightFairShare=20000
PriorityWeightTRES=CPU=1000,Mem=2000,GRES/gpu=3000

As a result gres/gpu was added to:

AccountingStorageTRES=cpu,mem,energy,node,billing,fs/disk,vmem,pages,gres/gpu

This functionality changes in SLURM 19+, but our current version is 18.08 which is the latest version packaged with OpenHPC 1.3.

Update: AcctGatherFilesystemType was also enabled for lustre.

Did You Know ... Slack Edition

2020-02-13T20:19:00.001Z

Did you know we have a Slack channel for HPC/Research Computing?

Signup to our Carnegie Institution for Science workspace (click here) and then join the #hpc channel. Please use your Google login, "@carnegiescience.edu", email address.

This channel is useful for a couple of reasons (as well as other Carnegie Science channels!):
- Anyone can share photos, Googe Drive documents, articles, issues and best practices.
- Anyone can send me an informal message to talk about HPC/Research Computing/Memex issues, software questions, usage statistics, and much more. If I can’t answer or address your concerns right away, a formal ticket can be created for a faster response ***.
- Messages can be directed to a particular user or posted to the channel for everyone to see. General issues can be shared to #hpc and searched by other users with similar questions or issues.

To join, click here!!

Other things you can do are create a new private channel, have a group call (type "/call @user1 @user2"), set reminders (type "/remind help" for options), connect with a Github (type "/github help" for options), and much more (formatting messages, built-in commands)!

*** Slack does not replace our ticketing system. It is a way for us to discuss research computing without the formality of a ticketing system (by emailing [email protected]" target="_blank" rel="noopener">[email protected] or typing "/freshservice-ticket" from Slack’s direct messaging system).

Did You Know ... Python Edition

2020-01-23T16:56:00.001Z

Did you know official support for Python 2 is over?

That said, we have Python 3 available on Memex by loading the module, "python/3.6.7". This Python version includes conda, R, Jupyter, IntelMPI, and many other packages. Most Python packages can be installed by sending your request to [email protected]" target="_blank" rel="noopener">[email protected].
Did you know you can install your own Python packages using conda?

For example, if you’d like the latest python version available, 3.8.1, first search for it here first (mainly conda-forge channel) or directly from the command line with:
```
 module purge \
#only needed to ensure a clean environment
module load python/3.6.7
conda search python \
#latest should be 3.8.1 as of last week
```
and use it to set up your own conda environment (both anaconda.org and the command line shows the proper channel, "-c", to use below).
```
 conda create -p /lustre/scratch/$USER/.envs/py38 \
-c conda-forge python=3.8.1
```
The above command assumes /lustre/scratch/$USER/.envs is where you want the conda environment, py38, to reside. It has to be in a directory you own.

To activate any conda environment (using example, py38, above):
```
 source activate py38
```
To deactivate any conda environment:
```
 source deactivate
```
To remove an entire conda environment:
```
 conda-env remove -n py38
```
Did you know you can list existing conda environments already on Memex?

To list the available conda environments (still using the module, python/3.6.7), simply type:
```
 conda env list
```
if you can see them, you can activate them but if you don’t have write permission, you cannot install, remove, or modify them. You can however, clone them (make the new name unique–> "foo", not the clone name --> “foo2”):

conda create -p /lustre/scratch/$USER/.envs/foo --clone foo2
Did you know you can list any installed Python package, in the base conda environment (activated by module load python/3.6.7) or a custom conda environment (activated by source activate foo)?

conda list packagename #replace packagename with real package name

where “packagename” is a placeholder for a real package (like python, or r-base). Of course,
```
 conda list
```
lists all packages installed in your current environment.
Did you know there are other useful commands (#replace packagename and X with real values):

conda search packagename=X.X.X --info #where X.X.X=version_number of packagename
conda clean #removes downloaded packages and caches
conda remove packagename #to remove package
conda list --revisions #to identify a revision to rollback to
conda install --revision X #rollback to revision number, X

Type conda -h and/or conda-env -h for more options.

Note: To manage where your cache directory is located (includes tar balls and downloaded packages), you can set the following in your bashrc:

echo CONDA_PKGS_DIRS=/lustre/scratch/$USER/.envs/pkgs >> \
$HOME/.bashrc
source $HOME/.bashrc

Or in your current environment only with:

export CONDA_PKGS_DIRS=/lustre/scratch/$USER/.envs/pkgs

Did You Know ... Storage Edition

2020-01-15T15:57:00.001Z

Did you know there’s a 256GB quota for all /home directories?

This policy will be fully enforced in the coming weeks. This requirement is needed to manage space and load for /home. Home directories should primarily be used to set up scripts, small software applications, and environment variables across the cluster. To that end, we will be enforcing a hard quota of 256GB. A consolidation of all /home directories to one storage device will take place this year. The exact date will be advertised but please contact me or [email protected]" target="_blank" rel="noopener">[email protected] with any other questions.

Note: If you are currently over the limit, you will have time to move data but start moving (or removing) data now.

Did you know we have about 950TB of storage right now?

This includes the Lustre filesystem (/lustre, 698TB) and the MemexNFS filesystem (/work, /scratch, /share/apps, and /home ~ 252TB). Check out our login banner (see "Mountpoint Information") to see how each filesystem should be used.

Did you know we now have a Memex Globus endpoint, "cisuser#carnegiescience"?

Globus transfers are typically much faster than our command line option. Instructions on how to setup and use Globus on Memex is in our FreshService ticketing system (Solutions --> Computation).
Did you know you can check disk usage in the following ways?

For most users, the new command
```
 $ zquota
```
will show your /home usage and your group’s usage for /work/DEPT. Shared directories such as /share/apps, /scratch, and /lustre usage can seen by the ‘df’ command. For example:
```
 $ df -h /lustre /share/apps /scratch
```
shows usage for all three filesystems. Of course, other commands like $ du -sh /home/username/directory/ for directories or $ ls -Shl filename for files can be used as well.

Memex unable to accept new user logins

2020-01-14T21:45:00.001Z

Issue 01/14/20:

While installing new packages on the login server, the file /etc/resolv.conf was overwritten and caused new user logins to fail. Once the file was replaced with the proper nameserver values, Memex accepted new logins again.

Did You Know ... SLURM Edition

2020-01-13T21:58:00.001Z

Did you know there’s a gui to view SLURM jobs? Inside a VNC or “ssh -XY …” session, type sview -a from the command line (check out the menu selection as well):

Did you know you can view the maximum resources of each node with:

sinfo -e -o "%20N %10c %10m %25f %10G"
Did you know you can view the maximum memory used for running jobs with (see “/bin/sacct -e” for format options):

/bin/sacct --format="JobID,CPUTime,MaxRSS" -j JOBID

better yet, try sstat --format="jobid,maxrss,avecpu" -j JOBID (“sstat -e” for format options)?
Did you know you can email yourself job status changes?

#SBATCH [email protected]
#SBATCH --mail-type=FAIL,BEGIN,END,SUSPEND

Here are a few more tips and you can contact me directly with any questions.