changelog Updates

hpc-internal.carnegiescience.edu

timestamp1590112920001

Master Node Rebooted

by Floyd Fayton, HPC Admin

Incident (5/21/20): While fixing issues with the GPU nodes, the master node became unstable because of several mount points that were damaged. All of the mount were runtime filesystems so a reboot was requested and fulfilled by SRCF...

timestamp1581625140001

Did You Know ... Slack Edition

by Floyd Fayton, HPC Admin

Did you know we have a Slack channel for HPC/Research Computing? Signup to our Carnegie Institution for Science workspace (click here) and then join the #hpc channel. Please use your Google login, "@carnegiescience.edu", email...

timestamp1579798560001

Did You Know ... Python Edition

by Floyd Fayton, HPC Admin

Did you know official support for Python 2 is over? That said, we have Python 3 available on Memex by loading the module, "python/3.6.7". This Python version includes conda, R, Jupyter, IntelMPI, and many other packages. Most...

timestamp1579103820001

Did You Know ... Storage Edition

by Floyd Fayton, HPC Admin

Did you know there's a 256GB quota for all This policy will be fully enforced in the coming weeks. This requirement is needed to manage space and load for > ==Note:== If you are currently over the limit, you will have time to move...

timestamp1579038300001

Memex unable to accept new user logins

by Floyd Fayton, HPC Admin

Issue 01/14/20: While installing new packages on the login server, the file /etc/resolv.conf was overwritten and caused new user logins to fail. Once the file was replaced with the proper nameserver values, Memex accepted new logins...

timestamp1578952680001

Did You Know ... SLURM Edition

by Floyd Fayton, HPC Admin

Did you know there's a gui to view SLURM jobs? Inside a VNC or "ssh -XY .." session, type Did you know you can view the maximum resources of each node with: Did you know you can view the maximum memory used for running jobs with (see...

timestamp1576776900001

User reported that rsync/cp/scp too slow on /memexnfs/apps,

by Floyd Fayton, HPC Admin

Temporary Resolution 01/02/20: Using rclone instead of rsync/cp/scp is 10x faster for large directory and possibly large file syncs to /memexnfs/ mountpoints. Although all reads from /memexnfs mounts are performing performing as...

timestamp1575316620001

Login Node Slowness (module command hanging on memex.carnegiescience.edu)

by Floyd Fayton, HPC Admin

Resolved 12/10/19: After tuning the NFS server and clients, the slowness was resolved. Although there were several adjustments, the RPCNFSDCOUNT variable in /etc/sysconfig/nfs was the change that made the biggest improvement (100x...

timestamp1571067600001

System Update & Failed disk in SureStoreHD, memexnfs ZFS pool degraded

by Floyd Fayton, HPC Admin

System Update 11/6/19: System was updated on November 8th, which includes updates to SLURM, ZFS (0.6 to 0.8, which improves time to rebuild failed disk), and CentOS (7.5 to 7.7). Resolved 11/2/19: Issue resolved. Replacement disk has...