changelog Updates

hpc-internal.carnegiescience.edu

timestamp1637184777411

Memex Login Hung on 11/17/21

by Floyd Fayton

Initial incident and probable cause: Broken pipe on the login caused by a file update on the login. The master node and login were both rebooted and “wwsh file sync” commands were automated memex_routecheck.sh (crontab and cron.hourly).

timestamp1593115740001

Login hangs after kernel message...

by Floyd Fayton, HPC Admin

Update (6/29/20): The login hang was caused by high I/O load which is returning after a weekend hiatus. Unfortunately, limiting the I/O on the login is not yet feasible due to the design of the system. The issue is not due to a lack of...

timestamp1590112920001

Master Node Rebooted

by Floyd Fayton, HPC Admin

Incident (5/21/20): While fixing issues with the GPU nodes, the master node became unstable because of several mount points that were damaged. All of the mount were runtime filesystems so a reboot was requested and fulfilled by SRCF...

timestamp1579038300001

Memex unable to accept new user logins

by Floyd Fayton, HPC Admin

Issue 01/14/20: While installing new packages on the login server, the file /etc/resolv.conf was overwritten and caused new user logins to fail. Once the file was replaced with the proper nameserver values, Memex accepted new logins...

timestamp1576776900001

User reported that rsync/cp/scp too slow on /memexnfs/apps,

by Floyd Fayton, HPC Admin

Temporary Resolution 01/02/20: Using rclone instead of rsync/cp/scp is 10x faster for large directory and possibly large file syncs to /memexnfs/ mountpoints. Although all reads from /memexnfs mounts are performing performing as...

Newer