changelog Updates

hpc-internal.carnegiescience.edu

timestamp1579103820001

Did You Know ... Storage Edition

by Floyd Fayton, HPC Admin

Did you know there's a 256GB quota for all This policy will be fully enforced in the coming weeks. This requirement is needed to manage space and load for > ==Note:== If you are currently over the limit, you will have time to move...

timestamp1579038300001

Memex unable to accept new user logins

by Floyd Fayton, HPC Admin

Issue 01/14/20: While installing new packages on the login server, the file /etc/resolv.conf was overwritten and caused new user logins to fail. Once the file was replaced with the proper nameserver values, Memex accepted new logins...

timestamp1578952680001

Did You Know ... SLURM Edition

by Floyd Fayton, HPC Admin

Did you know there's a gui to view SLURM jobs? Inside a VNC or "ssh -XY .." session, type Did you know you can view the maximum resources of each node with: Did you know you can view the maximum memory used for running jobs with (see...

timestamp1576776900001

User reported that rsync/cp/scp too slow on /memexnfs/apps,

by Floyd Fayton, HPC Admin

Temporary Resolution 01/02/20: Using rclone instead of rsync/cp/scp is 10x faster for large directory and possibly large file syncs to /memexnfs/ mountpoints. Although all reads from /memexnfs mounts are performing performing as...

timestamp1575316620001

Login Node Slowness (module command hanging on memex.carnegiescience.edu)

by Floyd Fayton, HPC Admin

Resolved 12/10/19: After tuning the NFS server and clients, the slowness was resolved. Although there were several adjustments, the RPCNFSDCOUNT variable in /etc/sysconfig/nfs was the change that made the biggest improvement (100x...

timestamp1562701260001

Intel Python Issue.. conda base corrupted

by Floyd Fayton, HPC Admin

On July 8th, the conda environment for modules, python/2.7.0 and python/3.6.0, was affected after an incomplete install for seaborn and pandas was aborted. Subsequent steps to fallback to a sane state and install those packages failed...

timestamp1555085220001

Error via getvnfs networking

by Floyd Fayton, HPC Admin

Fixed: by restart httpd on OpenHPC master node. Issue: Down nodes could not be reimaged because of the PXE process hanging at the getvnfs stage (seen in /var/log/messages, boot never makes it past getvnfs). Affected nodes: memex-c[014...

timestamp1552065900001

Login Stalled - SSH denial/System locked up

by Floyd Fayton, HPC Admin

Memex's login became unresponsive to established and new SSH sessions. A reboot was initiated shortly after. The initial concern is that a new package, abrt-gui, was installed, detected a system issue and halted the server. Admin...

timestamp1551141600001

Memex Up: Switch failure in Rack 2 (resolved)

by Floyd Fayton, HPC Admin

Issue: The switch in rack 2 has powered down for reasons unknown (probably failure). We will be inspecting the switch and if necessary, have Dell replace it. Apologies for the interruption of your work but we are diligently working on...