Temporary Resolution 01/02/20:
Using rclone instead of rsync/cp/scp is 10x faster for large directory and possibly large file syncs to /memexnfs/ mountpoints. Although all reads from /memexnfs mounts are performing performing as expected, all disk to disk writes to /memexnfs are not. As load increases, presumably from cluster jobs and transfers (syncs to /home), write performance suffers. The workaround is to use rclone and this email was sent out to all users:

Please use rclone for large local or remote transfers while using /memenfs/* filesystems, or /share/apps/dept, /home/username, /work/DEPT, or /scratch/username. There seems to be an issue with the common linux commands, rsync and cp, when transferring large directories (size and number of files).

The solution is to use rclone instead of rsync or cp or scp for large directories (size and number of files),

rclone sync /home/username/directory/ /scratch/username/directory/ -LP

This syncing issue currently affects write speeds but not read speeds for large directories to /memexnfs/*. This solution has been tested and should also work fine for small directories and files.

Of course, rclone is used to sync files to/from GDrive as well.

Rclone Tutorial:
https://carnegiescience.freshservice.com/support/solutions/articles/3000040389

Issue 12/19/19:
A user reported rsyncs were tool slow on the /share/apps mount of /memexnfs/apps. Since all /memexnfs/* mounts share the same disks/setup, it was determined the issue was not isolated to /share/apps, /work, /scratch, and /home (all /memexnfs mountpoints across the cluster).