Connecting to NERSC¶
Opening an SSH connection to NERSC systems results in a connection to a login node. Typically systems will have multiple login nodes which sit behind a load balancer. New connections will be assigned a random node. If an account has recently connected the load balancer will attempt to connect to the same login node as the previous connection.
Connect to NERSC Computational Systems¶
Please make sure you have configured Multi-Factor Authentication (MFA) prior to login.
To access Perlmutter via
ssh you can do the following:
Similarly, you can access Cori with
If you have configured sshproxy then you can run the following:
ssh -i ~/.ssh/nersc <user>@perlmutter-p1.nersc.gov # or 'ssh -i ~/.ssh/nersc <user>@saul-p1.nersc.gov'
ssh -i ~/.ssh/nersc <user>@cori.nersc.gov
This assumes your identity file is in
~/.ssh/nersc. The sshproxy route will be convenient if you have multiple ssh connections without having to authenticate every time.
X11 forwarding allows one to display remote computer to your local machine, this can be done as follows:
ssh -X <user>@cori.nersc.gov
Alternatively, you can make this change persistent by adding the following line in
All NERSC computers (except HPSS) are reached using either the Secure Shell (SSH) communication and encryption protocol (version 2) or by Grid tools that use trusted certificates.
The Secure Shell (SSH) Protocol is a protocol for secure remote login and other secure network services over an insecure network. The protocol is defined by IETF RFCs 4251, 4252, 4253, and 4254. Among the most widely used implementations of SSH is OpenSSH.
Connecting with SSH¶
On a system with an SSH client installed, one can log into the Perlmutter or Cori login nodes by running the following command:
ssh <username>@perlmutter-p1.nersc.gov # or 'ssh <username>@saul-p1.nersc.gov'
If the user has generated a temporary SSH key using
sshproxy, then this command will connect the user to one of Cori's login nodes. If the user has not generated a temporary SSH key, then SSH will challenge the user for their Iris password as well as their one-time password (OTP) before connecting them to a login node.
Users are strongly encouraged to use the most up-to-date versions of SSH clients that are available.
Password-less logins and transfers¶
Consult the documentation on using the SSH Proxy service in the MFA documentation section for ways to connect to NERSC systems without reentering your password and one-time password.
SSH certificate authority¶
NERSC generates SSH certificates for the primary login nodes using a NERSC SSH certificate authority. Most recent ssh clients support these certificates. You can add the following entry to the
known_hosts file to make use of this certificate.
@cert-authority *.nersc.gov ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2yKBpvRbdD9MWiu+7wg17vBsKy46AjjuL27DmdpDYiCqRE2mN0om9b0jn4eI91RGykbcRa9wUKJ2qaD0zsD08A8HM+R14H4UsZ5hi7S+xGqscJH7uTmXy5Igo5xEOahS9Z+ecgonDCgWKJnbd/FRu4vITYXrvTlIIGHGRBYj0GzbgLHBzedoMaGNRwhVyadH2SGRaZCgbH+Swevzy0GwYfZJA9zd7EX0jiAClkSYcflIOsygmI3gHv+b35mrvXcHDeQOR/wg8knfpSiFLCkVDpfgnj27Lemzxe6k61Brhv9CUiq+t7WApVDBovhdXZn6pBg+OKeDk1G1OLvRbxJ2bw==
NERSC may occasionally update the host keys on the major systems. Check here to confirm the current fingerprints.
4096 SHA256:Db9s2Fa4J3qx7An5oIMgUqUAdK7UWJGTPGoIKD44+Gs perlmutter-p1.nersc.gov (RSA)
4096 SHA256:35yiNfemgwzHCHFrPGWrJBCCqERqLtOVSrR36s1DaPc cori.nersc.gov (RSA) 256 SHA256:YOycBUgqcXq5ZiO45oG8JKNo9sekO7n0C1XoOMpQZtc cori.nersc.gov (ECDSA) 256 SHA256:/bLLKa0JDmbElrot7lvVlf+CQC3tFC+e9CkCCObtS+o cori.nersc.gov (ED25519)
All of the dtn nodes should have the same fingerprints:
2048 SHA256:/cIQwTFd8zgeZKVdzE5Jqscu3IX3mRBn7ikaAGH5h6k dtn01.nersc.gov (RSA) 256 SHA256:tIO6fLqc2dHa1o3IGmWA5mtxqOURTlxHm3E6lV9zIGg dtn01.nersc.gov (ECDSA) 256 SHA256:wirBRUHXris8lXH856CnJMg6JFO2zSWqogXsDmZnZo8 dtn01.nersc.gov (ED25519)
The fingerprints for NoMachine host are encoded differently, depending on where you view them. In both cases keys are hashed using SHA256. The NoMachine client displays key fingerprints with hex encoding.
SHA256 E7 1D 95 A6 9D B4 A4 99 5B 1A 77 8E 2C FD CF FF D3 5E ED 32 BC 63 9E EB A4 46 F1 76 0F 66 49 23 nxcloud01.nersc.gov (RSA) SHA256 E3 2B 16 1A 97 39 02 FA D0 A8 D8 78 CD F7 EE DB 15 F3 90 B3 55 B5 A9 1F A0 6A F0 F7 E7 68 57 16 nxcloud01.nersc.gov (ECDSA) SHA256 98 C7 54 41 60 76 9B 71 33 D3 B0 43 11 0B C0 D6 9A 70 8F C9 D7 4C FA AE D9 72 48 88 90 86 24 AD nxcloud01.nersc.gov (ED25519)
If you are troubleshooting your NX connection by trying to manually SSH to
nxcloud01.nersc.gov, SSH logs display key signatures using base64 encoding.
2048 SHA256:5x2Vpp20pJlbGneOLP3P/9Ne7TK8Y57rpEbxdg9mSSM nxcloud01.nersc.gov (RSA) 256 SHA256:4ysWGpc5AvrQqNh4zffu2xXzkLNVtakfoGrw9+doVxY nxcloud01.nersc.gov (ECDSA) 256 SHA256:mMdUQWB2m3Ez07BDEQvA1ppwj8nXTPqu2XJIiJCGJK0 nxcloud01.nersc.gov (ED25519)
Scan for public keys on a NERSC host and generate their key fingerprints via:
ssh-keygen -lf <(ssh-keyscan -t rsa,ed25519,ecdsa <NERSC hostname> 2>/dev/null)
.ssh/known_hostsfile, verify their fingerprints via:
ssh-keygen -lf ~/.ssh/known_hosts | grep <NERSC hostname>
Depending on the ssh client you use to connect to NERSC systems, you may see different key fingerprints. For example, Putty uses different format of fingerprints as follows:
ssh-rsa 4096 3f:6b:e2:c7:95:36:ef:7c:ae:3f:be:1e:13:97:3b:08
ssh-rsa 4096 f6:36:aa:ed:60:9c:d5:f1:29:af:35:81:58:f6:26:45
ssh-ed25519 256 09:11:1a:d7:fc:d9:db:cb:11:c2:ea:6e:61:96:c3:3e
You may see the following warning when connecting to Cori with Putty, but it is safe to ingore.
PuTTY Security Alert The server's host key is not cached in the registry. You have no guarantee that the server is the computer you think it is. The server's rsa2 key fingerprint is: ssh-rsa 4096 f6:36:aa:ed:60:9c:d5:f1:29:af:35:81:58:f6:26:45 If you trust this host, hit Yes to add the key to PuTTY's cache and carry on connecting. If you want to carry on connecting just once, without adding the key to the cache, hit No. If you do not trust this host, hit Cancel to abandon the connection.
These are the entries in
perlmutter-p1.nersc.gov ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRDn86LRaU3Hl9E+OTwHtyLhLRvjNCF3e/LAuRu/Ms06FNsXTK1iIb3QMnWjzzFd4CwQwrKYZy9umHIJOyCW/SbEI4GRPrr7NJpVOvkGKVBqSo5aYY3LhqCqKgsX7lbnhwAyvYvHoy9qd/XyzjUMwvUYf1G0dmkIN4AGauEUysn2ixgfhi6+sODsLI9fNrZftnTpYecw2NPP7cn8Dwknh0Fw3450FaXZpUryghdsZp7VvtGwVjQYy5C2PFj8GKqoPh5pIrq4R63hx0OmQOQlRp8cSLtElx1vNzbTRMh43Te3/SJQ66h3sG8AgsPejJgPYEVKJhLpC8CQFj3mQFygcBcp5jOfNnEH2PWZ2Xmzi1p3qrtj01ZML6tIZX9GSdIPif0N2/Zi7+h2wsC+/d4gWdHwX0wQ+o3gYVCKl1Ipf3HJbQ+ZkJ+SecV81nWmu0sGP5nrKALYrQjSwsoO1yFxNhzFk0+XztFoSrMVsfLGG0n4ahTc/yJdB9/VL2/TcujiH+u64cDE9HhOw/JrHtUr13KC/0giaSc9qbjjdhKaXTDcdz4v9NCCqa6jb9R9+MDuufCJy/hOvvYqg99iCGAS+fKPvxJQ3byXOmzZ/dNVw0iz9jQdyTMISmn5BW23YQBtu9RbA8uvzhlV5vsTjtjadbVrWx4VEFgDLtBy8E0kACWw==
cori.nersc.gov ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCsxGw19ZL8EN+NZ9HhD+O/nATuvgZxcEuy/yXnGqz5wMzJj6rK7TsrdU8rdJNrhZDe3yjpCiKvqkbSKp22jK2/iMAeWDQvYpMgC6KyiNd0hztowtMFJEwb8gVmtkVioqIaf9ufJnOO0LX5A5J/4fQhICfbyPiX8SsjX0p655/kIm3T6hr7t89b4IkRu19/uWufbNaV/mZSFWl7asLKXJNTMhzEn6bsTcAqlm55Tp4NvCe1hvv6OY/vU5luDz09UDmnDfr/uukmVm5aIjtlZBGqbOe7huNJGIWhoGCN/SoArRu9T9c9fjOlRMOHcf0QYMQmxFQnR0TkJZQoJ5N+EYNUIB9dvnJs2mlN0ZEuUU0RwAUOge7RwujiZ2AWp/dV/PNvLGmDVUxiyXC0Uuw57Ga2e49hYisYU/J/NPp9AbHqO8M6kZqYdqWKYueIsM3FDti3vUbjV4J6sL6mOBbxuJpUhUEX5UXxGbR39hDVx9Lsj4dszu+mcBFnDNcpRCDjw3z+hDqdNNpzhIRlbHQErLBWL3vnn2MLnb/3z163gyRtu1iTuR5myBIs9jLDAsX94VbBzKWdCFe22x4Eo6HwB6u+UHlXov0fnBXtAmgwRegc1gQwxi2FXB/ty0q1EO+PYo3fjUVRRb4uqBBIvpFarwtL0T6iYAYgHY11vH9Z2BFAHQ==
"Access Denied", "Permission Denied" or "Too many authentication failures"¶
Common causes of these are:
- You are using your laptop username instead of your NERSC username (SSH will by default assume that your remote (NERSC) username is the same as your local (laptop) one. You can avoid this by:
- Passing your NERSC username explicitly, for example if your NERSC username is
ssh -u elvis cori.nersc.gov, or
Specifying your NERSC username in your laptop
Host cori Hostname cori.nersc.gov User elvis
You mistyped or forgot your password. Check that you have the correct passowrd by logging in to Iris. Logging into Iris also clears authentication failures, in the event that too many failed authentication attempts have prompted the system to lock your account.
You are using PuTTY or MobaXterm and the NERSC system you want is currently unavailable (These Windows SSH clients are known to not display the pre-authentication banners which NERSC uses to indicate down status). You can check the system status on the NERSC MOTD page.
You have many SSH keys in your keychain or SSH agent, and your SSH client is trying them all one at a time. After too many wrong ones are tried in a row, the SSH server will stop accepting further attempts. You can check if this might be the cause by adding
sshcommand and look for failed authentication attempts in the output, eg
ssh -vv firstname.lastname@example.org.
If this might be the cause, you can instruct your laptop SSH client to only use a specific identity, for example:
ssh -o IdentitiesOnly=yes -i ~/.ssh/nersc email@example.com you are using sshproxy (the identity sshproxy generates is called
ssh -o IdentitiesOnly=yes -i ~/.ssh/id.rsa firstname.lastname@example.org you are not.
If you are still unable to login, contact NERSC Account Support at email@example.com.
This message may appear when a connection to a new machine is first established:
The authenticity of host 'cori.nersc.gov' can't be established. RSA key fingerprint is <omitted> Are you sure you want to continue connecting (yes/no)?
- Check that the fingerprints match the list above.
- If they match accept
- If they do not match let us know.
Host identification changed¶
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! Someone could be eavesdropping on you right now (man-in-the-middle attack)! It is also possible that a host key has just been changed. ...
Ensure that your
~/.ssh/known_hosts file contains the correct entries for the host and confirm the fingerprints using the posted fingerprints above. Add the certificate authority line to your known_hosts file if you version of ssh supports SSH certificates.
- Remove any lines referring the host in question and save the file
- Paste the host key entries from above or retry connecting to the host and accept the new host key after verify that you have the correct "fingerprint" from the above list.
SSH connection disconnects periodically¶
Some users experience their SSH or Globus connections to NERSC systems disconnect periodically and somewhat randomly. One cause of this is due to a user's SSH client using jumbo Ethernet frames, while a network switch or router between the user's computer and the NERSC network is configured not to forward jumbo frames, e.g., a home WiFi router or a router maintained by an institution's IT division.
Users who experience intermittent failures of their SSH connections to NERSC systems are encouraged to read the detailed description of the jumbo frame problem by ESnet.
Other SSH connection failures¶
If users encounter unexpected failure when attempting to establish SSH connections to NERSC systems, they are encouraged to collect verbose output from the SSH client where possible, and provide that output in a ticket submitted via the NERSC Help Portal. If using a variant of OpenSSH, verbose output can be obtained by adding the
-vvv flag to the
ssh command. Other SSH clients may have other ways of emitting verbose output, and users are encouraged to consult the relevant documentation for obtaining such output.