Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

A possible fix for slow SSH connections Network
I noticed that while SSHing to some of my servers at work that connections where taking forever. The few severs I could actually connect to would take over two minutes to connect to before I would get a password prompt. Other servers would time out alltogether after three minutes.

Most of this activity takes place over a VPN connection to a Watchguard. I thought I might have screwed something up with all of the config changes I made, so I reinstalled OS X. After a fresh install, I could connect to all of my servers. The connections still took a long time (30 to 45 seconds) to establish, but at least I could get in. After upgrading from 10.0 to 10.3.8, the same problem came back. After some more digging, I started running SSH in three levels of debug mode (i.e. ssh -v -v -v user@192.168.0.3). I found that it would hang at the following point:
...
debug1: Remote protocol version 2.0, remote software version OpenSSH_3.8.1p1
debug1: match: OpenSSH_3.8.1p1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_3.6.1p1+CAN-2004-0175
debug3: Trying to reverse map address 192.168.0.3.
...
Read the rest for my solution...

At this point, I tried adding the following entry in /etc/hosts:
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
192.168.0.3     sun     <---- NEW ENTRY
127.0.0.1       localhost
255.255.255.255 broadcasthost
::1             localhost 
And now I can connect without SSH timing out on me. Issuing either ssh user@sun or ssh user@192.168.0.3 brings back an instant password prompt. I tried adding a second server to the hosts list that would usually time out, and it instantly came back with a password prompt as well.

If anyone else's SSH connections have been timing out or taking forever I hope this helps.
    •    
  • Currently 3.38 / 5
  You rated: 5 / 5 (8 votes cast)
 
[66,451 views]  

A possible fix for slow SSH connections | 30 comments | Create New Account
Click here to return to the 'A possible fix for slow SSH connections' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
A possible fix for slow SSH connections
Authored by: legacyb4 on Mar 31, '05 10:51:38AM

Sounds like your sysadmin didn't set up reverse records on your internal DNS server where you are connecting to. This is a temporary fix, but not having those in place can affect a handful of other network services relying on the ability to do these lookups.

Cheers.



[ Reply to This | # ]
Static DNS
Authored by: lullabud on Mar 31, '05 12:39:08PM

If it is a DNS problem a static DNS entry in the current Network Location would fix it. Static DNS's get put ahead of the DHCP DNS's in the resolv.conf file.



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: ghinteclinn on Mar 31, '05 12:40:56PM

What legacy said.
You (or your site admins) need to fix your DNS records so that reverse lookups work.



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: Safar on Mar 31, '05 11:03:09AM

... i think you could change 'VerifyReverseMapping yes' to 'VerifyReverseMapping no' in /etc/sshd_config on the host computer. But i'm not sure...



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: rprestonatmitre on Mar 31, '05 11:21:55AM
Changing the line
VerifyReverseMapping yes
to
VerifyReverseMapping no
will also fix the delay. Because I'm the only ssh user on my system, I've also added the line
AllowUsers myShortUsername
to etc -> sshd_config

You may also want to create an etc -> hosts.deny containing

ALL: ALL: deny
to stop all incoming connections. To enable ssh logins, create etc -> hosts.allow,
sshd: ALL
sshd-keygen-wrapper: ALL

I'm not sure this is the best solution, but it works. It blocks all incoming connections except ssh. All ssh connections get passed to the ssh daemon, which allows only the user "myShortUsername" to connect, and there is no delay due to reverse name lookups.

[ Reply to This | # ]

A possible fix for slow SSH connections
Authored by: ptroot on Mar 31, '05 09:12:49PM

That would be in ssh_config not sshd_config. sshd_config controls the local daemon.

The problem is a dns timeout. ssh is trying to resolve the
address. The easiest fix is putting it in hosts. Next would be making sure dns is working.



[ Reply to This | # ]
sshd_config vs. ssh_config
Authored by: sjk on Apr 03, '05 07:12:10PM
What would be in ssh_config? VerifyReverseMapping and AllowUsers keywords are only valid in sshd_config.

[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: sapporo on Mar 31, '05 11:04:57AM

Could it be that ssh is trying to reverse map the server's IP to detect DNS spoofing? If so, you could try adding "CheckHostIP no" to ~/.ssh/config.



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: Lido on Mar 31, '05 01:16:07PM

I am having this problem as well. See thread here:

http://forums.macosxhints.com/showthread.php?t=37233

The thing is, if reverse dns is the trouble, shouldn't it go away if I login using the IP address instead of the domain name (of the site I'm trying to login to from Terminal)? It still hangs when I use the ip address.

---
-Lido



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: sahilt on Mar 31, '05 01:57:39PM
Your logic is reversed (pun most definitely intended). Refine your understanding of reverse dns; you'll realize that the problem exists exactly when you specify an IP. Reverse DNS refers to looking up a hostname that corresponds to a particular IP which, in this case, is specified as an argument to ssh.

[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: Lido on Mar 31, '05 02:08:28PM

Well, if reverse dns is needed whether you do:
ssh -luser domain.com

or

ssh -luser x.x.x.x

then that still might make reverse dns the issue.

---
-Lido



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: Lido on Mar 31, '05 02:10:35PM

I just asked my friend to login to my server from his mac and he can do it fine, so it's not something to do with reverse dns (unless somehow he turned his checking off on his mac). This problem seems to be isolated to my mac trying to ssh to a specific server.

---
-Lido



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: ghinteclinn on Mar 31, '05 03:00:51PM

Then most probably one of the two nodes, your machine or the remote machine, is not configured to do name lookups correctly. A common mistake is not including relevant domains in the resolver search lists.



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: Lido on Mar 31, '05 05:25:14PM

On my machine I can connect to other servers so I don't understand what would screw this one up. Also, other people can connect to the server that's giving me trouble (including me if I ssh through another server).

I don't see why I need to add anything to any config file to ssh connect somewhere. I thought it was like using a browser. Tell terminal the username and server you want to connect to and it tries. It shouldnt need to know anything beyond that should it?

---
-Lido



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: ghinteclinn on Mar 31, '05 05:46:29PM

Yes, it should. Look under "spoofing attacks" and ways to prevent them. Checking DNS records is a common first step in spotting a spoof attempt.



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: rprestonatmitre on Mar 31, '05 05:54:45PM
This may help, found in man ssh(1):
ssh automatically maintains and checks a database containing identifications for all hosts it has ever been used with. Host keys are stored in $HOME/.ssh/known_hosts in the user's home directory. Additionally, the file /etc/ssh_known_hosts is automatically checked for known hosts. Any new hosts are automatically added to the user's file. If a host's identification ever changes, ssh warns about this and disables password authentication to prevent a trojan horse from getting the user's password.
Perhaps deleting the entry for the machine you're trying to connect to will help.



[ Reply to This | # ]

A possible fix for slow SSH connections
Authored by: Lido on Mar 31, '05 07:04:39PM

What's weird is that added the server name and ip address to /etc/hosts and that fixed the problem. Then I read your post about /known_hosts and commented out the line I'd added to /etc/hosts and deleted the line in known_hosts related to my server. It also worked. The solution is as baffling as the problem.

---
-Lido



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: grumpy on Mar 31, '05 05:16:59PM

I had the problem originally described as well and came up with the same solution some time back.

Worth noting though is that of two machines I have, the 10.2.8 machine has always worked okay. The 10.3.X machine did work but then stopped working at one point when one of the updates for 10.3 was applied. Both these machines are on a home network behind a ADSL modem with firewall. I already tried the various suggestions about fiddling with the sshd options and none worked that I could find. The only thing that worked was the hosts file entry.

If I go onto a totally different BSD system on the Internet and go to the same target machine, it has no problem at all.



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: Lido on Mar 31, '05 05:50:35PM

Ok, I sudo'd and edited my /etc/hosts file and now it works. I don't like it that that is how it got fixed because it makes no sense to me. Did the recent security update disallow ssh connections to servers that don't have reverse dns set up? If so, that makes sense, but I still don't like it.

---
-Lido



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: ptroot on Mar 31, '05 09:19:22PM

but you're going to a private address. That means your dns server has to do the resolution. Since you are putting the IP
address in the command line, that tells me you don't have
the sun in any dns.

From a terminal, run nslookup. Then you can do a couple things. First, check if any dns server is handling the 192.168
network you are accessing.

set q=soa
0.168.192.in-addr.arpa.

will check the class C network 192.168.0.0 network.

and then

set q=any
3.0.168.192.in-addr.arpa.

will let you check the reverse lookup.



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: striker on Apr 02, '05 01:13:51PM

The problem is DNS lookup timeout. Editing /etc/hosts provides a quick fix for any given client, but does not eliminate the problem.

You need to check /etc/resolv.conf and make sure that it points to a valid name server.

There needs to be at least one line of the form:
nameserver: A.B.C.D

Where A.B.C.D is the IP address of a DNS server which responds. DNS servers are queried in the order in which they are specified, so if you already have nameserver entries, make sure they respond.

The other file you might want to look at is /etc/nsswitch.conf
This file tells the system where it should look in order to resolve various things. The line you care about is the 'hosts' line. It should probably read 'hosts: files dns' (Not sure about any mac-specific stuff that might be in there too. I'm a unix guy, not a mac guy.)

'files' tells the system to first check for host entries in /etc/hosts. 'dns' tells it to check /etc/resolv.conf for name servers to query.

Hope this helps.

Rached Blili



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: teefour on Apr 02, '05 06:13:03PM

tcpdump showed me that my mac won't try the order of preference which lookupd is supposed to use, it tries mDNS (Rendezvous) instead. Adding an address to /etc/hosts will fix it, but a better way would be to set the search to hosts first, followed by dns and mDNS afterwards.
So far i haven't bothered because the following in ~/.ssh/config

Host myserver
Hostname ip-address
Port 22
User foo

fixed it for me. (the reverse still doesn't work, but it's fast enough this way)



[ Reply to This | # ]
sshd, lookupd, reverse DNS
Authored by: sjk on Apr 03, '05 07:06:49PM
That might explain the "Apr 3 11:48:26 hostname sshd[1731]: reverse mapping checking getaddrinfo for example.org failed - POSSIBLE BREAKIN ATTEMPT!" warnings from sshd even when reverse DNS is properly configured. But I don't see any way to configure lookupd to avoid that; it's currently using:
LookupOrder: Cache FF DNS NI DS
_config_name: Host Configuration
Another symptom is the who command (for instance) displaying the IP address for remote hostnames, while DNS PTR lookups return accurate data.

Anyone know what needs to be tweaked so address-to-name lookups will work properly, without creating static host entries, when DNS is verified as correctly configured?

[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: rhodium on May 11, '05 01:03:10PM
After reading through this trying various 'fixes' I said the hell with it (figured apple was playing with more authentication methods than I needed [i.e. rsa/dsa auth only please]) and recompiled ssh. Worked like a charm - ssh is BACK!!

FWIW, I couldn't seem to get
 UseDNS no 
or
VerfiyReverseMapping no
on the server to effect anything from a Mac client. It does work on with other *NIX clients though. I actually believe the client was forcing it, and there isn't any client option to disable it.. This combined with a ton of GSAPI authentication methods was more overhead than I wanted to incur.


Here's what I did:
From a Terminal

> curl -O ftp://ftp5.usa.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-4.0p1.tar.gz  
> tar zxpvf openssh-4.0p1.tar.gz
> cd openssh-4.0p1 
# Now I don't want to replace the existing binaries - I want new versions.. 
# There may be a good reason for the old one so I'll keep the existing ones
# I just want a faster ssh..
> ./configure --prefix=/usr/local --sysconfdir=/etc 

This should result in a configuration that looks like this

OpenSSH has been configured with the following options:
                     User binaries: /usr/local/bin
                   System binaries: /usr/local/sbin
               Configuration files: /etc
                   Askpass program: /usr/local/libexec/ssh-askpass
                      Manual pages: /usr/local/man/manX
                          PID file: /var/run
  Privilege separation chroot path: /var/empty
            sshd default user PATH: /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
                    Manpage format: doc
                       PAM support: no
                 KerberosV support: no
                 Smartcard support: no
                     S/KEY support: no
              TCP Wrappers support: no
              MD5 password support: no
                   libedit support: no
       IP address in $DISPLAY hack: no
           Translate v4 in v6 hack: no
                  BSD Auth support: no
              Random number source: OpenSSL internal ONLY

              Host: powerpc-apple-darwin8.0.0
          Compiler: gcc
    Compiler flags: -g -O2 -Wall -Wpointer-arith -Wno-uninitialized
Preprocessor flags: 
      Linker flags: 
         Libraries:   -lcrypto -lz 
Then just make and your off and running..

> make
# move our existing configurations over..
> sudo mv /etc/ssh_config /etc/ssh_config.old
> sudo mv /etc/sshd_config /etc/sshd_config.old
> sudo make install
Then test it using the old method first....

> time /usr/bin/ssh <some local machine> date 
/usr/bin/ssh <some local machine> date  0.10s user 0.07s system 1% cpu 8.890 total
YUCK!!!
Now let's try our new ssh..

> time /usr/local/bin/ssh <some local machine> date
/usr/local/bin/ssh <some local machine> date  0.03s user 0.04s system 12% cpu 0.582 total
Rock on!!

[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: rhodium on May 11, '05 01:19:51PM
I added a few more things..

First, fix /usr/sbin directory..

cd /usr/sbin 
sudo mv sshd sshd.old
ln -s /usr/local/sbin/sshd .
Then fix the /usr/bin directory..

cd /usr/bin
sudo mv ssh-keyscan ssh-keyscan.old
sudo mv ssh-keygen ssh-keygen.old
sudo mv ssh-agent ssh-agent.old
sudo mv ssh-add ssh-add.old
sudo mv ssh ssh.old
sudo mv scp scp.old
sudo mv sftp sftp.old
sudo ln -s /usr/local/bin/ssh* .
sudo ln -s /usr/local/bin/scp .
sudo ln -s /usr/local/bin/sftp .
Then you're all set..

[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: name99 on May 27, '05 10:49:35PM
There is a very important PROBLEM with the above hint. While it appears great (and, for the most part works) DO NOT, REPEAT, DO NOT enter the lines

   cd /usr/sbin 
   sudo mv sshd sshd.old
   ln -s /usr/local/sbin/sshd .

The idea behind these lines is that we replace the built-in ssh server with the newly built one. Unfortunately, while the newly built ssh server is probably faster than the old one, it suffers from the grievous defect of not working. Who knows what's going on inside it, but it now rejects any passwords it is given. The new server does not work with either the old Apple client or the newly built client. So the bottom line with this hint is: (1) Download as above, but check for versions. openssh-4.1p1.tar.gz has just been released, rather than the 4.0p1 of the hint. (2) Config, build, install just like above. (3) Do NOT set up the server links as suggested. Continue to use the Apple supplied /usr/sbin/sshd. (4) Do set up all the links to change the /usr/bin client apps.

[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: Sefu on Aug 30, '08 04:48:35AM

...your suggestion just turns the new ssh installation off.

I also have the same problem though with refused passwords - I think the problem with this configuration lies with the old public/private key pairs remaining in a directory also in OpenSSH's file path directive - I will try to remove these come Monday, and let you know the results.

If anyone has found another solution (especially if the above is not it), then do please let us know. Thanks!



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: Sefu on Aug 31, '08 01:12:59AM

I noticed that there is no instruction here to change the 'startup items' sshd to the new one... still looking for possible reasons this is not working. Again, any input would be very helpful. Cheers.



[ Reply to This | # ]
A possible fix for slow SSH connections
Authored by: maestric on Aug 02, '07 06:12:41PM
A possible fix for slow SSH connections
Authored by: Sefu on Sep 08, '08 10:08:01AM

...nope, this configuration is still refusing passwords. Can anyone see the why of this?



[ Reply to This | # ]