This will limit remote access to those users who really need it. 

• Open the /etc/ssh/sshd_config file.
  • # vi /etc/ssh/sshd_config
• Add an AllowUsers line at the bottom of the file with a space separated by list of usernames. For example, user tecmint and sheena both have access to remote ssh.
  • AllowUsers david mark
• Now restart ssh service.