Passive scans that hit your port 22 can hardly be called a security issue, and changing your port number definitely does not add any sort of security. This is a confusing concept, since changing your port number might decrease the probability your server gets taken advantage of (temporarily). It's a trivial fix for attackers everywhere to scan multiple ports instead of one at marginal cost.