IP can get effected if it carries any bad domain. Since you are in VPS now, it's not possible. And also I have never heard of that an IP can effect a site or changing the host can be a problem to SEO.
I will suggest that check on Google updates and all. I believe you are being effected by any update or of such sort. Changing your hosting is purely coincidental.