Am I wrong? What should I have done?

No, you aren't wrong for trying to protect your children. If it were me and I seen that I would have done the same thing. I would have probably also asked the mother, just the mother and children, if they would step into a room together where no one else was around and tell her what they said and talk to her about it. And if she would have denied it then I would have definitely had investigated just to make sure that they were okay and in goods hands. You don't know the guy so you are just concerned about the fact that he is around your kids.