If you have a solicitor then I think you should take his advice. On here you can only get opinions. Looks like you are in a very sad situation. If your ex partner is being aggressive to you then a barring order sounds like the best option. It would presumably also help to prove that your ex would not be a suitable person to get custody.
Passport or no passport there are ways and means for people to get people out of the country so be very careful about access, not trying to worry you, but unfortunatly someone who is abused like yourself may be in a vulnerable position, and sustainable to emotional pressure from your ex. Also the fact that the HSE are involved, I presume this means your ex cannot see the child alone. Be very clear on what you are going to do if your ex comes back to try and get custody. And the best person to advice you on that is your solicitor and also of course the advice of the HSE.