Without seeing a plan of the two structures its difficult to say.
Say their wall was a separate wall and your fence was "inside it" and separate and not taking support from it.
Then it sounds as though someones wholly owned structure may have damaged your wholly owned structure.
If that is so, its possible he other party may be liable.
I think you would need a solicitor and someone like a chartered surveyor or architect to inspect, survey and swear the position in law and issue correspondence with the other side stating your position.
ONQ.
All advice on AAM is remote from the situation and cannot be relied upon as a defence or support - in and of itself - should legal action be taken.
Competent legal and building professionals should be asked to advise in Real Life with rights to inspect and issue reports on the matters at hand.