If you didn't sign a contract with the first agent, it's hard to see how they could claim a fee. They may be happy just to pass a client on to someone else or they may not realise that you have entered into a contract with another agency.
Agency Y's fee will almost certainly be payable even if they are not the ones who introduce the buyer, that'll be covered in the contract you signed with them.
Best inform agency X that agency Y have exclusive representation rights for the property, and if they wish, they can pass on the details of the interested party. They may look for an additional finders fee, it's up to you then whether it's worth paying that.