I was in a situation like this before where the seller tried to make us buy behind the EA's back. The EA's take on it was they had 'introduced' us to the house and therefore were entitled the selling fee.
It's not over just at Sale Agreed you know, there can be an awful lot of hassle involved in the whole sale transaction, where dealing directly with the buyer can get very tricky. You would need a very flexible and understanding solicitor to go ahead with this.
Also if this Sale falls through in two weeks time say, you are left with no estate agent. And you can be sure if you then try to use another estate agent, the first EA will be on the blower to let them know you tried to do the deal behind their back.