Basically, in order to justifiably retain the deposit/portion of deposit you have to prove that you have suffered financial loss. Extra people above those stated in a lease may cause extra wear and tear.
Did you do a complete detailed inventory of the property stating the condition of floors, walls, furniture (room by room), paint work etc?. Was it signed by the tenant? Failure to do this basic requirement may invalidate any claims with the PRTB you may have for retaining any of the deposit as you have no evidence of the condition of the property at the end of the tenancy as opposed to the start of the tenancy.
You do realise that the tenants have no obligation to move out at the end of a fixed term lease, should they wish to remain, as they have acquired Part 4 rights to remain in residence for a total of 4 years before a new agreement could come into force. They may remain without signing any new lease which then becomes a Part 4 tenancy.