Morally and ethically, yes. Legally, no.
Depends. This is one of the reasons why your licence should be in writing.
If you don't have it in writing, and your lodger is (let's say) a student who you both know will need accommodation until the end of the academic year, then the courts are likely to imply a term into your licence that says that you won't terminate the licence before the end of the year, unless the licensee is violating the terms of the licence in a way that make it reasonable for you to terminate it — e.g. bringing back guests when you're not there, smoking, damaging the property, misbehaving towards you or other licensees.
There's no default presumption that a licence is terminable at will for any reason or none at all. If there isn't an express termination provision the courts will imply a reasonable one, and what is reasonable will depend on the factors and circumstances of the case, the parties' shared intention or presumed shared intention in entering into the licence, etc. It's very, very unlikely that the courts will reckon the lodger intended to enter into a licence under which you would be entitled to throw him out because you decided you didn't like the colour of his eyes.
So, if you really want a licence that is terminable at will without notice, you have to put that in writing. And, even then, the courts will imply a term reflecting the parties' shared expection that they would each behave reasonably towards the other, so if you exercise your power to terminate at will in a capricious, oppressive matter, there's a good chance they'll find a remedy for the licensee.
You're probably better off with a licence agreement that sets a term and allows early termination without notice on stated grounds, or in stated circumstances — bringing in guests without permission, smoking, payment in arrears by more than 7 days, misbehaviour towards others in the house; that kind of thing. That creates a quite a clear set of specific shared expectations, and leaves much less room for the courts to imply terms that they think reasonable.