A lot simpler than that: "Pay us €500 within 12 hours or we make a criminal complaint to the police and reserve the right to sue you for the full costs of your disruption".
Ooh, blackmail!
I like it. But criminal prosecutions for misbehaviour on aircraft are rare, and behaviour which is merely "unruly" isn't necessarily a crime. I think what you're doing has to rise to the level of a threat to safety before you have to worry about prosecution; spoiling other passengers' experience of the flight is probably not enough. And leaving aside whether you
could be prosecuted for it, the observed fact is that people very rarely are, which might suggest that the guards are not keen on running these cases, which might make people think that the threat of reporting the matter to the guards is not that serious a concern.
But I think you're on to something with the hotel fine for smoking in the room. Not being a smoker, I've never had direct experience of this, but I assume it works something like this: your contract with the hotel includes a clause that says something like "I won't smoke in a non-smoking room and, if I do, I understand that I incur an additional charge of €X for the purification, cleansing and exorcism of the defiled room". The hotel typically doesn't know until after you've left that you smoked in the room, but they have your credit card details so they charge your card.
Similarly with car hire; you may have left the premises before they observe that the car you just returned is missing its hubcaps and requires panel work on both sides; they charge the excess not covered by insurance to your credit card. (You authorised them to do this in that document you signed without reading when you picked up the car.)
In both cases, they have your money; it's up to you to dispute the charge.
I guess Ryanair could adopt that kind of collection mechanism.