We have an ultrasonic device plugged in at home for the past two years. It seems to work, but before the device we usually only got colonised by a couple of mice once a year (as weather turned cold) and in some years we escaped their visits, so it is too soon to judge.
Certainly, the device has some effect: when we first plugged it in, the mice became far more visible - i.e. they ran across the kitchen floor in front of us, whereas we had previously only ever seen their droppings. This seems consistent with the alleged modus operandi of these devices: that they make a loud noise outside the spectrum of what we can hear, that this makes quite a racket for the mice, and that they are not able to listen for other sounds so well (hence blundering out in front of us, because perhaps they could not hear us in the room?) all of which perhaps makes the house an uncomfortable hangout for them.
The device is pretty cheap anyway, so there is little to be lost by trying it.