You'll regret it.
Forget that you have a couple of bedrooms free. How much of the current ground floor space do you use? If you currently use all ground floor rooms, then there's little point in moving.
There's only two of us and we have a large rural house, 4 beds, 2 reception and office...