Not at all. It is very easy for anyone to do, the info is all online.You gotta trust some people surely you are talking dedicated hacker here?
In a housing estate or urban area it would be very easy for an unauthorised user to be using your connection without your knowledge.