Its all about supply and demand; what you are looking for is what everyone is looking for, so prices will be high whether through agencies or not. Anyone sitting on land in that area knows the value of it and won't sell it cheaply even if you deal directly with the vendor.
Your best bet is to try to buy an existing run-down house and demolish/rebuild; it won't come cheap but will leave you a margin if you control the building costs well. Building on a greenfield site is almost impossible, usually premission only given for local need. If you buy an entire farm you will probably get permission, but again you are talking serious money per acre.