Just a few points regarding the planning potential of the land:
Check that the land is zoned for residential use or a use that would permit the building of a house - don't assume anything from the fact that there are houses nearby. There could be zoning boundary between this site and their land. This information will be in the Development Plan for the area, which you may be able to view on your local authority web-site. Alternatively, give the planning department a call and ask.
If the site is located outside a town on agricultural land, then building a new house may be restricted to local need, ie. are you from the area, do you work the land, do you have ties that require you to be in that place, etc. Again - a call to the planner should clarify.
You will also have to think about how to get services onto the land. Is there electricity near by, water supply, drainage? You will probably have to pay for all of these at some point, so it would be worth looking into the costs involved now. Drainage, in particular. If there is no public system, and it sounds unlikely from your description, you will require a septic tank or similar. These have very specific requirements that can be affected by soil type, distances to boundaries, water courses, buildings and other septic tanks. I'd recommend getting this checked out, as without a drainage system you wont be able to build at all. Also depending on the soil, you may have to go for an alternative system, which may or may not affect your decision.
Although, there is expense involved in getting services fed direct to a site, it is worthwhile. I'd avoid doing deals with neighbours etc. about accessing their water supply, drainage system etc. as you wouldn't have complete control over these resources and it could affect the value of the property should you ever decide to sell.
Bear in mind that the planning process takes a minimum of 12 weeks from submission to grant of permission, if you're making an offer subject to this. It could take much longer, if any of the neighbours object - an appeal to An Bord Pleanala will take a minimum of 18 weeks.
Finally, sounds like the seller is playing hard to get with his 'don't care if I sell or not' - a shrewd operator! Why not phone up a few local auctioneers and tell them you're thinking long-term of buying in that particular area and want to get a feel for land prices at the moment. You don't have to commit yourself and they should have a better idea of where the market value is.
You're taking the right approach - do your homework, get a feel for the market and then, if you do decide to go ahead, bear in mind that values are projected to drop a further 10% this year, so make an offer that builds in a bit of security for yourself.
Best of luck!