OP - do you not have this specified or anyone signing off the build who can advise you? A builder, bless them, will tend to suggest what's easiest/quickest/most profitable for them. Not the people to ask for independent advice.
"One builder is telling me to put minimal insulation in the cavity between the inner and outer external walls" - eh? WAT?
"to completely fill the cavity with the full 100mm insulation" - don't believe this is a supported build method with rigid insulation
"He reckons that the 70mm would be more than sufficient" - it's not down to the builder's opinion, this must comply with building regulations
"should be more worried about the u value of our windows" - again this is defined in the building regs.
"He prefers to skim the inner walls rather than dry line" - well he would it's much cheaper!
Not sure when you got your permission but the regs. are very strict on the U-values for walls, floor, roof, windows, etc.
FWIW - we have 60mm Kingspan in the 100mm cavity with 50mm insulated boards on all external walls. The main argument against pumping the cavity is coverage and the moisture path. The main arg. against the insulated boards is the complexity around sealing sockets, windows, etc.
There's loads of threads on this topic on here and on boards - it's almost a religious debate for some people.
SSE