The problem, as you’ve discovered is that the builder can’t claim VAT relief, only you can. Some companies are well up on how to do this (when my kitchen and bathroom were initially installed, the company just had me sign the declaration and they did the rest), but others don’t have the knowledge or the systems in place to do it on your behalf.
So the only solution in this case is for you to pay for materials from the source and claim VAT relief on the hardware. I did this when the bathroom was changed to a wetroom. So I paid the company who supplied and laid the floor, whilst the builder who did the rest of the work wasn’t VAT registered so there was no VAT. Undoubtedly, he paid VAT on some of his supplies but I basically had to just live with this (i.e. tiles, plasterboard etc).
This may of course mean you have to supply your builder with a separate copy of the VAT relief form for each supplier. Or you contact every supplier and arrange to pay for everything yourself, but each time you’ll need a new VAT relief form.
It might be a beggar of a way to do things, but 20% off all the materials will be worth it.