I realise you might not be able to get into too many specifics, without identifying yourself, but I just wonder, accused by whom?
Is it some kind of officialdom, like the school, or a social worker, or is it a remark by a neighbour? If the latter, then yes, it’s legal - they can hold what views they like. Doesn’t make it true, though, and they probably wouldn’t get anywhere with any complaint to the authorities.
But if it IS some branch of the authorities, presumably there is some formal process to contest this allegation? You would need expert advice (i.e. lawyer, most likely).
I don’t have children, so no, it hasn’t arisen. But how many parents can genuinely offer the “perfect” childhood, anyway? Is there any such thing? Family illness, relationship breakups, money worries are just some of the things that can go wrong. But that doesn’t imply all children affected by such things are “neglected”. Challenging circumstances and neglect are not the same thing.