I have PoA over my father’s finances so can tell you that this does not give any power over your father and who he is allowed to see. Legally the PoA is required to use the money in his charge for the sole benefit of the person whose money it is and whilst they can claim reasonable expenses and for legal advice necessary to run the PoA they are not allowed to use the money for their own personal benefit.
The practicalities are however a little more complicated because a care home will tend to work on the instructions of the one actually paying them.
Therefore the care home legally cannot prevent your father leaving on his own or with whoever he so chooses to go out with. The care home would have a duty of care and would be able to deny admission to anyone your father did not want visiting him and would probably not let him leave the home in circumstances they felt unsure about.
I would suggest what you do is try to make an appointment with the care home manager and discuss things to see if an amicable arrangement can be come to. It is a whole lot cheaper than taking action via the court of protection against your brother.
A lot of this is also about money in that some people need locking up for their own protection but if the state does that say under the Mental Health Act then they have to fund such care. They do not want to do that so they arrange for care or allow relatives to arrange care on a voluntary basis and then ignore the fact that these people become locked in care homes against their will.