The only restriction on family is between parent & child - you can't claim rent supplement if you're living at home with your parents. There should be no problem in renting from your brother.
You will have to provide a tenancy/lease agreement in order to make the application (same as you would for any application). You will have to make a brand new application for Rent Supplement if you move address (whether it's to your brother's place or not).
If the rent supplement is approved, your brother will be asked to provide his PPSN as landlord to the HSE, which in turn is forwarded to the Revenue Commissioners each year, along with the amount of Rent Supplement paid in respect of the tenancy.