I have hear both being used, do they mean different things or are they both used in the same way?
No, they’re different.
“Oh, bother” is an expression of annoyance and frustration at something that isn’t going right. It’s like saying, “Oh, shoot” or it’s more vulgar variant. For instance, you’re in a hurry and late for an appointment, so you park, gather your things, get out of the car, slam the door locked and closed, and realize you’ve just locked your keys in the car. At that point, you would stamp your foot on the ground and say “oh bother!”
“Oh, brother” is usually said while you’re rolling your eyes at something that’s so stupid, it’s hard to believe. It’s more like “for crying out loud” or “for Pete’s sake”. For instance, if you see someone in line to view a casket and they’re laughing and talking with the person next to them, but when they get up to the casket, they burst into tears and wailing for a minute…all you can do is roll your eyes at the phony grief and say, “Oh, brother.”