As someone who has been fostering and rescuing about 17 years, I know exactly what you went through. Most of the dogs I have had were great, but every once in a while you get one that does not work out in your home. This is not your fault. For whatever the reason, it's not a good fit.
Sometimes it is a physical condition; sometimes it's pack order; sometimes it's hard to know. As a foster, your input is extremely valuable so the rescue knows how to proceed with the dog in question.
IMHO, fosters are like gold. They, quite simply, save lives. There are more rescues that need fosters than not. If you are not appreciated where you are, there is some other group that would be glad to have you.
Bottom line: your family, and this includes your pets, have a right to feel safe in their own home. It is unfair for you and your pets to feel threatened because one animal is not working out. Face it: some dogs are better off being the only dog. All rescuers know this. And people giving up their dogs don't always mention negative things about the dog, putting others at risk.
So don't feel bad at 'inconveniencing' the rescue. You did nothing wrong.