So sorry about your brother. I know its hard. Maybe this will help you out with bringing yourself to make the quilt.
When my mother passed away I inherited her wedding rings. My father said she wamted him to make sure I got them. I certainly couldn't wear them so I put them in a drawer and would take them out every once in a while, hold them, think of her and even cry sometimes. Ten years later when dad passed away I recieved his wedding band. I started to put it away and it hit me thats not what they would have wanted me to do, so I had them made into a lovely piece of jewelry which I can wear and whenever I go anywhere fun or travel I wear it and feel like Im taking a part of them with me.
I have never regretted doing this and I dont think you will ever regret making this quilt once it is done. You will have fond memories every time you look at the quilt. Maybe you could make blocks and use a square of the clothing as the center of each block. I certainly hope this helps