I think the key to this conversation is "older man", I firmly believe you look at your past history with rosy colored glasses. You sort of round the edges, make things softer, if you know what I mean. Yes, people used to use old clothing, because that is all they had. The women welcomed bits and scraps of new fabric, but it was not readily available.
Your quilt is splendid!!