I would be just like you. It would be great to just ignore those five years. You are in a tough position. If you say no, your husband may be resentful. If you say yes, it will be hard to be friendly to the kid even though nothing that happened was his fault.
The only bright side I see to this is the boy calling and wanting to visit is a nice testament to the impact your husband had on his life. Many step kids are glad to get rid of the step parent, especialy a short term one. Your husband must be a good father figure.
I hope whatever decision you come to, allows peace in your house.