You were right to tell them to contact your listed referees. This woman was not your employer or line manager and as such should not be contacted for a reference.
You said the right thing..... she is not the person to ask about your abilities. Don't say that you did not get on as the interviewer may have a good relationship with the other person and then it is possible that they would blame you for the conflict.
I think that you handled the situation correctly.
Good luck getting the job, I will keep my fingers crossed for you.
Hugs
Caroline