I did the same thing when I got my first Janome. I thought it was lint. I just put it back in the hole and the machine is going strong. That is the only place that you still have to oil. I don't think it is something that will damage the machine.
Sue