If hawks or owls start coming around, and they survive the attempted attacks, they will leave for sure. There are other birds that will kill them for their nest. So, basically, they will book if everything isn't just so-so. I have a house, where 2 pair seem to just hang out from time-to-time, but I don't think they are nesting here. I have heard that if you can get a colony started, it can sometimes take 5 years!