Suppose for some reason you lose your phone number or retire it for something local. Then suppose it is recycled to people with bad intent. Could the number be used to fetch passwords? Could they get into your e-mail?
You bet they could!
Millions of people get new phone numbers, some of which have been used in the past by someone else.