Suppose for some reason you lose your phone number. Then suppose it is recycled to people with bad intent. Could the number be used to fetch passwords? Could they get into your e-mail? You bet they could.
Millions of people get new phone numbers, some of which have been used before by someone else.