.Google.com's Gary Illyes discloses Googlebot collects web links rather than "adhering to" all of them, negating formal documentation.Googlebot picks up links instead of observing all of them in real-time.Gary Illyes discussed this info in Google's 'Search Off The File' podcast.Google.com's documentation opposes the real spider behavior.