The explanation could be a combination of experience, details, survivor bias and true scottman.

That is with experience you can select something on the ocean far away that you can track (kelp, etc), with experience and focus to accurately track it and take into account its own movement. Then the method is obviously only promoted by those who successfully survived using it, as the potential nay-sayers who used it and failed are no longer there to give a counter-point. Finally, those who did not use it successfully are probably characterized as "not good navigators", in circular logic.