I recommend setting a silent ringtone by default and then manually setting ordinary ringtones for your contacts.

Alternatively, if you’re on iOS, there seems to be a setting that automatically does this at:

Settings > Phone > Silence Unknown Callers: Set this to ON.