It's a little arbitrary. Look at the graph on page 6, there's no steep gap in the spectrum there. 16 just about the balance point
But there is a steep gap in the spectrum at 16 on page 7
That's the spectrum of LoRAs, which are LoW RAnk by design.
Yes. But from their paper: “In our analysis, we present compelling empirical evidence for the existence of universal subspaces
within LoRA adapters across different modalities and tasks.”
I also don’t understand what they write under figure 2, since resnet50 has 50 layers, not 31.
There’s lots of hockey stick charts in the paper that might answer this visually, if that’s of interest.
or is it just that 16 was arbitrarily chosen by them as close enough to the actual minimal number of dimensions necessary?