Yes, I guess it’s curious that the information lost doesn’t seem very significant (this also matches my experience!)
Pray tell, which dimension do we lose when we normalize, say a 2D vector?
Before normalization, the vector lies in R^n, which is an n-dimensional manifold.
After normalization, the vector lies in the unit sphere in R^n, which is an (n-1)-dimensional manifold.
>>> Magnitude is not a dimension [...] To prove this normalize any vector and then try to de-normalize it again.
Say you have the vector (18, -5) in a normal Euclidean x, y plane.
Now project that vector onto the y-axis.
Now try to un-project it again.
What do you think you just proved?
But the intuitive notion is that if you take all 3D and flatten it / expand it to be just the surface of the 3D sphere, then paste yourself onto it Flatland style, it's not the same as if you were to Flatland yourself into the 2D plane. The obvious thing is that triangles won't sum to 180, but also parallel lines will intersect, and all sorts of differing strange things will happen.
I mean, it might still work in practice, but it's obviously different from some method of dimensionality reduction because you're changing the curvature of the space.
> triangles won't sum to 180
Exactly. Spherical triangles have the sum of their interior angles exceed 180 degrees.
> parallel lines will intersect
Yes because parallel "lines" are really great circles on the sphere.
Consider [1,0] and [x,x] Normalised we get [1,0] and [sqrt(.5),sqrt(.5)] — clearly something has changed because the first vector is now larger in dimension zero than the second, despite starting off as an arbitrary value, x, which could have been smaller than 1. As such we have lost information about x’s magnitude which we cannot recover from just the normalized vector.