I think the problem is that in spots where the concepts build on one another, you need to memorize the lower level concepts or else it'll be too hard to make progress on the higher level concepts.

If you're trying to expand polynomials and you constantly have to re-derive multiplication from first principles, you're never going to make any progress on expanding polynomials.