Before our rule engine has a chance to touch the document, we run several pre-processing steps that imbue semantic meaning to the words it reads.
> LLMs have completely overshadowed ML NLP methods from 10 years ago, and they themselves replaced decades statistical NLP work, which also replaced another few decades of symbolic grammar-based NLP work.
This is a drastic oversimplification. I'll admit that transformer-based approaches are indeed quite prevalent, but I do not believe that "LLMs" in the conventional sense are "replacing" a significant fraction of NLP research.
I appreciate your skepticism and attention to detail.
1. https://jalammar.github.io/illustrated-word2vec/
2. https://jalammar.github.io/visualizing-neural-machine-transl...
3. https://jalammar.github.io/illustrated-transformer/
4. https://jalammar.github.io/illustrated-bert/
5. https://jalammar.github.io/illustrated-gpt2/
And from there it's mostly work on improving optimization (both at training and inference time), training techniques (many stages), data (quality and modality), and scale.
---
There's also state space models, but don't believe they've gone mainstream yet.
https://newsletter.maartengrootendorst.com/p/a-visual-guide-...
And diffusion models - but I'm struggling to find a good resource so https://ml-gsai.github.io/LLaDA-demo/
---
All this being said- many tasks are solved very well using a linear model and tfidf. And are actually interpretable.
Indeed, before that there was a lot of work on applying classical ML classifiers (Naive Bayes, Decision Trees, SVM, Logistic Regression...) and clustering algorithms (fancily referred to as unsupervised ML) to bag-of-words vectors. This was a big field, with some overlap with Information Retrieval, lending to fancier weightings and normalizations of bag-of-words vectors (TF-IDF, BM25). There was also the whole field of Topic Modeling.
Before that there was a ton of statistical NLP modeling (Markov chains and such), primarily focused around machine translation before neural-networks got good enough (like the early version of Google Translate).
And before that there were a few decades of research on grammars (starting with Chomsky), with a lot of overlap with compilers, theoretical CS (state-machines and such) and symbolic AI (lisps, logic programming, expert systems...).
I myself don't have a very clear picture of all of this. I learned some in undergrad and read a few ancient NLP books (60s - 90s) out of curiosity. I started around the time where NLP, and AI in general, had been rather stagnant for a decade or two, it was rather boring and niche, believe it or not, but was starting to be revitalized by the new wave of ML and then word2vec with DNNs.
I'm just a bit skeptical about this quote:
> Harper takes advantage of decades of natural language research to analyze exactly how your words come together.
But it's just a rather small collection of hard-coded rules:
https://docs.rs/harper-core/latest/harper_core/linting/trait...
Where did the decades of classical NLP go? No gold-standard resources like WordNet? No statistical methods?
There's nothing wrong with this, the solution is a good pragmatic choice. It's just interesting how our collective consciousness of expansive scientific fields can be so thoroughly purged when a new paradigm arises.
LLMs have completely overshadowed ML NLP methods from 10 years ago, and they themselves replaced decades statistical NLP work, which also replaced another few decades of symbolic grammar-based NLP work.
Progress is good, but it's important not to forget all those hard-earned lessons, it can sometimes be a real superpower to be able to leverage that old toolbox in modern contexts. In many ways, we had much more advanced methods in the 60s for solving this problem than what Harper is doing here by naively reinventing the wheel.