Step one: add log severity to your log messages (pretty much every log library supports this out of the box).
Step two: add a log archive (you should have this anyway so that logs can be retained past the initial retention period of your log querying tools. Eg you might have a compliance requirement to keep logs for two years but you obviously wouldn’t want anything that old stored in your expensive fast log search)
Step three: create a way to ingest your archived logs (again, something your business should have, otherwise what’s the bloody point in having an archive)
Step four: have a rule that pushes logs of high severity straight into your log ingestion pipeline, and logs of lower severity into your archive.
Step four seems to be the piece that most people are oblivious too. But it’s generally really easy to implement. Particularly so if you’re using a reputable observability platform.
People who think “log everything” means “log PII” or “stick everything in the same log ingestion pipeline” are simply doing logging wrong. I’m not normally one to say “you’re doing it wrong” but when it comes to logging, these tools are long since mature now. The problem isn’t the tooling, it’s people’s awareness of it.
Step one: add log severity to your log messages (pretty much every log library supports this out of the box).
Step two: add a log archive (you should have this anyway so that logs can be retained past the initial retention period of your log querying tools. Eg you might have a compliance requirement to keep logs for two years but you obviously wouldn’t want anything that old stored in your expensive fast log search)
Step three: create a way to ingest your archived logs (again, something your business should have, otherwise what’s the bloody point in having an archive)
Step four: have a rule that pushes logs of high severity straight into your log ingestion pipeline, and logs of lower severity into your archive.
Step four seems to be the piece that most people are oblivious too. But it’s generally really easy to implement. Particularly so if you’re using a reputable observability platform.
People who think “log everything” means “log PII” or “stick everything in the same log ingestion pipeline” are simply doing logging wrong. I’m not normally one to say “you’re doing it wrong” but when it comes to logging, these tools are long since mature now. The problem isn’t the tooling, it’s people’s awareness of it.