If you can do that, do that. It's great.
That being said, deploying and loading these certs is a fun adventure, even in somewhat modern software.
We're discovering a surprising amount of fairly modern software which handles certs in stupid ways. For example, if I recall right, NodeJS applications tend to load certificates to secure a connection to PostgreSQL into memory and never bother to reload. libpq on the other hand loads them on connection startup. Even Spring wasn't able to properly reload certificates 3-4 years ago and this was only added in recent versions with dynamic certificates - a colleague patched this in back in the day. Our loadbalancers folded the ticket of "Hey, reload these three certs" into the issue of reloading the entire configuration, including opening, closing ports, handing TCP connections over transparently, and a billion other topics.
Funny enough, if there is enough stuff running, there's not even an agreement on how to store stuff. Some stuff needs PEM-files, some stuff needs PKCS8 stores, some stuff wants 1-2 PKCS12 keyrings with certs and keys in there in separate entries. I even had to patch a PKCS12 handling library because one ruby library needed everything in one entry in a PKCS12 key store and there was no go-code under the sun to find to do this.
So there is a sunny place in which PKI is indeed solved. And besides that there is a deep dark hole that goes deeper than it should on earth.
Also, I used to do IT, I get it but what do you think the fix here is? You could also run your own CA that you push to all the devices and then you can cut certificates as long as you want.
> PKI isn’t a solved problem.
PKI is largely a solved issue nowadays. Software like Vault from hashicorp (it's FIPS compliant, too: https://developer.hashicorp.com/vault/docs/enterprise/fips) let you create a cryptographically-strong CA and build the automation you need.
It's been out for years now, integrating the root CA shouldn't be much of an issue via group policies (in windows, there are equivalents for mac os and gnu/linux i guess).
> Just because someone’s homelab is fully cert’d through Caddy and LE that they slapped together over a weekend two years ago, doesn’t mean the process is trivial or easy for the masses.
Quite the contrary: it means that the process is technically so trivial the masses can do it in an afternoon and live off it for years with little to no maintenance.
Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology.
You have no idea the environment they work in. The "skill issue" here is you thinking your basic knowledge of Vault matters.
> Software like Vault from hashicorp (it's FIPS compliant, too: https://developer.hashicorp.com/vault/docs/enterprise/fips) let you create a cryptographically-strong CA and build the automation you need.
They didn't tell you their needs, but you're convinced this vendor product solves it.
Are you a non-technical CTO by chance?
> there are equivalents for mac os and gnu/linux i guess
You guess? I'm sensing a skill issue. Why would you say it's solved for their environment, "I guess??"
> Quite the contrary: it means that the process is technically so trivial the masses can do it in an afternoon and live off it for years with little to no maintenance.
I'm sensing you work in a low skill environment if you think "home lab trivial" translates to enterprise and defense.
> Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology.
Absolutely meaningless statement.
* Yes, I have experience with Vault. I have deployed it internally, used it, loathed it, and shelved it. It’s entirely too cumbersome for basic PKI and secrets management in non-programmatic environments, which is the bulk of enterprise and business IT in my experience.
* You’re right, the organization is the problem. Let me just take that enlightened statement to my leadership and get my ass fired for insubordination, again, because I have literally tried this before with that outcome. Just because I know better doesn’t mean the org has to respect that knowledge or expertise. Meritocracies aren’t real.
* The reason I don’t solve my own PKI issues with Caddy in my homelab is because that’s an irrelevant skill to my actual day job, which - see the point above - doesn’t actually respect the skills and knowledge of the engineers doing the work, only the opinions of the C-suite and whatever Gartner report they’re foisting upon the board. Hence why we have outdated equipment on outdated technologies that don’t meet modern guidelines, which is most enterprises today. Outside of the tech world, you’re dealing with comparable dinosaurs (no relation) who see neither the value or the need for such slick, simplified solutions, especially when they prevent politicians inside the org from pulling crap.
I’ve been in these trenches for fifteen years. I’ve worked in small businesses, MSPs, school campuses, non-profits, major enterprises, manufacturing concerns, and a household name on par with FAANG. Nobody had this solved, anywhere, except for the non-profit and a software company that both went all-in on AD CA early-on and threw anything that couldn’t use a cert from there off the network.
This is why I storm into the comments on blogs like these to champion their cause.
PKI sucks ass, and I’m tired of letting DevOps people claim otherwise because of Let’s Encrypt and ACME.
I've deployed Vault both at home and in two different companies, doing anything from pki, mutual-tls, secret storage and other stuff.
> > Software like Vault from hashicorp (it's FIPS compliant, too: https://developer.hashicorp.com/vault/docs/enterprise/fips) let you create a cryptographically-strong CA and build the automation you need.
> They didn't tell you their needs, but you're convinced this vendor product solves it.
It was an example from the ecosystem of available tools but in general yes, Vault can do that. Mentioning FIPS compliance was about letting you know that the software can be used also in governative environments. It's not just a "homelab toy".
> Are you a non-technical CTO by chance?
Senior cloud engineer here. Worked anywhere from not-so-small companies (250 people, 100 engineers) to faangs (tens of thousands of engineers).
> > there are equivalents for mac os and gnu/linux i guess
> You guess? I'm sensing a skill issue.
You're attacking me on a personal level because you can't argue otherwise. That's a common logical fallacy ("Ad Hominem" - https://www.britannica.com/topic/ad-hominem). You basically have skill issue at debating =)
> Why would you say it's solved for their environment, "I guess??"
When you account Windows, Mac OS and Linux you're accounting for pretty much the totality of the desktop computing landscape. The last two macbooks I had for work came with the mac os equivalent of group policies with certificates installed etc etc. Enterprise-tier Linux distributions can do that as well (eg: Red Hat Enterprise Linux).
> I'm sensing you work in a low skill environment if you think "home lab trivial" translates to enterprise and defense.
Again, worked anywhere from companies with 250 people to FAANGs. You have skill issue at sensing, it seems.
To get back to the point: homelab "triviality". In a way, yes. Large enterprises and even more defense can spend all the money not just for software but even for consulting from various company that can bring the skills to implement and maintain all the services that are needed, and train your people at that. Things become non trivial not on the base of technical issue, but on the base of organizational issues...
If we talk government and defense... Do you know the US government has dedicated cloud regions (eg: https://aws.amazon.com/govcloud-us/)? Do you really think that cloud providers offer those services at loss? Do you really think a few vault enterprise licenses are the issue there?
And by the way, Vault is just an example of one of the possible solutions. It was meant to be an example but you clearly missed the point.
> > Hence, if a large organization is not able to implement that, the issue is in the organization, not in the technology.
> Absolutely meaningless statement.
I think it's very meaningful.
It's not 1995, cryptography isn't arcane anymore. We had hardware crypto acceleration in cpu since at least 2010 (AES-NI). The tooling is well established on both servers and clients. The skills are on the market ready to be hired (either via employment or via contracting).
The issue is not technical in nature.
Oh and by the way: I've worked closely with engineers working for the US government. I wasn't close to the US government (because I am not an US citizen) but they were. They were "close enough" that they had to work in a SCIF and could only interact with me via phone. The systems they were working on... Those systems had their own private CA (among other things).
It's feasible. It's not a technical issue. If it's not done then it's an organizational issue.
My username is literally a cryptographic mode of operation. But you didn't know that, because you have a low skill issue.
> Do you know the US government has dedicated cloud regions (eg:
This is a joke, right? You're just an LLM going through training.
Again, ad hominem attack... You proved yourself to be quite a fool. You can't argue, you can't back your own opinions, you are only capable of attacking on a personal level.
> This is a joke, right? You're just an LLM going through training.
My account is from 2015 and has ~11k points. Your account is 4 months old and barely has 150 points. It's more likely that you're a poorly trained LLM (whoever trained you had skill issues :P) rather than me.
I'll be dropping this useless conversation. Farewell.
That's not what ad hominem means. Ad hominem doesn't mean I can't insult you
> My account is from 2015 and has ~11k points. Your account is 4 months old and barely has 150 points. It's more likely that you're a poorly trained LLM (whoever trained you had skill issues :P) rather than me.
Content matters more than age you goofball
How do you do this on a proprietary device from the late 90s that runs a WindRiver VXWorks RTOS with 1 MB of SRAM? The updated (full color!) Panelview HMI is running Windows CE 6.0, so it's perhaps more likely to be compatible, but I don't think the same group policies exist on that platform.
The masses can do it in an afternoon because they can choose to only install modern equipment that's compatible with the new requirements. Some of the "heavy iron" castings for the machines we have to work with were built more than a century ago, and only later automated with tubes and relays, and then again with the first PLCs that could do the job. But now "SSL everywhere" policies and certificate expiration timelines that don't make a distinction between firewalled OT networks and Internet-facing webservers don't allow anything to run for a decade without major, risky rewrites that cost tens of thousands of dollars for highly specialized engineering services and minimal downtime. Sure, adding a cert to the SCADA server is trivial, it runs Windows Server and has a NIC that can access the Internet, but on the other NIC...there's a menagerie of 30 years of industrial oddities.
If your homelab is still working after 2 years, that's great, but if it's not running after 100 years would you call that an organizational failure?
I've automated certificate upgrades on everything from dodgy network appliances to IBM System i servers.
I've used everything from headless browsers to scripted 5250 terminal emulators. Generally, if you have the will, you can automate anything that you do manually. The solutions are often janky, but they can usually be verified easily by other tools (curl output can be parsed to verify cert details).
You weight the cost of incurring in that thing getting hacked against the cost of that thing being rebuilt on modern hardware with modern technologies and enough processing power to do TLS.
Then you pick the "rebuild route". Easy.
Anyway, it's crazy how on this forum it goes from "REWRITE EVERYTHING IN RUST!!! ANYTHING THAT'S NOT RUST IS INHERENTLY UNSAFE" to the complete opposite of "why doesn't anybody think of the poor WindRiver VXWorks RTOS!!".
> Sure, adding a cert to the SCADA server is trivial, it runs Windows Server and has a NIC that can access the Internet, but on the other NIC...there's a menagerie of 30 years of industrial oddities.
Not gonna lie, all i hear is "i'm annoyed i was able to ignore the problem for 30 years but now i have to actually fix it".
> Not gonna lie, all i hear is "i'm annoyed i was able to ignore the problem for 30 years but now i have to actually fix it".
You're not entirely wrong, but you're talking to the wrong guy. Clean-slate rewrites are some of my favorite best projects, and I hate dealing with legacy junk - but there's just not budget to keep everything brand new all the time.
Realize that when you flush the toilet, the reason the water level goes down is likely to be a municipal sewer system and waste treatment plant that must never stop...they replace mechanical wear items once in a while, but some of the controls are well over 30 years old. Same story at the clean water plant that fills the tank back up. An average consumer might replace their phone every 2 years, but industrial processes and infrastructure have much, much longer timelines.
Hey, I get it.
The thing is, budget is largely artificial. Unless a company is on the brink of bankruptcy, there can be budget (but the company has to eat that from profits).
The thing is, eating budget for maintenance and system upgrades is a known playbook companies use. That doesn't make it right though.
That is why i wrote that the issue is organizational, not technical...
Just because someone’s homelab is fully cert’d through Caddy and LE that they slapped together over a weekend two years ago, doesn’t mean the process is trivial or easy for the masses. Believe me, I’ve been fighting this battle internally my entire career and I hate it. I hate the shitty state of PKI today, and how the sole focus seems to be public-facing web services instead of, y’know, the other 90% of a network’s devices and resources.
PKI isn’t a solved problem.