I do agree that genuine software excellence and craft is rare in corporate settings. If you want to make a piece of excellent software like curl or sqlite you will be doing it on your own, and you most likely will not be making a living off of it either.
1)
Whenever a new framework is introduced this usually requires broad consent, clear scope, and agreement on committing to this new direction, doubly so when it's introducing a new language. Do you embark on working on these projects without discussing what you'll be doing?
If I was managing a RoR project and someone went off for a week or two and came back with projects written in Prolog I'd be livid. Who's going to maintain that? Why wasn't this discussed?
Have you even considered code reviews? How are you going to have useful code reviews if no one else knows the language and _knows it well_?
I understand writing one-time use tools in whatever language you want if you're the only one who's going to be using it, but otherwise, it makes the most sense to stick to the team's strengths when in a commercial project.
2)
It took me time to come to terms with the fact that not everybody working professionally in software is passionate about tech/software/languages. Some people just view it as a dayjob and prefer to stick to the known, well-trodden paths rather than exploring the field (an old boss would often make the distinction between dayjobbers and technologists aka people that are in love with tech).
Often great is the enemy of good enough and the most important part of working professionally in dev, for the vast majority of cases, is hitting the mvp and ensuring maintainability. Sometimes the time investment is better used elsewhere.
Channel your passion towards your personal projects and/or open source projects, not at work (unless the environment is conducive to that).
Now, startups are a wonderful place for mixing passion with work and a big exception to the above, usually there's a lot of room for experimenting and coming up with clever/complex/out-of-the-box solutions to problems.
Like I mentioned above. If I want quality, then I alone make this decision. If anyone wants to join, they join on my terms. I will make my terms uncompromising and uncomfortable for people who want consensus deliberately because I don't want to work with people who want consensus -- they suck at programming.
For my day job, I work with people who want consensus, an easy way to slack, pretend to work, hit the lowest bar available to pick the paycheck and go home, watch TV, play guitar, whatever. They don't produce anything that resembles quality work. Never will. That's not their goal, nor do they feel bad for not accomplishing that.
> Who's going to maintain that? Why wasn't this discussed?
Whoever knows how to program will.
It wasn't discussed because I wouldn't care about an opinion coming from someone who doesn't know how to program. Same way how I don't ask the neighbor's cat when I make programming decisions.
> Have you even considered code reviews? ... if no one else knows the language and _knows it well_?
Yes. I considered. This is their problem, not mine. If you want to be a programmer, it's your job to know your tools well. If you want a fat paycheck and be a useless blight on the bloat of the corporate world... well, make up the rules that are even more convenient to you...
> in love with tech
This has nothing to do with being "in love with tech". This is about the quality of output of people who are employed as programmers but aren't. One can be "in love" and suck, while other can "hate" and be very good at what they do. The reason why the situation the way it is that that human nature which pushes everyone towards a place where they can be lazy and ignorant meets no resistance.
While in many other professions there's a market for high-quality products, like very expensive and very high-quality watches, cars, photo equipment, clothes, food... in programming there's no market for anything that's trying for quality rather than time to market, price or reach. There's no niche, when it comes to programming market that would pay tenfold or hundred times more for a higher-quality product. That's why in industrial setting nobody is trying to make high-quality products, even though some, naively, come with this idea into the trade, they are quickly shown the reality where nobody cares.
Even if I didn't need the C++ part, HaXe way of doing things is very different from Flex. Also, the code for SWF generation required quite a bit of linear algebra, which scared the original developers (the code had to take various properties of display objects and convert them into matrices of affine transformations, because that's how SWF format natively encodes it).
Even after all that, the code would lose a significant portion of performance gained from it being written in HaXe. HaXe made two significant improvements on MXMLC: since it was able to prove type correctness at compile time it removed type checks at run time (that's almost like removing half of the bytecode). And, on top of that, if it was able to resolve a reference during compile time, it generated a more efficient code for lookup (i.e. MXMLC generated code that stacked all contexts within function visibility and whenever a non-local variable was referenced it'd put a call to a function to look through this stack; this especially penalized the code with nested functions).
> Likewise I doubt that the parser being written in C was the problem
Doubt all you want, but it was the problem... Why did you choose to second-guess me if you were never there, never interacted with people in question, never seen the program in question? Like, on what authority do you think you know better?
> the problem was probably that (I assume?) you didnt also provide the convenient python bindings
You guessed wrong...
> Otherwise it's hard to believe that someone would throw away finished code.
You must be new here... I've seen this more times than I can count. Sometimes it's even a good thing.
> seems a bit too lazy.
Being lazy and ignorant is the name of the game. This is what people in general are made of, but when it comes to programmers, they have fewer checks that make other professionals resist being lazy and ignorant.
> don't want to deal with it.
Because, let me repeat it: ignorant and lazy.
> Also I wouldn't want to inherit Prolog code either because it's an ancient niche language, IDE and docs and everything else is probably terrible.
You are also demonstrably ignorant and lazy: you didn't bother to check, and decided to spew a bunch of nonsense.
* What difference does it make in what year was a language created? English is older than Esperanto, does this make English worse? Hebrew is older than English, is Hebrew worse?
* Prolog is alive and well. Multiple language implementations released new versions in the last year. A lot of academic research happens in Prolog because, conceptually, it's newer and more advanced than, say, Java or Python.
* Documentation is fine. Python documentation is a lot worse for example, because it's written by idiots... Python documentation writes simply don't know how to do it well, so it doesn't matter if there are many of them, or how much effort they put into it -- what comes out is garbage.
* I haven't experienced technical problems programming in Prolog. Installed SWI Prolog, wrote some code, ran it, fixed errors, ran it again... nothing out of ordinary. Editor support was not a problem, not anymore than editor's support for Python or any other more popular language.
I sympathize with your manager here. If someone under me, ostensibly working on a Python app, wrote a component in C while I was away on leave, without clearing it with me first, I'd be pissed off too.
You decided to use a programming language that other people on your team didn't know, and therefore nobody other than you could maintain, debug, or extend what your wrote. And you did it underhandedly while your manager was away. You deserved the ire here.
There are lots of other ways you can make things closer to C-level performance while sticking closer to something a team of Python devs could maintain, e.g. using Cython or even running a Python program with PyPy (especially if it involves a lot of looping and basic string operations).
And did you even benchmark the Python implementation? How much faster was the C version really? Was it even a bottleneck in the system to begin with? How many developer-hours did you spend on the C version of this component, and how many would you have spent on the equivalent Python version?
Also, C is a really hard language to learn and use effectively, because of the loosey-goosey types and absence of memory safety. Your decision imposed a tremendous burden on the rest of the organization, which might not have been worth the performance gain in this one component of your system.
This was IMO a bad decision on your part, because it's overly fixated on the benefits of achieving a narrow technical objective, disregarding a variety of short- and long-term costs. At minimum, it's not at all obvious that your decision aligns with the broader goal of your team consistently delivering value over a longer period of time. There are countless war stories to be told, of overzealous junior- and mid-level ICs pulling shit like this and it ending up badly for the org.
It's one thing if another team actually committed to using a different programming language, and then backed off their commitment, as in the Haxe example. That's on them, not on you. But being a cowboy and writing stuff in hard-to-learn languages that other people on your team don't know, without any org-level buy-in for getting people trained up on it, is not at all a good habit to be in. Consider that you are the only common factor among all these problematic situations at different organizations.
If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate it.
As others have pointed out, a lot of your failures sound more political than technical. Dealing with idiots and jumping through political hoops to get them to go along with an idea that they didn't think up themselves can be extremely tiring. I cannot claim any particular expertise in that area myself, and I can't think of any way around it (aside from working alone, of course). I figure cooperation is the cost of progress.
My conclusion so far is that if you want to make things work well, you shouldn't be working on a commercial project, use a unpopular language with a steep learning curve to filter out those who'd be a drag on your project. Maybe you don't have to be a jerk, but being blunt helps.
Below are some examples of initiatives that were meant to improve things and how they failed due to other programmers being lazy and / or ignorant.
When ActionScript was a thing it competed with HaXe. A similar (also ECMAScript-related) language with a small but dedicated community, a compiler that was hugely superior to to MXMLC (official Adobe compiler for AS3) and a bunch of features intended to improve code correctness and performance.
I was hired by a company making a "PowerPoint online" kind of product. The main system component was a large Flex (AS3) applet that was hugely inefficient especially in terms of how it utilized network. It had to load huge shared libraries with assets (mostly clip art) every time users wanted to either edit or watch a presentation. The AWS bill was growing dangerously big. My mission was to find a solution to reduce the network activity.
My idea was to create a separate player component that would extract the relevant assets from the libraries server-side, compile them into individual SWFs. The reason was that the final presentations were loaded a lot more often and by first-time users (i.e. no caching). HaXe was the ideal language because it already had a library that could generate a large subset of SWF, and it could compile both to AS3 and to C++, so that generation could also be done on a server using a more efficient implementation.
After several month of work, I produced a set of programs that could generate SWFs both server-side and client side and showed how this would improve the network activity. The other programmers on the AS3 team, who earlier promised to get familiar with HaXe, since they had to incorporate the new player component into the existing Flex applet... didn't hold their part of the bargain. No matter the amount of help I provided, they simply wouldn't do anything to incorporate the new component, instead making claims that grew more bizarre and more untrue as time went by.
Having spent more time trying to convince the team to adopt my code rather than writing it, I decided to look for a different place to work at. In the end, this entire effort went down the drain.
----
In a very similar way, I had to solve a problem created by using Google's Protobuf Python bindings which required generating Python modules in order to function. We needed to have an API server that could simultaneously serve multiple versions of the same Protobuf API from similarly named modules. Since Google's implementation didn't allow this, I wrote my own (while my manager was on maternity leave). I improved parsing speed, network load, reduced the amount of maintenance the component needed by making it possible to add new Protobuf message definitions at run time...
The problem was I wrote the parser in C. This is what enabled good performance. When my manager came back to work, she realized she declared that she doesn't know C and will never learn (even though she wasn't related directly to the project), and the project was thrown to the dogs.
----
I have a similar story about extracting and aggregating Web interface from a RoR app, producing a Swagger definition... written in Prolog, which was also thrown away because Prolog. Similarly, had written an I/O tester for distributed filesystem in Prolog, which was thrown away for the same reason... And this answer will eventually hit the character limit if I keep listing things that were discarded simply because programmers didn't want to learn how to do their job.