The goal here is to see if the candidate understands the domain (generic distributed systems) well enough on their own. For more senior roles I look to make sure they can then communicate that understanding to a team, and then drive consensus around some approach.
This is why I’m stuck at Senior lol. I can craft incredibly deep technical documents on why X is the preferred path, but when inevitably someone else counters with soft points like DX, I fall down. No, I don’t care that the optimal solution requires you to read documentation and understand it, instead of using whatever you’re used to. If you wanted to use that, why did you ask me to do a deep-dive into the problem?
Making the dependencies of "it depends" explicit is the whole point.
This also means there are 100 builders in my area who only build stick frame houses and won't even talk to you if you want something else and only 1 or 2 who will even think about those other options. (they do compete with the other builders so costs are not unreasonable)
You run into questions of how well a candidate remembers a project, which may not be perfect. You may end up drilling into a project that is trivial. The candidate may simply parrot things that someone else on the team came up with. And when candidates say things, you really have no way to understand if what they're saying is true, particularly when internal systems are involved.
I have found system design interviews specifically much much better at getting signal. I have picked a real problem we had and start people with a simplified architecture diagram of our actual system and ask them how they would solve it for us. I am explicitly not looking for people to over design it. I do give people the advice at the start of every skills interview tp treat this as a real work problem at my startup not a hypothetical exercise.
I have had a lot more luck identifying the boundaries of people's knowledge/abilities in this setting than when asking people about their projects.
And while everyone interviewing hates this fact, false positives are very expensive and can be particularly painful if the gap is "this person is not a terrible programmer, just more junior than we wanted" because now you have to either fire someone who would be fine in another role if you had the headcount for it or have a misshapen team.
I have found system design interviews specifically
much much better at getting signal. I have picked a
real problem we had and start people with a simplified
architecture diagram of our actual system
I believe this will tend to be true even in the ideal case, which is when the interviewer is focused on "is the candidate asking great questions and modifying their proposed design accordingly" rather than "is the candidate coming up with the 'right' solution."
Because, if the candidate his already built/designed a similar system, they will naturally be able to ask better and more focused questions about the design goals and constraints.
I have tried giving the experiential interviews Casey
describes, but I find it quite hard to get signal out of them.
[...] when candidates say things, you really have no way to
understand if what they're saying is true, particularly when
internal systems are involved.
https://youtu.be/gZ2V5VtwrCw?t=2068
The host discussed work he did for Amazon on an event ticket ordering system. Now, I don't think you need to know much about the ticket sales business to think about the challenges here!
The most obvious challenge to me is some sort of distributed lock to ensure the system doesn't don't sell the same seat to multiple people. Another obvious challenge would be handling very bursty traffic patterns, ie an onrush of people and bots when popular event tickets go on sale. Another challenge, brought up by the host, that I wouldn't have thought of is that ticket sales need to avoid "fragmentation" as much as possible. Most people don't want single seats, so you want to sell tickets in a fashion that doesn't leave scattered, "orphaned" single seats that nobody wants to buy.
Those are interesting challenges, and if the interview is an experienced developer, I don't think a candidate could really bullshit through them to a significant degree.
The candidate may simply parrot things that someone else on
the team came up with.
You may end up drilling into a project that is trivial
You run into questions of how well a candidate remembers
a project, which may not be perfect
Yes, this is not an IQ test, we are trying to see how people react to problems in our domain not measure some generalized form of reasoning. The advantage of picking a problem as close to our real problems as possible is that I don't have to worry how they generalize from the interview to work.
In general, my experience with system design interviews is that people make bad designs, and when you drill down on them they give bad rationales. Similar to coding screens, people just out themselves as not very good at their jobs regularly.
> Those are interesting challenges, and if the interview is an experienced developer, I don't think a candidate could really bullshit through them to a significant degree.
It's not really about "bullshit" per se, but about whether their understanding of their context is correct or not. They can tell you fully reasonable sounding things that are just wrong. In a mock interview, you can see if they ask good questions about their context.
> I personally wouldn't care if the ideas originated with the candidate vs. another member of the team. I'd be looking for how well the candidate understood the problem and the solution they implemented, the other solutions considered, the tradeoffs, etc.
I totally disagree with this. It is very different to be able to remember the design doc for the project and parrot the things that were talked about vs actually writing it.
If I want to hire someone who can design things well from scratch and I get someone who makes bad decisions unless someone is supervising them, I will be very disappointed.
In general, I have given both interviews to the same candidate and after saying a bunch of reasonable things about their existing work, when I ask them how to do design something I quickly find that they are less impressive than they seem. Again, maybe I'm bad at giving experiential interviews, but being hard to administer is a point against them.
My experience of hiring is also that I am generally not looking to take chances, unwinding bad hires is super painful.
Actually, this is a case of Peters principle[1], clear as day.
Recruiters should not recruit programmers if they, the recruiters, have not worked in a programming context. The best software development recruiter is another programmer.
It also extends, of course, to technical managers and is another factor in how one should approach recruitment interviews - is this placement going to result in a high Peter principle, and if so - is the candidate going to be capable of rising above it, or dealing with it in a way that is conducive to the organizations goals?
Because Peter principle is not always a 'negative' - its more of just a condition that results from a lack of communication between parties who really should know each others jobs, better.
People who know how to understand other peoples jobs as well as their own, work great together.
[1] - not Flynn effect
Ran into this the other day, a company reached out, and while I wasn't job hunting, the senior role they were trying to fill was basically exactly a perfect fit for me from a skills and background on a Venn diagram.
The first two calls were young, non-technical women (they shared linkedin links), and they clearly did not understand what they were hiring for and couldn't answer questions. They insisted on their scripted questions, and didn't want to talk about the companies where I did the exact role they were hiring for.
I was not rude or arrogant about this but the next day got the "Unfortunately, we've..." email. It's actual pretty funny, I'm just glad I didn't really need the job.
Companies, stop letting HR be your first contacts and screening before technical folks. It doesn't work. No wonder your pipelines are full of the fakers and liars like many of you lament.
Indeed, this is what I believe is endemic to the recruitment industry - the scripts are there to get the "# of interviews done" statistic in line with some expectation; the interview itself being successful, is secondary to the purpose.
I have personally saved myself huge headaches by flunking a recruiter who wasn't really doing the job, just pretending to.
(I agree screening by non-programmers is flawed, but there's always a compromise. Involving programmers at this early stage can be a source of frustration and wasted time).
Imagine the recruitment industry around Munich or Melbourne or Singapore. Similar fish, very different kettles.
In my career, some have and some haven't. In my current job, there's a screening by non-tech recruiters, then managers, then me and possibly others for the real tech interview. My EM is technical but cannot go in-depth as we can.
Are there companies that skip the tech interview step?
> Recruiters should not recruit programmers if they, the recruiters, have not worked in a programming context. The best software development recruiter is another programmer.
I don't think recruiters must be programmers, or even that that's a good idea. Recruiting and doing the tech interview are different steps.
Also, this is changing now, radically, with AI/ML tooling - we programmers can indeed cut out a lot of middlemen now.
"recruiters must be programmers" - this absolute is not a policy, rather a test against the nature of the relationship being offered. If I, as a developer, am mis-represented by the recruiter to the tech interviewer, then it is an inadequate recruitment.
If however, a tech-savvy recruiter does the tech interview and understands the nature of the systems being built/technology needed for specific cases, then the process is smoother - not just for the company, but also for the individual.
This pre-supposes, of course, that the interview candidate is taking the process seriously, and identifies faulty onboarding practices. If you're not asking pointy questions about why a recruiter is being involved, or finding out whether or not the recruiter understands your Resume/CV well enough to represent you, then you're putting yourself on the wrong foot.
Developers/technologists being managed by non-devs, is a huge source of friction - and bloat - in the software industry.
The point is that if you're getting interviewed by someone who cannot understand your job, they are in the throes of a Peter principle manifestation, and you are very likely to end up in the same situation if you don't identify the incompetence - you'll inherit it.
Overall shallow knowledge is not a positive signal, in my opinion. If they really are a firefighter who constantly jumps around, the interviewer should lean in to the organizational challenges they face when identifying and fixing problems across a variety of projects and domains. There's always a way to drill down with more specific questions.
And I'm not interested in the sorts of specific details that go fuzzy over time. I'm interested in a big picture view of choices made and alternatives discarded. We should talk about some implementation details as well, but "I don't recall" is definitely an acceptable answer for some decently large percentage of those choices.
If you're talking about a Rails project you did ten years ago, I don't really care if you used Sidekiq instead of Resque or vice-versa. I mostly want to know if you know what sorts of jobs should be moved to background tasks and how you structure those jobs and what are the tradeoffs etc.
Also, if I (the interviewer) selected one of the candidate's more "boring" roles, I'd happily let them suggest we focus on one that is juicier and/or one where they were more involved in the design process.
OTOH, I'd hate to be in a position where I had to think carefully about every aspect of that stuff during an interview. Even if nobody noticed an inadvertent classified spill, it could still suck.
I definitely empathize with your perspective. Simultaneously, I can also see how it would be most ideally fair to candidates to have some method of evaluating them without relying on intimate details of recent job experience.
E.g., let’s pretend that Steve Wozniak took 5 years off to raise children, are you going to pass him up because he has no job experience from the last 5 years? You’d be well within your rights to, but as a hiring manager I want to choose the best candidate, not the candidate with the best resume.
Everyone in GIS knows all the geometry stuff, but also if you say "in Postgres" you're strongly implying PostGIS.
That is the entire point of the technical interview.
Someone who is already experienced in a particular subject is also at least qualified to ask the right questions to the candidate related to the subject.
It all can be solved by a system where 1 company hires programmers then other companies can rent specific programer by hour and leave reviews.
This way a programmer doesn't need to interview, he's always employed and getting paid.
If they find no company to bid on your time, they can fire you after sometime.
It’d also make job hopping far less painful (see above: cheaper for the candidates) which is why they don’t.
So we may conclude: the point of leetcode interviews is wage suppression.
Like you said, tech companies need candidates to feel like they barely passed a grueling interview because it makes them wary to jump ship and have to go through that again, not that they are well qualified industry professionals who have the credentials to move between jobs and work anywhere they are certified.
If they find no company to bid on your time, they can fire you after sometime.
That's much less efficient for big software projects. The knowledge built up over time is one of the core things making you effective, so you have a big productivity loss if you don't retain the same people for a long time. With a continual middleman, the overhead from that "hiring" process is usually proportional to work done, so after a period of time it will have been more efficient for the employer to hire directly (even at a cost of tens of thousands of dollars) rather than to go through the middleman.
> leaves reviews
We've seen how well that works in every other part of the market.... Interviews are necessary because of a lack of trust. In your system, every party has an incentive to lie, and the incentives are strong enough that it's worth paying people to facilitate your lies. You see that with various contractors all the time, having somebody with good reviews take the contract and somebody with fewer skills actually do the work, the employer understating the work to be done, etc. Good people exist, but they're hard to find in a low-trust environment.
> always employed and getting paid
Interestingly, that usually benefits everyone except the programmer. As a rule of thumb, the person taking risks will have higher returns, so if you can afford those risks you should choose to take them yourself. Examples include various forms of device "warranties" (if you're paying, it's insurance, not a warranty, and I can afford the financial hit when a thumb drive fails, so I'm not going to pay a premium to insure against that possibility), some forms of actual insurance (full coverage on used cars is an example -- if my car is ever totaled I'll just buy another -- I have liability insurance out the wazoo, but full coverage isn't worth it), etc. Programmers often make gobs of money, so except for potentially the very beginning of your career you should be able to easily weather a few years without work. The exact contract details vary, but that would suggest then that programmers are better off on average cutting out the middleman providing those risk guarantees (and they're not really guarantees in the first place, right? people are currently without jobs because there are fewer programming jobs than there used to be, and no middleman scheme is going to fix that; you have to wait for this economic blip to blow over).
I could go on. Contracting is fine, but it's not a replacement for all salaried work.
I can't think of a reason why a highly skilled individual would sign up for that.
Another problem, which TFA hints at, is bias. System Design interviews are often terrible for this reason. If you present me with some scenario that I know is trivial to handle with a single server, I’m going to want to discuss that, but you are probably expecting me to talk about a message queue, a caching layer, etc. Both are valid depending on the situation, but if you’ve only ever known one type, you may dismiss others out of hand.