Per Penny Arcade [1], anonymous accounts being jerks get less leeway from me than people being respectful and/or standing behind their words.
All comments here are my own. I do not speak for my employer.
[1] http://www.penny-arcade.com/comic/2004/03/19
- wpietriI should add that the bond between relational databases and spinning rust goes back further. My dad, who started working as a programmer in the 60s with just magtape as storage, talked about the early era of disks as a big step forward but requiring a lot of detailed work to decide where to put the data and how to find it again. For him, databases were a solution to the problems that that disks created for programmers. And I can certainly imagine that. Suddenly you have to deal with way more data stored in multiple dimensions (platter, cylinder, sector) with wildly nonlinear access times (platter rotation, head movement). I can see how commercial solutions to that problem would have been wildly popular, but also build around solving a number of problems that don't matter.
- As a friend described the first generation: they're way worse than hand vacuuming, but way better than not vacuuming. So they're really only valuable for people who don't enjoy cleaning and don't have servants clean for them.
- And that's such a good summary of the toxic dynamic around estimates!
- Look, software estimates are a whole studied field and have been for decades. Barry Boehm comes to mind here, as well as McConnell. When I talk about estimation, I'm talking about that. And I'm also talking about the bullshit-production process which yields similarly-shape artifacts that also gets called estimation. Something also discussed extensively in the literature.
If your point is that making things has a temporal reality to it, sure. But when people ask me for estimates, I give them two options: we start a formal estimation practice, where we do activities that produce what actual engineers would recognize as reliable numbers, or we do it kanban-style, which does not involve an estimation practice, but gives product stakeholders direct control over time/cost tradeoffs.
Some people, here and elsewhere, argue for a third option, which is going through various levels of rigamarole that gives horseshit numbers so that the more mendacious managers and executives can put on a performance of rigor while telling people with power what they want to hear. I refuse.
And I think it's that bucking the primate dominance hierarchy that makes people absolutely freak out when people talk about no-estimates approaches. Per POSIWID, the purpose of standard corporate "estimation" is not to produce reliable numbers. It's to avoid conflict. I get why people would rather freak out at me than make their bosses unhappy. But I would suggest that's a short-term gain for long-term pain. For organizations that want to actually get things done, hallucinatory project "plans" are bad for everybody. I think ultimately software engineering should be an actual profession. It's not looking great, but I live in hope.
- I have to keep responding to your posts because you keep grossly mischaracterizing what I'm saying. I have great relationships with my stakeholders. Just this we we delivered something greatly desired by the organization. We are being reliable and professional, just in a way that apparently confuses and frightens you and others. Indeed, I'd claim I'm being more professional that people who give nonsense estimates just because bosses ask for them.
- I am always fascinated when people tell me it is impossible to do something I have done. That plenty of people have done. For decades. But I shouldn't be surprised. Kuhn was not kidding around.
And yes, I strongly believe in accountability to business stakeholders, customers, and colleagues. I just think the better way is not paper fantasies, but demonstrated realities. Focusing on working software, customer collaboration, and responding to change, one might say. But people have said it before and it didn't make much difference.
- Yes, this sounds very familiar to me. I started with dates and went to story points used to estimate dates. Then as we turned up the release cadence, we eventually dropped estimating altogether, even in points, because it wasn't really helping anything.
That doesn't mean we refuse to solve the problems that estimates are used to solve. E.g., "Can we have something by the big conference," or "Can we keep our customers happy." We just solve them other ways.
And totally agreed about the corrosive effect of treating estimates as commitments. It's such a negative-sum game, but people play it all the time.
- It is not a schedule, it's a standard. One I normally try to exceed. We ship when things are ready, which for my current team is ~2-3x/week, but in the past I've had teams that were faster.
We know that there will be things to ship because we try to break the work down into small units of deliverable value by focusing on seeking the highest-value things to do. Large requests are typically composed of a bunch of things of varying value, so we split them until we find things that advance the needs of the user, the customer, or the business. One that's often not intuitive to people is the business need for getting information about what's truly valuable to some part of the audience. So we'll often ship a small thing for a particular audience and see how they react and what actually gets used. (You can save an amazing amount of time by not building the things nobody would end up using.)
Sometimes we can't figure out how to break down something smaller enough that we have something to release right away. Or sometimes a chunk of work surprises us and it drags out. We avoid that, because compared to working on smaller things, it's much less comfortable for both developers and business stakeholders. But if it happens, it happens. We try to learn from it for the next time.
Regarding deadlines, we sometimes have them. Broadly, efforts break down into two categories: driven by date or driven by need. For the former, releasing early and often means we adjust scope to hit the date. For the latter, scope is primary and they get stuff when they get it. Either way because the business side sees steady improvement and has fine-grained control over what gets shipped, they feel in control.
This can be a learning experience for business stakeholders used to waterfall-ish, plan-driven approaches. But I have never once had somebody successfully work this way and want to go back. I have, however, had some product managers get thrown back into document-driven development and tell me how much they missed working like we did.
- This is a gross and misleading caricature of what I'm saying. I prefer this approach precisely because it increases accountability. If you'd like to learn what I'm actually suggesting, I'm happy to answer questions. Or you can read many of the things that have been written by other people on the topic.
- I want to be clear that I am being entirely practical here. This is not navel-gazing. I am describing something that works. That has worked for me and others for decades.
And yes, if you are in an environment where people with power want things, you have to take that into account.
But no, we don't have to just blindly do what people with power ask. The difference between being a professional an a minion is that professionals know much more about the work and how to do it than the people paying them. Personally, I think we are professionals, which gives us a variety of responsibilities not just to the person with the money, but to the profession and society at large.
Does that mean I never have given estimates? Not at all. But it does mean that if somebody asks me to do things in a way I think suboptimal, I'm at least going to say, "Hey, there's a better way to satisfy your goals here." And then I'm going to help them learn enough about the better way that they're at least willing to try it.
- I'm not upset. But as far as I can tell you've wandered off into some sort of philosophical space when I'm speaking of practicalities, without apparent recognition of the transition. I thought that was a bit ridiculous, something from the "wow dude have you ever really looked at your hand" school of conversation, so I made a joke. Apparently the joke didn't land for you. Alas.
- I addressed the rest elsewhere, but done well a lack of estimates makes people more accountable. If I am shipping something every week (or as is common for my teams, more often), stakeholders can directly evaluate progress. There's no snowing them with paperwork and claims of progress against semi-fictional documents. They see what they see, they try it out, they watch people use it.
The reality of use is what we in software are ultimately accountable to, and that's what I am suggesting people optimize for. Closing that feedback loop early and often builds stakeholder trust such that they stop asking for elaborate fantasy plans and estimates.
- None of which requires estimates. And the bit about working software as the primary measure of progress is specifically targeted against the estimate-driven culture of the time, where people would treat GANTT charts, "percentage complete" numbers, etc, as meaningful measures of progress.
- I feel like I've already answered the how question elsewhere. But the short version is a product-driven kanban-ish approach with frequent releases, small units of work, and engaged product management focused on actual real-world goals. This approach originated 25 years ago in Kent Beck's "Extreme Programming". Which did start with estimates, but teams doing well at it quickly realized (via said regular reflection) that they could do without it and still deliver well.
That some random person on the internet who wouldn't have hired me anyhow will now not hire me is a blow I'll have to learn to live with. But I have literal decades of stakeholders happy working this way. One way that happens is that they pick a date and then we built what fits between now and then. So they get the date, but no promises about what will get delivered.
In practice people like this better because beyond a certain very coarse level, estimates are about feelings of engagement and control. What this approach gives them instead is not just feelings, but actual engagement and control. What they get from me is not some unreliable bullshit date, but a commitment to deliver useful things early and often, so that we can together discover something better than their initial visions.
And yes, of course I accept this from people who do work for me. My side hustle is starting a pinball museum. When volunteers take on repairing a just-arrived machine, I never ask them for estimates. I ask them to focus on doing it right. They discover problems, figure out how to fix them, and try it out to see if there are more problems. The work takes as long as it takes. Or I hired a friend to design the logo. I never asked for a date. We iterated on it, discovering an number of things along the way, including through user tests. It took a lot longer than I would have guessed, and that's great, because it turned out better than I could have hoped.
I understand that this is unfamiliar to people, and apparently it's quite upsetting. But I swear this works. There are more things, Horatio.
- If you're saying there's some sort of wisps-and-moonbeams notion of estimation in everything that we do, sure. I'm not going to argue with that. One world, brah.
What I am talking about here, though, is a practice of software estimation where programmers produce detailed numbers on the amount of time requested work will take. Which is certainly the common meaning of estimating around here, and also what the original article is about.
- On the contrary, constraints often mean you don't need formal estimates. (I'll come back to prioritization in a sec.)
Startups are a great example. When you raise your first chunk of money, the size of that isn't really driven by a carefully considered, detailed plan with engineering hours estimated per task. What you get is basically determined what's currently fashionable among angels and small-end VCs, plus who's doing your fundraising. (If you're Jeffery Katzenberg and Meg Whitman, you can raise $1bn. [1] https://en.wikipedia.org/wiki/Quibi But the rest of us have to make do with what we can get.)
So at that point you have a strong constraint (whatever you raised) and some relatively clear goal. As I said, cost isn't nearly as relevant as ROI, and nobody can put real numbers on the R in a startup. At that point you have two choices.
One is just to build to whatever the CEO (or some set of HiPPOs wants). Then you launch and find out whether or not you're fucked. The other is to take something akin to the Lean Startup approach, where you iteratively chase your goal, testing product and marketing hypotheses by shipping early and often.
In that later context, are people making intuitive ROI judgments? Absolutely. Every thing you try has people doing what you could casually call estimating. But does that require an estimation practice, where engineers carefully examine the work and produce numbers? Not at all. Again, I've done it many times. Especially in a startup context, the effort required for estimation is much better put into maximizing learning per unit of spending.
And how do you do that? Relentless prioritization. I was once part of a team that was so good at it that they launched with no login system. Initial users just typed their names in text fields. They wanted proper auth, and eventually they built it, but for demonstrating traction and raising money there were higher priorities. It worked out for them; they built up to have millions of users and were eventually acquired for tens of millions. On very little investor money.
Being great at prioritization makes estimation way less necessary. The units of work get small enough that the law of large numbers is on your side. And the amount of learning gained from the things released change both the R and I numbers frequently enough that formal estimates don't have a long shelf life.
So I get what you're saying in theory, but I'm telling you in practice it's different.
- Thanks. It was hard won. I spent maybe a decade naively thinking that if we just made software methods that worked in service of stated business goals and values, they'd get adopted and we'd all live happily ever after.
It took me a long time to come to grips with the POSIWID [1] version of the purpose of planning and estimates. One of the things that really blew my mind is Mary Poppendieck's story about how they built the Empire State Building on time and under budget even though they didn't have it fully designed when they started. [2] Different, more effective approaches are not only possible, they exist. But they can no longer win out, and I think it's because of the rise of managerialism, the current dominant ideology and culture of big business. [3]
[1] https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...
[2] Talk: https://www.infoq.com/presentations/tyranny-of-plan/ And transcript: https://web.archive.org/web/20140311004931/https://chrisgagn...
[3] See, e.g., https://www.amazon.com/Confronting-Managerialism-Business-Ec...
- Right? For those who want to check, the four values: https://agilemanifesto.org/
And the 12 principles: https://agilemanifesto.org/principles.html
But to my dismay, the Agile world quickly got colonized by certification schemes and consultants targeting large companies, so it rapidly turned into something that a lot of the early people were very disappointed with. I wrote about the dynamic some years back: https://williampietri.com/writing/2011/agiles-second-chasm-a...
- I have had many successful projects where we spent approximately zero time on estimates. The fact that a successful approach is culturally seen as illegitimate to even talk about is a great example of why I wrote that last paragraph.
- I can say with some confidence, having been involved in the movement since before the term "Agile" was coined, that it requires neither.
I grant that both of those are common, but that's because the median "Agile" implementation quickly devolved into mini-Waterfall with more hip names.