> worthwhile software is usually novel.
The typical reason is that developers in large teams often don't know all the places where a field may be used - and their IDE could very well miss some. So automated testing or QA finds an issue, and then the unaccounted-for, time-consuming bugtracking begins.
CRUD: Create, Read, Update, Delete. Basically just inputting and displaying data from a database.
This is an interesting assumption. I’d argue that the overwhelming majority of software is the most boring LoB CRUD apps you can imagine, and not novel at all. Yet, people need to estimate the tasks on these projects as well.
If something is truly boring in software, it gets turned into a library or a tool for non-programmers to use. Our value is always driven by the novelty of the need.
And no, people don't need to estimate the tasks. My dad did LoB apps in the 1970s to the 1990s. E.g., order entry and shop floor management systems for office furniture factories. His approach was to get something basic working, see how it worked for the users, and then iteratively improve things until they'd created enough business advantage and/or cost savings to move on. Exploratory, iterative work like that can at best be done with broad ballpark estimates.
I grant that people want estimates. But that is usually about managerial fear of waste and/or need for control. But I think there are better ways to solve those problems.
[1] e.g., https://en.wikipedia.org/wiki/DBase
And yes, if you are in an environment where people with power want things, you have to take that into account.
But no, we don't have to just blindly do what people with power ask. The difference between being a professional an a minion is that professionals know much more about the work and how to do it than the people paying them. Personally, I think we are professionals, which gives us a variety of responsibilities not just to the person with the money, but to the profession and society at large.
Does that mean I never have given estimates? Not at all. But it does mean that if somebody asks me to do things in a way I think suboptimal, I'm at least going to say, "Hey, there's a better way to satisfy your goals here." And then I'm going to help them learn enough about the better way that they're at least willing to try it.
I think it's important that our "work culture" allows us to critique why we do certain tasks. When push comes to shove, I guess a manager can say: "Because I say so!", but I also hope those kind of managers are few and far between.
Also, I wholeheartedly agree that we do need to question the work culture we follow and the measures we make, and that managers with control issues shouldn't dictate them.
On the other hand, the point I was getting to is that a critique of estimation that amounts to "the work I do is so bespoke and unique and novel and important that I can't be bothered to estimate how long it'll take", is just… ignorant. Most software engineers are not lone wolf 10X wizards without any accountability, have managers and other departments to report to, and thus are not eligible to make that point.
Just because something hasn't been done the exact same way you're doing, that doesn't mean you can't apply a generic solution. I have never changed a tyre on an SUV before, yet I do know how to do so based on my previous experience with a sedan. The same applies to a car mechanic; even if I bring a brand new car to the workshop they have never seen before, I can and should expect them to be able to (at least roughly) estimate how long a tyre change is going to take.
This is so good.
It took me a long time to come to grips with the POSIWID [1] version of the purpose of planning and estimates. One of the things that really blew my mind is Mary Poppendieck's story about how they built the Empire State Building on time and under budget even though they didn't have it fully designed when they started. [2] Different, more effective approaches are not only possible, they exist. But they can no longer win out, and I think it's because of the rise of managerialism, the current dominant ideology and culture of big business. [3]
[1] https://en.wikipedia.org/wiki/The_purpose_of_a_system_is_wha...
[2] Talk: https://www.infoq.com/presentations/tyranny-of-plan/ And transcript: https://web.archive.org/web/20140311004931/https://chrisgagn...
[3] See, e.g., https://www.amazon.com/Confronting-Managerialism-Business-Ec...
This is what "agile" is: https://agilemanifesto.org/
More specific methodologies that say they are agile may use concepts like estimates (story points or time or whatever), but even with Scrum I've never run into a Scrum-imposed "deadline". In Scrum the sprint ends, yes, but sprints often end without hitting all the sprint goals and that, in conjunction with whatever you were able to deliver, just informs your backlog for the next sprint.
Real "hard" deadlines are usually imposed by the business stakeholders. But with agile methods the thing they try to do most of all isn't manage deadlines, but maximize pace at which you can understand and solve a relevant business problem. That can often be just as well done by iteratively shipping and adjusting at high velocity, but without a lot of time spent on estimates or calendar management.
It seems like you acknowledge some amount of estimating is needed and I agree that there is an overemphasis on estimation in many places, but I'll ask you the same thing I asked others, which is:
How do you do either of the following without spending any time at all on estimates?
"Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."
"At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."
The reality of use is what we in software are ultimately accountable to, and that's what I am suggesting people optimize for. Closing that feedback loop early and often builds stakeholder trust such that they stop asking for elaborate fantasy plans and estimates.
I will ask this here: If you are shipping code to production on a weekly basis, is that not a schedule, also known as a deadline for delivery?
If you expect to ship code to production every week, how do you know whether there will be something to ship without doing any estimation of the effort and time required?
> "Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."
This is 'just' bum-standard continuous delivery (which is where most organizations should be heading). You pull the next todo from the backlog, start working on it. If it takes more than a day to commit something, you split the task into something smaller.
You don't need to estimate ahead of time at all as long as the task is small enough, all you need is to be able to put the near-term backlog of work into a good priority order of business value.
If the high-value task was small it doesn't prevent you from doing more work, because the next unit of work to do is the same either way (the next item on the backlog).
If the high-value task was too big, it can cause you to take a pause to reflect on whether you scoped the task properly and if it is still high-value, but an estimate wouldn't have saved you from it because if you'd truly understood the work ahead of time you wouldn't be pausing to reflect. An estimate, had you performed it, would not have changed the priority.
But this Kanban-style process can be performed without estimates at all, and organizations that work to setup an appropriate context for this will find that they get faster delivery of value than trying to shoehorn delivery into prior estimates instead. But there are people who work faster with the fire of a deadline under their tail so I can't say it's unilaterally better.
> "At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."
If it's hard to do the work as a team, you should be able to tell it was hard retrospectively, with or without having done an estimate ahead of time.
You might say that failing to hit your prior schedule estimates would be a good topic to discuss at a retrospective session, but I would tell you that this is a self-licking ice cream cone. If your customers are happy despite missing internal schedule estimates you're in a good spot, and if your customers are unhappy even though you're "hitting schedule projections" you're in a bad spot.
There's a lot more productive discussions to be done when the team reflects on how things are going, and they typically relate to identifying and addressing obstacles to the continuous flow of value from the product team to the end users.
But that is rarely how it works. In the dozens of different projects across ten or twelve companies I’ve had insight into, “doing Agile” is analogous with “we have a scrum master, hold stand ups, and schedule iterations” while the simple reality is “Agilefall.”
You might be speaking a little more broadly than I am interpreting.
There is no scenario where it's appropriate or necessary when developing software professionally or even as a side project where others are expecting you to complete work at some point.
One of the many misconceptions in the original comment in this thread is that "worthwhile software is usually novel", which is not the case without a very specific and novel definition of worthwhile that I don't believe was intended.
I grant that both of those are common, but that's because the median "Agile" implementation quickly devolved into mini-Waterfall with more hip names.
And the 12 principles: https://agilemanifesto.org/principles.html
But to my dismay, the Agile world quickly got colonized by certification schemes and consultants targeting large companies, so it rapidly turned into something that a lot of the early people were very disappointed with. I wrote about the dynamic some years back: https://williampietri.com/writing/2011/agiles-second-chasm-a...
"Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale."
"At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."
If I was your colleague, had invested in your company, or was considering giving you money to deliver some product and you refused to tell me when you think it'll be done, it would be the last time any of the above ever happened.
You wouldn't accept that from any service you were attempting to hire, so I don't know why it would be acceptable for software development.
That some random person on the internet who wouldn't have hired me anyhow will now not hire me is a blow I'll have to learn to live with. But I have literal decades of stakeholders happy working this way. One way that happens is that they pick a date and then we built what fits between now and then. So they get the date, but no promises about what will get delivered.
In practice people like this better because beyond a certain very coarse level, estimates are about feelings of engagement and control. What this approach gives them instead is not just feelings, but actual engagement and control. What they get from me is not some unreliable bullshit date, but a commitment to deliver useful things early and often, so that we can together discover something better than their initial visions.
And yes, of course I accept this from people who do work for me. My side hustle is starting a pinball museum. When volunteers take on repairing a just-arrived machine, I never ask them for estimates. I ask them to focus on doing it right. They discover problems, figure out how to fix them, and try it out to see if there are more problems. The work takes as long as it takes. Or I hired a friend to design the logo. I never asked for a date. We iterated on it, discovering an number of things along the way, including through user tests. It took a lot longer than I would have guessed, and that's great, because it turned out better than I could have hoped.
I understand that this is unfamiliar to people, and apparently it's quite upsetting. But I swear this works. There are more things, Horatio.
"Working software is the primary measure of progress."
"At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly."
As you say, worthwhile software is usually novel. And to justify our expense, it needs to be valuable. So to decide whether a project is worth doing, we're looking at some sort of estimate of return on investment.
That estimate will also, at least implicitly, have a range. That range is determined by both the I and the R. If you don't have a precise estimate of return, making your estimate of investment more precise doesn't help anything. And I've never seen an estimate of return both precise and accurate; business is even less certain than software.
In my opinion, effort put into careful estimates is almost always better put into early, iterative delivery and product management that maximizes the information gained. Shipping early and often buys much clearer information on both I and R than you can ever get in a conference room.
Of course all of this only matters if running an effective business is more important than managerial soap opera and office politics. Those often require estimates in much the same way they're required from Star Trek's engineers: so the people with main character syndrome have something to dramatically ignore or override to prove their dominance over the NPCs and material reality.