The Planning Fallacy

It is pretty epic how the human mind (read: me!) manages to both ‘know’ something as a fact and yet not internalize its relevance. A rather pertinent instance of this is with regards to the planning fallacy, as popularised by psychologist Daniel Kahneman.

The topic has been on my mind for a bit as I have been reading the absolutely amazing ‘Thinking, Fast and Slow’ for about a month now, and yet have seemingly not managed to move from intellectual acknowledgement and appreciation of these biases to fully utilizing its lessons. This comes to the fore at the end of, what was to me, a pretty underwhelming CS3216 Assignment 3. Underwhelming because of the last-minute rush for completion, rather unpolished state of the submission, and an extremely low completion rate of features in comparison to what we initially planned.

It is important to note, nevertheless, that this low feature-completion rate was not out of the norm at all. This has been the case for most, if not all, significant scale software projects I have been a part of. While the software engineering course I took did try to highlight this common over-optimism on the part of developers, I did not take away any important suggestions on managing or mitigating this bias. The only suggestion I can recall was to add a time buffer to the schedule to allow for inevitable extensions and bug-fixing.

In practice, this has not proved at all effective as even WITH the buffer, we as developers prove overambitious. My general observation has been that while many developers have experienced this and grown to expect it, they still fail to account for it fully when planning. To paraphrase Richard, my Assignment 3 groupmate, we are ‘convinced that Programming is easy’, and see no reason why anything should go drastically wrong. This is despite most developers having faced scenarios with various bugs, difficulties and errors stopping development in its tracks and delaying the project significantly. We conveniently forget this during planning, restricting our imagination of doing the project to simply learning the framework, doing the features and minor bug testing.

On considering the amount of things that are necessary to go right in order for the best-case scenario to occur and our plans to be accurate, we should realise that we are setting ourselves up to expect a minor miracle. However, in addition to the overconfidence of the planning fallacy, an anchoring effect (also covered by Kahneman) also occurs to exacerbates the issue. Since the probability of each particular aspect (e.g development setup, library install, algorithm working as expected) succeeding is pretty high, we extrapolate the probability of everything going right (the conjunctive event) to be pretty high as well. This of course ignores the reality that the successive multiplications steadily erode the high probability, such that the probability of everything going right is actually pretty low.

While such fallacious optimism has much more serious effects in industry, it also has insidious effects on school or side projects. When we set a high bar of achievement, it ensures that most ambitious projects end with a feeling of disappointment. Personally, I think it is rather disheartening if after every major project, we are doomed to wallowing in the sorrows of ‘What could have been’.

Getting rid of this over-optimism is likely to be difficult, though. A major issue is that pessimism at the planning stage would possibly be interpreted by team-mates or authority figures as a lack of confidence in one’s ability or lack of ambition, which understandably most students are reluctant to signal. In fact, even ignoring the signaling to others, I am convinced that our innate optimistic bias is too high to be constrained even when we are actively aware of this over-optimism.

This belief attracts me strongly to the MVP school of thought, where the idea seems to be to push out a ‘Minimum Viable Product’ of core functionality as fast as possible and then iterate on it continuously, negating the need for a full/ long-term plan. This brings with it its own challenges, of course - what is ‘Minimum’ and what is ‘Viable’? It also may not be feasible for projects where a plan is expected due to the whims and fancies of the bureaucracy, who are understandably averse to the idea of going into a project with no estimate of schedule, time or costs, or if a plan is part of the grading rubrics for your project (hint!).

Kahneman’s recommendation for avoiding this fallacy is to look at previous similar projects and take an average of that as a base figure, moving up or down if you can justify why your current scenario is different from the average of past projects. Attempting to do this on previous CS3216 projects is rather futile, as only a small selection of projects are still live to date, which would likely bias the sample to the more succesful ones. Moreover, there is also a significant disparity between the apparent quality / difficulty of projects within the sample. Comparing a current scenario to the past average is also difficult, as it is difficult to make judgements on the skills and resources of past groups to judge if we are significantly better / worse off.

Finally, all this analysis ignores that over-optimism is what seems to run the whole startup ecosystem, so maybe the solution is to just ride the rollercoaster of ambition and disappointment for every project, ignoring the harsh realities of history and experience.