The problem with Scrum?
Agile is a very trendy word in the tech space lately, and of all the children of mother Agile, Scrum is the most famous and infamous lately.
The web is filled with articles extolling the benefits of SCRUM or equally warning painfully of its problems in equal measure. A quick net search shows you certification authorities, books and success/failure stories.
Depending on which side you read, you might be confused a little.
Is using SCRUM ultimately a net positive to the team, the project itself and the clients or isn’t it?
Well the answer to that is a not black and white even though i have my own settled conviction on the matter (spoiler alert i like Scrum), but as Deckard Cain would say in the sleepless twilight stricken Diablo Town of Tristram, stay a while and listen..
First i need to be clear here that i am sharing my own opinion on the subject matter, and evidently as such it is just my subjective take and not a statement of objective truth. I will also not be using this article space as a means to teach Scrum rules there are much better resources, but rather to shed a certain perspective on some examples of key pain points.
Secondly i am assuming if you are reading this article you have heard of SCRUM already, be it as a passing phrase you read about previously, something you actually practice or practiced at work with your team who might like it or dislike it, or even as someone who is already specialized in SCRUM and is a certified Scrum Master(SM) or Product Owner(PO) and is curious about another blogger philosophizing about Scrum.
This opinion piece article has a bit for everybody i believe.
First full disclosure, i am a certified SM and PO from both the Scrum Alliance and Scrum.org. Additionally i have been on all sides of this topic, both as a software engineer working on ‘Scrum teams’, and as a unofficial then officially certified Scrum Master and later Product Owner dealing with Scrum teams of developers and managing stakeholders.
This background has let me experience Scrum through different angles and lenses over the years and my current conviction is based on the chronological order and evolution of my own involvement.
Initially when i worked in ‘Scrum’ teams as an engineer we did ‘sprints’ very passionately similar to how King Leonidas kicked the messenger of Xerxes into the bottom of the well in the movie 300. The words Agile and Scrum were all still quite novel fresh and abuzz back then.
Still i remember i had a very distinct bitter taste in my mouth from the experience though. I was fully convinced after a few sprints that Agile and Scrum in particular is a useless management style for people who didn’t know how to properly plan and manage as ultimately i couldn't see what were the benefits.
I had many reasons to feel that. For example there was no rhyme or reason for what was selected as tickets for each sprint. No one asked if or how something was doable. It was just another tablet with a list commandments Moses from the Sales department brought down on us from the mountain each ‘sprint’.
Additionally, each sprint had flexible length duration decided either after brief talks with the team and the tech lead, or more usually just put forth as a fact by the business and sales team who were eager to appease or win a client.
It felt like a weird demeaning dance ceremony we had to go through that didn’t involve any serious phase of planning or forethought and which had a different length each time dedicated to our indirect confusion and torture, for the sole purpose of working on things we had no clue why they were selected with hard-locked deadlines that were set in stone effectively making each sprint a race.
I didn’t know much about the details of proper Scrum back then, we didn’t have dedicated Scrum Masters or Product Owners, nor did i care to be honest like most coders would, i just tried to do my personal best coding-wise but ultimately was resentful of the repetitive visionless, lazy, and stressful spirit each sprint took.
If we didn’t meet the deadline of one ‘sprint’ it would be pushed naturally a little further and we would try our best to meet the new one. Normally we would do plenty of overnight overtime as a matter of making each deadline work, and when the time finally came to deliver the work no one knew in the team who was supposed to deliver it to the client, and more ironically we didn’t know if the client would even accept it after the deliverables ‘were done’ as there was no clear mechanism of detailed requirements and approval until after the client had long played a game of ‘Tin Can Telephone’ with the sales which replayed the game with the tech lead saying ‘this is what we want and this is a vague document about it’ and left it to his/her own interpretation as to what that means and we had spent a couple of good nights coding it at the office instead of actually living like humans. Oh the pride of being a fighter.. this never lasts long though.
Mind you back then i was working in a mid sized company as an associate level developer with around 40 developers working on different projects. The clients were serious from big industry sectors, not small mom and pop shops, yet it was normal to be Agile like that.
Prior to that i had worked also in a much bigger company who didn’t pretend to be Agile and who had a ratio of 1 manager per 1 coder with around 200 engineers who where following a Waterfall plan instead and had no idea when they would finish the project nor the amount of remaining resources they need to do it.
So from my perspective, i switched from the horrors of Waterfall management to the horrors of Agile management and as a coder i had nothing good to say or think about managers back then.
Regardless of the shared managerial hell, what is of note between these two projects styles and companies is interesting.
The first was doing Waterfall, and in waterfall no one was under the delusion that the project was going to be straightforward. As everyone fumbled forward painfully, it was clear that documents were missing, managers were over their heads, roadmaps were wrong, the budget had did a prison break and left the building etc.. But, in the second scenario with Scrum, even though the fire was similar yet on a smaller scale, there wasnt an assumption of something missing in doing Agile Scrum, as ‘you just write down tickets and do them in a sprint’ and that was supposed to be good enough. Managers blamed coders and coders blamed managers. The project management aspect wasn’t part of this story as Agile was supposed to be the new viagra for managing programing and it was supposed to work by itself and instantly out of the box.
Both of these mismanaged and badly implemented systems created burnout with coders quitting, churn with clients leaving , missed deadlines, confused requirements, engineers who were doubting their life decisions, but only under the Scrum Agile was this tsunami of problems casually, and in unaware fashion, pushed under the rug simply with the dependable ‘the coders didn’t estimate or code good/fast enough’ verdict or ‘managers are pampering idiots’ coding truth statement.
What is there not to love with this version of Scrum? It brings everyone together in harmonious disdain and hatred.
Perfect if you are a masochist or sadist. Sadly most of us aren’t.
In later projects as i grew in seniority and took on the mantle of management in some teams and projects ‘joining the enemy camp’, my first instinct was to try waterfall planning with all its beautiful Gantt breakdown timeline that made so much sense to my data structured oriented mind.
The lucid dream of being able to properly manage the project using this approach was short lived though, for when i managed to get a project successfully planned and executed using Waterfall, i failed in others.
The main difference between the projects were Waterfall worked or not was how clear the requirements was at the start and how the project was susceptible to change or not. If change was a constant element that i couldn’t push against, then my beautiful Gantt chart would turn into a deadly sliding quicksand scene where the beautiful Artax white horse was drowning in the movie The Never Ending Story. Positively shocking.
So i started to think to myself, try to forget about using projections in a Gantt chart for now in future projects, and try again some Agile techniques. Keep in mind though i was still very skeptical based on my past experience. But desperate times require desperate measures. I was convinced Waterfall wasn’t going to cut it for fast changing SAAS and mobile projects so why not.
And so it was with new teams and projects i set up Kanbans on Trello and both myself and my team of coders all started writing tickets. Sooner than later tho, even though we were doing some work, there didn’t seem to be a guiding work vision, a regularity or improvement of our capacity for work. Some tickets would get stuck in ticket hell, some wouldn’t, we never felt true closure about anything, priorities weren’t clear and there was weak alignment with business.
In short we didn’t feel like a performing football team confident in our estimations and capacity, nor were we aligned with delivering somewhat predictable and good enough quality deliverables for the business. Clearly there was room for improvement in this process.
Since we already had stepped in the land of Agile, and we needed to seek improvement, which framework comes first and foremost when you search online? You guessed it, it is Scrum! So now after a few google searches that re-confirmed to us(me) that we need to do sprints and some ceremonies like planning and review myself and the team were abuzz with new energy to try it out! (Mostly just me though as my team was experiencing the same level of existentialist dread from mismanagement as i did years previously and i was the cause of it this time)
And try it out we did, i played Scrum Master and we did sprints and everything started to get better immediately and the team cheered.. and no none of this happened, actually when we did the ‘sprints’ all went to hell much more quickly!
Deadlines weren’t being met again, i would extend deadlines and the burnout was real, deliverables were low quality, we had more meetings than we coded on some days both internally and with business, everyone could talk to everyone during the sprint, scope could change during it, nothing was being tracked on burnup and burndown charts, all ticket estimations meant nothing.. as they say online in meme world, task successfully failed. And it was all under my watch. Problematic.
This is just one example of Scrum going wrong, other coders, teams and projects have other not so bright stories, but our story doesn’t end here.
I was equally upset and curious at that point, why wasn’t agile and Scrum working (again). It actually made things worse it seems, but why are some people saying it actually helps! There was something wrong somewhere, i was missing information, i didn’t have a clear understanding of what was happening.
This is when i decided like a true tech bro version of Hercules doing his Twelve Labors to go to further into the belly of the Scrum beast but instead of rescuing King Augeas’s daughter i would try to rescue my team and projects by taking training courses to take the Scrum Master exam and officially to get certified.
Luckily for me i found some epic good teachers online both on Udemy and Youtube, and what was made clear to me at that point and that i was unaware of til this point, was that Scrum while being a framework is more of a spirited framework. This was key. You have to understand the spirit of why Scrum does what it does so you can better implement things.
For example transparency and courage to communicate is very important, because it allows us to build trust, and with that we can build in time more performing teams and better products. We do 15 minute standups max in the morning (it could be shorter than 15mins but never longer) just go have a quick alignment call and see if anyhow has a blocker that they need help with. Any longer than this and it starts to affect team focus and backfires. Even in the new editions of Scrum in order to avoid wasting time they don’t require the team to talk about what they did yesterday and what will to today anymore. As for the Product Owner he/she can be present to answer questions, but he is not there to preach/manage each morning, if a team does this ‘tradition’ it is not Scrum anymore.
The ceremonies of Scrum serve a purpose, they are not bureaucratic creations. For instance Sprint planning dedicates some time to plan together and puts estimates with ‘safe’ margins. Sprint review is needed to align with stakeholders to get their feedback on the done features they already approved in the sprint planning through the Product Owner for the next sprint and provide clear closure or changes. Sprint Retrospectives are to discuss together as a team honestly as to what can be improved next time about the processes. The definition of ‘Done’ is important so that no one murders another during a sprint.
The product owner is a diplomat. He/She is not a manager. He doesn’t lead directly. He is an expert in communicating with everyone including business and stakeholders, and all forms of requests/notes should flow thorough him mainly if not only. He knows the business like the coders know their tech stacks.
Scrum allows us to incrementally and reliably deliver value for a product in sprints. Not all tickets in a sprint need to be done. So sprints are small steps in a marathon and not a race. That is their spirit. When treated differently this is no longer Scrum.
Selected tickets have business priorities and groomed by the PO in alignment with the needs of the stakeholders. The team doing the work doesn’t take on more work during a sprint than they think they can handle.
The cadence of the team is their regular ability to deliver ‘something’ of value, good quality and without quitting. As the team cohesion increases, with time their performance will increase with it and so will the amount of tickets/stories they can do naturally like professional motivated athletes.
The Scrum Master is responsible for teaching, reminding and monitoring so that everybody sticks to the spirit of Scrum or else things can back fire.
Stakeholders talking regularly to team members backfires, they should talk to the PO instead.
Changing the sprint length or extending them doesn’t make any sense. That isn’t Scrum even though many companies do it like i once did.
The Scrum Master and Product Owner are very distinct roles. They can be the same person but that is a clear conflict of interest as the Scrum Master’s priority is that the team stays ‘agile’ while the Product Owner priority is to ‘deliver value’ and both work in the same diplomatic spirit of not leading directly or managing anyone directly like in Waterfall.
Scrum has a spirit, i honestly didn’t know that. And after i understood the spirit i liked it. I wanted to learn more and took also the Product Owner exam and passed it to understand better that perspective.
Now armed with this new knowledge, i came back to my team and i told them that i as their tech lead and manager was doing Scrum and sprints a bit wrong that is why they weren’t working. I took a month or so training them on the new proper Scrum that we will switch next to, during which i explained to the stakeholders also how things will be different and what to expect as net positives and compromises.
The dev team will be protected from external interference with proper Scrum through the SM and the PO and will have ceremonies that will help them plan, execute, showcase and improve without ever feeling in a race. Alternatively the SM and PO will provide more transparency and better refined requirements and priorities management so that the stakeholder have a simple yet elegant system in their hands to make requests, put them in a priority queue through the PO, and get to see/check on a regular controlled interval/cadence the value work being done and give their feedback for the items and priorities of the next cycle. Clockwork.
This system compared to the other options discussed of forest fires with nothing clicking together with wrong Scrum implementations or an impossible Waterfall, is a net positive win for everyone. But it has to be implemented correctly and in the spirit of Scrum. Key to this implementation is a good SM and PO. Without them doing their roles as they should the full potential of Scrum is not realized.
Scrum management cannot always be compared to a classic management scenarios where business and management get to write down and dictate how a franchise should be built and work down to a T.
A tech startup is not a bank or a burger franchise, the products they create evolve, the tech, art and marketing is very challenging, the talent working on these types of projects is very different from the workforce in other sectors. So whichever management style is employed it must take into account all these flowing elements.
After implementing the improved corrected Scrum it took a few month after which the dev team and the business people both got used to it and considered it an improvement in practice and surveys. It wasn’t perfect, but nothing is for fast moving projects. And since then i got the chance to train and implement it again constructively with other teams and projects. But each time i do, i am very careful to start with explaining the spirit of Scrum first. If someone tech company wants me to train an army of fanatical digital ninja assassins then perhaps Scrum isn’t the best framework.
Scrum doesn’t micromanage people. It empowers them and encourages cross functionality as long as the team size is relatively small (less than 10). I can’t speak about Scrum at scale as i haven’t experienced it yet.
The people who do the work do the estimates and help each other with T shaped cross-functional skills and grow together. Trust is increased, transparency is maintained. It is in line with the Agile manifesto of people before processes (note that processes still exist but the order is people first)
That is why i like it. It is a good tool if used properly. But as you can see the problem with Scrum is that it can very easily be used incorrectly or partially. It’s supposed simplicity is very misleading. It is only simple after you get its guiding spirit and logic.
It gives the impression that it is easy to implement, if i personally didn’t take the courses and certifications to dig deeper i wouldn’t have understood what i was doing wrong with it.
So this is an issue. But on the other hand, if someone wants to be a Scrum Master or Product Owner, then it is also reasonable to assume that he/she will want to get properly trained and certified on the subject matter if the current version of Scrum being used just isn’t yielding the best outcomes.
Now that i shared and explained why i am a proponent, do i think Scrum is for everyone and every project ? No i don’t. It is just a tool. A hammer must be used properly and a hammer can’t be used for everything, same with Scrum.
Some well performing teams won’t need it, they already have an efficient way of communicating. Many great projects were done without Scrum.
Some projects won’t need it as they are clearly defined enough before hand, Waterfall is better suited for those.
But when the team and project scenarios are compatible for Scrum, Scrum can help definitely. It becomes advantageous.
It enables an organized, human centric, repeatable, and feedback loop work system to incrementally create, with good quality and without breaking people, complicated long running projects with a vague initial scope.
That’s a mouthful but that’s cool also.
So what is the problem with Scrum?
In my opinion there is no problem, it is one of the good useful management tools if used properly and when actually needed.