[Excerpts from the book introduction]
Can we ever get perfect software requirements?
The title of this book was very deliberately chosen to highlight the potential for misunderstanding in the written word.
“Perfect Requirements in Two Hours”
Does it mean that you can create perfect requirements in two hours?
Maybe it means you can perfect your requirements in two hours?
Perhaps it means that you can read this book in two hours and learn to get perfect requirements?
Maybe it means you can read this book in two hours?
Or all of the above?
That is the point. The written word is open to interpretation. And the more words you add to aid in clarification, the harder it is to understand the whole as there is more to assimilate.
I’m a walking, talking, history of the modern computing era – from punch cards to green screens, assembly to low code, computers that needed a building to house them to vastly more powerful computers fitting in your pocket. 45+ years of my life (so far!) have been development of transactional business management systems, based on user requirements. As I reflect on that experience, not much has changed since all those years ago. As my role changed from traditional analyst and developer to an independent software company offering custom business systems development, my needs evolved too. I was often bidding for fixed price development work, and that was all based on initial requirements and my understanding of what it really took to build solutions that met stakeholder needs.
After a couple of early client projects where I ‘rolled the dice for price’ and almost lost my financial shirt, I found that my modeling approach worked! I’d begun tracking where my team’s time had been going in the major elements of the projects, such as design, development, acceptance testing, deployment, training and adoption, etc., and found that broadly speaking, one third of the time was in requirements and design, one third was in actual code development, and the remainder of the project was deployment, documenting, testing, adjusting, and rollout. That was a pretty consistent finding for my team, so I started changing how I priced and bid on projects. I could quickly approximate the code complexity and get to a pretty accurate number for how long that would take. Then I multiplied that by four! Once, for the code work, once for the database and application architecture design work, once for documentation, training, and rollout, and then once for my profit margin. And I was consistently within 10% of my estimate.
Agile practices, including Scrum, Kanban and more, broadly speaking, accept that software requirements are never that good, take far too long to gather and document, and will change during development, and that early delivery of anything working, followed by iteration on the requirements with working code, is a good model.
I challenge this notion and believe you can indeed have perfect software requirements, in a very short space of time, that will endure and guide your project no matter what business domain you are working in.
On Waterfall and Agile
We’ve all seen the changes in the way that software projects are defined and built over the last few decades, right? The swing from ‘waterfall’ approaches to a variety of ‘agile’ methodologies? Indeed, if you’ve got less than 20 years experience in software development, you may have never been involved in a waterfall managed project!
In waterfall paradigms, we spent a lot of time on the Requirements Phase before progressing to the Development phase. We produced reams of paper describing in great detail what each screen, form, dialog, and operation would do. Crikey! We had people whose sole job was upkeep and maintenance of the written requirements in 3” thick D-ring binders, cross-referenced to a requirements change log, and ultimately produced a completely unusable document. It was too big to grasp in a sitting. And we still discovered new requirements while building code. And we managed scope change by reference to the written word. Which itself led to challenges of interpretation. Waterfall got a bad name as being inflexible to changing requirements.
Today, in 2024, we have a similar problem! Systems are described in small snippets of User Stories. These are filed in ‘ticket’ systems in Epics and Features and Stories and Tasks. We replaced the 3” thick D-ring binder and a thousand pages with… a digital directory with a thousand “as a persona, I want to be able to…”. Again, the challenge is that no-one can assimilate all those separately collected stories. There’s no cohesion, and it’s super easy to slip in a story that has no relation to anything else, but someone thought it was a good idea. And it is all still words, but we moved them into a database and can search and filter, and if we read enough, we’ll be able to figure out the big picture. Not a very effective way of getting to understand a set of requirements!
Scope? Is it “in” or “out”? From loosely written words, our stakeholder would contend that a particular feature was ‘in scope’ because it makes business sense to them so it should be obvious to you. Our developers would contend that it was ‘out of scope’ because they hadn’t thought about it, or it had not been discussed or accounted for. It was always and still is a bun fight. Unless you have a simple yet comprehensive approach to resolving the eternal debate. Or better still, your stakeholder understands the scope at an intuitive level, because they developed the model with you!
Great requirements work for agile too!
A study in 2024 reported “that software projects adopting Agile practices are 268 percent more likely to fail than those that do not”. Perhaps more significant is that “According to the study, putting a specification in place before development begins can result in a 50 percent increase in success, and making sure the requirements are accurate to the real-world problem can lead to a 57 percent increase.” You can find a summary review of the report and link to the detailed report at https://www.theregister.com/2024/06/05/agile_failure_rates/
With great requirements you can give your project a much better chance of success!
That’s what this book is about. In this shift to agile working practices, we still do all the same total amount of requirements discovery and documentation work (some would argue that in the agile model we do even more work!), but we changed where and when in the project lifecycle that this work gets done. At the outset of an agile project, we may have a project brief; a short business statement about what we want to accomplish. It might include the reasons and hoped for business outcome, along with a graphic design deck for the ‘happy path’, but it will be light on the details that the development team needs. Whatever way its organized for your team, an underpinning of the agile approach is that we acknowledge that we will learn more about requirements during the development phases, with an emphasis on early delivery of something working that stakeholders can poke a stick at, refine, and developers can now learn a little more and build a little more. The result is Refactor, Rinse, and Repeat until complete. How will we know when we’re ‘done’? In this model, it is open ended, we cannot guess how many refactors and reviews will be needed to get it right. If you are funding this project, it presents a serious financial risk
“If I knew then what I know now, I’d have built it differently.”
I’ve heard this feeling shared by just about everyone in software engineering at some point. This book and approach to requirements gathering directly addresses this problem. The fastest and least expensive project is one where there is no rework, no iterating on work, no missing requirements, and no ‘aha!’ moments after something has been built. There is no temptation to go with what is already built, even though the newly discovered requirements indicate otherwise, just to stick to a schedule.
This, for me, is the bottom line for why agile methods don’t work. By the time you realize that you should have built it differently, it’s already looking attractive to your stakeholder, and there will be pressure to go forward with the version you’ve been learning and iterating on. Put differently, you have discovered that your foundation work is no longer appropriate for the mission at hand, but the stakeholder or project schedule and funding pressure will be to hack at the existing code until it ‘works’.
What if we could get high quality requirements and minimize the rework, refactoring, and product management and stakeholder deliverable “how’s this look?” demos and reviews? The agile workflow will require less oversight and can make progress much more rapidly.
Data Driven Requirements
I have been using this Data Driven Requirements approach since 1984. Systems architectures and user experiences have changed over the years, from mainframes, minicomputers, to client-server, and the web and mobile. Yet this approach to requirements gathering and management has been consistently useful and valid no matter the technology stack being used, and no matter the UI/UX design approaches for any project. In fact, we’ll discuss later just how little the user interface design decisions matter when adopting my approach.
I grew up and learned my trade in the era where you hired a small army of business analysts, did multiple classic interviews, and the output was one or more of the aforementioned 3” thick D-ring binders, that would be stuffed with hundreds of pages of written word specifications. Super thorough but completely impractical. Indeed, at one early career employer, one of the then big-five consulting teams was leading a major project and when I landed, it was still in the requirements phase. They’d been working on the requirements for six months and produced a mighty written word output. It was unusable. We had low confidence in the documentation, it was hard to understand the big picture, and the fact that we had a couple of analysts whose sole job was maintenance and upkeep of the written Word as new requirements were found or evolved, was a sure sign this project would fail. There had to be a better way.
One of the biggest challenges faced in communicating requirements is that the written word is often open to misinterpretation, leading to misunderstanding. Worse perhaps, the narrative can be incomplete, and you have no way to know that. There’s nothing to ‘test’, and you need to keep the whole plot in mind when assimilating understanding. Page 300 of your requirements may (unintentionally!) contradict something on Page 151. Or introduce a vagary. Or seem like it’s clear until you realize it impacts several other areas of your design. The bigger the project, the higher the risk your requirements will be a problem.
Can we do it better?
Absolutely! And this book is all about how I have practiced an approach to requirements gathering that offers both rigor and clarity to all parties, can be understood by anyone in a couple of hours, and provides a touchstone of truth for anyone involved in the development and delivery of the finished product.
And it’s taken me my working career so far to realize this? No, I realized it years ago. It has taken me this long to understand that what I have been doing and take for granted as a working practice, is not commonly understood, and then get around to writing about how I do it, my way.
Start with the data
So how do we communicate and document software system requirements between stakeholder and developer?
This is where the rubber hits the road.
This book will show you how a very simple modeling approach can capture very complex business requirements that both your UI/UX design team and your software development team can easily comprehend and use as a touchstone for further discussion and development. Everyone sets out on the right foot!
But what if the application is bigger than a whiteboard? Won’t it take more than two hours? Of course. Two hours is a long time to be in heavy thinking mode though. Especially in this era where many of us work virtually, two hours on a video call is probably the maximum before fatigue and attention-drop sets in. For those projects that are clearly larger, the answer is to do what you can in two hours, and then begin breaking down separate areas of the application into smaller business domain sub-systems.
What you’ll find in this book
We start with a very simple business model for a simple ‘ToDo’ list application and use that as our sandbox for learning about how to think and use Data Driven Requirements. When you are learning to play guitar, you don’t start out with ‘Eruption’!
Then we slice it apart and discuss why it works and make some very simple ‘rules’ that you and your stakeholder can apply. There is no complicated modeling language like UML or RML other than those simple rules. We’ll talk about how you ‘read’ a model and how the model itself asks all the right questions! Anyone can do this and be successful. You’ll do all this with your stakeholders. The whole point is your stakeholder is making scope and complexity decisions with you, and their understanding of feature needs and any previously unseen complexity is greatly enhanced.