Stories from the "Knowledge Acquisition" curve, part 1

If you've ever attended any of Alistair Cockburn's recent presentations on the nature of software "engineering", you'll have heard him discuss the early part of the design process as a game of knowledge acquisition (link - scroll down to the section titled "Chapter 2 of the Story, 2010).

Essentially what he is saying is that it is a successful strategy to treat initial design as a "pay to learn" period. No, you're not handing out piles of cash to trench-coated agents that are slipping you unpublished USDA reports. The nature of your coin is the effort of the team. What you are paying for is information about the success and/or failure of the product you are designing.

Although there is no simple formula to identify exactly what nuggets of knowledge are needed to determine the success and/or failure of a product, Alistair does provide us with a map to the territory. Here are his 4 categories of knowledge worth "paying" to learn:

1. Business Risk (Are we building the right thing?)
2. Social Risk (Can our team build it?)
3. Technical Risk (Will our solution work?)
4. Cost/Schedule Risk (Do we understand the cost / timing?)

As the title implies, the point of this (and subsequent) blog entries is to tell stories about how this game of knowledge acquisition was played. Depending on where you are at in your Shu Ha Ri path you will hopefully gain something useful even if it is the certainty that yours truly isn't playing the "knowledge acquisition" game with a full deck. 

This first story is about gaining knowledge about Cost/Schedule risks. At the risk of sounding boastful I've chosen to tell this story first because it was the impetus for Alistair to include the Cost/Schedule risk category to the above list.

Our story begins with the decision to create a new application that would allow medical residency physicians to study for their board examinations. Medical board examinations are typically case-centric, presenting the examinee with specific medical cases that they would then identify, diagnose and discuss treatment options.

As you might guess, the key to providing a useful learning product is having high quality content, something that Amirsys is known for within the medical community. Not only did we have existing content, but the content itself was structured in such a way that re-using existing content for new purposes (such as this learning application) is something that we are very good at.

Our skill in reuse of existing content wasn't an accident. Early on in my career with Amirsys I was surprised to discover that the timeline for the creation of high quality medical content significantly exceeded software development timelines. In other words, the most significant cost of developing a new product wasn't the development work, but the content creation for that product.

Considering that we had a pretty good idea of the costs associated with content creation, any new products requiring new content had to have a pretty convincing ROI to justify the content creation costs. Either that, or we had to come up with product ideas that wouldn't require new content creation. Thus the idea for this learning application was born.

At the time of product conception we had data from roughly 60,000 medical cases in our content repository. In order for a learning product to be successful you need to present both content to learn from and assessment about that content (questions). There was no lack of learning content, but at that time we had no authored questions over that content. 

The big break came when it was pointed out that our cases were structured in such a way that it may be possible to automatically create relevant questions about the cases without requiring a medical expert to perform any authoring. On the strength of this realization the development of the product was given the green light.

Shortly after work began on the product I was participating in a conversation about an unrelated product when one of our content authors happened to mention that the cases they authored didn't have all of the content I had assumed to be required for authoring. Given how dependent we were on this content being sufficiently complete to allow for the automated generation of assessment questions, this was a significant risk to the delivery of this product.

Unfortunately there was no easy way to evaluate the suitability of the content outside of actually showing it within the context of the new product and having a medical expert interact with it. Considering that the success of the new product hinged on the fact that we didn't have to explicitly author all of these new questions we had to do something to "acquire knowledge" about the extent of this new risk.

Our solution was to build a walking skeleton of the application that would allow a user to "take a quiz". No bells and whistles - just the presentation of these auto-generated questions and the ability to select an answer and see if the answer was right or wrong. It took members of the team approximately two weeks to build this tool and put it in front of our medical experts.

Within 10 minutes it was clear that there was a significant amount of inconsistency in our case data, to the point where a paying customer would not receive a useful learning experience from the content. Armed with the knowledge that our content was not in the condition we had assumed it was, the business needed to make a decision.

Fortunately the story has a happy ending. It was determined that although our case content wasn't consistent enough to auto-generate assessment questions, it was good enough to allow us to significantly speed up the authoring process by "prepopulating" assessments that authors could then edit rather than create from scratch.

Had we not known about Alistair's pattern of "Paying for Knowledge" it is entirely possible that we would not have discovered our content challenges until it was too late to meet key delivery dates or be required to disrupt other revenue generating activities. As it turned out the product was delivered on time, with the only negative consequence being a smaller set of content shipping with the new product. 

As of the time of this writing this product has been generating revenue for close to a year now and has given us the tools to build similar products for external customers who have similar expert content learning application needs.

Stay tuned for Part 2 - A tale of Social Risk