Monday, November 7, 2011

The high cost of failing to introspect

The high cost of failing to introspect

Disclaimer: I am not authorized to speak in any way, shape or form for Walmart, and Walmart Labs, the opinions expressed here in this blog  are entirely my own, and if history is any sort of indicator, half-baked.

I've been at my new position as Director of Engineering for the Mobile group within Walmart Labs for roughly 10 days now, which usually would mean that I'm barely qualified to find my way to the bathroom unaided. But since I'd like to make a good impression with my bosses I've done my best to hit the ground running as fast as I can

Right into a brick wall, as it turns out.

From the moment I walked in the door I was aware of a certain amount of tension between individual client application teams and a services team that was responsible for providing back end services for those client teams. Knowing that this was was the most visible area of organizational pain within my new professional home I shifted to investigative mode.

The first theory that I tested was that there were personality and/or work ethic issues between the teams. On the surface this seemed to be a good one to start with given that the services team had been in place prior to the leadership change that brought in our "Lean Startup" oriented management and client teams.

It didn't take long to discard this theory. When I first met with the services team their sense of frustration and anxiety over the recent changes around them was clearly evident. But what was interesting is that their passion about getting the job done was also equally evident, which means that the real problem wasn't in that room. Since I was fresh out of theories the next step was for me to sit with the team and watch them do their job.

It worked. 

The specific job I was most interested in watching was how the team went about handling the production deployment process. As luck would have it the team was set to do a production deployment that same night. In an effort to understand what I'd be seeing during the production deployment that evening, I sat down with one of the team members and had him outline the steps in the process for me.

An hour later we had to get another team member to come in and fill in some of the details that the first team member was unsure of. 30 minutes after that the two of them had to go get yet another guy to fill in a couple of blanks that the others didn't have enough information about. Are you starting to get the picture? 

Watching for myself it was as complex as the description. Responsibility handoffs were "over the wall" style, meaning that the person shepherding the changes through would have to start from scratch every time a new person entered into the deployment process. Scheduled event times meant little to outside teams as there was no visibility on our side as to what else was on the deployment schedule, or what was causing a deployment schedule to slip if deployments were being pushed back. Even tools created to facilitate the deployment sometimes caused more problems in the deployment than they solved.

Translation, we were losing more than 50% of the productivity of the services team due to the "cost" of interacting with the production release process.

What was most amazing to me was the high degree of tolerance within the larger organization of the pain of this deployment process. Now understand that I come from a much smaller team, where knowledgeable and trained people took the place of process. In a larger company we don't always have this luxury, thus the reason why the process had been created in the first place. It was clear that every single individual working within the process was working hard and doing the best they could within their specific responsibilities, but had become accustomed to the fact that this was how things worked, and it wasn't going to change. In some cases they didn't even seem to recognize this process as being "painful" from an organizational perspective. Speaking as the "Agile Sadist", I was surprised at the amount of organizational and individual professional pain that was being endured.

You're probably thinking that my next question would be "How could this have happened?". Surprisingly, it isn't. I am sure that the actual story of how the current status quo evolved would be interesting and informative, but I believe that the actual cause is far easier to diagnose - a lack of sufficient introspection.

Introspection allow you the opportunity to review what is and isn't working, and make changes based on that information. It is one of Alistair's core principles of the Crystal family of methodologies, and rightly so. It provides a thoughtful change agent, one that is based on first-hand knowledge and experience of the team that is performing the work.

Without introspection there is no good measurement of whether the application of changes are needed in the first place or if they are working to address the original issue. In the case of our production release process you can discern a number of causative elements if you look closely enough, but the changes implemented to address them in most cases not only failed to solve the original issue, but actually spawned new issues by their introduction.

Again, understand that this is not a rant against how bad production deployment seems to be at the 'ol workplace. It has certainly worked for them to this point, and allowed them to reach their business goals. But the business goals of our mobile group are different, and require a much more "agile" release capability. To be honest, it is exciting to see the possibilities for improvement here, and you can be sure that whatever we end up doing, introspection will be a key part of it.

No comments:

Post a Comment