Over thinking Agile

A trend that I have been noticing over the last little while in the Agile BI debate is that people are certainly over thinking the concept. Perhaps it is just me but I like to stick to basics and continuously improve things over time as needed. So what is the most basic framework?

1. Understand the big picture. Where is it all headed?
2. Plan work for an iteration according to business priority (we use 2 weeks)
3. Meet daily as a team to discuss the following:
• What did you do yesterday?
• What will you do today?
• Do you have any roadblocks?
4. Demo the teams work to those that care after each iteration. Depending on where you are at in your development cycle, certain demos may not be stunning to all audience members.
5. Have the team discuss how you they improve the next iteration based on things that did and did not go well in the previous iteration. This is a lessons learned and lean continuous improvement ceremony.

In my experience, agile is a highly collaborative methodology that requires all participant be open and honest with each other.

Do not expect immediate results! Expect frustration and a desire to fall back to old habits. Building a team, refining your systems development life cycle and finding rhythm takes time. For more around this topic see a post of mine titled "Religion and Agile BI – Defeating zealots and discounting silver bullets"

You gotta start somewhere!

Earlier this week in my Twitterverse I ran across this tweet from @paulg that got me thinking...

"Startups in 137 chars: Make something someone specific needs, launch fast, let users show you what to change, change it, repeat last two."

There are two things that I take from this statement that should help BI programs. One is that BI Should be managed and run like a business. And second is that as we develop our product, that we should iterate and continuously improve future iterations. Sounds a little like the principles behind agile. 

Sometimes people struggle with where to start and this can be especially true when building BI applications. Where do we begin? It is often not immediately apparent but at some point you just gotta start as there is no value being generated by not starting. Now do not get me wrong. Some thought around potential solutions and direction is helpful but sometimes we get too stuck in trying to diagram perfection before we start anything.

So if you find yourself stuck with where to start, pick a direction and just go. But do not go too far. Check in an validate the direction that you are headed. Sometimes this feels like a game of hot and cold that we all played as kids but as you and your team being to improve the art of iterating, the quicker and smoother this game will become. You will learn to guide each other and trust in the process.
#iodgc #baforum

Religion and Agile BI – Defeating zealots and discounting silver bullets

In any movement or philosophy that has existed over time, the zealots have always been the ones we hear from the most. They are just more flashy and radical and the pragmatists just are not as loud and extreme but they and are the ones that are generally left to defend whatever movement they support. The Agile movement is no different.

If you adopt Agile, hopefully it is with good reason. The success of your program will not (and should not) be measured by how closely you stuck to Agile Principles but by how adaptable and responsive you were on behalf of the business while delivering the right things of value. If that means departing from strict Agile principles, then so be it. They are just a guideline, a mindset and a way of structuring a set of tasks. Fear not, the Agile boogie man will not jump out and chastise you, except perhaps at the occasional conference, but in any instance you should not really not care if you have been delivering value. They will fly home and do their thing and you can go back to serving your satisfied user community.

I prefer to take a much more pragmatic approach to all of this and believe in the spirit of continuous improvement as we evolve in order to maintain relevance within our business. At breakfast this morning a gentleman was surprised when I said we had been at Agile for 4 years and still need to improve things. My response was this: “Toyota has been using lean principles (roots of Agile) for 50+ years and to this day they are still continuously improving to remain competitive”. So with that context I see four years as being pretty good.

As I see it the issues is this. Far too many of us are looking for that silver bullet that just simply does not exist. You need good people processes and technology to make any BI program successful. There are lots of ways to make your BI program as a whole more Agile using the strict dictionary.com definition and that is what should be targeted. The methodology is but one way, one part.

As an Agile advocate (not zealot) I can see the concern that some folks have around “Tipping the sacred cows of BI” as Evan Levy would say. However that is where we are at today. We need to be more adaptable and as a BI program manager, I can say that is a tight rope I enjoy walking and have accepted on behalf of my organization.

At the end of the day, I just want to deliver the right things better and not necessarily faster.

Countdown to TDWI 2010 Summer Conference

Agile comes of age for the Business Intelligence industry at the TDWI 2010 Summer World Conference in San Diego.

In just under two weeks, the TDWI 2010 Summer World Conference will be in full swing. The theme is "Creating an Agile BI environment - Delivering data at the speed of thought". Personally it is exciting to see how the uptake and interest in Agile has taken off over the last 6 months or so and that Agile, which once on though of as the realm of software developers, has now hit the main stage of the BI industry at a TDWI conference. I am also glad that they used the term "environment" in the title because it transcends development.

This years conference features numerous agile related courses but features Ralph Hughes, author of Agile Data Warehouse and a keynote from Wayne Eckerson. See the conference schedule for more details.

In addition to this, I will be sharing my thoughts on our four years using and refining the Agile method on behalf of IBM. Here are the session details:

Title: Architecting BI for Agility - Embracing Agile principles for Competitive advantage

Abstract: Recently Agile has become one of the big Business Intelligence industry buzz words but what does it really mean to be Agile? Keeping pace with the changes in business requires a different mindset. See how WestJet is weaving Agile thinking into the very fabric of its BI program culture to enable it to harness competitive advantage.

Date: Wednesday August 18th
Location: Manchester A
Time: 11:55-12:15pm

Resources:
Check out my contributions to the following TDWI publications that are available to TDWI members only:

• BI This Week August 4, 2010 - Reflections on an Agile BI Program
• Business Intelligence Journal Q2 2009 in the experts perspective section under a column titled Agile Development
  

Education and awareness has come along way in almost four year as this years conference is a far cry from the one course that was offered at the TDWI fall conference in Orlando back in 2007. Larissa Moss taught that course and at that time there were only about 3 of 100+ members of the audience that were using the methodology or really knew anything about it. Larissa incited some great group discussion and it was at that time I knew that I had entered an interesting space based on the questions I got after the seminar.

My agile journey began in early 2006 when our Data Management group was moved under the direction of the Software development director. At the time we had no methodology and were doing things "at random". Given he had no idea what to do with a BI group, his first ask was for us to adapt to the current methodology that the software development team used, which happened to be Agile.

The first year was tough as we tried to shoehorn BI development and its inherent differences into the methodology and in the end it took almost two years before we hit sustainable rhythm. And to put things in perspective (not to scare anyone) it has taken an additional 1.5 years to improve and I estimate by the end of 2010 we will be nearing peek efficiency. Now over this time period we had several major enterprise changes going on over this time which made it difficult to do quicker but at a minimum we were able to demonstrate continuous improvement which is really the important part. This is a journey after all.

Agile takes team work and I truly work with some bright minds that enable this to succeed. So just remember it is a people process and that it takes time to hit rhythm so be patient. There is no out of the box Agile.

I plan to re-plan

Say this out loud a few times. And then say it again.

Lets face it, this is the new reality for all project teams today, not just those in Business Intelligence. Sometimes we just do not realize how much of this occurs on a frequent basis with the multitude of new information that is constantly coming at us. Those that are accepting of this new axiom and are prepared to manage the multitude of work that comes with re-planing and coordination have a greater shot at being successful. Who works somewhere where things are not constantly changing? Business is constantly changing as influenced by both the internal and external environment. When things change in either environment, back office support must be able to adapt as accordingly.

But this is often an uncomfortable position to be in, and after all, not many people are comfortable living with constant change. The culture and team support must be in place and this is where agile philosophies can help.

One of the core principles in the agile manifesto states the following:

"Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage."

To stay relevant as a BI program, you must welcome change and handle it gracefully. In order to allow this to happen in a controlled and known fashion, we rely on the numerous ceremonies that are part of the methodology. One of these is a daily re-planning session called the scrum. See the CIO blog post titled "Agile Development, Project Management and Five [Easy] Questions" for more on the daily scrum ceremony and how progress information is kept in the open. As mentioned in this blog, planning and re-planning takes considerably more work and communication. Agile is just not "run and gun".

As the old saying goes, "Change is the only constant". We must be prepared to plan and re-plan until we deliver the right things. This must be part of our mindset and should baked directly into our processes. And this where the strength of the Agile methodology is most evident.

What does it mean to be Agile?

Agile has been one of the big buzz words of the Business Intelligence industry for the past few months but what does it really mean to be Agile? Agile as an adjective can be and probably has been applied to anything and everything in BI because it sounds good and sells. “Our software enables Agile BI!” says the vendor. And after all, who does not want to be more agile, especially this day in age? Just saying the word elicits feelings of quickness and adaptability.

But before diving into the nuts and bolts of the methodology over the life of this blog, it is important to stand back and understand what it means to be agile, what needs to be agile and then how to go about improving the agility of those parts.

Here are some definitions of Agile from dictionary.com which we will use as a starting point

1. Characterized by quickness, lightness, and ease of movement; nimble.
2. Quick and well-coordinated in movement; lithe: an agile leap.
3. Active; lively: an agile person.
4. Marked by an ability to think quickly; mentally acute or aware: She's 95 and still very agile.

So what are the parts that we want to exude these attributes? Usually we only think about the development side of the puzzle but it applies to many more things such as the following subject areas below. I will go into each of these in more depth in future blog posts as they are quite large but for now, here is a flavor.

Organizational and Team Culture: The organization has to be willing to accept, be patient with and embrace the methodology. Agile represents a new way of managing the overall BI program work stream and it takes time to adapt to this.

Architecture: Solutions must be architected in a manner that allows for adaptability. This requires experienced designers that have been caught by lack of adaptive solutions and that have the creativity to prevent this in the future. Nobody in the business likes to hear that any change that is perceived to be simple will take 4 weeks to roll out. It is very important to determine how to roll out a program that has the attributes of Lego. See Architect Big and Deliver Small - Applying Lean Agile Techniques to BI Program Management.

Development planning and prioritization: This need to become a living, breathing process that is constantly reviewed, reworked and adjusted for business changes. If you have every heard the phrase "The Business is always Changing", this where adjustments need to be made.

Development process: This is what people typically think of when it comes to Agile. We can structure our work and the development team to follow the Agile Methodology to better enable the team to deliver the right things better.

Business Analysis: Analysis needs to be Agile as well. As the overall program needs to be designed for agility, so do the analytic processes that surround it.

So would you use any of the above definitions to describe your Business Intelligence program? If not, take it as something to aspire to. As you will see, one of the core philosophies is continuous improvement. So no matter where you are on your journey towards being more Agile, there is always room for improvement.

And by the way, people and not software enable Agile BI.