One Step at a Time

There’s a tendency that still somehow exists across all information technology disciplines to try to complete a project all at once, and to release it, replacing and forswearing all other technology, on a single day. This Big Bang approach has been tried a gazillion times and has succeeded approximately zero. The problem is that there is far too much to do, and implementation projects take long enough that your data environment, your market, your corporate situation and much more all change during the course of a project.

Luckily, more modern approaches are more popular these days, including Agile approaches to projects. And these types of approaches are applicable not only to delivering software solutions, but data solutions as well.

Agile methodologies encourage the completion of smaller pieces of production-quality systems, built bit-by-bit, fully tested, but focused on the areas that will provide the most business value. Agile further encourages the idea that priorities can be regularly revisited, reordered, or completely upended, depending on the evolving needs of the business.

We can take this type of approach even when developing, say, a data model for a warehouse or other data solution to fit our reporting, campaign, or analytic needs. Starting at the desired outcomes, we can identify what the necessary data are - for example, if we need a reporting system for our brick-and-mortar toy store business, we can recognize that we need that reporting system to summarize sales by store, by date, by product, by brand, and by customer.

This type of information informs our data model - we can now design a model that includes all these ideas, and build out stories in our Agile methodology that represent each piece of work necessary to complete just that portion of the data warehouse.

Maybe we get through building out the sales fact and the product dimension, when our executives come to us and tell us that what’s more important now is being able to build marketing campaigns for customers who haven’t used a coupon in three months or more.

In a traditional life cycle, this would require a massive update to a massive document that outlines the massive requirements for the overall massive system. It would need to go through some massive number of reviews before being made available to the engineers, who now would have to add more dimensionality to the warehouse (e.g. coupon) to make it support the new use case. All of which would need to be done in addition to the prior requirements.

But with Agile, these new requirements are documented (mostly) separately, reprioritized to the top of the queue. The warehousing work for sales and product aren’t tossed - in fact, the sales data is a requirement for our new use case - and the team builds out the necessary coupon and date dimensions to support the latest use case and ship.

The original use case isn’t done - but that’s okay. The business had specified that it was of lower importance now anyway, they’re getting value out of the coupon use case, and our technologists can now revert their attention back to the items that got deprioritized.

The data model can grow incrementally, the data engineering and prep work can occur incrementally, and value can be productionized incrementally.