What is Developmental Evaluation?

November 19th, 2011

Developmental evaluation (DE) is a problematic concept because it deals with a complex set of conditions and potential outcomes that differ from and challenge the orthodoxy in much of mainstream research and evaluation and makes it difficult to communicate. At a recent gathering of DE practitioners in Toronto, we were charged with coming up with an elevator pitch to describe DE to someone who wasn’t familiar with it; this is what I came up with.

Developmental evaluation is an approach to understanding the activities of a program operating in dynamic, novel environments with complex interactions. It focuses on innovation and strategic learning rather than standard outcomes and is as much a way of thinking about programs-in-context and the feedback they produce. The concept is an extension of Michael Quinn Patton’s original concept of Utilization Focused Evaluation with concepts gleaned from complexity science to account for the dynamism and novelty. While Utilization Focused Evaluation has a series of steps to follow (PDF), Developmental Evaluation is less prescriptive, which is both its strength and its challenge for describing it to people (things I’ve discussed in earlier posts).

So with that in mind, our group was charged with coming up with a way to explain DE to someone who is not familiar with it using anything we’d like — song, poetry, dance, slides, stories and beyond. While my colleague Dan chose to lead us all in song, I opted to go with a simple analogy by comparing DE to a hybrid of Trip Advisor and the classic Road Trip (due to lack of good vocalizing skills).

Trip Advisor has emerged as one of the most popular tools for travellers seeking advice on everything from hotel rooms to airlines to resorts and all the destinations along the way. Trip Advisor is averaging more than 13 million unique visitors per month and, unlike its competitors, focuses on user-generated content to support its service. Thus, your fellow travellers are the source of the recommendations not some professional travel agent or journalist. At its heart are stories of varies tones, detail and quality. People upload various accounts of their stay, chronicling even the most minute detail through photos, links to their blogs, video, and narrative. If you want to get the inside details on what a hotel is really like, check Trip Advisor and you’ll likely find it.

However, like any self-organizing set of ideas, the quality of the content will vary along with the level of reportage and the conclusions will be different depending on the context and experience of the person doing the reporting. For example, if you are a North American who is used to having even the most basic hotel chain offer a room with full-service linens, a bathroom, closet, desk and separate shower, you’ll have a hard time adjusting to something like EasyHotel in Europe.

The Road Trip part (capitalization intended here to denote something different than a regular trip by road), denotes the experience that comes from a journey with a desired destination, but not a pre-determined route and only a generalized timeline. A Road Trip is something that is more than just traveling from Point A to Point B, which is usually accomplished by taking the shortest route, the fastest route or a combination of the two; rather it is a journey. Movies like National Lampoon’s Vacation (and, European Vacation), Thelma and Louise, Harold & Kumar Go to White Castle, and (surprise!) Road Trip all capture this spirit to some effect. I suppose one might even find a more grim example of a Road Trip in the Lord of the Rings Trilogy or The Road.

Road Trips have a long history and are not just a North American phenomenon as this article from the Indian Newspaper, The Hindu reports in some detail:

“Road trips are fun when they are not planned point-to-point. As long as you have accommodation booked, that is enough. Its better not to have agendas; get as spontaneous and adventurous as you can. My friends and I went on a road trip to Goa last year. It was loads of fun as it was the first time we took off on our own without parents. To me, it was more than just a trip with friends. It showed that I could take care of myself and that I was now a grown-up, free to do what I wanted,” says Siddharth, who is doing his engineering.

The idea of spontaneity and adventure are part of the process, not an unexpected problem to be solved like in a traditional evaluation. Indeed, some of these unplanned and unusual departures are not only part of the learning, but essential to it. It is akin to what Thor Muller describes as planned serendipity; you might not know what is going to come, but it is possible to set the conditions up to increase the likelihood of and preparedness for moments of discovery and learning. This is like setting out on a journey with a mindset of developmental and strategic learning to fit with what Louis Pasteur stated about discovery:

Chance favours the prepared mind

Thus, as Developmental Evaluators and program implementation leaders we are creating conditions to learn en route to a general destination, but without a clear path and an open mind towards what might unfold. This attention to the emergence of new patterns and then the sensemaking to understand what these new patterns mean in the context to which they emerged and the goals, directions and resources that surround the discovery is a important facet of what separates Developmental Evaluation from other forms of evaluation and research.

So in describing DE to others, I proposed combining these two ideas of Trip Advisor and the Road Trip to create: Road Trip Advisor.

Road Trip Advisor for Developmental Evaluation

Road Trip Advisor would involve going on a journey that has a general destination, but with no single path to it. Along the way, the Developmental Evaluator would work with those taking the journey with him — likely the program staff, stakeholders and others interested in strategic learning and feedback — and systematically capture the decision points to take a particular path, the process that unfolded in making decisions, the outcomes or events connected to those decisions inasmuch as one can draw such linkages, and then continually dialogue with the program team about what she or he or they are seeing, sensing and experiencing. This includes what innovations are being produced.

Returning to the article on road tripping from The Hindu:

“Road-tripping is a great way to bond with the people you are travelling with and I would strongly recommend it to people. It not only makes you appreciate yourself as an individual but is an amazing experience as you get to meet new people, know different cultures and sample different cuisines. I can never forget biking on sleet, riding though torrential rains, gobbling hot rotis at dhabas, the beautiful snow-capped mountains and guy talk with friends on the trip,” says Dheeraj, who recently went to Ladakh.

Here the focus is on relationships, learning new things and taking that learning onward. That is what DE is all about. My colleague Remi illustrated this in our meeting by having us all spread out throughout the room and go through a pantomime-type skit where he collected information from each participant about where the wisdom was and then bringing this person along for the journey. So as he started out alone as the Developmental Evaluator, he wound up at the destination of wisdom with everyone.

Road Trip Advisor requires documenting the journey along the way, sharing what you learn with others, and continuing learning and revisiting your notes — while checking out what notes others have (including use of evidence from other projects and academic research) — and integrating that together on an ongoing basis.

But as my other colleagues pointed out in their presentations, the journey isn’t always about feeling good. Sometimes there are challenges as the Hindu article adds:

all is not hunky dory during these trips. You have to be way about accidents and mishaps. And, realise that freedom comes with responsibility. Says Arjun: “I had borrowed my friend’s bike for the trip, and though it looked good, it gave problems on the foothills of Kodaikanal and we couldn’t do the climb. Being a weekend, there were no mechanics. It helps to know your machine. A passion for road-tripping is not enough. You need to be equipped to take care of yourself also.”

Here, the story parallel is about being prepared. Know evaluation methods, know how to build and sustain relationships and to deal with conflict. A high tolerance for ambiguity and the flexibility to adapt is also important. Knowing a little about systems thinking and complexity doesn’t hurt either. Developmental evaluation is not healthy for those who need a high degree of predictability, are not flexible in their approach, and adhere to rigid timelines. Complex systems collapse under rigid boundary conditions and do evaluators working with such restrictions in developmental contexts.

So why do people do it? “Well, my memories of my favourite road trip were an injured leg, chocolates, beautiful photographs and a great sense of fulfilment,” recalls Arjun.

It is youngsters like these who have transformed road-tripping from just a hobby to an art.

After all, friendship and travel is a potent combination that you can’t say no to.

In DE, the “youngsters” are everyone. But as we (my DE colleagues) all pointed out: DE is fun. It is fun because we learn and grow and challenge ourselves and the programs that we are working with. It’s collaborative, instructive, and promotes a level of connection between people, programs and ideas that other methods of evaluation and learning are less effective at. DE is not for everyone or every program and Micheal Quinn Patton has pointed this out repeatedly. But for those programs where innovation, strategic learning and collaboration count, it is pretty good way to journey from where you are to where you want to go.