This post is cross-posted from Lucy Chambers‘s blog.[clear]
Widely-held Opinion (to which I subscribe): There is a huge amount of potential power to be unlocked in non-profits using technology in their quest for positive change in the world.
Comic Book Reference, ergo Fact: With great power comes great responsibility.
Personal Opinion: As tech projects become more and more widespread at NGOs, and their databases ever larger, we should reflect on whether impact they are likely to achieve will always be positive.
This post tackles the question: “Can we provide simple templates to help people understand whether there are any potential negative impacts of their technology project?”
Some confessions
Confession 1: I’m a big Theory of Change fan.
Confession 2: I really like the Theory of Change template from the DIY Toolkit and it has become a staple of mine for creating lightweight, easy to read theories of change for projects.
Update: Reactions to Feedback on this post
I got some great thoughts on this post. I wanted to clarify a couple of things specifically about the process for creating ‘Utopian’ theories of change which didn’t come across in the original version. Basically, these are:
- Creating a full theory of change is a much bigger piece of work than simply filling in a template. A theory of change should start from an organisational strategy and include all of the activities you are hoping to engage in.
- At the same time, I have seen so many organisations be overwhelmed by having to produce a full theory of change straight off. Worst case scenario, this leads to them not producing one at all. The DIY toolkit template I consider an accessible compromise to getting started with theories of change.
I am mainly addressing point 2. in this post. Getting creative juices flowing and hopefully making theories of change seem a little less overwhelming. The Dystopian theory of change is also a hack of the standard methodology, the antithesis of what it was originally intended for. Both templates serve as an absolute minimum of questions one should ask oneself before starting, rather than a replacement for a deeper thought process.
Going deeper into theories of change? Some useful resources:
As ever, thank you in particular to Dirk Slater for his thoughts on the topic!
A flash of lightning: genius or smiting?
Last year at the Responsible Data Resource Sprint in Hungary – I had a thought: Many of the theory of change templates I had seen all assumed that the change that the project would bring in the world would be positive. But what if the project went wrong? Really wrong? What if the data you collected fell into the hands of someone who wanted to do something bad with it?
I thought,
What if people had to write a dystopian theory of change for their project before starting it?
What if they had to answer the question: “What’s the worst change that I could possibly unleash by doing this project?”.
With great power comes great responsibility
Again, I’m talking specifically about technology projects here. The reason being that I have seen very few mechanisms or training to help funders or implementers (a few great exceptions aside) to catch out a bad technology project before it starts. Technology is still often seen as something mystical and a fix-all, and I have seen people with huge amounts of sector expertise and common sense suddenly set those things aside because there is the possibility of producing a flashy visualisation, or a dashboard with bells and whistles.
I hacked the DIY-Toolkit template (CC-BY-NC-SA) to give an example of what these questions could look like for planning the worst case scenario. The original template is quite good, in that it asks you to question your assumptions at each stage of the process. However, I personally have never found exposing assumptions very easy when I am so close to a project. I find the only way I can really expose what my assumptions are is to play through a few scenarios (ideally with someone else for sanity checking) and see if there is anything they reveal about how I am planning to conduct my project.
Here’s the template I came up with in 5 minutes:
Poor eyesight? Zoom original image
(Yes, this is optimised for prettiness rather than practicality. Remix at will!)
Note, I was thinking in particular about two possible failure modes for projects when I wrote this (inspired by the types of examples people were talking about at the sprint):
- Genuine evil intentions: Data collected by the project gets into the hands of someone who has intentions less noble than the project leaders.
- Power / user check: What if the intended users of the site / app / data are not the only/main ones who use it? I am thinking here particularly of cases where something generic is entered into the “Who is the audience?” box such as ‘all-citizens’ (shiver). I can tell you now that I am still searching for a project that ‘all-citizens’ are interested in. It is far more likely that a given project will appeal only to a particular subset of a population, and that may have specific implications for who the information empowers.
Roadtests
Our group did a lightweight roadtest to see what happened when we tried different types of projects in the template. Try it yourself! Here’s an example which might seem silly, but I have seen quite a few projects which have not even gone into this level of detail in thinking through their intended impact…
Cleaning up crime
Utopian version
What is the problem you are trying to solve?
Gun crime in my neighbourhood is on the rise. Many innocent people are dying and the younger generation is getting sucked into the crime loop.
Who is your key audience?
Citizens in high crime areas and police officers.
What is your entry point to reaching your audience?
An online visualisation of crime statistics by neighbourhood accessible to all and prominently linked from the police website. An accompanying application for mobile users.
What steps are needed to bring about the change?
- Collect data about crimes in the local area.
- If not already done, categorise this data somehow and geolocate it
- Produce an interactive map
- Publish and promote
- Update data regularly
What is the measurable impact of your work?
- Lots of site visits
- Media coverage
- Evidence of citizens making informed choices about where they want to live based on the data. (Could collect case studies etc)
- Evidence of parents keeping children away from dangerous areas.
What is the long term change you wish to see in the world?
Safer neighbourhoods with less crime. Better informed policing strategies meaning a more efficient police force.
The dystopian version
Problem statement is the same.
What if others were more interested in using your work than your intended audience?
Actually, it turns out that the most interested group is made up of wealthier, more socially mobile members of society. Poorer people either cannot access the visualisation as they do not have computers / the time [add more here…].
How do they find out about your work?
Your website is featured in the Guardian (its only media reference), lots of nerdy blogs and widely on social media. The mobile app version trends on the app store and appears in the top downloads, meaning many more people are exposed to it (leaderboard syndrome).
What are the measurable effects?
- Lots of site visits, (If you have baked in a mechanism for tracking demographics, you will discover it is being used largely by middle-class, socially mobile people).
- Media coverage
- Evidence of citizens making informed choices about where they want to live based on the data. This genuinely happens but not all people equally able to move. The richer people move out of the area, the poorer are left behind.
- Evidence of parents keeping children away from dangerous areas. Perceivable stigmatisation of particular areas as dangerous creates ghettos.
Asking more questions before starting a project
The Schooloscope example also made me think that there are so many other questions that people should ask themselves before starting e.g. ‘what is my sustainability plan?’ and ‘what happens if I need to shut down?’. (More on that topic is captured in the Data in the Project Lifecycle section from the same resource sprint).
For a cheat sneak-peak into what actually happened with the Schooloscope project, see Mushon Zer-Aviv’s great blog post.
Other theory-of-change-like ideas for the same purpose
I love the exercise of trying to come up with a headline in 10-30 years time. What it will look like if you are really successful?
“No child will ever go hungry ever again.”
You could equally well do a dystopian version 🙂
Conclusions
The theory of change template is meant as a lightweight sanity check for implementers starting out on a technology project. It is not a crystal ball and will not catch everything that could possibly go wrong with a project.
Mitigate – don’t stagnate
This method is not intended to discourage experimentation. I personally believe that it is important that we don’t let abstract worries and “what ifs” inhibit pushing the limits of tech for good. We should not clamp down before we have seen potential, but we should apply common sense.
What the template does is ask the implementer to reflect before starting a project on whether there is anything they are overlooking and whether any mitigating steps are required. It should take as long as a piece of string to complete. If that piece of string is at least as long as it takes to drink a cup of tea and to talk over an idea with a friend, I think we are making progress.
A delicate balancing act
You may (and probably will) decide that the probability of your dystopian reality coming true insignificantly small. It will hopefully be outweighed by the benefits and there will be mitigating steps that can be put in place to cover your proverbial for the worst case scenario.
Even in the example about crime statistics above, I believe a way could be found to successfully deliver on the initial problem statement. We could start by looking more closely into who exactly the users would be, what power they would have to act, how we would reach them and what would happen if they were not being reached.
Perhaps, in this example, if the original Theory of Change template had been used correctly, and it had been possible to correctly outline assumptions, we would not need to run the dystopian version. Half of the problem here was that the user had not been defined accurately enough and the assumption which was made was that the site would be of equal benefit to “all citizens” (shiver). Power mapping and empowerment were big topics at the sprint.
Keep swapping stories
One of the best parts about the Responsible Data Forum was story exchange. Participants were given a safe space to share examples of types of ways in which projects had gone awry.
In that sense, the participants of the Responsible Data Forum are in a priveledged position. It is always easier to imagine worst case scenarios and tease out assumptions when you are armed with a wealth of stories and patterns for how things might go wrong.
Talking about these issues can be hard, but I hope that this community will find a mechanism of sharing these in a way that encourages more people to come forward and share their stories. I look forward to following the Responsible Data community closely for more action on this front in the future!
Try it yourself
Need some examples? Here’s a random selection to try:
- Wikileaks releasing the diplomatic cables
- An application to allow flight attendants to take videos of people who they believe might be being trafficked
- Schooloscope. (Hint: if you put something generic like ‘all parents’ in the users for the Utopian version, try being more specific with specific income or social mobility levels in the dystopian version).
Feedback
I love feedback. Drop me a line on lucy [at] fedia.net.
Thanks
Thank you to Beatrice Martini for her thoughtful feedback on this post and to Zara Rahman for providing numerous examples to test.
