Delib's 3 quick start articles on data analysis in Citizen Space
So, the responses have been flooding in, and now it’s time to get down to working out what they mean!
Have a look at our quick start guide if you need to jump into the process straight away.
Looking at and handling your data
When to start?
You may want to start carrying out analysis before your consultation closes. This saves you time if you’re on a tight schedule. There’s no reason to wait until you close to start tagging responses, but we would suggest you wait until you have at least a good number of responses, so you can accurately break down the qualitative answers you’re getting into qualitative tags (there is more on tagging/coding in the first article in this series, and below)
What are you trying to find out?
What analysis you do very much depends on what you are trying to achieve with the consultation.
Some consultations have a fairly ‘open’ objective – so simply to seek the views of the public, or a select stakeholder group, on a particular issue. A report on a consultation like this should probably seek to extract a number of conclusions from the data, by summarising the most prominent points that your respondents have made.
Alternatively, a ‘closed’ consultation seeks to gauge the public or stakeholders’ response to closed questions. This could be a yes or no question, or one with a limited number of responses – ie. Which plan do you prefer? Analysis in this case will, as its primary goal, seek to clarify the answers to these questions
In practice, most consultations will have elements of both. It is important when reporting, however, to focus on the key question.
Let’s look at some examples:
1. BIS (Department for Business, Education and Skills) wanted to consult on the Sharing Economy. Their consultation was centred around an objective of gathering views, in a broad sense – their end product was to produce a ‘review’ of the sharing economy – a fairly open-ended document exploring the issue - which could then inform future policy.
This is an example of an ‘open’ objective. A good report from the consultation would highlight some of the most prominent themes arising from respondents’ qualitative inputs in its headlines, and then support this by breaking down some of the data behind these themes – eg. what type of respondent was more likely to make what point? This interpretive data then serves as a reference for the review.
2. TFL (Transport for London) needed to consult on the plans for new ‘cycle superhighways’ going through Central London. Their broad objective was to seek public approval for the plans, on a stage-by-stage basis. In contrast to the BIS consultation above, their objective is quite ‘closed’ – how many people approve each part of the plans, and how many disapprove? Quantitative analysis can then reveal correlations in the data – which groups were more likely to approve the plans? Were there correlations between approval of one part of the plan and disapproval of another? These figures can then be supported by qualitative data – people’s reasons for approving or disapproving. This can be analysed by coding/tagging to support the ‘headline’ numerical figures.
Keep the context of your responses in mind
Remember not just to look at the numbers in front of you. When reporting numerical responses to closed questions ask yourself what the base size is (the number of respondents who were asked that question), as well as the number of people who have given a response to that question. For example, saying 35 out of 225 respondents disagreed with a policy gives much more context than saying 35 people disagreed. Due to their nature, consultations are not always fully representative - using appropriate terms such as “40% of responses received” rather than “40% of people” is important. If the number of non-respondents to a question is large, it's also important to indicate this in your analysis as well as in any tables or charts.
Pull data from multiple sources before drawing the overall conclusions
What you can and can’t do analysis-wise depends on the questions you asked in your survey. This means you can only draw overall conclusions initially from the survey data. If you are interested in developing a more holistic approach consider triangulation. Survey data + focus groups + stakeholder events for example can provide a wider overview. Assisted digital is important within this.
Manipulating and Interpreting data
Where to start?
It can sometimes be tricky to know where to begin with interpreting your responses. Citizen Space allows you to either view responses by respondent (ie. all of the questions that one respondent answered) or responses by question (showing the way all respondents answered one of the questions). Which way is more useful depends on what you are looking to find out.
Going to 'Analyse responses' allows you to go through the responses one by one. You are able to tag the responses and also to append commentary to each question.
Qualitative data - Set up coding directories & append comments
Before analysing qualitative (i.e. text) consultation responses, ensure you go back to the aims of your consultation, which may not be the same as your consultation questions.
For example, you may want to gauge which policy option respondents favour whilst also looking at how different groups within your population have responded. If you are interested in the views of your respondents who are responding from an organisation, against those who are responding as an individual, you will need to compare the two groups.
Coding or 'tagging' in Citizen Space can help turn qualitative information into quantitative, helping to track key themes that are emerging from the consultation. However, coding is not the same as collecting quantitative information. Coding each response into these categories will give you an idea of the numbers in each group. Whilst coding you can also use the 'analyst only' questions in Citizen Space to help select illustrative quotes. This article explains more about tagging/coding and how to do it.
Basic coding can take the form of 'agree/disagree'.
Thematic coding identifies themes within the qualitative data - you can either decide which themes to look for in your data in advance, or explore your data to see what themes emerge in response to your questions, or combine both approaches.
You may want to tag or code responses under the coding/themes that you’ve identified. One of our customers has taken a combined approach, reviewing the first 10-20% of responses first before deciding on the relevant codes to add. Using a google doc which is shared with colleagues to create a code book can help with this, it also helps ensure codes added are less subjective. Planning codes in advance can also ensure you keep the number to a sensible limit - too many codes can lead to issues of not being able to see trends.
Question: What improvements would you like to see in the surgery?
If the responses broadly fall into three categories: a) those proposing shorter waiter times b) improvements to the waiting room area and c) longer opening times you could use the following codes:
Code 1 - Shorter waiting times
Code 2 – Waiting room improvements
Code 3 – Longer opening times
Code 4 – Other response
Another function available to you in 'Analyse Responses' is adding analyst commentary to responses. You will see that there is a text box below each question that the respondent has to answer - this is a space for you to add analyst commentary. Clicking 'copy' above the text copies out the response into the box, allowing you to redact it or insert your interpretative comments directly into the response text.
Use grouping (cross-tabulation) and filtering to your advantage
Cross tabulation or 'grouping' provides an excellent way of comparing two subgroups of information. Cross tabulation allow you to compare data from two questions to determine if there is a relationship between them. There's more about the practicalities of doing this in Citizen Space here.
Filters allow you select specific subsets of data to view. Unlike a cross tab, that compares two questions, a filter will allow you to examine all questions for a particular subset of the responses. By viewing only the data from the people who responded negatively, look at how they answered other questions. Find patterns or trends that help define why a person answered the way they did. You can even filter on multiple questions and criteria to do a more detailed search if necessary.
Filters do not permanently remove the responses of those people that do not match the specified criteria; they simply eliminate them from the current view of the data, making it much easier to perform analysis. By looking at the same question with different filters applied, differences between the various respondents represented by the filter can be quickly seen.
Top Tip: Always remember to clear filters from your previous query as they do not clear automatically
Benchmark & look at using ‘longitudinal’ analysis
Benchmarking helps establish a baseline number. If this is the first year you are collecting information make this year your benchmark. Being able to track year after year is called longitudinal data analysis. Once a benchmark is established you can determine whether and how numbers shift. You could also look at the trends within different sub-groups.
Exporting data - Keeping your house in order & using the right tools
When you export the results of a consultation from Citizen Space, they'll be in the form of a .csv file. These can be opened and manipulated in Excel, and for most cases, the analytic functionality in Excel is sufficient for analysis the data that comes out of a consultation on Citizen Space. This article describes how to export your responses.
Some of our customers use other, more specialised analytic tools, such as IBM's SPSS Statistics. To get the most out of packages like SPSS, you or your colleague who is doing the analysis should have some understanding and experience of statistics and technical analysis. We'll discuss it more below and in the next article on reporting, but you should note that it's important to do the right analysis for both your data and your audience. Simpler can sometimes be better - particularly if your data is reasonably simple, or your audience is only interested in quite a basic and high-level interpretation. So, whilst it can be tempting to perform as thorough and technical an analysis as possible on your data, think about whether it is really necessary before you commit your resources to it!
Making sure data in organised and correct is very important to getting analysis right. A couple of good principles to follow are:
- In Citizen Space, you can download the full response data at any-stage, whether the consultation is still open or not. Make sure that the data you are analysing is up to date and the latest version.
- Once you have downloaded your data, keep a ‘clean copy’ of the data so that this first sheet is ‘read only’ - this ensures you always have a copy of your raw data to hand, which can be useful if you accidentally make changes or delete the data as you manipulate it.
- If you are not familiar with Excel, or need a source for refreshing your knowledge, the following sites provides lots of short ‘how-to’ videos!:
- If you're using SPSS, there's also a host of good tutorials online: this one from the LSE's website is useful.
How to deal with campaign responses
If you receive a large number of standard responses, for example as part of a campaign, you need to include all responses in your initial analysis. However, you may also provide additional analysis that excludes the standard responses in order not to skew the results. You may also highlight the views of particular groups separately. For example, if your policy affects hill farmers more than other types of farmers, you should provide analysis of responses for all respondents, and also analyse responses of hill farmers against those of other farmers.
Be aware of which stakeholders did not respond to the consultation, and consider whether they need to be engaged through other means.
Including a table of campaign responses as an appendix as Department of Health did with their report on standardising tobacco packing can help. Referencing the different campaigns which were run and if relevant by which organisation can also be relevant.