Getting started with analysis
Whether you're an analyst by profession or analysing a consultation for the first time, this guide aims to give you an overview of how to run a successful analysis process, alongside tips and tricks for getting the right mix. We've developed these articles to provide guidance for our customers and to share key lessons we've learnt. If you are an analysis expert and feel this guide is missing anything please feel free to get in touch - we love continuous improvement.
What is Analysis?
Analysis involves taking data collected and turning it into meaningful and actionable information. This information can be used in order to make effective adjustments and improvements. Analysis is one of the most important parts of the consultation process. Respondents have taken valuable time to respond to your consultation, you owe them an effective response back. Don’t worry, sometimes the simplest of data analysis can turn out to be the most useful.
We’ve split our analysis success guide into three parts to break down the process:
There's also our quick start analysis guide, which should be your first port of call if you need to turn the results of a recently closed consultation into meaningful data and don't quite know where to start!
Preparing for your survey
The reason some people feel they don’t get on with analysis is that often the planning stage is missed out. Effective planning and preparation makes it easier to produce useful analysis. It’s important to approach the task of analysis in a structured way. You should think carefully about who will actually be doing the analysis, what they’re going to do and when they’re going to do it. Time can be a constraint and good preparation will help you focus on doing what needs to be done in the time available. Another part of good preparation is in the design of the survey itself – if you don’t ask the right questions in the right way, you’ll find it very hard to get useful data out of your responses. Lastly, there are a lot of things you can do to improve your capacity to analyse, and get yourself and colleagues in a good condition to deal with data.
Taking a structured approach
Figure out what resources you have available
Resources, in this context, means people and time. Analysis will always involve an element of people time. If you know you are not going to have much resource available to analyse, and your consultation allows for it, try and maximise your quantitative data collection. Do you have the skills to analyse the data you’ll get for your responses, or is there someone in your department such as a social researcher who may be able to help advise?
Top Tip: Depending on the length of your consultation, we have found that for up to 300 responses, it will take two people two days to do the analysis. If you are looking at more than 500 responses, it will be a larger piece of work and you may want to consider approaching an external company, allocating more time or drafting in additional resources.
If you are analysing the data yourself or within a small team, think about what tools you have at your disposal. If you have access to a second screen, use it - Excel can become mind boggling after a while. If you can keep a clear record of everything you are doing - perhaps via an audit log for example. If your system permits it (it can often be blocked by security protocols), Google Docs can be a great tool for collaborative working or assigning different colleagues to different tasks.
Be clear about what you want to achieve, and establish success metrics.
Before you start your survey, you should be clear about what you want to get from the exercise. It sounds simple – but often organisations fail to do this properly. These considerations should play a big part in how you design the survey. Remember that an unnecessary or poorly-thought out question will not only reduce the chances of a respondent completing the survey, it will also create more work for you at the analysis stage.
To guide and focus your activities, it’s useful to establish some success metrics for your consultation. Example metrics could include:
- Number or responses: Think about whether you need get a very high volume of responses or a smaller number of high-quality responses.
- Quality of response: If you are aiming to reach a very large audience, you may want to focus your analysis on quite high-level figures, (ie. 80% of 750 respondents were in favour of the plans, except for group x, of whom only 40% of 55 respondents approved the plans). If you want to focus on a smaller group, you might want to get above a certain number of qualitative statements from each (see qualitative vs quantitative questions below).
- How many respondents completed the whole consultation, (this is another measure of response quality). Using analytics tools can help you find out your dropout rate.
- If you have one or two ‘key’ questions in the consultation, how many people answered these questions?
- How well did the responses reflect the demographics of the audience – so, were hard-to-reach groups engaged enough?
- How well planned was the consultation and what could be improved upon for next time?
- How long did the analysis take and how did this compare to the planned allocated time? If you audited each others’ work, (such as reviewing codes applied), how did this go?
Designing an effective survey
Make sure the survey fits your purposes
As mentioned above, it is important to make sure your survey questions really match what you want to get out of the exercise. Avoid any unnecessary questions: everything you ask should be geared towards producing useful data outputs, which really answer your organisation’s need. It can be tempting to ‘fill out’ a survey with extra questions if it feels a bit short – avoid this. Your respondents will not enjoy answering questions they don’t need to. If there are only a few questions you need to ask, focus on making them as high quality as possible. If the person/team designing the survey and the person/team producing the analysis are not the same, make sure you are joined up and talking clearly about this.
When asked, you should be able to easily articulate why a consultation and survey is being run. Has anything already been run in this area before which could be included as a reference point? If the consultation is part of a phase in the policymaking cycle make this clear. How much difference will this consultation actually make?
Be aware of your audience – and use analyst-only questions to your advantage
Be open about the possibility of bias in your results, and seek to address it. It is unlikely that you will get a representative selection of responses through a consultation. Those with strong views on your policy are more likely to take the time to respond. Alongside being mindful of your population sample, using analyst only questions in Citizen Space can help bring out key information. They will allow you to add additional data to help you catalogue responses e.g. if you get a disproportionate amount of data from one group in your audience. This article explains more about analyst-only questions. Analyst only responses can also be used in order to indicate whether a response is from a 'key respondent' - a person or organisation who is a particularly important stakeholder for you.
Quantitative vs Qualitative questions
When designing your survey, it’s important to distinguish whether you want to get quantitative and qualitative data out of a question. Think about the different responses you will get to the same question if it is posed qualitatively or quantatively, as below:
In your responses to question 4, you'll be able to clearly see which is the preferred choice of respondents, the correlation between each of the choices (as you can tick more than one), and so on. Question 5, on the other hand, will get free text responses that will need to be read by a human to be interpreted. They could all say completely different things.
This is essentially what we mean here by 'qualitative and quantitative'
Sometimes it is very clear, but it can also sometimes be a hard choice whether a question should have a quantitative or qualitative phrasing.
Things to consider are:
- In general, qualitative questions are not as well-suited to producing statistical data, such as charts, graphs, and numerical figures.
- However, qualitative questions give the respondent much more control over their answer, and allow for a much wider range of responses and sentiment.
- Using the tagging feature of Citizen Space (which performs what is called ‘coding’ in statistics), qualitative data can effectively be turned into quantitative data. However, you should be aware that:
- This can require a lot of time and effort from analysts.
- Tags will not show up on the PDF summary report, so qualitative questions will not show anything except their response rates, but you can query and export your tagged answers.
- The tags on a response will be listed within a single cell in the .xlsx file that exports from Citizen Space. Some software may be able to automatically separate them – but it is not in Excel’s basic functionality.
- Qualitative questions are suited to questions that have a limited number of answers. Therefore before making a question qualitative and ‘open’, think about whether it really is open, or whether there are only a few answers that you will get. This can save you time tagging responses after they come in.
- In surveys where you are looking for general and open responses (for example on a detailed document, plan or new regulation), qualitative questions can be good for drawing out a nuanced response. This is particularly case where you have an audience of ‘experts’. In these cases, it may help your analysis to give some guidelines in the question to guide responses – for example, ask respondents to answer specific questions rather than just ‘do you have any comment on x’?
- At their best, qualitative questions will let you tell a compelling story of what your respondents think. The best quantitative questions will produce numerical data that can be turned into compelling visualisations and subjected to detailed statistical analysis.
There are a lot of things you can do to make sure you and your organisation are as well-equipped as possible to analyse consultation responses:
Audit your existing capacity to analyse
- Could it be that someone in your team already has the skills that you are looking for, or is there someone has an interest in the topic, and could be made a ‘lead’ on data analysis?
- Have a look around the likely places to see if anyone in your organisation has produced useful documents or guidance on analysis.
Document your work
We do this at Delib all the time, we rely on repeatable processes and we save time by writing down how we did something tricky, so that our team mates don't have to spend time doing the same investigations that we had to!
- If they are not already available, create some useful documents that can be disseminated around your organisation
- Produce a consultation analysis guidance booklet, which references what internal resources are available to help. Widely promote this.
- Create a standard report template, if you don’t have one already. This can help if people without an analytic background will be in charge of reporting on consultations, and they don’t know where to start.
Run drop-in sessions or promote online tutorials
We hope these articles are a useful starting point, but there is plenty more information online: Youtube videos, Microsoft tutorials, etc. For a start, in terms of general best practice, you can have a look at this from the Scottish government.
Seminars or drop-in sessions can be a great way of passing what you have learned onto colleagues. They may even offer to teach you something in return! You can promote ‘skills swap’ in your organisation, where people volunteer to teach each other a skill. Alternatively, a ‘learning hour’ – where you take out an hour every week for one team member to teach everyone about something they are an expert in – can be both useful and fun, (depending on what your team members’ expertise is in…).
Work collaboratively, Audit each others' work & evaluate
- Having at least two people working on data analysis is a good way of avoiding your analysis becoming too subjective, especially if you have a lot of qualitative data. Getting two eyes on the data will make your interpretation more objective, and help avoid errors.
- Analyst only questions can be used in Citizen Space in order to indicate who has analysed a particular response, creating accountability.
- Analysis is an inherently subjective process and can take a large amount of time to complete. Ensuring more than one individual in an organisation is familiar with the data can help reduce the risk on this.
Top Tip: As a quality and objective check - ensure a colleague who has not done the analysis, but is involved in the consultation project is available to check every 10th response analysed.