In his book, Doing Qualitative Research, Silverman writes “no method of research, quantitative or qualitative, is intrinsically better than any other”[fn]Silverman, D., 2000. Doing Qualitative Research. Sage Publications Ltd.[/fn]. However, there may be a perception that quantitative research is ‘better’, because it is ‘bigger’; it is based on ‘bigger’ samples which is better than the smaller samples upon which qualitative research is based.

This article sets out some of the arguments for why this isn’t necessarily the case, and why small may be beautiful - not only beautifully insightful but also beautifully robust.

There are several commonly heard criticisms about qualitative research. Primarily it is criticised for lacking scientific rigour. As Mays and Pope point out, labelling an approach as ‘unscientific’ is particularly damning in an era when “scientific knowledge is generally regarded as the highest form of knowing.’’[fn]Mays, N., and Pope, C.,1995. Rigour and Qualitative Research. King's Fund Institute.[/fn] Silverman also points out that the phrase qualitative research seems to carry with it an implication that ‘we will avoid or downplay statistical methods and the mechanics of … quantitative methods used in survey research’.

Mays and Pope summarise the other criticisms as follows:

• Qualitative research is merely an assembly of anecdote and personal impressions, strongly subject to researcher bias
• Qualitative research lacks reproducibility – the research is so personal to the researcher that there is no guarantee that a different researcher would not come to radically different conclusions
• Qualitative research lacks generalisability. It is said that qualitative methods tend to generate large amounts of detailed information about a small number of settings.

In this report we will look at the strategies that can be applied to the qualitative research process in order to overcome these issues.

A brief background about quantitative and qualitative techniques – and what qualitative research offers
Generally speaking, quantitative methods achieve scope and qualitative methods achieve depth and detail. There is considerable power in bringing both methods to your research question.

Quantitative surveys are designed to be used on much larger samples than qualitative interviews, which is what allows inferences to be made to wider populations. Quantitative research also has in-built standardised reliability measures aimed at establishing the ‘facts’ with which the study is concerned, whereas qualitative researchers tend to use a non-positivist model of reality – that is to say, they hold a dynamic view of reality which will change according to the individual’s perceptions.

There isn’t the space in this editorial to discuss the advantages and benefits of qualitative research in detail, but essentially, qualitative research is about hearing from your audience in their language, exploring complex and often seemingly contradictory attitudes, beliefs, behaviours and responses in an open and exploratory way. It is deep, case-oriented analysis which puts the audience at the heart of the research, allowing you to gain a deeper insight or understanding about why people feel or behave the way that they do.

The crux of the starting point of any sound research project lies in correctly identifying the research question, exploring it from all angles and then deciding on the best methodology or combination or methodologies to answer it.

Don’t be blinded by science!

In the same way that the phrase ‘qualitative research’ does not automatically mean ‘unscientific’, it is worth mentioning that the phrase ‘quantitative research’ does not necessarily mean an automatic guarantee of validity or reliability.

There are two aspects to this. The first regards the issue of whether there is actually any “presumed underlying ‘truth’” which can be measured[fn]Mays, N., and Pope, C.,1995. Rigour and Qualitative Research. King's Fund Institute.[/fn]. In that sense, it is not possible for any type of research to capture a ‘literal truth of events’.

This leads us to the realisation proposed by Fielding and Fielding, that “ultimately all methods of data collection are analysed ‘qualitatively’, in so far as the act of analysis is an interpretation, and therefore of necessity a selective rendering”[fn]Fielding and Fielding, quoted in Mays, N., and Pope, C.,1995. Rigour and Qualitative Research. King's Fund Institute.[/fn]

The second aspect reflects that the credibility of any form of research, whether quantitative or qualitative, depends upon the skill and judgement of the researcher. A quantitative researcher can no more guarantee that the questions, categories and language used in a survey are shared uniformly by respondents, than a qualitative researcher can get his own interpretation or presence in the research interview out of the way.

As Silverman says, “So as you prepare your qualitative study, you should not be overly defensive. Quantitative researchers have no ‘golden key’ to validity”

Let’s go on to look at the ways in which we can bring rigour to the qualitative research process.

Bringing rigour and robustness to your qualitative research

There are numerous strategies available within qualitative research to protect against bias and enhance the reliability of findings. Mays and Pope provide a helpful summary of the underlying strategy which should be applied from the very beginnings of the research design to the presentation of the findings:

“The basic strategy to ensure rigour in qualitative research is systematic and self-conscious research design, data collection, interpretation and communication”.

Beyond this, Mays and Pope argue that there are two goals which touch on the issues of reliability (or reproducibility) and validity:

• To create an account of method and data which can stand independently so that another trained researcher could analyse the same data and come to essentially the same conclusions
• To produce a plausible and coherent explanation of the phenomenon under scrutiny

Ritchie and Lewis set out a list of questions which can be used to ensure the quality of each of these stages, and they have been referenced where appropriate in the sections below.

Reliability and validity in relation to the qualitative research process

Ritchie and Lewis argue that, “mathematical and scientific definitions of reliability and validity are wholly inappropriate for qualitative investigation. But in their broadest conception, reliability meaning ‘sustainable’ and validity meaning ‘well grounded’, they will have relevance for qualitative research since they help to define the strength of the data.”[fn] Ritchie, J., and Lewis, J., 2003. Qualitative Research Practice. Sage Publications Ltd.[/fn]

Reliability

According to Ritchie and Lewis, reliability is generally understood to concern the replicability of research findings and whether or not they would be repeated if another study, using the same or similar methods, was undertaken. Given the debate surrounding the issue of whether there ever is a ‘single reality’ to be captured, more useful language includes looking for confirmability of findings, trustworthiness, consistency and dependability.

Validity

The validity of findings or data is traditionally understood to refer to the ‘correctness’ or ‘precision’ of a research reading. As with reliability, there has been some attempt to move away from the concept of validity and to use instead other terms which are more appropriately related to the ‘correctness’ of qual evidence. These include credibility or transferability.[fn]Lincoln and Guba; Glaser and Strauss, quoted in Ritchie, J., and Lewis, J., 2003. Qualitative Research Practice. Sage Publications Ltd.[/fn]

Although the validity of measurement is seen as a primary concern of quantitative research, and of positivist research more broadly, it is recognised that it is an equally significant issue for qualitative research. However, Ritchie and Lewis describe how the questions posed are different and relate more to the validity of representation, understanding and interpretation, posing an overarching question to keep in mind as follows: Are we accurately reflecting the phenomena under study as perceived by the study population?

There are a number of strategies that can be used to measure validity. Here are a few of the ‘external’ methods.

• Triangulation – where data is collected from a deliberately wide range of different, independent sources and often by different means
• Feeding findings back to participants for verification – to see if they regard the findings as a reasonable account of their experience
• Continuing the interviews or groups as analysis unfolds – in order to incorporate their reactions to the evolving analysis in the research data

Explanation

Mays and Pope also discuss the importance of considering explanation in relation to validity and pose a similar observation; “the report should carry sufficient conviction to enable someone else to have the same experience as the original observer and appreciate the truth of the account”.

In addition, Mays and Pope highlight three simple yet compelling questions that can be asked of the research:

• How well does this analysis explain why people behave the way that they do?
• How comprehensible would this explanation be to a thoughtful participant in the research setting?
• How well does what the explanation is advancing cohere with what we already know?

In as much as these questions underpin internal quality control processes for those doing the research, a client may well feel empowered to judge a piece of qualitative research for reliability and validity by referring to these questions and measuring the findings against them.

1. Systematic and self conscious research design – methodology & sampling

The process starts with selecting the most appropriate qualitative methodology. Having arrived at the decision of using qualitative research rather than quantitative, it is then necessary to determine what kind of qualitative techniques are best suited to the research question. These range from face to face to online, groups to in-depth and many combinations in between.

The second task is to design the research sample. This is a key part of the process hence the importance given to working with a client to agree a recruitment screener. The sampling techniques differ for quantitative and qualitative research, but the process for qualitative sampling is no less considered and fit for purpose than quantitative techniques.

Quantitative research uses statistical sampling, such as random sampling, in order to address the concern with similarity and difference in classifying behaviour and awareness and maximise external validity or generalisability (as put forward in Mays and Pope).

Whilst there is no theoretical reason why random sampling couldn’t be used for qualitative research, it is not as practical. In addition, statistical representativeness is not normally a priority. The alternative sampling method in qualitative research is systematic, non-probabilistic sampling, or purposive or criterion based sampling.

This means that the purpose of the sample is to identify specific groups of people who either possess the same characteristics or live in circumstances relevant to the social phenomena being studied - not to establish a random or representative sample from a population.

As explained by Ritchie and Lewis, “members of a sample are chosen with a ‘purpose’ to represent a location or type in relation to a key criterion. This has two principle aims. The first is to ensure that all the key constituencies of relevance to the subject matter are covered. The second is to ensure that, within each of the key criteria, some diversity is included so that the impact of the characteristic concerned can be explored”.

The decisions about which criteria to use for selection are made in the early design stages of the research, and this will be informed by the aims of the study, existing knowledge or theories about the field of study, hypotheses to be explored and gaps in knowledge about the study population.

As Ritchie and Lewis point out, although purposive sampling involves deliberate choices, “this should not suggest any bias in the nature of the choices made. The process of purposive sampling requires clear objectivity so that the sample stands up to independent scrutiny. So although the researcher or funders may well have hypotheses they want to test, the opportunity for these to be proved or disproved needs to be equal”.

The questions posed by Ritchie & Lewis to ensure reliability of sample coverage: Was the sample design/selection without bias, ‘symbolically’ representative of the target population, comprehensive of all known constituencies; was there any known feature of non-response or attrition within the sample?

2. Systematic and self conscious data collection

In brief, this refers to the need to collect direct sources of data, allowing the participants closest to the research questions being addressed to speak for themselves from their own perspective, experience, particular area of expertise or any other relevance to the sample. This avoids a situation where opinions are collected which are on behalf of another group or individual, or speculative in nature.

It also refers to the quality of questioning, and the skill of the moderator in writing and using an effective guide which allows responses to unfold in full, and moderates the same guide fairly across the whole sample. This reminds us of the importance of the discussion guide phase and the client sign-off ahead of starting the research.
 
The questions posed by Ritchie and Lewis to ensure reliability of data collection: Was the fieldwork carried out consistently, was the quality of questioning sufficiently effective - did it allow respondents sufficient opportunities to cover relevant ground, to portray their experiences

3. Systematic and self conscious interpretation – ensuring reliability of analysis

Whilst we acknowledged earlier that ‘quantitative researchers have no ‘golden key’ to validity’, with their in-depth access to single case studies, qualitative researchers do have to ‘overcome a special temptation’[fn]Silverman, D., 2000. Doing Qualitative Research. Sage Publications Ltd.[/fn]raises the question, ‘how are they to convince themselves (and their audience) that their ‘findings’ are genuinely based on critical investigation of all their data and do not depend on a few well-chosen ‘examples?’” – and in so doing, lay the qualitative research open to the kinds of criticisms we looked at earlier.

Document the analysis process

Silverman’s answer in part is, ‘for reliability to be calculated, it is incumbent on the [researcher] to document his or her procedure’. Mays and Pope agree when they propose that the main ways in which qualitative researchers ensure the ‘retest reliability’ of their analysis is by maintaining ‘meticulous records of interviews and observations and by documenting the process in detail’. This includes logging the grounds or criteria for including certain examples and not others, so that it is possible to determine the typicality or representativeness of instances and findings generated from them.

Very often the raw data collected in qualitative research is in relatively unstructured forms such as tape recordings and transcripts of conversations. It is the transparency and record of the analysis process which both helps the client to evaluate the quality of the findings they have been given and also lends itself to independent evaluation if appropriate.

Silverman goes on to acknowledge that detailed data presentations which make minimal inferences are always preferable to researchers’ presentations of their own (and therefore high inference) summaries of their data. A ‘low inference’ descriptor means ‘recording observations in terms that are as concrete as possible, including verbatim accounts of what people say ... rather than researchers’ reconstructions of the general sense of what a person said’[fn]Seale quoted in Silverman, D., 2000. Doing Qualitative Research. Sage Publications Ltd.[/fn]

Researchers shouldn’t be afraid to use numbers in qualitative research

There may be occasions when it’s appropriate to use counts as a means of analysing and interpreting the data. It is perfectly feasible to count respondents’ own categories (i.e. in their own language) as used in naturally occurring places.

Silverman summarises this well:

“There is no reason why qualitative researchers should not, where appropriate, use quantitative measures. Simple counting techniques, theoretically derived and ideally based on members’ own categories, can offer a means to survey the whole corpus of data ordinarily lost in intensive, qualitative research. Instead of taking the researcher’s word for it, the reader has a chance to gain a sense of the flavour of the data as a whole. In turn, researchers are able to test and revise their generalisations, removing nagging doubts about the accuracy of their impressions about the data”

Relating this to the kind of research nfpSynergy carries out for our clients, might mean, for example, ‘counting’ results for a research project on a new name for an organisation or product, or evaluating different design routes for a new campaign. By collecting data during the research which measures order of preferences, such as favourites and least favourites it is possible to reflect back the general sense of preference.

As Kirk and Miller remark:“By our pragmatic view, qualitative research does imply a commitment to field activities. It does not imply a commitment to innumeracy”[fn]Kirk and Miller quoted in Silverman, D., 2000. Doing Qualitative Research. Sage Publications Ltd.[/fn]

The questions posed by Ritchie & Lewis to ensure reliability of interpretation:

• Was the analysis carried out systematically and comprehensively, were classifications, typologies confirmed by multiple assessments?
• Have the phenomena been identified, categorised and ‘named’ in ways that reflect the meanings assigned by study participants?
• Is there sufficient internal evidence for the explanatory accounts that have been developed, i.e. is interpretation well supported by the evidence?
• Did the design/conduct allow equal opportunity for all perspectives to be identified or were there features that led to selective, or missing, coverage?

4. Systematic and self conscious communication - presentation of research findings

As Mays and Pope point out, it is vital that the presentation of the research allows the audience as far as possible to distinguish the data, the analytic framework used, and the interpretation. The challenge of presenting qualitative analyses objectively is ‘the sheer volume of data customarily available and the relatively greater difficulty faced by the researcher in summarising qualitative data’.

Although it is possible to make full or partial transcripts available, we appreciate that we are being paid by our client for the task of distilling vast amounts of data into something concise, actionable and insightful. Our goal is to tell a story of the data in sufficient detail in order that the conclusions and recommendations make sense, and to use our judgement to edit and filter the data so that our clients don’t hear everything that everyone said, but do hear the content that matters.

The questions posed by Ritchie & Lewis to ensure reliability of data presentation: Have the findings been portrayed in a way that remains ‘true’ to the original data and allows others to see the analytic constructions that have occurred?

Conclusion

Through this study of the strategies that can be applied to ensure quality in qualitative research, it is possible to see how the process of systematic and transparent research design - choosing the right qualitative technique, and sampling; to high quality questioning and data collection; to systematic and thoughtful interpretation of the data in a way which is meaningful both to the study participant and to the wider context of what we already understand about the subject being researched, overcomes the criticisms levelled at qualitative research that it is merely an assembly of personal impressions, which are subjective and biased to the individual conducting the research.

What about the question of generalisability? Some might hold a perception that qualitative research is limited in its scope because it can’t be generalised to a wider population. Ritchie and Lewis contend that there is a misunderstanding that small samples do not permit statistical generalisations, and that there is in fact a strong link between the validity of qualitative data and the extent to which generalisation can occur – but in time honoured tradition of finishing with a cliff-hanger, that is a story for another day!

Silverman provides a pithy conclusion to this debate. “Deciding to do qualitative research is not a soft option. Such research demands theoretical sophistication and methodological rigour. Just because we do not use complicated statistical tests or do much counting does not mean that we can wallow in comforting hot baths of ‘empathetic’ or ‘authentic’ discussions with respondents. After all, if this is the limit of our ambitions, can we do better than a talk show presenter?”

Hopefully this report has addressed any concerns you may have had about the rigour of qualitative research. Not only that, hopefully it has empowered you to be able to judge good qualitative research for yourselves, not only when you are receiving a debrief or report but even when evaluating a proposal, to check that some of these strategies are in place. And if you don’t think they are, ask!

What I haven’t had been able to address in detail is why and when you would use qualitative research and what you get from it. For those who are unfamiliar with qualitative (or quantitative) research, please do not hesitate to contact us for more information.

For further information contact insight@nfpsynergy.net