lectures_natalija_2026

1. Reliability, validity, and triangulation in cultural studies research

Reliability,
validity,
and
triangulatio
n in
cultural
studies
research
Natalija Majsova
University of Ljubljana
Cultural Studies Research - 1

2. Welcome to the Kick-off meeting!

Welcome to the Kickoff meeting!
• What do we know?
• Who are we?
• What do we want?
• What are we up against?
• How do we proceed?

3. Our 3 sessions

Session 1, March 16:
Reliability, validity, and triangulation in qualitative research.
Session 2, March 23:
No class; please attend one of the round tables organized in Akademska čitalnica Aleša Debeljaka as
part of the Aleš Debeljak Symposium. Choose one of the following
• 13.00 Natalija Majsova & Mitja Velikonja: From the »Postmodern Sphinx« to the
»21st-Century (Digital Boy)«
Luka Arsenjuk (Maryland), Iva Kosmos (Zagreb), Maja Manojlović (Los Angeles),
Nina Rajić Kranjac (Ljubljana)
• 15.00 Aljoša Pužar: Academic knowledge and contemporary society – the role of
the intellectual class in the time of artificial intelligence
Konstantin Haensch (Berlin), Peter Klepec (Ljubljana), Katarina Peović (Reka),
Ksenija Vidmar Horvat (Ljubljana)
We will discuss impressions in class on March 30.
Homework: Please contribute a project idea to:
https://docs.google.com/document/d/1t2Qpc6Kic7TaB1FglegKJccg3O8S5X2dvr1KxgBuRPI/edit?usp=sha
ring , by March 26.
Session 3, March 30:
Project writing and planning: initial project proposal brainstorming.

4. Cultural studies research objects (the list is non-exhaustive)

popular culture (from video spots and
film to fashion)
art
food
Angela McRobbie:
“the three Es”:
sports
cultural technologies
imaginaries
politics of aesthetics
structures of feeling
empirical
ethnographic
experiential

5. Raymond Williams (1921-1988): The Social Production of Art or “Marx was simply not materialist enough!”

An exploration of the concept of ‘production’ itself.
Accusation of sociology of art:
Raymond Williams
(1921-1988): The
Social Production of
Art or “Marx was
simply not materialist
enough!”
Reductionistic (sociological) conceptions of art: as
either ideological epiphenomena of the economic
base or simple ‘reflections’ of society.
A reification of both economic systems and art,
treating each as discrete self-sufficient objects of
radically different orders.
Accusation of art history as formalist.
BOTH of these are a product of modern capitalism
(a specialization and fetishization of economic
practices; a marginalization of those practices that
do not subscribe to a capitalist logic.)
MEDIATION as an alternative concept (against
“reflection”)? Bohuslav Vasulka, The Art of Memory
(1987).

6. Cultural studies research methods

Aim: to chart ‘empirical
changes in culture and society
on living human subjects’ and,
secondly, inviting ‘these same
human subjects . . . to reflect
on how they live through and
make sense of such changes’
(McRobbie 1997).
• How do we grasp
experience?
Experiences as modalities of existence; as sites of struggle, pleasure, and agency

7. What about the numbers?

• Cultural studies has held a perennial
distrust toward quantitative methods
where you count things and use
statistics to make sense of big
numbers.
• Why?
• And why is this a problem?
• However, what are we missing out on?

8. The stories we tell …

“Stories are central to the ways in which
people make sense of their experience
and interpret the social world. In everyday
life and popular culture, we are
continually engaged in narratives of one
kind or another. They fill our days and
form our lives. They link us together
socially and allow us to bring past and
present into relative coherence.” (Michael
Pickering, 2008).
The stories we tell …
• How can we use stories to grasp
experience?
• Whose experience are we grasping?
• How is our own entangled with it?

9. Narrative research – how?

• By examining the ways in which stories circulate socially as cultural
resources, how they operate in our everyday lives as organising
devices through which we interpret and constitute the world.
• By approaching and interpreting these narratives (= consisting of
characters+action+plot) as researchers. But how?
• Does it matter whether narratives about the social world are true in
the sense that they refer, in however mediated a manner, to an
empirical world ‘out there’? Are narrative truths local and
contingent, rather than universal and absolute? In what ways are
truth claims politically significant? (Steph Lawler in Pickering, 2008)
• Examples: (auto)ethnography; media narratives; (false) memoirs;
memories.

10. Discourse analysis

Problem of DA for cultural studies purposes:
We are not only interested in the narratives and
rhetoric, but also in the ways they have an effect
on people!
• How are language and power intertwined?
• Discursive formations: words, entangled with actors, structures, media, institutions.
• Discourse analysis one of the privileged methods for ideology critique
• But how do we know that what we are researching will bring any meaningful insights?
• Martin Barker (2008): if quantitative methods use the standards of validity, reliability,
and generalizability, qualitative methods resort to trustworthiness as the key
touchstone.
• Trustworthiness in discourse analysis relates back to:
• having a defensible corpus of material for use in discourse analysis;• using defensible methods as we move from text to context or back and forth between them;
• taking responsibility for implied claims, particularly about reception.

11. Cultural production – what and how?

• Political economy (capital flows, networks of decisionmakers, producers, distributers, media technologies
and channels, and their targets)
• Textual analysis (analysis of audiovisual and written
texts as narratives, representations, archival
documents, symptoms of a certain cultural condition,
…)
• Sociological/ethnographic work (looking first-hand at
the practices and conventions that are involved in
cultural production)
• Examples. What is art production/news
production/musical genre production as a social and
cultural process?

12. Cultural consumption and reception

• What is consumption as a cultural practice?
• Media consumption
• Fashion
• Music, audiovisual culture, museums
• Food&drink
• Games, sports
• Relationships?
How?
Interviews
Focus Groups
Ethnography
• Examples: readers’ consumption of newspaper
discourses on paedophilia and the ways this
affects their understandings of the issue
(Anneke Meyer 2007);
• Female consumers of self-help literature and
how this literature is bound up with gender
identity (Wendy Simonds 2008).
Problem: “the lines between production
and consumption are increasingly blurred as
everyday cultural consumers may also be
cultural producers of one kind or another.”
(Pickering 2008)

13. Ethnography, (participant) observation

• Going back to the question of experience …
• Cultural studies research typically rejects the
idea of universal experience and universal
subjects, privileging particularity and plurality.
Ethnography,
(participant)
observation
• Therefore, while categorization and
classification are important, they are always
immediately questioned. What are they based
on? Whose experience, which circumstances,
which assumptions?
• Virginia Nightingale (2008): Observation matters
because it brings the participants’ worlds of
experience into closely considered view. The
communicative exchange on which it is based
means that neither side’s version can any longer
be considered paramount or as necessarily
carrying greater authority than the other.

14. Visual analysis

• Analysis of contemporary cultures as
equally visual and textual.
• Methods from art history; visual
anthropology; visual sociology; cultural
analytics?
• Sarah Pink: “Ethnographic research consists
of visual and sensory embodied experience
and knowledge, and this can best be
communicated by combining images and
words”.

15. Back to experience

• Sarah Pink (2008): the uses of visual media and visual methods,
among both researchers and research participants, are located in
particular social relationships and cultural practices. Institutional
and contextual meanings are in turn embedded in visual images, in
the conventions that inform their production, and in the role of
situated human agents as viewers and interpreters of images.
• Examples: research into a community garden project connected with
the Slow City (Cittàslow) movement in the UK; visual representations
found on a Spanish website (close examination of Spanish social
relations and cultural values).
• One of the concerns of cultural studies is to develop collaborative,
non-hierarchical methods.
• Whose ‘voice’ is being heard, and possibly privileged, in any specific
research project or sample of cultural analysis?
• Pickering (2008): “It is desperately easy to assume what is involved
in the experience of research participants, and perhaps as well feel
one has the intellectual authority to configure this on their behalf.”
Back to experience

16. Research designs in cultural studies

Research perk 1 (since the 1970s): the interplay
between lived experience, texts or discourses, and
the social context.
1970s-1980s: ground-breaking research into media
audiences, subcultures, working class youth
cultures.
Research anxieties (via Saukko 2003):
Has our interest in cultures that are radically
different from our own, such as working-class or
non-Western cultures, been warranted, and can we
understand and do justice to these cultures?
How can we critically analyze culture in a situation
where we as scholars, and research as an
institution, are an integral part of this culture and
its struggles?
Is culture the most important topic to investigate in
the face of gruelling global economic inequality and
exploitation?

17. Further research anxieties

18. Reliability and Validity in (Social) Science?

19. Margaret Mead and the problem of Validity

• Margaret Mead (1929): Coming of Age in Samoa: A
psychological study of primitive youth for Western
civilization:
“Are the disturbances which vex our adolescents due to
the nature of adolescence itself or the civilization?”
(Mead, 1929: 11)

20. Derek Freeman (1983): Margaret Mead and Samoa: The making and unmaking of an anthropological myth

21. How could it happen that the two anthropologists arrived at such different conclusion?

Temporal frame
Informants
Intentions and paradigmatic background

22. Culturalism (Mead) vs. Sociobiology (Freeman)

• Mead: behaviours (such as adolescence or
sexuality), which have been thought to be
shared by all humanity, have turned out to be
the result of civilization, “present in the
inhabitants of one country, absent in another
country, and this without a change of race”.
(1929: 4)
• Freeman: referring to certain violent events in
Samoa, argues that in such circumstances
conventional behaviours are dropped and
people are taken over by “highly emotional
and impulsive behaviour that is animal-like in
its ferocity.” (1983: 301). I.e. aggression is a
proof of “much older phylogenetically given
structures” that define behaviour in addition
to culture.

23. Complications of “truth” about Samoa …

• fluidity of Samoa itself (different opinions, groups,
historical change, and so on). There are young girls,
village elders, myths, customs, different rules,
institutionalized and informal trespasses, rank-based
and gendered social and political divisions, struggles
and perspectives, all constantly evolving and
transforming.
• commitments that frame the research of the scholars
(historical, political and theoretical investments). The
anthropologist’s vision is coloured by her and his
personal gendered, raced and aged inclinations and
paradigmatic and political allegiances. Both Mead and
Freeman render Samoa a parable or allegory for the
West, and their oppositional readings end up
encapsulating the classic juxtapositions harboured in
the Western notion of the ‘primitive’: Apollonian
sensual paradise and Dionysian violence and danger.
• language (impressionistic or realist genre) used to
describe Samoa. Mead’s broad-brushed
impressionistic style paints a dreamy, softshaped
portrait of Samoa. Freeman’s use of hard-core
objectivist realism presents us a police-report on the
aggressive Samoans and a court-case against Mead,
ending up no less ideological and political than
Mead’s writing.

24. What are validities in the ocean of qualitative methodologies?

Research design basics: what do we want to achieve?

25. 1 Dialogic validity. Capturing the “truth” of others and representing it.

Truthfulness. Research should do justice to the
perspectives of the people being studied, so that
they can, in the main, agree with it. This entails
collaborative forms of research, such as measures to
allow the people being studied, such as Samoans, to
have a say in the way in which they are studied and
represented (in traditional research parlance the
latter is referred to as ‘member check’ (e.g. Seale,
1999)).
Self-reflexivity. Researchers should be reflexive
about the personal, social, and paradigmatic
discourses that guide the way they perceive reality
and other people. This entails that scholars need to
try to become aware of the cultural baggage, such
as notions of the ‘primitive’, that mediates their
understanding of different worlds.
Polyvocality. Researchers should be conscientious
that they are not studying a lived reality but many.
This means that they should make sure that they
include the views or voices of major ‘stakeholders’,
such as young girls as well as village elders (Lincoln
and Guba, 1985), trying to be true to their diversity
as well as relations and tensions between them.

26.

Dialogism does not view research in terms of describing other worlds from the
outside, but in terms of an encounter or interaction between different worlds. The
main criteria of validity of this approach then is how well the researcher fulfils the
ethical imperative to be true to, and to respect, other people’s lived worlds and
realities.

27. 2 Deconstructive validity. Unravelling social tropes and discourses that, over time, have come to pass for a ‘truth’ about the

world.
Postmodern excess. The postmodern or Baudrillardian (1980;
also Lather, 1993) notion of ‘excess’ of discourses points out
that there is a potentially infinite number of ‘truths’ or ways
of approaching the reality. Thus, research is assessed in terms
of how it manages to highlight the multiple ways in which a
particular phenomenon can be understood, to destabilize any
‘fixed’ understanding of it. E.g. Freeman–Mead controversy.
Genealogical historicity. Genealogy, associated with the work
of Foucault (1984), challenges truths by exposing their
historicity. Thus, research is evaluated in terms of how well it
unravels the way in which certain taken-for-granted truths
are not universal or timeless but products of specific
historical and political agendas.
E.g. the analysis of the historical, political and theoretical commitments of
Mead’s and Freeman’s works.
Deconstructive critique. Deconstruction, associated with the
work of Derrida (1976), aims to question the binaries that
organize our thought, to expose their hidden politics. Thus,
research is evaluated in terms of how it manages to unearth
the constitutive binaries that underpin our understanding of
a particular phenomenon.
E.g. an analysis of the constitutive binary between the sensual or aggressive
nature of ‘primitive’ societies and the ‘civilized constraint’ of the Western world
that interlace both Mead’s and Freeman’s works.

28. Isn’t this just about bias?

Poststructuralist critique argues that there
is no ‘unbiased’ way of comprehending the
world.
Therefore, its notion of good research is
twofold:
Isn’t this just about
bias?
1)
good or valid research is understood
to expose the historicity, political
investments, omissions and blind
spots of social ‘truths’;
2)
good or valid research is also
understood to be aware of its own
historical, political and social
investments, continuously reflecting
back on its own commitments.

29. 3 Contextual validity. Locating the phenomenon within the wider (social, political, global) context.

Sensitivity to social context. The duty of scholarship is
to carefully analyze, for example, historical events,
statistics and developments, using and com- paring
different resources and views. Research cannot be
haphazard or based on a hunch.
Studying Samoa from this perspective would mean to
carefully analyze the history of the islands, their social
structures and interaction with the outside world
through commerce, missionaries, even
anthropologists. Even if both Mead and Freeman
discuss the social context of Samoa, this fades into the
background against their project of capturing the
‘ethos’ of a relatively timeless ‘primitive’ society.
Awareness of historicity. Research should be able to
understand its own historicity. Social science and its
object, historical society, cannot be separated, and
analyzing the social context also enables research to
become aware, and be able to critically evaluate, its
role in it.
Research on Samoa would need to be aware of the
ways in which it is implicated in the social context of
which Samoa forms a part, such as structures of
colonialism or anti-colonialist struggles (Bhaskar,
1979).

30. (Beyond?) Triangulation

Four basic types of triangulation:
(1) data triangulation, which involves time, space, and persons;
(2) investigator triangulation, which consists of the use of multiple rather than
single observers;
(3) theory triangulation, which consists of using more than one theoretical
scheme in the interpretation of a phenomenon;
(4) methodological triangulation, which involves using more than one method
and may adopt within-method or between-method strategies.

31.

There is also multiple triangulation, whereby the researcher combines in one investigation
several observers, theoretical perspectives, data sources, and methodologies.
Critical or interpretive triangulation can be viewed as an alternative or incitement to traditional
postpositivist forms of validation.
Interpretive triangulation opens the space for conversations about how a text authorizes or
legitimizes itself through the use of multiple voices and representational forms.
These forms may act as catalysts to transgressive validities and to a politics of resistance.
Problems with triangulation
(1) locating a common subject of analysis to which multiple methods, observers, and
theories can be applied;
(2) reconciling discrepant findings and interpretations;
(3) novelty, or the location of a problem that has not been investigated before;
(4) restrictions of time and money.

32. Beyond Triangulation?

In social research, triangulation refers to combining different
theories, methods, sources and material’ (Denzin, 1989; Flick,
1998).
Problem: triangulation derives from a positivist take on science
that presupposes access to universal truths … but how can we
coherently combine research methods that, rather than searching
for one truth, aim at achieving different perspectives?
Laurel Richardson (2000): different methods offer different prisms
of looking at reality. “Crystals not only ‘reflect externalities’ but
‘refract them within themselves’ (2000: 934). The metaphor of
crystals highlights the way in which reality changes when we
change the methodological angle or perspective from which we
look at it. (Saukko 2003)
E.g. A poem Richardson (1992) has written on the life-story of a
woman, ‘Louisa May’, whom she had interviewed as part of a
project on unwed mothers. Through the poem, Richardson
wanted to convey Louisa May’s life in her own terms and in her
own Southern rhythm, without reducing her to statistical,
sociological categories of class, educational level and so on. Thus,
methodologies and writing strategies are not seen as means of
reflecting reality, presumably ‘objectively’, but as devices that the
scholar uses to create and convey different realities.

33.

34. Mediating between universalism and prisms: diffraction

35. Paula Saukko (2003: 25): Aligning triangulation with cultural experience. Methodological dialogues?

(Ontology – nature of reality
Epistemology – nature of knowledge)

36. Recap

• Research in CS
• Qualitative over quantitative methods of data collection,
processing, and analysis
• Data: Narratives, discourses, sounds, visual representations,
affects
(grounded in theory and context analysis)
• Data sources: cultural texts (apart from documents and
media accounts, films, buildings, video games, clothes, etc.
also count as texts), events, people …
Recap
• Collaborative research method development to break the
“tyranny of the interviewer”
• Aims:
• Deconstruct and reconstruct meaning, point to its
embeddedness in value systems, norms, and systems of
oppression
• Give voices to the silenced, giving agency to the noncounted
• Offer insights into “social impact” from a non-economybased perspective
And to thereby produce positive change.

37. Session aims

Translate ideas into data and methodologies
Look at several research designs
Translate ideas and research designs into a project
proposal

38. Where to start? Intro to an Idea-based approach

Before setting out to write a project
proposal, identify
• the idea itself;
• what it can produce (knowledge, practice,
structural change, ..);
• a broad estimate of project resources;
• a preliminary sense of project timeline.
Writing a project proposal, in turn, means:
• the essence of your idea is integrated into
its intellectual, academic, and/or applied
contexts;
• the idea is made more specific and palpable
in the outlined description of methods and
proposed activities;
• the idea is revisited from the vantage
point of resources and needs.
What does the Gdoc say?

39.

Sketching Your Idea:
Practice-based, Knowledge-based,
and Output-based sketches
What kind of a project is this?
(In terms of size, complexity, s

40. Practice-Oriented (Impact-Driven) Alternatives

Practice-Oriented (ImpactDriven) Alternatives
Subject?
History – Language?
Guidelines for study of
historical personae
and events?
Training program for
teachers and archivists?
What is the added value of brainstorming?

41. Practice-based idea brainstorming

Practicebased
idea
brainstor
ming

42. Knowledge-Oriented (Research-Driven) Alternatives

Knowledge-Oriented (ResearchDriven) Alternatives
What is the added value of group discus
Expanding teaching approaches!
Chronological vs thematic
Attempt at an integrative approach
Guidelines for other teachers

43. Research-based idea brainstorming

• What do we want to
understand/achieve, and
why? (Problem/Challenge)
Researchbased idea
brainstorming
• To understand this, what
do we absolutely need to
find out? (Goal/Objective)
• What do we need to do to
find out that one crucial
thing? (Methods)
• Is this feasible within
the given time frame and
resources?

44. Thinking in Terms of Outputs

Book deterioration prevention
development of a center specialized in
thoughtful, high-quality preservation se
Output: thinking factory.
Outcome (effect of the output): book pre

45. Imagination, Outlook, Capability: ideally, we start from what we know

Imagination
,
Outlook,
Capability:
ideally, we
start from
what we
know

46. Status Update/s

47. Sketch  Research project proposal

Sketch Research project
proposal
identify your
idea in terms
of descriptors
a brief
description of
the key
elements of the
project
(starting point
and major
aspect), and an
emphasis of a
particular
aspect of the
project
(research,
organization,
scale,
relevance,
etc.).
idea must be
re-written to
match the
topical
interest and
secondary goals
of the class.

48. From Idea to Proposal

49. Proposals

The core idea: WHAT AND HOW, WHY, WHO BENEFITS
Project description/ purpose / summary
Example:
teaching history through a series of readings
both forward and backward in time to arrive at
a balanced historical interpretation
a contribution to the practice of teaching
history
students develop a fuller understanding of the
value of source material along with a
heightened ability to sift for veritas (truth)
history, language, economics, and social
science can all be pursued simultaneously
local historical societies and local

50. What to tell them?

• Basic structure of a
Project proposal (for most
funders):
• Core idea: What you are
proposing, why, with what
benefits to practice or
knowledge?
• Context of your idea: What
is its relationship to the
work of others (literature
and other reviews)?
• Methods: How will you know?
• Activities: What will you
do?
• Products: What will be the
immediate outputs and
longer-term outcomes?

51. Context

State of the Art / Context / Literature
Review
Describe how things have evolved
historically and/or how things have been
practiced in the past.
Cite work in each of these arenas, show how
our work is related, and describe how our
work builds on the work of others and
represents a rather new and promising point
of departure.
Are we extending the work of others, taking
a new tack, rechecking findings, reaching
toward a new synthesis, operating at a
different scale, or working with different
populations?

52.

Context

53. Idea

54. Methods

What is our Corpus (data)?
Research
How will we access the data, where is it?
How will we collect the data? How much will we
collect and why this much?
How will we organize the data?
How will we process the data?
How will we arrive to conclusions?
Method development
How will we conduct our experiments? How will
tour activities be organized?

55.

56. RQs (mixed methods)

57. Data corpora

Qualitative

58.

Mixed

59. Proposals and criteria

60. Example of topical questions

• How many students can usefully take an
archive-based course at any one time?
• What opportunities exist for individual
and joint student archival research,
and what are the costs and benefits of
each?
• Which inquiry topics lend themselves to
the archival approach, and which do
not?
• How much time is required (average and
range) to reach stable historical
findings for different types of inquiry
topics?

61. Results and Deliverables: Outcomes and Outputs

What will we
produce?
Whom will it
benefit?
Why is this
important?
What about this
project
can/should be
evaluated?

62. Work Plan / Implementation

Timeline of activities
-per project
-per work package
-per phase
(whichever model suits our project best)
Example:
1 Presentation of and training in basic
archival research techniques
2 Handling of archival materials and archival
protocols
3 Development of archival research assignments
and their variations as individual or group
projects
4 Uses of classroom discussion