Students ask: How do you read your own work and criticize it? How can I read like an examiner?
People usually read research papers because they have some interest in the topic, and because they want to know about other relevant work that’s been done. Academics are taught to read critically – ideally, the more we read, assess, and compare research literature, the more like examiners we become. As readers, we’re asking: Is this interesting? Am I convinced by the evidence and claims? What are the strengths and weakness of the reported research?
Start by considering how you read someone else’s paper. What are the questions you ask? Don’t just absorb the content; consider how convincing the material is, and why. Consider the writing as well as the content: what annoys or appeals to you in their writing? What signals a strong or a weak paper to you as a reader? (See the table at the end of the post for examples of the sorts of indicators that examiners pick up on when reading a dissertation.)
What can you do to reach a critical view of your own writing? Here are some practices to help you improve your critical reading and consider your own writing from different perspectives.
Write early enough to take a break between writing and reading – and then return with ‘fresh eyes’. It can be tough to change roles properly, to set aside the Writer role, with all the associated aspirations, emotional investment, and background knowledge, in order to take on the Reader role and see just what’s on the page.
Check the flow and accessibility of your argument. One trick is to use a highlighter: mark the key sentence for each paragraph. Is there one? Is there only one? Where is it – is it buried in the paragraph? If you only read the highlighted sentences, how clear is the argument? (If you give only the highlighted sentences to someone else to read, do they feel they have a good overview of the argument?). Another trick is to diagram your argument – or have someone else do so. Is there are clear flow, or are there too many branches? Are there elements that are left dangling? Is the sequence clear and logical?
Read out loud (and listen while you read). It will slow you down and force you to parse the text word-by-word. Writing that is too dense or convoluted tends to be difficult to read aloud. Repetition tends to ‘jump out’. You’re likely to spot deficiencies in phrasing or punctuation. Etc.
Read with a specific purpose. Read for specific qualities, issues or flaws, such as assessing whether the methods are expressed in a clear sequence, spotting unsubstantiated assertions, considering whether the use of literature is accurate and distinguishes clearly between what others wrote and how you interpret it, detecting language glitches you often make … and so on.
Find some ‘critical friends’, people who are good readers who will be honest about what they see. Most people will read something for you, if they can count on you returning the favour. Reading and critiquing other students’ writing – including dissertations – is a great way to gain insight into what works or doesn’t, and why – so everyone benefits. And it’s usually easier to read someone else’s work than your own. Have a conversation between reader and writer. What does the reader understand as the important elements of the research? Can the reader replay the key ideas or argument? (You can ask the reader to diagram the argument.) Was anything unclear or problematic? What would help the reader read and understand the work more easily? Were the conclusions justified by the evidence? Remember that you can ask your critical friend to read for specific things (e.g., sanity check, structure check, clarity of language, description of methods and execution, use of statistics, anything that doesn’t contribute to the argument and should be deleted, etc.).
I never publish material without having it read by someone I trust. More often than not, a good critical reader will spot precisely where I struggled with the writing – and be able to suggest an improvement. Sometimes the reader will have such a different perspective, that I’m provoked to reconsider my work in unexpected ways. Sometimes the reader will spot my ‘babies’, those precious ideas or paragraphs I’ve nurtured for so long that I can’t let go of them, and will help me delete unnecessary material. Sometimes the reader will surprise me and say “Just submit it.”
The trick is to learn how to detach from our own investment in what we write, and take on the persona of an external reader who is simply engaging with what’s on the page.
Here are some examples of the sorts of indicators that examiners pick up on when reading a dissertation. Note that dissertations may be held to a higher standard than submissions to a conference or journal (and that different venues have different standards, and different assessments of ‘rigour’ and evidence). Notice that ‘bad smells’ and ‘good smells’ are often ‘two sides of the coin’; the elements are similar, but the execution makes the difference.
| Examples of ‘bad smells’ in dissertations: | Examples of ‘good smells’ in dissertations |
| Failure to introduce and justify the research question (or problem), leaving the reader wondering ‘Why am I reading this?’. | Clear, early ‘framing’ of the research, for example, early expression of the over-riding research question (even if there is further specification and development of the research question later), what motivated the researcher to ask it, and why it’s worth asking. |
| A dense narrative (often with overly long paragraphs and sentences) that is difficult to navigate and parse. (implies: lack of focus, lack of critical reflection, lack clarity in the thesis.) | Clear structure, clear signposting, meaningful headings. |
| Lots of low-level errors (implies: lack of attention to detail – which might apply to all aspects of the research). | |
| Unsubstantiated assertions (implies: don’t understand scholarship or scientific research). | [See blog post on The power of ‘because’: “‘Because’ is the fulcrum for informed debate. It’s not enough to have a position; one must explain, substantiate, and defend it.”https://pragmaticphd.comthe-power-of-because/.] |
| Undefined terms, and terms used with different meanings in different places (implies: lack of clarity and precision in the underlying thinking). | Clear definition of key terms, usually arising in the framing of the research – and then consistent use of those terms. |
| Too many weasel words (implies: lack of precision, potential lack of critical depth, potentially weak or equivocal evidence, sloppiness in execution or interpretation). | |
| Inaccurate reporting, selective reporting (in a way that tends to misrepresent the source), or uncritical reporting (implies: failure to master the discourse, lack of critical depth, potential issues of intellectual integrity). | Clear situation and grounding in literature and theory, accurately summarising the relevant ‘state of the art’, so that the reader knows what the seminal sources are, which ideas are being used, with what interpretation, and from what perspective. A good review adds value by identifying patterns and gaps relevant to the research. |
| Undifferentiated lists of citations (implies: couldn’t be bothered to read properly, or to report accurately, or don’t really know what’s specific to each source). | Complete citations including page numbers for quotations. Distinguishing the contexts of different studies that report similar results or reached similar conclusions. |
| Failure to distinguish between ‘what the source said’ and ‘what I think’. | Accurate (even if concise) reporting of prior research that respects the source in its own terms before examining it through the lens of the current research. |
| Use of secondary references without reference to the relevant primary sources (implies: can’t be bothered to go to the source, content to let others do the work). | |
| Failure to justify research decisions, such as why a particular method was chosen and what alternatives were considered (implies: potential lack of critical thinking in the study design, or reliance on others’ choices without deep understanding). | Clear audit trail, allowing the reader to follow each step in a process (i.e., clear articulation of methods and protocols) or argument (i.e., clear and explicit line of reasoning). |
| Lack of clarity in the description of methods, e.g., reliance on invocation of a method name without a clear description of how the method has been applied (implies: application without reflection, potential lack of understanding, lack of vigilance in the research design and execution) | Clear description of methods, and justification of why they were chosen. |
| Research methods that are fit-for-purpose, e.g., appropriate, accurate application of statistics; appropriate use and presentation of qualitative analysis that is well-grounded in the data. | |
| Ill-structured, selective, or biased presentation of data (implies: potential bias). | Clear presentation of data. |
| Failure to distinguish between evidence and its interpretation – and presenting interpretation as evidence (implies: potential bias) | Clear presentation of findings, distinct from discussion and interpretation, allowing the reader to consider the evidence. |
| Overclaiming – overstating ‘contributions’, making generalisations not warranted by the evidence, failure to recognise and discuss potential bias and limitations. | Conclusions that are justified by the evidence – no exaggeration or amplification, and a good sense of context and limitations. |