Peer Review

Thanks for posting in the forums, it seems I might have skipped this episode cause I didn’t remember hearing it. Reading your comment really got me to engage with the episode in a different way. There’s a lot in which I agree and disagree with you. But your reply was so extensive and well thought out that I thought It merited being acknowledged and commented on. For reference I’m a 4th year Medical Student, I’m one test away from an MD.First of all, I do agree with you that a researcher’s job is primarily to contribute to a body of science that has some very complex language on the basis of need. What I interpreted from Josue/Kelli in the podcast is that in an ideal world Researchers would also take on the role of educators for the general public, especially because there is a lot of misinformation, from both people outside of the professional field as well as reporters who oversimplify study findings. (i.e. X research shows that protein x is expressed in high amounts in people who have cancer and people who like kittens. Headline: Research shows Kitten cause cancer). In fields like mental health and medicine specifically it’s a lot more important because research implies possible life saving applications and clinicians are taught early on that public education is as much their responsibility as research (From an epidemiologic standpoint). I think it’s an ethical distinction to be made between different fields of study.I agree that the more specialized your field of study becomes the more jargon you have to master. A radiologist has to constantly talk in terms of T1 or T2 weighted MRI Scans. (Something that probably sounds like gibberish to most people) and when applied to research that Jargon is necessary to actually simplify the language of research. Something I worry about (I don’t know if this was what Kelli was thinking about) is that In fields with clinical relevance, a lot of researchers actually double as physicians and they carry that jargon with them as they talk to their patients. In a clinical setting this can be very problematic, and it is in part because institutions do not stress much on educating clinicians on how to distill complex ideas into easily digestible ideas for patients.
As you mentioned, common reasons to reject a paper are:
a) It’s not actually new.
b) It needs more experiments.
c) The experiments are not convincing.

But I think it lacks one of the most vital reasons why some papers are rejected. Because it does not fit well with the journal’s views, perspectives or goals. I could write an article on how Videogame assisted therapy helps kids work through trauma but I would likely be rejected by a lot of publications because they do not view videogames as a profitable factor that would contribute to the journal’s set goals. Perhaps the board members find such research childish or inappropriate. Much like the legendary supervisor that inspired Josué to start Geek Therapy denied the use of videogames as a suitable therapeutic alternative. Similarly many papers on religiously sensitive forms of therapy are rejected on the grounds of the journal being preferably secularized from religious ideology. I think this might also be something where robotics and medicine/psychology diverge because sometimes institutional review boards can have. A lot of political/philosophical biases that are not accounted for. For years transgender medicine was not a researchable subject because review boards did not deem it a respectable subject.

We are taught that one of the most important parts about publishing research is to find a journal where it fits in. Something as fringe as videogame use in therapy is unlikely to find many places where they can be published outside of Videogame centric journals only read by professionals actually interested in that type of research, creating a bubble that doesn’t really extend the conversation throughout the field.

In truth I’m with you 100% on the value of peer reviewed research. Not too long ago I think it was Elon Musk that suggested there be a website where articles could be peer reviewed by the internet. And that idea chilled me to the bone. If you spend one day in Twitter you will find Thousands of people who agree on rather effed up or just blatantly incorrect things and that continue to validate it. For doctors, the anti-Vaxxer movement is a great example. It has been validated by sooooo many people, none of which went through a successfully peer reviewed process.

I think public education is important and I also believe there are instances where peer reviewed science is not appropiate. In Medicine we have Case Studies. These aren’t peer reviewed but they can be REALLY REALLY REALLY useful and they are glorified versions of some things you might see in youtube videos. I think one of the most important things in this discussion is to define validity. Peer-reviewed research is valid in the sense that it is reproducible and standardizeable. Take for example research that finds a medication that helps treat depression. If the peer reviewed research is successful, I can feel safe as a physician to prescribe this medication to my patients because I know when and under which circumstances the studies have shown that such an intervention would result in betterment of depressive symptoms. On the other hand sometimes there are experiences that cannot be easily standardized and vary from a case to case basis. If playing Journey with a kid with social issues helped him better socialize with peers that doesn’t mean that I can consistently prescribe Journey to every kid that presents with social issues. In part I have to be culturally sensitive to the background of the kid and determine subjectively if that sort of intervention could be repeated. In a institutional review board you would likely be rejected because your results are not repeatable or standardize. However that does not mean that publishing that case study would not be a benefit to the clinicians that learn choose to study that experience. I believe this is the sense in which both formats of publishing research are both extremely valid and necessary. Making the clear distinction that peer review is needed to make something reproducible. A youtube like or agree does not count as a peer review because the distinction is that I can like a youtube video because it agrees with my views even if it is not based on evidence. A peer review implies that the professional will return your paper and tell you exactly why your research could be flawed taking into consideration the current knowledge base of the fields. It implies that the review board is always looking at your paper critically and will not readily accept it without a thorough vetting. Because again, peer review means standardizeable and reproducible and so the consequences are likely large scale. Another thing they addressed was how the only way to peer review something is for it to be written in journal format. Perhaps people who dominate a video or spoken format over a written format would have done research with sound science and procedures that would otherwise find it’s way to a journal if the expression of their ideas was done is a written medium. I understand that procedure is important for standardizability, but it is none the less restrictive of possible valid contributions.

Again thanks for posting and looking forward to how this topic evolves in discussion.

Hi @sterlingm! Thanks for taking the time to respond. Posting here is a great way to keep the discussion going! You can also find ways to reach us at headshotspodcast.com.

I think @Gianminni covered some of what I wanted to say about how psychology/behavior peer review is the world I’m most familiar with and since I’m also an engineer, I can understand some of the differences in your peer review experience.

What I was trying to discuss the idea of peer review and how it could be applied in the world of games. I was thinking: Who are the scholars? And why do we only listen to them? In games in particular, does the opinion of the people who experience the games the most not matter? And who decides that? Can a million likes and 500 million views elevate an idea or theory to the level of some person who writes an article that 30 people have read?

I’m using the term peer review liberally and literally. What is the value of having a peer… review your work (or theory/idea) and who are your peers?

I agree that formal and informal platforms can co-exist. The truth is that they do, simultaneously, but I don’t think there is enough collaboration. If “formal” research is siloed and “informal” research is seen by the masses, what effect does that have on our fields? In psychology, we often complain about how research is reported. It’s infuriating how sometimes a research article is completely misreported and then, most people will never read the journal article or white paper or whatever is available.

I guess the last thought I had was that I understand your point about the experts talking to the experts, I’m not saying that’s right or wrong, I’m saying that a lot of people are hungry for information and in some cases could benefit from it, if it were more accessible. My other point is that in some fields (like psychology, behavior, mental health), I think there is value in allowing a diversity of voices, especially those who may not have earned the “right” to join the discussion and yet have insights we’ve never considered.

Please feel free to keep this conversation going, react to more episodes, or start a new thread @sterlingm!

• Josué