Dr. Umar Toseeb Hello, welcome to the Papers Podcast series for the Association for Child and Adolescent Mental Health, or ACAMH for short. I’m Umar Toseeb, Professor of Psychology. My research focuses on special educational needs and mental health in childhood and adolescence. In this series, we speak to authors of papers published in one of ACAMH’s three journals. These are the Journal of Child Psychology and Psychiatry, commonly known as JCPP, the Child and Adolescent Mental Health, known as CAMH, and JCPP Advances. All listeners to this and indeed, any of ACAMH’s podcasts, are eligible for a free CPD certificate. Do please visit acamhlearn.org for details of this, together with information on how you can access hundreds of hours of free talks, lectures, interviews, all of which you can also get free CPD certificates for. The web address is acamhlearn.org. That’s a-c-a-m-h-l-e-a-r-n.org. If you’re a fan of our Papers Podcast series, please subscribe on your preferred streaming platform, let us know how we did, with the rating or review, and do share with your friends and colleagues. Today, I’ll be speaking to Dr. Franjo Ivankovic, Lead Author of the paper “Optimisation of Self or Parent-reported Psychiatric Phenotypes in Longitudinal Studies” published in the JCPP. Franjo, thank you so much for joining me. Dr. Franjo Ivankovic Thank you for having me. Dr. Umar Toseeb So, let’s start with some introductions. Do you just want to introduce yourself, what you do and what your research interests are and where you work? Dr. Franjo Ivankovic My name is Franjo Ivankovic. I am currently a Postdoctoral Research Fellow at Mass General Hospital/Broad Institute/Harvard Medical School. And my primary research interests are really genetic and phenotypic variability in psychiatric disorders, and I’m fairly disorder agnostic, but I think I would say my research tends to focus more on obsessive compulsive and related disorders, so OCD, Tourette Syndrome, and then eating disorders, as well. Dr. Umar Toseeb When you say ‘disorder agnostic’, is that you’re not really fussed about which ones you look at or you’re interested in mechanisms that underpin lots of different disorders? Dr. Franjo Ivankovic Well, if I bring up construct validity, I think we might rearrange a territory that’s entirely of the primary topic for this podcast, but I, kind of, view psychiatric disorders that, let’s say, not super well defined. So, I don’t tend to hyperfocus on a particular disorder. I tend to, kind of, see them more as a, let’s say, multidimensional spectrum, and I’ll go to the datasets that can answer interesting questions, essentially, regardless of what kind of psychiatric disorders have been phenotyped in that dataset. Dr. Umar Toseeb Give us some context. So, the paper that we’re going to talk about today, what’s the problem that you’re trying to address in this paper? Dr. Franjo Ivankovic Yeah, so, this actually wasn’t really an intentional project from the beginning. I got my PhD in genetics and genomic, and so, my training has been fairly squarely in the genetics and genomic aspect of psychiatric disorders, but my mentor at the University of Florida is a Psychiatrist and we were essentially collecting data from our ABCD study and defining phenotype which of interest for us was OCD at the time. We saw that there was a general over-representation of OCD among adolescents in this study, particularly if you consider the fact that the sampling for this study was meant to be representative of different demographic characteristics of United States, but the recruitment was in schools and it wasn’t targeted for psychiatric symptoms, right? So, at the time, I believe – at the release of the data that they had, I believe, the prevalence of OCD was 13.5%, and that’s, kind of, high for a bunch of ten year olds that are not sampled to have psychiatric symptoms, much less OCD. And so, this, kind of, brought up a discussion of how to do a proper quality control on the phenotypes, and as it happens, concatenately, another member of the lab, Luis Sordo Vieira, was working similar approach in brain health registry data, and that dataset is a little bit different from ABCD dataset. It is mostly self-report, however, there’s a subset of BHR that has been clinically validated. So, essentially, what they looked at in their paper was what is the best way leveraging longitudinal assessments in BHR to, essentially, QC the psychiatric phenotypes? And so, what they found is that if they take any endorsement of a disorder as a threshold versus endorsements at most timepoints at which they were examined, or temporarily, kind of, consistent ones, what they found is that most and temporarily consistent ones are – and all, essentially, have about the same rate of validation in the subset that was clinically examined, as well. And then, we essentially decided to do similar thing in the ABCD dataset in this mostly or entirely paediatric sample, and that was the project. Dr. Umar Toseeb So, when you said ‘temporarily consistent’, so, my understanding of what you’re saying is, if you ask young people a number of times about their symptoms of OCD in this example, like every year, for example, and if you find a threshold and see how many people are above a threshold, and if a child is above the threshold at any one of the end timepoints, three timepoints, let’s say, that would give you the number of people who meet the threshold at any one time point. But the people who meet that threshold is much higher than what you would expect in the general population. So, what you’re suggesting is, what this other study did was they looked at multiple timepoints and the people who consistently scored above a threshold, and in doing that, that gave them some indication of what the prevalence was of whatever condition you’re interested in, in the study? Dr. Franjo Ivankovic Yeah, correct. So, I’m not too familiar with the way that the BHR works, but I believe it was also a self-report of having different disorders. So, I don’t think it’s based on, like, questionnaires. I know for a fact that in ABCD data, it’s based on a computerised form of the Kiddie Schedule for Affective Disorders and Schizophrenia, and there, of course, what you have is the K-SADS, which is normally a semi-structured interview administered – meant to be administered by Clinicians, that had been essentially, kind of, turned into a questionnaire form and then computerised and essentially given on a tablet to parents/children. Modules were given differently. So, some disorders were assessed in both parents and children, some were assessed specifically in parents, and some were assessed specifically in children. OCD, for example, was only parent-assessed, and they would essentially, go through the questionnaire on the tablet and the Research Assistants at the ABCD study would, kind of, do the calculations on the backend and determine what the diagnosis class is or diagnosis value is for a given timepoint. And these are assessed, essentially, cross-sectionally at – or well, longitudinal over a period of time, but they’re assessed at each timepoint. They don’t take into account, like, values of the previous assessment. Dr. Umar Toseeb Okay, and in the introduction of your paper, you talk about the ‘low agreement’ between parent report, child report and Clinician report. Why might that be? Why do the different informants report different levels of mental health difficulties if the target child is the same? Dr. Franjo Ivankovic Yeah, that’s a great question. I think in terms of parents and children, right, there’s certainly an aspect of understanding what constitutes a functionally impairing level of symptom experience. I know certainly – well, a lot of us are familiar, certainly, with the fact that people colloquially tend to use certain psychiatric disorders and/symptoms to describe their everyday that are not necessarily right at that threshold of functional impairment. For example, people tend to say all the time that they have OCD because they like things organised, which is not necessarily the case, and so, that can certainly be a source of over-endorsement. And then, of course, I feel like parents can have maybe a slightly distorted perception of what their children are experiencing, and they are not always trained Clinicians, and they’re not always necessarily trained to detect symptoms, right, as they are. So, they might misinterpret certain things as well, and I think that’s, kind of, where you get over-endorsement or under-endorsement, depending on, like, if the child is experiencing internalising or externalising type of symptoms. Dr. Umar Toseeb So then – so, you’ve described the problem, which is people tend to over-endorse symptoms of mental health difficulties if they’re reporting about themselves and for various reasons, which – and parents might not also be able to get to the objective level of symptoms. And that’s problematic because it leads to an inflation of the prevalence of apparent mental health difficulties in large population-based studies, like the ABCD. So, what did you want to do in this study? What were your research questions? Dr. Franjo Ivankovic Yeah, so, our research question was primarily, how can we take this dataset, that is a great dataset, that has so many different aspects of these children’s development assessed longitudinally, and how can we, essentially, overcome this quality control issue in terms of psychiatric phenotype determination so that it is useful for certain downstream analyses where sensitivity and specificity specifically are of paramount importance and can impact them significantly? Dr. Umar Toseeb Excellent, thank you, and you’ve already mentioned this a bit. So, let’s just – hopefully, you can describe it in a bit more detail. You’ve already talked about the ABCD study. Can you just tell us a bit more about what the study is, what its aims were, what the sample size is like, who was recruited and what the follow-ups are like? Dr. Franjo Ivankovic So, Adolescent Brain Cognitive Development Study, (ABCD) study, is a American NIH initiative where they wanted to track longitudinal development of children. And so, the study targets subjects or participants when they are just about to, essentially, enter adolescents, about the time when they’re nine to ten years old, and then tracks them for the period of ten years, and essentially, they exit when they’re about 19/20 years old. And I believe the study was primarily meant to assess risks of development of substance use disorders. However, their approach in assessment is very broad. So, they’re assessing, you know, psychiatric disorders through K-SADS. They assess that every year on an annual basis, and in the early arms of the study, most of the assessments are given to parents, some are given to children, but as the study progressed, and of course, there’s extensive neuroimaging that’s also involved in the study, so, MRI. They’re genetically assessed as well. So, at the beginning of the study, they collect their blood/saliva and they essentially genotype this group of people. They administer a lot of quantitative type of scales to children, such as, for example, a Child Behaviour Checklist (CBCL), which assesses also a – which assess also psychiatric symptomatology. They have pretty extensive demographic assessments, so, you know, socioeconomic status, racial background, ethnic background. They assess sex, gender, sexual orientation. So, it’s – they’re fairly thorough in terms of capturing different aspects of these kids’ lives, and they do this, essentially – I believe – it depends on the specific data modality, of course, but psychiatric phenotypes. Some batteries are also administered, like, at the half year point, some are administered every year, and with psychiatric phenotypes specifically, for example, they don’t get the entire breadth of K-SADS, all the time. For example, OCD is assessed once every two years, psychosis is assessed every year. They do assess, like, drug and alcohol use disorder. These are assessed, I believe, every other year in parents, and then now, they’re assessing them every year in the children, right? So, it varies a lot how they’re administering these assessments, but it’s wide breadth of data. There’s, like, about, I think, like, few hundred to couple of thousand, like, different modalities that you can look at. Dr. Umar Toseeb Yeah, it seems very extensive. So, you’ve mentioned Kiddie-SADS, so K-SADS. So, is that the only measure that you used? And just give us an indication of which mental health conditions we’re going to talk about in terms of your analysis. Dr. Franjo Ivankovic So, the ABCD study assessed psychiatric disorders through the computerised form of K-SADS, and they’ve essentially, assessed pretty much the entire K-SADS battery. So, that includes, you know, mood disorders, like depressive, bipolar disorder, disruptive mood regulation disorders, psychosis, and anxiety disorders like panic disorders, separation anxiety disorder, social phobias, specific phobias, some general anxiety disorder, OCD, eating disorders, ADHD, the conduct disorders like oppositional defiant disorder and conduct disorder, tic disorders, and autism spectrum disorders, which are neurodevelopmental disorders, alcohol use disorder, drug use disorder, PTSD, sleep problems, and then also, suicidality and homicidality. The ones that they don’t administer were enuresis and encopresis. I believe that’s probably because they’re passed the developmental period where they would experience these, and then also selective mutism, and I think that about covers all of the K-SADS modules. I’m not entirely sure. Dr. Umar Toseeb Very comprehensive. Yeah, we’ve got the ABCD study, it’s a longitudinal study and there’s been lots of measures. What did you find? Dr. Franjo Ivankovic Interestingly, at the time of the assessment, this was a release version four when I did these analyses. They had pulled several modules back because they have noticed similar issue where there was over-endorsement of psychiatric phenotypes. So, they had pulled a depression module, a disruptive mood regulation disorder module, agoraphobia module, eating disorders module, and then ADHD and autism modules. Essentially, what they did is that they had adjusted the programming on how the thresholds were calculated in order to improve the classification, right, of individuals. So, those were not analysed in this paper because they were essentially removed from the study to be reprocessed, and so, our question was, “Are the rest of the disorders following similar trend and should they also be reconsidered in terms of programming errors?” And so, when we did what we’re calling in the paper ‘broad disorder classification’, that is, if they have – so, if an individual who has been assessed, let’s say, three times for a disorder, if they endorse having this disorder at any timepoint, so, you know, at the entry, follow-up one or two, they would be classified as having this disorder. And then, when we did that, what we found is that, essentially, across the board, the rates of endorsements were higher than what you would anticipate in children in this specific population that was considered in ABCD, based on the literature report. So, what we did is we tried to pull, essentially, as best as possible, data from the reference materials to assess the prevalences and compare that to what the prevalences were in ABCD study. And what we found is that, essentially, across the board, except for, like, a couple of disorders, like panic disorder and bulimia nervosa, I believe, the rates were higher, and in some cases, you know, like, ten times higher than what you would actually anticipate in kids of this age. That, kind of, led us to think about how we correct this, and so, one way that we’ve come up with is, essentially, setting a narrow diagnosis construct. And the narrow diagnosis construct is predicated on the assumption that you need to have endorsed this disorder over 50%, not at least 50%. So, if you were assessed twice for a disorder, for example for OCD, they have only taken it at two timepoints, they would have to endorse OCD at both timepoints in order to be classified as having the disorder. Dr. Umar Toseeb Did you see any evidence that the extent to which people were over-endorsing was for the conditions that there’s high levels of public awareness of the symptoms of those conditions? Dr. Franjo Ivankovic That’s a great question. So, not in particular. You know, I think the disorders that were most over-endorsed, and I mean, in a relative way, were psychosis and bipolar disorder, and I would not anticipate people to be more familiar with bipolar disorder than they are with, let’s say, ADHD, I mean, in general public, ADHD or autism or OCD, for example. OCD was still over-endorsed, about – I think about – the rates of OCD reported compared to what we considered the reference rate, which was 2.5%, was about five to six times, or I think BD1 was, kind of, almost ten times more prevalent than the reference rate. But then, if you think, like, specific phobias, for example, which are also fairly well-known in terms of symptomatology, right, was maybe, like, 1.8 times the reference prevalence rate, so nothing extreme or, you know, social phobia was actually about well-matched to the reference prevalence rate. Tic disorders, which I guess, at the time were not as familiar, ‘cause I think in about 2020, we had that swathe of TikTok tics, where the Tourette syndrome, kind of, became fairly well known among the general public, but at the time was not, it was also fairly over-endorsed. It doesn’t seem like there is a awareness-driven association, I guess. I would imagine if that’s – I mean, I haven’t asked this specific question, but I would imagine that the drive for the over-endorsement in a lot of cases was maybe over-estimation of some externalising symptoms, let’s say. Dr. Umar Toseeb So, your working hypothesis is that it’s not that people are necessarily more aware of certain symptoms and they’re reporting those more, it’s the symptoms that people are aware of, they’re interpreting the severity of those more? Dr. Franjo Ivankovic Yeah, exactly. Dr. Umar Toseeb Okay, excellent. What are the implications of this for other Researchers who use other studies, like myself? So, like, we use population-based studies all the time where we have self-report measures of common mental health conditions, like depression and anxiety, and then, from that we infer certain things. Is this likely to be a problem across the board on this kind of cohort study population level investigations, and if so, is the recommendation that we use a similar approach to what you’re suggesting, which is if you’ve got multiple timepoints, see what’s been endorsed at multiple timepoints? Dr. Franjo Ivankovic I do think that studies that rely on – and in fact, I will actually say, even in electronic health records data – I guess, maybe a little bit of background there. So, electronic health records data, right, are data that are fetched from – in different databanks, where people contribute their medical history in terms of codes that are associated with specific disorders. And what we see is that even in those data, which are essentially determined primarily by Physicians, right, they’re the ones assessing diagnoses, you also see, not as extreme, but some rate of over-endorsement of psychiatric disorders. So, it is certainly a pervasive challenge with respect to psychiatric disorders data, and I do think that it definitely indicates that some degree of quality control of the phenotypes is absolutely necessary. However, of course, a lot of data collections are cross-sectional. That is, they’re assessed at a single timepoint, and the issue there, right, is of course, we lack the longitudinal data, right? So, we don’t know how to best QC that, but with respect to longitudinal data, at least, I think, one way to look at the quality of phenotypes would be to consider the longitudinal trajectories of endorsements and see if your particular – in your particular study, the longitudinal endorsements are consistent or if they vary a lot. And I guess, if they do vary a lot or they’re inconsistent, then I’d be worried a little bit, and there’s various reasons, right, why longitudinal assessments would be inconsistent. I mean, obviously, yeah, a person could have a false endorsement of a psychiatric disorder or false positive, but you could also have a false negative, right? Like, recall bias is certainly a thing and people can, you know, forget that they had experienced a specific disorder, especially if they’re in remission or are no longer experiencing symptoms and are far away from that period of life where they are bothered by the symptoms that they were experiencing. They can miss that they had it, yeah. Dr. Umar Toseeb Why might people who work for a National Health Service or the health service or Clinicians who are not doing it for a research study, why might those estimates be biased upwards, because those are the people who are entering the data onto the administrative datasets? Dr. Franjo Ivankovic That’s another great question. In the – I’m not very familiar with British healthcare systems, so I can’t really speak to UK, but in the US, certainly, the way that healthcare system works, there’s fairly limited duration in the clinic in terms of contact between a Physician and a patient. And putting aside, let’s say, coding errors, which can certainly occur, right? So, when you have a diagnosis in mind and you go enter the ICD code and you start typing something, for example, let’s use the case of anorexia, medical anorexia, which is just a absence of appetite, and then anorexia nervosa, which is a psychologically-driven thinness and absence of appetite, right? So, let’s say you go in and you start typing ‘anorexia’, and then you accidentally might choose anorexia nervosa instead of medical anorexia, which can occur. But I think, the limited duration of contact can lead to, you know, maybe not as thorough assessments, which people on the research side have the privilege of, right, essentially administering multiple batteries and multiple tests and different, kind of, assessment tools to triangulate the diagnosis. In a medical setting, you might get, you know, PHQ or Patient Health Questionnaire, which will assess for, you know, depressive symptoms and you can get some qualitative interview, which would lead a Physician to establish an MDD diagnosis or something like that. But in reality, patient could have a different mood disorder, to which depression is just an aspect of the diagnosis, but there’s not enough time to, essentially, assess that, or they haven’t spent enough time to reach the relevant questions that could lead them to assess a more appropriate diagnosis, you know, such as, maybe, bipolar or whatever. Dr. Umar Toseeb It’s really interesting because I think that in the UK at the moment, there’s – more and more large administrative datasets are being linked and being made available for Researchers to use for research. Dr. Franjo Ivankovic Hmmm hmm. Dr. Umar Toseeb And, you know, everybody is aware that there’s no such thing as a perfect dataset. So, it’s interesting to have this conversation around some of the limitations of those – potential limitations of those administrative datasets, ‘cause I think the challenges that you described for Physicians in the States, I imagine it’s a similar – some of those are similar to the UK and other parts of the world, especially those ones around time limited or limited time during a consultation. Let’s move on to the future. So, I know you said your training is behaviour genetics and the genomics, and what – on this topic around measurement of psychiatric phenotypes, do you have more work planned, or in general, more broadly, what have you got coming up? Dr. Franjo Ivankovic Yeah, so, I think – well, nursology has become one of my passions through this specific project, but also assessment of psychiatric phenotypes and phenotype-related questions. So, they have become an integral part of my genetic studies, as well, and I think they will, kind of, thread through in that aspect, certainly, for a fairly long time. Thinking off top of my head, I don’t have specific projects that are of the same nature as this one queued up yet, but if I – I will probably run into a dataset that will have similar type of issues and are – is going to present similar type of challenge and – yeah. Dr. Umar Toseeb And what’s your take-home message for the listeners? Dr. Franjo Ivankovic For the Researcher listeners, I would certainly say it’s important not to take phenotypes at face values. Even when we think of these batteries, as well, or specific questionnaires or batteries, right, that are well established and have been used for a very long time, they are not perfect. And, you know, I would caution against over-relying on these assessments, because again, a timepoint to timepoint variability can be present. And especially for the Researchers who also do genetics or neuroimaging studies where slight changes in the specificity and sensitivity of your phenotyping can, kind of, lead to cascading effects down into your, you know, neuroimaging or genetic study, taking phenotypes at a face value can have drastic and negative effect that cannot be really picked up unless you ask specifically if your phenotypes are what they’re supposed to be. Dr. Umar Toseeb Thank you, that’s fascinating and really, really interesting conversation there around phenotypes – psychiatric phenotype measurement. Thank you so much. For more details on the paper, please visit the ACAMH website, that’s www.acamh.org, and Twitter @ACAMH. ACAMH is spelt A-C-A-M-H, and don’t forget to follow us on your preferred streaming platform, let us know if you enjoy the podcast, with a rating or review, and do share with your friends and colleagues.