Substance Use Disorder Journal Club

1. The study may be underpowered -that is, a difference may have been missed due to small sample size and the loss to follow up.

If you want to consider the gorey details, see what a power calculation on sealed envelope.com suggests:

Assume we want 80% power to detect a difference (ie, 20% chance of a false negative). Also assume (based on other ADHD trials) a 30% symptom reduction will occur with medication (=12pts on the ASRS in this study). We might consider a 40% symptom reduction (=17, d=5) worth notice. To detect that kind of a difference between treatments you would need 100 people (with 100% fup ).

If we were ok with finding a 47% symptom reduction from one treatment, compared to a 30% reduction in the other, a sample size of 40 would be sufficient. (This sample size was 51 with 68% fup.)

The authors do not describe their power calculation, which is not typical of published work.

1. We don’t know what happened to the data for those who did not complete the study, and this can affect findings (either missing a relationship, or discovering one that is not there). Understanding how the missing data was dealt with can help us draw more valid conclusions about whether the loss to follow up affected the results. In addition, doing a sensitivity analysis can be helpful – ie, reporting what happens when the missing data are treated in a variety of ways such as with missing data entered as no change, as the worst outcome, and/or using last data point available).
2. Patients, and maybe their treating clinicians, were not blinded to the treatment received. It is hard to know what the impact of this is given (my) lack of familiarity with this population, but it is curious that the investigators did not blind subjects. Lack of blinding can affect outcomes that are largely subjective – in what way might depend on what this population tends to believe about the two therapies..ie, if pts believed that atx was a better treatment than stimulants they might answer questions about improvement favorably. This would affect even a blinded rater’s assessment, and could overestimate the effect of atx. (and same could be true for MPH).
3. This is a short term study for a therapy that will be long term.   This study does not provide evidence about potential long term benefits (functional outcomes like employment, relationships) or harms (ie, substance misuse, tolerance, cardiovascular outcomes).

4. Finally, let’s consider external validity – how we apply these findings to populations other than the one studied. This was a study done in a university setting in Korea, with a different care system and culture. The patients were relatively young, largely male, and carried a diagnosis of MDD on SSRIs. Importantly, people with substance use disorders were excluded (but those termed ‘drinkers’, or prescribed benzodiazepines, were included).  How would these differences likely impact the effect (and safety) of a stimulant versus non stimulant? If one of the therapies is affecting ADHD sxs by treating depression, than those without depression may not reap the same benefits. And, given the high likability of stimulants, people with substance use disorders may have higher risk of misuse in the long term (though that outcome is not assessed in this evaluation). 

It would be valuable to conduct a head to head trial of stimulants versus not stimulants in a larger population without depression, in a double blind fashion and over a longer period, in order to overcome the limitations described.  In the absence of that, this study suggests that MPH may not be superior to ATX in this population. And if it is superior (and we don’t see that because of lack of power), the margin of superiority does not appear to be large.