This is interesting, but the conclusion sounds pretty extreme and hard to believe. Are you saying people can’t respond differently to the same substances (and/or other interventions) based on genetic differences? I heard that people with a genetic predisposition to alcoholism tend to be more energized by alcohol etc. It certainly seems to affect different people’s behavior in different ways. What about people who claim they can fall asleep after drinking coffee? Are they lying or confused? What about people who say cilantro tastes like soap? Am I just completely misinterpreting the concluding paragraphs? Because it sounds like it’s saying these things shouldn’t be possible.
Not "can't", but "in most cases will not". In probability terms of the heterogeneity distribution, the standard deviation is very narrow, i.e., most people's effects will be very close to the population average.
I don't understand this - are you claiming that the standard deviation of treatment response (so the standard deviation of the difference between, for example, numerical disease score before and after treatment) is usually very narrow? If so, that seems to be contradicted by the entire field of psychiatry - consider, for example, response to antipsychotics, where many patients will see no response at all and many will see complete remission of symptoms. Or cancer immunotherapy, as the other commenter pointed out. I don't know if your central thesis about interaction effects is necessarily wrong, but I really don't understand how you conclude from that that treatments rarely "have dramatically different effects for otherwise similar people".
Varying outcomes from treatment does not imply interactions are present. Can you link me any convincing study showing an interaction between psychiatric treatment and patient characteristics?
Considering my response to this has made me really doubt your thesis.
Some people will be started on antihypertensive drugs. Some of them will have a significant reduction in blood pressure - more than 10mmHg systolic, say. Some will have a modest reduction in blood pressure - only 3mmHg systolic. An unlucky few might even have an increase in blood pressure. I expect the change people see in their blood pressure would follow a normal distribution.
It's really hard to predict who will respond to which antihypertensive drugs. There are blood tests that have some ability to predict. For example, for people who have hyperaldosteronism (which will generally show up on a blood test), any blood pressure drug other than an antimineralocorticoid like spironolactone is basically a waste of time. So in the case of hyperaldosteronism, there is an interaction between drug response and patient characteristics.
But wait - isn't a patient's response to an antihypertensive drug itself a patient characteristic? You might object that it's not predictable. However, that's not the case. It can be predicted: just do a short (individual) trial of the antihypertensive drug, and measure the reduction in blood pressure. In the case of a patient with hyperaldosteronism who is tried on something other than an antimineralocorticoid, they'll see no change in their blood pressure. But the reason doesn't really matter, at least from a clinical perspective - what matters is just the patient's response, whether it be due to something identifiable like hyperaldosteronism or some complex mess of factors we don't understand.
Though I'm not aware of anyone studying this rigorously, I would think with near certainty that this trial would very strongly predict how well the patient would respond to this drug in the future. In other words, how well a patient has responded to a drug in the past is a patient characteristic that strongly predicts their future response to that drug. This is an interaction between patient characteristics and clinical outcomes.
I think you're right that, currently, claimed interactions between patient characteristics and clinical outcomes rarely turn out to be useful (with important exceptions). However, it seems clear to me that this is only due to poor understanding of the factors that impact clinical outcomes; these interactions do exist, but are very difficult to measure or quantify in most cases without doing a trial of the drug. But if you do a trial, then in many cases you can measure a proxy for the overall effect of the patient characteristics on response to the drug.
Is personalized medicine a waste of time, though? I'm hesitant to say that it is across the board, but I haven't seen many cases where there's really convincing evidence for it. I don't think we really understand medicine well enough for personalized medicine to be useful outside of relatively rare circumstances.
(To answer the question you posed about patient characteristics in psychiatry, the obvious example is blood levels of psych drugs, which vary significantly from patient to patient; some patients will have very low blood levels relative to other patients on the same dose of the same drug with the same route of administration. This paper, for example, discusses response to the antipsychotic clozapine: https://pubmed.ncbi.nlm.nih.gov/34374073/ )
The point is that the normal distribution of responses you see will include random noises plus the true effect of the drug. My claim is that the distribution of the true effects is very narrow, and that most of the apparent variation is due to random noise. This is why attempts to find interactions/moderators almost invariably fail because it is impossible to find interactions in most cases.
Consistency of observed patient response to the same drug in multiple trials would help make the case, yes. However, I think these are generally small because the random noise component is so large relative to signal. For placebo effects, at least, it seems consistency can be large. https://www.sciencedirect.com/science/article/abs/pii/S0022399907004461 But this is maybe not that interesting as it is kind of a measure of the impressionability of people.
This study with a real drug to treat migraines showed not that strong patient stability of treatment effects. https://journals.sagepub.com/doi/pdf/10.1111/j.1468-2982.2008.01806.x Figure 3. But it's not really how I would model the data. I would use response to time 1 treatment to predict response to time 2 treatment to gauge consistency. I don't think this value would be particularly high. In the case of perfect within-patient stability, all values would be in 3/3 success and 0/3 success. The authors talk about the within-patient stability, but I cannot really be confident based on the results they reported.
You can claim obesity/body size as a interaction for every drug dose that isn't adjusted for body size. That's not generally what people have in mind when they argue for large interaction effects. The study you linked is cool as it used ng/ml dose. However, if you read the statistics, it is hardly impressive. If anything, it suggests a hormesis response function, as the high dose failed to look better than the regular dose, while the moderately high dose was better. Again, with serious uncertainties due to the small sample sizes. The more important questions based on that study: 1) are drug responses in terms of blood levels even consistent for patients over trials? Assume doses are in ng/ml, not ng. If yes, 2) can drug responses in blood levels be predicted? If yes, then this approach will work to establish a potentially clinically useful interaction if these predictions can be made reliably.
Recall that in my post I said cancer therapy is an exception to the point because of unique mutations in each cancer necessitating personalized treatment.
It's quite intriguing how the more I learn about statistics, the more my trust in medical science seems to waver.
With that in mind, would you mind taking a look at ivmmeta.com? Ivermectin has undoubtedly been contentious during COVID. However, my primary concern is not whether ivermectin works or not, but rather the epistemological issue it raises.
I now find myself in a position where I must choose between a worldview in which the establishment can effectively insulate itself from undesired information, and a distrust in scientific research. If the establishment is correct, then most of the 95+ studies on ivmmeta.com must be fraudulent. Conversely, if the establishment is wrong, then large-scale studies like RECOVERY and ACTIV-6 must be fraudulent. Neither option is very appealing.
I tend to lean towards ivmmeta.com for reasons similar to those you discuss in this blog. (The quality of the establishment's arguments did not help.) I would greatly appreciate it if you could examine the site and share your thoughts on whether it is fraudulent or if their arguments hold merit.
Some of the studies on that site are fraudulent, but most are probably just studies that suffer from small numbers and/or failure to control for any number of relevant variables.
Just as a side note, the idea that it's a good heuristic to always disagree with the establishment's opinion is pretty silly, considering the establishment also endorses electricity, internet, toothpaste, bridges, and everything else the contrarians use. In the case of medicine it is particularly silly, because while obviously the medical establishment can get something wrong, the claim made by anti-vax people that doctors and medical scientists want to kill people or not help them should lead one to abandon modern hospitals and modern medicine altogether, and yet, you don't see much anti-vax activity targeting organ transplants, pacemakers, paracetamol, cataract surgeries, cancer treatments, antibiotics, insulin medication, kidney stone removal, baby incubators, anesthesia, and so on.
Even if personalized medicine is borderline useless, I don't think a public debate with Topol would be the greatest idea. His sheer amount of knowledge regarding medicine is enough to steamroll nearly anyone else in a conversation (at least in the eyes of the audience).
This is interesting, but the conclusion sounds pretty extreme and hard to believe. Are you saying people can’t respond differently to the same substances (and/or other interventions) based on genetic differences? I heard that people with a genetic predisposition to alcoholism tend to be more energized by alcohol etc. It certainly seems to affect different people’s behavior in different ways. What about people who claim they can fall asleep after drinking coffee? Are they lying or confused? What about people who say cilantro tastes like soap? Am I just completely misinterpreting the concluding paragraphs? Because it sounds like it’s saying these things shouldn’t be possible.
Not "can't", but "in most cases will not". In probability terms of the heterogeneity distribution, the standard deviation is very narrow, i.e., most people's effects will be very close to the population average.
I don't understand this - are you claiming that the standard deviation of treatment response (so the standard deviation of the difference between, for example, numerical disease score before and after treatment) is usually very narrow? If so, that seems to be contradicted by the entire field of psychiatry - consider, for example, response to antipsychotics, where many patients will see no response at all and many will see complete remission of symptoms. Or cancer immunotherapy, as the other commenter pointed out. I don't know if your central thesis about interaction effects is necessarily wrong, but I really don't understand how you conclude from that that treatments rarely "have dramatically different effects for otherwise similar people".
Varying outcomes from treatment does not imply interactions are present. Can you link me any convincing study showing an interaction between psychiatric treatment and patient characteristics?
Considering my response to this has made me really doubt your thesis.
Some people will be started on antihypertensive drugs. Some of them will have a significant reduction in blood pressure - more than 10mmHg systolic, say. Some will have a modest reduction in blood pressure - only 3mmHg systolic. An unlucky few might even have an increase in blood pressure. I expect the change people see in their blood pressure would follow a normal distribution.
It's really hard to predict who will respond to which antihypertensive drugs. There are blood tests that have some ability to predict. For example, for people who have hyperaldosteronism (which will generally show up on a blood test), any blood pressure drug other than an antimineralocorticoid like spironolactone is basically a waste of time. So in the case of hyperaldosteronism, there is an interaction between drug response and patient characteristics.
But wait - isn't a patient's response to an antihypertensive drug itself a patient characteristic? You might object that it's not predictable. However, that's not the case. It can be predicted: just do a short (individual) trial of the antihypertensive drug, and measure the reduction in blood pressure. In the case of a patient with hyperaldosteronism who is tried on something other than an antimineralocorticoid, they'll see no change in their blood pressure. But the reason doesn't really matter, at least from a clinical perspective - what matters is just the patient's response, whether it be due to something identifiable like hyperaldosteronism or some complex mess of factors we don't understand.
Though I'm not aware of anyone studying this rigorously, I would think with near certainty that this trial would very strongly predict how well the patient would respond to this drug in the future. In other words, how well a patient has responded to a drug in the past is a patient characteristic that strongly predicts their future response to that drug. This is an interaction between patient characteristics and clinical outcomes.
I think you're right that, currently, claimed interactions between patient characteristics and clinical outcomes rarely turn out to be useful (with important exceptions). However, it seems clear to me that this is only due to poor understanding of the factors that impact clinical outcomes; these interactions do exist, but are very difficult to measure or quantify in most cases without doing a trial of the drug. But if you do a trial, then in many cases you can measure a proxy for the overall effect of the patient characteristics on response to the drug.
Is personalized medicine a waste of time, though? I'm hesitant to say that it is across the board, but I haven't seen many cases where there's really convincing evidence for it. I don't think we really understand medicine well enough for personalized medicine to be useful outside of relatively rare circumstances.
(To answer the question you posed about patient characteristics in psychiatry, the obvious example is blood levels of psych drugs, which vary significantly from patient to patient; some patients will have very low blood levels relative to other patients on the same dose of the same drug with the same route of administration. This paper, for example, discusses response to the antipsychotic clozapine: https://pubmed.ncbi.nlm.nih.gov/34374073/ )
The point is that the normal distribution of responses you see will include random noises plus the true effect of the drug. My claim is that the distribution of the true effects is very narrow, and that most of the apparent variation is due to random noise. This is why attempts to find interactions/moderators almost invariably fail because it is impossible to find interactions in most cases.
Consistency of observed patient response to the same drug in multiple trials would help make the case, yes. However, I think these are generally small because the random noise component is so large relative to signal. For placebo effects, at least, it seems consistency can be large. https://www.sciencedirect.com/science/article/abs/pii/S0022399907004461 But this is maybe not that interesting as it is kind of a measure of the impressionability of people.
This study with a real drug to treat migraines showed not that strong patient stability of treatment effects. https://journals.sagepub.com/doi/pdf/10.1111/j.1468-2982.2008.01806.x Figure 3. But it's not really how I would model the data. I would use response to time 1 treatment to predict response to time 2 treatment to gauge consistency. I don't think this value would be particularly high. In the case of perfect within-patient stability, all values would be in 3/3 success and 0/3 success. The authors talk about the within-patient stability, but I cannot really be confident based on the results they reported.
You can claim obesity/body size as a interaction for every drug dose that isn't adjusted for body size. That's not generally what people have in mind when they argue for large interaction effects. The study you linked is cool as it used ng/ml dose. However, if you read the statistics, it is hardly impressive. If anything, it suggests a hormesis response function, as the high dose failed to look better than the regular dose, while the moderately high dose was better. Again, with serious uncertainties due to the small sample sizes. The more important questions based on that study: 1) are drug responses in terms of blood levels even consistent for patients over trials? Assume doses are in ng/ml, not ng. If yes, 2) can drug responses in blood levels be predicted? If yes, then this approach will work to establish a potentially clinically useful interaction if these predictions can be made reliably.
Immunotheraphy for cancer has a ~15% response rate. You can't just assume things like that.
Recall that in my post I said cancer therapy is an exception to the point because of unique mutations in each cancer necessitating personalized treatment.
It's quite intriguing how the more I learn about statistics, the more my trust in medical science seems to waver.
With that in mind, would you mind taking a look at ivmmeta.com? Ivermectin has undoubtedly been contentious during COVID. However, my primary concern is not whether ivermectin works or not, but rather the epistemological issue it raises.
I now find myself in a position where I must choose between a worldview in which the establishment can effectively insulate itself from undesired information, and a distrust in scientific research. If the establishment is correct, then most of the 95+ studies on ivmmeta.com must be fraudulent. Conversely, if the establishment is wrong, then large-scale studies like RECOVERY and ACTIV-6 must be fraudulent. Neither option is very appealing.
I tend to lean towards ivmmeta.com for reasons similar to those you discuss in this blog. (The quality of the establishment's arguments did not help.) I would greatly appreciate it if you could examine the site and share your thoughts on whether it is fraudulent or if their arguments hold merit.
Some of the studies on that site are fraudulent, but most are probably just studies that suffer from small numbers and/or failure to control for any number of relevant variables.
Just as a side note, the idea that it's a good heuristic to always disagree with the establishment's opinion is pretty silly, considering the establishment also endorses electricity, internet, toothpaste, bridges, and everything else the contrarians use. In the case of medicine it is particularly silly, because while obviously the medical establishment can get something wrong, the claim made by anti-vax people that doctors and medical scientists want to kill people or not help them should lead one to abandon modern hospitals and modern medicine altogether, and yet, you don't see much anti-vax activity targeting organ transplants, pacemakers, paracetamol, cataract surgeries, cancer treatments, antibiotics, insulin medication, kidney stone removal, baby incubators, anesthesia, and so on.
Refreshingly logical. I would love to see you debate someone like Eric Topol on the utility of personalized medicine.
Even if personalized medicine is borderline useless, I don't think a public debate with Topol would be the greatest idea. His sheer amount of knowledge regarding medicine is enough to steamroll nearly anyone else in a conversation (at least in the eyes of the audience).