Partially dubious research reports


Dear readers,
A term that actually originated from science, but which is also often used outside of scientific research, is the term “clinical study”. Products for personal care, creams or dietary supplements are often advertised with the “clinically tested” rating, without making clear what exactly clinical refers to. The tests that are carried out in case of such studies are often very variable, and it is often difficult for a layperson to understand how the manufacturer of the product wants to have proven the effectiveness. In most cases – to take this as a prelude – “clinically tested” simply means that the advertised product is not harmful. Looking at what has been used in the course of history to counteract hair loss (see article about myths concerning hair loss) this be considered as an advantage, but that does not mean that the product has a positive effect.

Considering this, I got the impression that the pedicat “clinically tested” is especially used for hair care products. The “clinically tested” property is then often to suggesting that the product has positive effects on hair growth and hair texture or protects from hair loss. However, such promises should be treated with caution, since the tests do not always prove what is suggested by advertising. Therefore, I would like to take the opportunity to present to you what clinical trials actually are, and what features distinguish them from the kind of studies that are designated as “clinical” for advertising purposes. In this way, you should get a better feeling for which advertising messages you can trust and which you should be better off against.

Therefore, enjoy reading!
Yours, Angela Lehmann

What are clinical trials and what characteristics do they differentiate from other forms of scientific research?

A clinical study initially describes a particular form of data collection or, in general, the investigation of a newly developed treatment procedure, such as a medicament or a medical device. In general, such studies can be divided by several charateristics. On the one hand, it is possible to make a rough distinction between aspects of the study design which influences the scientific significance of the study. These aspects include:

  • Is it a placebo-controlled trial? Was there a group of participants who had received a treatment or a drug that could have had no effect and was this group compared to those who received the actual treatment or drug?
  • Is it a randomized trial? Is it therefore randomly determined who is in the experimental group or is the allocation to the treatment or placebo group based on previously defined criteria?
  • Is the study conducted in a single- or double-blinded manner or is it open? Open indicates that both researchers and participants in the study know whether who is in the placebo and who is in the treatment group. In the case of a single-blinded studies, only the participants are unaware of the group they are belongig to, and in a double-blinded studies, it is also not known to the researchers whether a patient whose data is currently being evaluated was in the treatment or placebo group.

If necessary, other aspects of the study design can also be distinguished, but the three aspects outlined above summarize the most important aspects and should be sufficient for now.

© NIAID | Participant of a clinical trial is injected in the arm with an active drug. © NIAID under Creative Commons Attribution License

© NIAID | Participant of a clinical trial is injected in the arm with an active drug.
© NIAID under
Creative Commons Attribution License

Furthermore, clinical trials can be classified according to their phase. These phases are first divided into a preclinical and a clinical section. The preclinical part of the study is used for basic research about the presumed mode of action of the new drug or the comparison of different drug candidates. Here, model systems, which have a different predictive value in preclinical experiments, play an important role. Furthermore, a distinction can be made between in-vitro tests and in-vivo tests. In the case of the in-vitro tests, the novel active substance is produced, for example, by bacteria or its influence on the metabolic dynamics of yeast cells is investigated. All in-vitro tests have in common that they are carried out in a reaction tube (Latin: in vitro). On the other hand, the new active substance is tested on animals during in-vivo experiments to investigate its effect on a more complex organism. In both cases, however, it must be noticed that it is difficult to simply transfer the findings from in-vitro and in-vivo tests to human beings. In this preclinical phase, as described above, basic research is addressed.

If all these trials are successful, the treatment method or drug may be approved for clinical testing. This is further divided into four phases, some literature also mentions a phase 0, which is not important here. In phase I, the drug is mostly administered to young, healthy, male adults. This first phase addresses primarily safety questions concerning new the drug. In the case of female subjects, for example, one would risk damage to the egg cells, whereas in male subjects, the germ cells are constantly formed up. Furthermore, the subjects should be healthy in this phase because otherwise no conclusions can be drawn as to whether an observed effect is based on the disease or on the administered drug. In phase I it is also determined how long the drug remains in the body, how it is metabolized and whether it has side effects. In phase II trials, the drug is first tested in persons suffering from the target disease. Here it is examined whether the theoretical therapy concept, which was developed on the basis of preclinical data, is correct (phase IIa). In addition to the investigation of side-effects, an attempt is also made to find the optimal dosage (phase IIb). In the so-called phase III studies, the drug must prove that it is effective in a larger number of diseased subjects. If there is a significant positive evidence of efficacy for the drug (phase IIIa), the marketing authorization for the drug can be requested, which then corresponds to phase IIIb. In the final phase IV, the approved drugs are ultimately examined for interactions with other drugs or whether the use of the drug causes very rare (less than 1 out of 10,000) adverse effects.

This procedure is based on a close monitoring of all results by independent institutions. The implementation of such clinical trials is usually associated with costs of several million dollars as well as several years of development and research. The approval of the preparations and the monitoring of the studies is carried out in the USA by the Food and Drug Administration, in the European Union by the Commission and the member states, and in Switzerland by Swissmedic, based in Berne.

back to table of contents

What are the criteria by which you can assess the quality of clinical trials?

In the following, we will look at a number of factors that enable you to assess the quality or seriosity of clinical trials. In principle, all you need for this is the internet.

back to table of contents

Was the study published and if so, where?

In the case of the predicate “clinically tested” and its synonyms, it is initially a non-regulated advertising statement, and not a quality predicate. However, someone who advertises a product with a clinical test will also refer somehow to the result of the test. For example, in TV advertising for so-called anti-aging creams, it is more frequently mentioned that a large proportion of the users is very satisfied. Accordingly, for example, some providers of hair care products point to such studies on their web domains. A quality criterion is that the study was accepted by a scientific journal for publication. This can be checked, for example, by searching the study title in scientific data bases. You can find there information on the publishing journal, the edition in which the study appeared and the year of publication. If such information is available, it can be assumed that it is actually a scientifically published study. On the other hand, any producer on the radio or on his website may claim that his product has been clinically tested. If, however, no publicly accessible data exist, this remains merely a claim.

back to table of contents

How were the findings obtained in the study?

This is a very wide field on which many factors have to be considered. For example, whether the study was performed with subjects or whether it is merely the above-described in-vitro test. This difference is important because there are also in-vitro studies which, for example, examine the hair growth of individual scalp biopsies. It may not be surprising that the hair grows when a piece of scalp, which contains healthy hair follicles, is placed in a suitable nutritient solution, and then hair growth is measured; but this experimental design is highly artificial and has not much in common with natural hair growth in an organism. Nevertheless, such experiments may produce results which testify an effect to the product which it does not have in the whole human being.

The situation is somewhat different if a study was conducted with subjects. First of all it is important how the participants were recruited. Were they randomly selected and assigned to a treatment or placebo group, or are they more likely to have a condition that they might want to eliminate by participating in the study? Here, the motivation of the participants plays a crucial role. Care should also be taken in case that the participants show a particularly large age range. If so, the participants may also suffer from other diseases that may affect the tests. Are such diseases indicated or are they simply omitted?

Furthermore, it should be noted how many persons have participated in the trial. Since scientific investigations are usually very expensive, it can not be a any-big population. However, on the other hand, it must not be too few, since the investigation hardly has any value. The ideal case is that the examiners have made a calculation of how many subjects they have to examine in order to demonstrate an effect.

One of the most important points addresses the experimental design itself and means the objectivity or subjectivity of a test. Especially in the case of studies on non-medical care products, self-assessment questionnaires are often submitted to the participants for their answers. For most people, it would be difficult to assess the aspects questioned there on a scientific level. For example, according to the impression of the subjects, it is asked if they have more or less wrinkles after applying a cream. Their answer should not have any meaning in a scientific manner. In particular, if the subjects had increased wrinkles at the beginning of the study and possibly suffered from them, the image underlying such a self-assessment is so strongly distorted and no information can be derived from it. In order to mitigate this fact, frequent medical experts who accompany such investigations are asked for their professional assessment. This should be less distorted, but it does not provide an objective result.

back to table of contents

Are any side effects reported or information about how the product should be handled?

Any study that claims to be scientific must consistently report about unexpected side effects and effects that occurred during the trails. At best, such effects have not occurred. But even that would have to be noted somewhere. Then, the reader can conclude that at least no known negative consequences are associated with the application of the product. Its use is therefore not harmful, but this is not a statement about the effect, since the effectiveness or ineffectiveness of a medical treatment does not necessarily correlate with the side effects it can have.

Furthermore, the reader of such a study should examine whether the application, as carried out within the investigation, is practically feasible. This means, on the one hand, that the experimental conditions correspond as far as possible to “physiological” circumstances. This criterion would not be met, for example, if the active ingredient of a food supplement only produced an effect at such high concentrations that this would require the intake of several kilograms of the drug in one day. Another example would be a substance against hair loss (alopecia), which must remain in contact with the scalp for several hours in order to exert an effect. This would be a period of time that goes far beyond the normal duration of a hair wash. On the other hand, any potential user of such products should consider whether the application is compatible with his lifestyle and circumstances at all. This might be the consideration of whether the costs of the continued use of a skin cream against wrinkles is sustainable in long-term; or whether it requires to change the previous hair-washing habits fundamentally, when a shampoo against hair loss is used.

back to table of contents

Can the results of the study possibly lead to inadmissible generalizations?

It often happens — and not only in “clinical studies” – that the authors of scientific publications derive inadmissible generalizations from their results. This is not intended to imply anyone to do so in a deliberate or even malicious manner. It is enough to deal with a research area very intensively for years, in order to quickly lose sight of the point at which a causal conclusion is meaningful. In order to make this clear, an analogy example is to be used.

The manufacturer of medicines (medications, lotions, shampoos or devices) to prevent hair loss tests his product and finds in the measurement that the rate of hair growth increases when the medicine is taken. Subsequently, it is suggested to the potential buyers as an advertising statement that the clinically tested preparation stimulates hair growth. The statement is actually true because the product causes existing hair to grow faster. What the advertising message, however, is not mentioning, is that this applies only to healthy hair follicles. Already dead follicles can not be “re-lived”. However, recognizing this is only possible if the corresponding publication has been studied.

back to table of contents

Is there any methodological or financial uncertainty about the published data?

A final point, which makes it easy to see whether a study is a serious one, involves potential methodological or financial entanglements. In order to prove, or even to recognize this, it is often necessary to make more extensive enquiries.

Methodically, it is a critical aspect, for example, when the examiner has developed the examination procedure. This point may not seem suspicious at first, but it is often the case that not all the details of an experiment are given a publication. This means a richness of detail which would allow any reader with access to the appropriate test equipment or chemicals to carry out the experiments themselves. However, the reference to earlier publications is much more common like for example with “As it was described earlier (another publication).” If the reader studies this new literature, he can sometimes trace that both methods of investigation as well as the results obtained with this method are derived from one and the same person. Depending on the individual case, this may restrict the meaningfulness of the published data. This holds to be true, without subordinating investigators in general to have disreputable motives.

A study may be even more questionable when reading the so-called conflict of interest. This is a special section of a scientific publication in which the researchers must indicate whether any financial or professional interest affected the publication or if any kind of interests of third parties are affected by publication of the results. In this case, the authors of the study are prompted to make these transparent in the conflict of interests section. But sometimes conflicting content can be found in the conflict of interests. This may be the case, for example, when it is explained that the study has been financed by a company and the researchers have also received financial grants from the same company. Even more difficult is the case when the company is also the manufacturer of the medicine or device, which is tested for its efficacy there. Such constellations are by no means uncommon and in many cases also unproblematic because it is not surprising that many companies around the world have an interest in assessing the effectiveness of their products by scientific means. It should also be pointed out that such information or such constellations between researchers and companies are not illegal. But they might be ethically questionable. However, an attentive reader should sufficiently involve this information in his considerations when thinking about whether a non-medical product is still considered to be “clinically tested”.

back to table of contents

Are you still uncertain about whether a publication is reputable?

If a publication satisfies the quality requirements discussed so far in this article, you can safely assume that it is a serious publication. However, if you still have doubts about how data has been collected, here is a last tip for you:

In addition to companies that promote of scientific studies, there is a corresponding control authority for the award of research funds in almost every country in the world that is engaged in scientific research. In the application process, many of these institutions are forced to develop so-called data management. This aims to make the primary data from the experiments available to so many other researchers and examiners as possible so that they can be verify and validate it. If necessary, ask if you can access the raw data; then you can calculate for yourself whether the correct conclusions are drawn from what has been calculated in the experiments. Otherwise, you have the opportunity to ask the researchers themselves; because, finally, there is a reference address on each publication, that names the publishing researchers. Sometimes it is worthwhile to ask them and to take a second look at the data!

back to table of contents