Uncontrolled Testimonials about the effectiveness of a health treatment are important, but ONLY as a source of questions and ideas for further rigorous testing, because there is no way to determine whether the reported treatment outcomes were actually caused by the treatment or by one or more of these alternative explanations:   The Placebo Effect - a perceived/actual health improvement caused by the expectation of a beneficial outcome   Spontaneous Improvement - many health problems get better over time regardless of any treatment or lack of treatment   Confusion of Causation and Correlation - just because one event occurs before another does not necessarily mean the first one caused the other   A Misdiagnosis of the Health Problem - both traditional and alternative medical practitioners can be mistaken about the cause of a health issue   Undocumented Treatments - people may try many treatments to improve their health but only remember those that were more dramatic or expensive   Subjective Symptoms & Outcomes - it is difficult to know what really happened with a treatment without keeping rigorous, well defined health records   A Sample Size of One - how can you ever know what would have happened if you had not tried a given product? Cognitive Biases - these are distortions in the way we perceive reality based on beliefs, experiences, training, expectations, etc. Unscrupulous Marketing Practices - people have been known to 'distort the truth' to sell products Only Positive Outcomes Are Reported - those selling a placebo-based product or service are not likely to provide testimonials from people who have not had positive experiences   A Desire to Bypass Regulations - The FDA and other regulatory agencies limit what a company can legally claim about their product, but they can report user experiences The Bottom Line We are constantly bombarded by marketing claims (whether they are media ads or in-person demonstrations) that try to provide compelling evidence that will convince us that a product or service is safe and effective.  It is extremely important to understand differences between the two main types of evidence you will encounter as you try and evaluate claims and determine whether any given product or service is actually safe and effective or just a feel-good 'remedy' that does not actually work as advertised. Uncontrolled Testimonials or Anecdotes - people's subjective observations, opinions and descriptions about how they believe a product or service has helped them.  These personal testimonials can be extremely passionate and compelling if the story-teller believes a product or service has cured them or a loved one of a serious health problem.  However, personal testimonials are scientifically unreliable because they have NOT been collected using processes that control for and minimize errors and bias.  Consequently, there is no way to prove that an action (use of a specific product or service) actually caused a specific outcome (an improvement in health).  Eleven specific reasons that uncontrolled testimonials provide unreliable evidence of effectiveness are described in this discussion.  This quote by Steven Novella, "I think anecdotes are worse than unreliable - they tend to lead us to conclusions we wish to be true rather than those that are true, and they can cause a false sense of confidence in the unwary." provides an important perspective on testimonials.   Experimental Evidence - specific, measurable (objective) information (data) about a product's performance and effectiveness.  The evidence has been collected in a controlled manner to minimize errors and bias.  This evidence reliably demonstrates that an action (use of a specific product or service) actually caused a specific outcome (an improvement in health).  The methods of evidence collection must be well documented so that anyone can repeat the experiments (or observations) and reach the same conclusions.  This page will describe why the methods used to collect scientifically reliable experimental (and observational) evidence are more trustworthy than uncontrolled testimonials.   A question frequently asked: If a placebo product or service seems to make me feel better what's the harm in using it, and what's the harm if the company promoting it uses false, pseudoscientific claims that have no basis in reality?

This discussion will examine Uncontrolled Testimonial Evidence and my contention that: In the absence of valid, reliable experimental evidence to support health claims made for alkaline water, homeopathy, other enhanced water products and hundreds of other 'health' products and services, promoters of the products or services must rely on what I call Uncontrolled Testimonials:  the stories and experiences that are collected specifically (and selectively) to support product claims that cannot be validated by any sort of controlled process like a scientific experiment, or even a well-designed survey.

Because uncontrolled testimonial evidence is completely subjective, different people might well come to very different conclusions about whether a product worked for them, so there is no way to actually assess a product's real effectiveness.

Think about it --- if you talk to three people at a sales meeting; one relates a positive experience with a product, another describes a negative experience after using the same product and another reports no change, how can you determine which testimonial is an accurate description of the product's effectiveness?  Testimonials are important, but ONLY as a source of questions and ideas for further rigorous testing (blinded experiments where practical).

One person's (or even many people's) uncontrolled reaction to a treatment that is claimed to produce health benefits is completely unreliable as a proof the treatment actually has any effect.  The problem with uncontrolled testimonials is that there are many alternate, and often more plausible, explanations for the observation of health benefits that have nothing to do with the 'health' product or service being promoted.

The 11 reasons to be skeptical about testimonials discussed on this page can apply to any of the water products or treatments for which health-enhancing claims are made including, but not limited to, water that has been: ionized, clustered, structured, magnetized, energized, oxygenated, vortexed, hydrogenated, intentionalized, catalyzed, m-activated, infused with noble gasses, exposed to pi ceramics, or serially diluted with succussion (shaking).  This warning to be skeptical of uncontrolled testimonials applies to ANY product or service that does not have valid experimental evidence to support claims of effectiveness.

A good reference that discusses Complementary & Alternative Medicine (CAM) including testimonials.  This essay also provides an interesting discussion of anecdotal evidence.

11 reasons to be skeptical of Uncontrolled Testimonials about health benefits of any product or service:

In this context 'be skeptical' does not mean be cynical, it means be a cautious and curious detective.  As you read through this list, particularly if you tend to believe stories people tell you about how effective some product or service is at treating a health problem, consider these other possible explanations:

1. Placebo Effect:  Observed results can be caused by the placebo effect - a perceived health improvement caused by the expectation of a beneficial outcome.  How does one determine whether perceived health benefits of drinking alkaline water, drinking oxygenated water or sipping a homeopathic remedy are due to actual biochemical effects of the product and not a result of the placebo effect?  The placebo effect and the "power of suggestion" are remarkably compelling phenomena that can lead to the perception of greater energy, reduced pain, and a number of other health benefits that have been well documented over the years.
   
   The placebo effect can even lead to mind-generated biochemical changes in the body that mimic the effect of some drugs.  There is no way that reported health benefits of alkaline water, homeopathy or other enhanced water products can be shown to be different from manifestations of the placebo effect without rigorous, unbiased experimentation.  If you take the time to locate and critically examine any alleged scientific evidence provided to support some of the altered &/or enhanced water claims, you will find either that no health benefits have been demonstrated or that, in carefully controlled experiments, any benefits can be shown to result entirely from the placebo effect.  According to this article in New Scientist evolution might explain these real biochemical placebo responses to expectations.
   
   An interesting question to consider: If a treatment causes either a perceived benefit to someone or actually results in biochemical changes that reduce pain or anxiety, is that a bad thing?  Even though the claimed process has no scientific validity and the treatment itself has no actual, direct, measurable biochemical effect on the body what's the harm in using and promoting these treatments?
    
2. Spontaneous Improvement:  Spontaneous, natural improvement in symptoms is attributed to a treatment.  In these cases, people will often try a treatment when their symptoms are the worst - which often just happens to be when the symptom severity will normally begin to improve.
   • Many diseases, like colds, flu, cough, allergies, etc., spontaneous improve over a week or two in most people regardless of treatment.
   • Symptom severity in patients with chronic ailments like anxiety, back pain, migraines, etc., may naturally fluctuate over time.
      
   If perceived healing can be caused by timing of the treatment, what observations or measurements could one person make to demonstrate that a given healing episode was due to the treatment and not to ordinary healing processes?  It has be said that if you have a cold you will normally get over it in a week, but if you take a special treatment like alkaline water or a homeopathic remedy, you will be cured in only seven days. 
   Spontaneous improvement of symptoms can lead to confusing causation with correlation - in other words, two events might be linked in time but there may be no cause and effect relationship between them.  This is an important specific example of point number three that needed its own discussion.
    
3. Confusing Causation with Correlation is one of the many logical fallacies employed by those who promote pseudomedicine and pseudoscience and can be illustrated by the example, "Many people who have strokes have been taking blood pressure medications. Those blood pressure pills must be causing strokes, therefore to be safe use our completely natural, μαγεία drink to lower your blood pressure instead of those dangerous medicines!"
   
   Attributing the cause of an observed health effect (or any event for that matter) to some incident that precedes it, as described above, is just one example of confusing Correlation (a relationship in time) with Causation (a real cause and effect relationship).  In other words, what may appear true on the surface because events are related in time, may lead to a completely false, though apparently reasonable, conclusions.  This rather extreme illustration demonstrates why eating pickles is dangerous and should be avoided at all costs.  While this is a parody, it clearly illustrates how events that are correlated in time (eating pickles followed by serious illness and death) can be misconstrued and used to try and prove causation.  Another interesting report links the installation of each additional cell-phone tower to the birth of 18 additional babies.  The confusion between correlation and causation is evident when marketers use uncontrolled testimonials to try and demonstrate their product actually caused a positive health outcome when, at best, there may only be a correlation.
   
   The correct interpretation of the first example is that one of the risk factors for stroke is high blood pressure, so those who take blood pressure medication will already have an increased risk of strokes, and the blood pressure medications may actually reduce the stroke risk.  To discover whether or not specific medications or other treatment options used to reduce blood pressure will also lower the risk of strokes, experiments must be performed to control for the many potential factors that can complicate the interpretation of simple observations.
   
   
4. A Misdiagnosis:  The symptoms or illness could be misdiagnosed.  For example, a person is informed by a care giver they have cancer (or other specific disease).  They are told (or believe) the condition is incurable or would require a drastic intervention to cure.  They begin to drink alkaline water, take a homeopathic remedy or try some other treatment because a friend tells them it can help cure cancer.  Two months later they have another checkup and are told there is no cancer.  Is it more likely the cure was a result of the treatment, a miracle or a misdiagnosis?
   
   A personal example: In June 2013 my 90 year old mother was told she had untreatable lung cancer and perhaps three months to live by her primary physician based on an x-ray. In this case, we did not believe it and asked for another diagnosis from another doctor based on a MRI scan. That scan found no evidence of cancer.  It would have been easy, in desperation, to try some alkaline water or other alternative treatment in the weeks before the MRI was scheduled and then attribute the absence of cancer to to the treatment's effectiveness.
   
   It is also quite possible, in some cases, that people simply decide they have a specific disease (based on some symptom and something they hear on TV, a friend's comment, an Internet article, etc.), embark on a treatment protocol like drinking alkaline water, feel better after a few days/weeks, and believe the treatment cured them of their self-diagnosed, nonexistent illness.

 
5. Undocumented Treatments:  Switching or adding treatments or changing behavior may cause a real improvement in symptoms, but the "cure" is attributed to the original treatment.  An example might be if someone who has been working hard lifting heavy objects develops lower back pain and begins taking a homeopathic pill that's "guaranteed to reduce pain".  They would probably also stop lifting things and they might go to bed and use ice and heat treatments - might even sneak an aspirin or two.  After a few days on the homeopathic remedy their back feels better.  It's likely that the homeopathic treatment will receive credit for the cure (because they had to actively pursue and purchase the novel treatment) and the contribution of the other life-style changes will be overlooked or minimized.  With an uncontrolled testimonial there is no way to know what other changes the person might have made to try and improve their health.  Scientific experiments, on the other hand, control for other possible contributing factors.
   
   
6. Subjective Symptoms & Outcomes:  Subjective evidence is the testimony of how a product performed based on the perception of the subject.  The quality of subjective evidence depends upon the honesty of the subjects, and their ability to accurately perceive reality.  These testimonials almost almost never include measurable reference points or targets for the symptoms or treatment goals.  In other words what does "increased energy", "increased mental acuity", or "increased heart health" or other common claims like "better hydration", or "increased detoxification" really mean?  What measurable characteristics can be documented before and after drinking alkaline water?  In the absence of specific, documented before and after measurements, the mind can create any story it likes based on a desire to become healthier.  Even if an individual created and documented measurable reference points and recorded some health benefits after drinking alkaline water or trying another enhanced water product, that would only suggests a correlation, not a cause and effect relationship - and could not rule out the placebo effect or some of the other reasons discussed here. 
    
7. A Sample Size of One:  How does anyone know what would have happened if they had not tried alkaline water, a homeopathic remedy, another enhanced water product, or some other treatment?  I submit that it is impossible to know "what might have been if only I had done something differently".  A testimonial I received about alkaline water, for example, claimed that "Except for the roughest allergy season I've experienced in my 45 year memory, I haven't been sick in almost 5 years now."  Does that mean alkaline water is not effective against allergies - perhaps alkaline water causes periodic severe allergic reactions?  Might some of the allergic symptoms actually have been a cold?  Might there be other reasons to account for her lack of sickness?  Might memory have failed and a minor cold four years ago been forgotten?  Can anyone really know what their health would have been over five years if they had not been drinking alkaline water?
   
   Testimonials can be hard to ignore, particularly when they come from a family member, trusted friend, someone you accept as an authority, or within the context of a meeting of excited individuals who have similar testimonials.  Humans are 'hard-wired' to pay attention to stories from others - that is how everyone learns virtually every thing they know about the world (besides their own personal experiences, of course).  Information is communicated from a person who has knowledge and experience to someone who does not. The problem is that a testimonial, even from a trusted source, may not be accurate because, even though the person may be sincere about their experience, they may be completely wrong about the cause and effect relationships for any of the reasons discussed here.  
    
8. Cognitive Biases:  There are a number of unconscious psychological phenomena that conspire to influence people's perception of reality and can lead to the reporting of an effective treatment (or otherwise skew reality) when there was no actual physical change.  Examples include:
   • Confirmation Bias (people naturally seek information that supports their beliefs and disregard disconfirming evidence - self-fulfilling information gathering)
   • Bandwagon or Herd Effect (people tend to follow the crowd and interpret their experiences as others in their circle do)
   • Social Desirability (the tendency of people to answer questions in a way that will be viewed favorably by others - related to the bandwagon effect)
   • Selective Perception (the tendency for expectations to affect perception - a component of the placebo effect where there is no actual physical change)
   • It is informative to go through this list and see what other biases might fit.
     
9. To put it bluntly, Some People Use Unscrupulous Marketing Practices - particularly when there is a profit to be made.  There is usually no way to determine whether testimonials used in marketing materials are sincere reports of people's experiences (or whether those experiences were influenced by the points described above), paid endorsements, or complete fabrications by those selling the product/process.
   

← To illustrate this point, I went to an Amazon sales site and copied a several anti-testimonials for the Enagic Kangen Water Leveluk SD501.  These testimonials will obviously not make it into the sales brochure or be promoted at distributor meetings: I was fooled and duped into buying one of these machines, and two years later my health problems are exactly the same. . I did everything the company said to do, like giving up soda and all "acidic" beverages like certain juices and other drinks. I thought at first I was seeing real results, so I bought a machine, but all I did was saddle myself with debt for the machine. After visiting the doctor, it was clear that my health had not improved. The person who sold me the machine told me to be patient, that it would probably take me a full year for my body to become "alkaline". So I kept it up for two entire years, but nothing ever improved (and some things got worse).  Sera I've had a Kagen water machine for two years. Before that my family and dogs were healthy. In the last 6 months my husband was diagnosed with Melanoma and my 5 year old Maltese who gets Kagen water daily with Mast Cell Cancer. I've seen no benefit what so ever from this product. Do not waste your money.  MichelleY  I would LOVE to start a Class Action Suit, and get my hard earned money back, for believing this machine and the water it produces, have healing affects.  Earthie Earth

10. Only Positive Outcomes are Reported (Cherry Picking):  Testimonials (at least as used by those marketing placebo products or services) only document and report positive outcomes. There is no way to know how many people used the product and experienced no positive benefits or got worse because their condition was not effectively treated.  Marketing for FDA approved medications must include all reported side effects - so at least there is some transparency in traditional medicine.
    
    On the other hand, there is no requirement to demonstrate the effectiveness of alternative medical treatments or report the number of patients who do not improve, who experienced side effects or who got worse after treatment - or who died.  Also note the interesting reference to 'Undocumented Treatments' in the first anti-testimonials to the right that clearly shows how the company representatives suggested making additional life-style changes that, by themselves, could contribute to perceived changes in health.
     
11. Used to Bypass Regulations: Testimonials are regularly used by placebo marketers to bypass FDA and FTC regulations against using untested, unsupported claims about specific disease treatments. A company seldom directly states that drinking alkaline water, for example, can cure cancer, diabetes or other specific disease, but many distributors list testimonials from consumers who claim to have been cured or alternative practitioners who claim to have treated or cured a specific disease (search Google for "kangen testimonials").
    
    The only requirement necessary to market alternative products and treatments that claim to provide such vague health benefits as: Greater energy and vitality, Increased hydration, Enhanced oxygen delivery, Better overall health, Decreased stress levels, A more positive attitude, Boosted metabolism, Detoxification of the body, etc., etc. is to add the Get Out of Jail Free Disclaimer everyone has seen, "This statement has not been evaluated by the FDA.  This product is not intended to diagnose, treat, cure or prevent any disease"  to their product.
    
    Check out the websites of most large manufactures of enhanced/altered products.  Their actual claims - the ones for which they can be held legally responsible - are all vague and meaningless.  It is left to the distributors and the testimonials they employ, which are 'outside the control of the parent company' to make specific claims about how the product really does cure arthritis, diabetes, cancer, etc.

The points above illustrate why I am skeptical about relying on uncontrolled testimonials to any support health claims for any product.  The Controlled vs. Uncontrolled Testimonials section will demonstrate how scientific experiments are designed to minimize the problems associated with uncontrolled testimonials and enable cause and effect relationships to be established.

This table briefly outlines how blinded experiments (described in more detail here) can produce controlled testimonials (and other information) that minimize the problems associated with uncontrolled testimonials - Specifically: 1.  The Placebo Effect - Blinded study participants do not know whether they are receiving the real treatment (or product) or a placebo, so any placebo effects will be similar in both groups and cancel each other out. 2.  Spontaneous Improvement - Blinded study participants are similar in age, health status, etc., so health conditions and spontaneous improvement rates would be expected to be similar in both groups and cancel each other out.  Improvements can be demonstrated to be caused by the treatment. Confusion of Causation and Correlation - Blinding does not directly help the process of sorting out cause and effect relationships, but it allows the process to take place with minimal bias and expectations that can cause either the researchers or the subjects to incorrectly conclude a correlated event is actually a cause. 4.  A Misdiagnosis of the Health Problem - Always a possibility, but since multiple subjects are followed over the course of the experiment and have as close to the same health status as possible (or the same symptoms & diagnosis) the chance for significant misdiagnosis is minimized. 5.  Undocumented Treatments - Part of a well designed blinded study is to instruct all participants in both groups not to try other treatment methods during the experiment and to follow the same routines when taking the treatment. 6.  Subjective Symptoms & Outcomes - Part of a well designed blinded study is to design objective outcomes that can be clearly identified and measured (or rated) and instruct participants in both groups how to identify and record the outcomes of interest. 7.  A Sample Size of One - A well designed blinded experiment has a number subjects in each group - the more the better - so data can be collected (and results compared) from many people.  The results are analyzed in a way that best demonstrates if there are any real differences between outcomes in the two groups or if any differences observed are probably due to chance. 8. Cognitive Biases - All of the biases described above are minimized because neither the subjects nor the experimenters know which group is taking the real, active product or treatment and which is taking the placebo, so any biases in reporting or analysis should be random and cancel out between the experimental groups. 9. Unscrupulous Marketing Practices - Sometimes scientists will falsify their evidence and produce fraudulent findings.  Usually, however, other scientists will identify and expose the fraud.  Science is designed to be self-correcting - pseudoscience is not. 10. Only Positive Outcomes Reported - A well designed and conducted blinded study will collect all data, both positive and negative, from all subjects.  Since neither the participants nor those conducting the experiment know which group is taking the active product until after the analysis has been completed, there is no benefit to withholding evidence. 11.  A Desire to Bypass Regulations - Obviously, where there is lots of money to be made, there is a desire by some (inside and outside of the scientific community) to break the rules.  At least there are checks and balances for those selling FDA regulated products that may catch those who try and circumvent the regulations. There are no equivalent checks and balances for those selling products and services for which health claims are made that are not FDA regulated.

As noted, testimonials are actually used to collect data from participants in valid, scientific experiments.  These controlled testimonials (or data) collected from experimental subjects are entirely different from uncontrolled testimonials with the limitations that were described.  They are records of the experiences (and can include specific biological measurements) of individuals who participate in experiments, they are not selectively collected only from those who have experienced a positive outcome.

A good experiment is designed so that the eleven limitations described above are minimized, and the testimonials can actually be used to help establish a cause and effect relationship (if one exists) between treatment and a health outcome.  The most effective experiments are blinded - they compare the effectiveness of a product or service as a treatment for some health issue with a placebo (a product or service that is as similar as possible to the product or service that is being tested, except that it has no real biological action - the famous 'sugar pill').