Whenever you are presented with claims that a product, service or idea is valid, carefully check to see if the supporting evidence has been collected without bias and experiments have been repeated by others who do not have a financial or ideological motivation for proving the claims true.  Engage the process of Critical Thinking summarized here (no Java).  This page will describe why the methods used to collect scientifically reliable experimental (and observational) evidence are more trustworthy than uncontrolled testimonials: An introduction to the methods of science and some terminology The difference between Quantitative and Qualitative Data Control Groups, Experimental Groups and Placebos Experiments - Description   - Open Experiment   - Single Blind Experiment   - Double Blind Experiment Double Blind Experiment - Examples  - Product Effectiveness  - Product Comparison Surveys Scientific Observational Evidence - The Bottom Line We are constantly bombarded by marketing claims (whether they are media ads or in-person demonstrations) that try to provide compelling evidence that will convince us that a product or service is safe and effective.  It is extremely important to understand differences between the two main types of evidence you will encounter as you try and evaluate claims and determine whether any given product or service is actually safe and effective or just a feel-good 'remedy' that does not actually work as advertised. Experimental Evidence - specific, measurable (objective) information (data) about a product's performance and effectiveness.  The evidence has been collected in a controlled manner to minimize errors and bias.  This evidence reliably demonstrates that an action (use of a specific product or service) actually caused a specific outcome (an improvement in health).  The methods of evidence collection must be well documented so that anyone can repeat the experiments (or observations) and reach the same conclusions. Uncontrolled Testimonials or Anecdotes - people's subjective observations, opinions and descriptions about how they believe a product or service has helped them.  These personal testimonials can be extremely passionate and compelling if the story-teller believes a product or service has cured them or a loved one of a serious health problem.  However, personal testimonials are scientifically unreliable because they have NOT been collected using processes that control for and minimize errors and bias.  Consequently, there is no way to prove that an action (use of a specific product or service) actually caused a specific outcome (an improvement in health).  Eleven specific reasons that uncontrolled testimonials provide unreliable evidence of effectiveness are discussed here. Two other methods of obtaining evidence that are more reliable than uncontrolled testimonials (anecdotes) are discussed on this page, but evidence collected by these methods is not generally used in marketing programs.  Surveys are a way to collect what amounts to testimonial data in a manner that provides some organization and control, however there are shortcomings that can limit the reliability of the results. Scientific Observation where events are carefully observed and documented (using processes that minimize bias) to determine patterns and deduce cause and effect relationships.  Observations are used in scientific disciplines like astronomy and paleontology where experiments are impossible to conduct.  Like experiments, scientific observations can be used to describe and understand natural processes and develop predictions (hypotheses) that can be tested for accuracy by further observation or experimentation.

This discussion will examine Experimental Evidence (collected using scientific research techniques) and my contention that: To provide reliable evidence that supports health claims about any product or service (particularly altered/enhanced water products that are allegedly energized, vortexted, structured, clustered, ionized, succussed, magnetized, oxygenated, etc.), it is necessary to control the way the evidence is collected, analyzed and presented.  This requires establishing and recording objective outcomes that can be clearly defined, tested and measured.  Three important characteristics of collecting controlled outcomes are that:
(1) all outcomes (positive and negative) are collected and clearly documented - not just outcomes that support the claim,
(2) a clear description of how the data was collected and analyzed is recorded, so anyone can evaluate the experimental process and try to reproduce and verify the results, and
(3) the process (experiments and/or observations) to collect the evidence and analyze the results is designed to minimize personal bias and expectations - which always contaminate poorly designed studies.

I discuss elsewhere how uncontrolled testimonials, which are generally solicited to help market a product or service or support other pseudoscientific claims, cannot be trusted to provide accurate, unbiased information about product effectiveness. 

If we can't necessarily trust what people tell us about their experiences with a product, though, how can we discover useful information about a product and determine whether a claim made about some product or service is accurate and trustworthy or whether it is pure marketing fantasy?  Testimonials are not all worthless, and they can actually provide useful information – but only if collection of the testimonials is controlled in a way that minimizes the potential problems outlined in the uncontrolled testimonials discussion.  The scientific experiment is one process specifically designed to collect information in a controlled manner and provide reliable information that allows an accurate evaluation of the product's effectiveness.

A brief introduction to the methods of science and some terminology:
The fundamental goal of science is to understand the natural universe in which we live.  Much of this understanding involves the discovery and description of causality (cause and effect relationships) in nature.  When natural cause and effect relationships are understood it is often possible to manipulate the causes and thus modify the effects.  If the causes of a disease are understood, for example, processes can be developed to remove or reduce exposure to those causes and/or modify the body's response (with vaccines or treatments) and minimize (or cure) the normal disease effects. 

The products of interest in this discussion (altered/enhanced water products) are marketed with claims that they can have some real, significant effects on health that are caused by REAL biological effects on the body and not just placebo responses.   If those claims are valid, there must be a way to collect valid supporting evidence using the methods outlined here.

Over the last several centuries scientists have developed a methodology (often referred to as the Scientific Method) that formalized guidelines for discovering possible cause and effect relationships (hypotheses) and then testing them to see if they can be validated.  There is lots of interesting (to me anyway) information available about about how science works.  However in the context of this very specific discussion, I will mostly focus on how to use the methods of science to collect reliable information about whether a specific product has a real effect on a person's health.  This process is often described as Evidence Based Medicine - this article provides an excellent introduction to clinical trials.  If you would like additional general information, you can read brief discussions on the goals of science and the scientific method.  Wikipedia and Berkeley also have good descriptions of the scientific method.

The key difference between the collection of uncontrolled testimonials to support a product's effectiveness and the scientific collection of information to determine whether a product is effective, is that scientists do not set out to collect only data that supports their hopes, expectations or beliefs about how a product should work.  Scientific data collection is (or should be) designed so all relevant data are collected, recorded and analyzed in a manner that minimizes ANY biases, expectations or beliefs scientists have which might influence how experiments are designed and how the data are collected, analyzed and interpreted.  Similarly the effects of the biases, expectations or beliefs of those who will use the product (the experimental subjects) should also be minimized.

It is important to realize that both the scientific researchers and the experimental subjects DO HAVE biases, expectations and beliefs about any experiment in which they are involved.  If the researchers were not interested in something about the product, and if they didn't have some expectations about the product's effects, they would not bother to conduct an experiment - if they were hired by a company to test a product, they might have a bias to produce favorable results.  If the subjects didn't have some expectations and beliefs about some claim made for a product they might not participate in the experiment. The examples below will illustrate how several scientific data collection methods can minimize the effect of all the biases everyone has and let the data 'speak for itself'.  

Other scientists judge whether an experiment is valid and has successfully demonstrated a cause and effect relationship in part by how well an experimental design, the methods and data analysis have successfully minimized all potential sources of bias.  High quality scientific journals strive only to publish results of studies where effects of bias, and expectation have been successfully controlled (a good starting list).  Research results submitted to these journals are reviewed by other researchers who are respected in their field, and papers are only approved for publication if all aspects of the experiment are high quality and meet stringent requirements to minimize bias and maximize objectivity, accuracy and reproducibility.  This process of Peer Review is employed by all high quality scientific journals. 

In recent years, however, a number of 'journals' have been established that may have impressive names but almost no quality control measures, and their peer review processes are non-existent or poorly controlled.  These 'journals' may be business ventures that require payment from the authors to be published &/or they may collect and publish 'research' that supports very biased positions that are not accepted by the scientific community as a whole.  Those who publish these suspect 'journals', and those who depend on them to become published, will claim there are conspiracies by the all the standard, high quality journals to suppress the findings of those researchers who are outside mainstream science.  They also claim traditional journals are completely biased toward traditional theories, only publish papers that support traditional scientific and medical beliefs and discriminate against anyone with novel ideas by ignoring their research.  An interesting example played out in late 2012 and early 2013 when a researcher from the Sasquatch Genome Project claimed to have sequenced Big Foot's DNA and was unable to get her paper published in a reputable journal.  She apparently purchased rights to an online journal, DeNovo, and published the paper.  The saga can be followed, here: a, b, c and d, from the Dallas Observer, the Huffington Post, io9 and the author herself.

Quantitative and Qualitative Data:  There are two basic types of information (or data) about experimental subjects that can be collected and analyzed in scientific studies and used to form a conclusion about the product's effectiveness:

• Quantitative Data - Specific, objective information that can be directly measured and collected as numbers.  Examples include; height, weight, age, temperature, the pH, O2 saturation and glucose levels of blood, the number and size of tomatoes picked from the garden, etc.  This information is fairly straight forward, and as long as it is collected accurately, the data will be trustworthy and can be used to compare the attributes of multiple subjects in a study or from the same subjects over time as they use a product.
  
• Qualitative Data - Descriptive, subjective information that cannot typically be directly measured or recorded as numbers.  Examples include; a person's reported status for: gender, race, nationality, energy, stress, hydration, health, mobility, emotional state or pain level. Reports people's religious, political, economic, persuasions as well as the taste, color and odor of tomatoes picked from the garden are also qualitative data. 
  
  When qualitative data is collected during a scientific experiment or survey, the possible responses are often limited to a coded scale - in other words, instead of asking 'what is your energy level after drinking OmyGod Oxygen Water?' (and receiving a virtually unlimited number of possible responses to try and analyze), a limited amount of control is imposed by presenting a response scale - for example a 10-level energy scale where the subject is asked, 'How would you rate your energy level five minutes after you drink a bottle of OmyGod Oxygen Water?  Please answer with a number from 1 to 10, where 1 = I  actually had significantly less energy and wanted to go to bed, 5 = I noticed no change in my energy level, and 10 = My energy level was significantly better and I felt like running around the block'. 
  
  Coded scales can be used to collect both quantitative and qualitative data and can include: Gender; 1=Female, 2=Male, 3=Other or Political Affiliation; 1=Democrat, 2=Republican, 3=Libertarian, 4=Green, 5=Other.  Probably no group has taken qualitative data coding to greater and more creative lengths than the wine industry that has catalogued well over 100 descriptors for the flavor, nose, finish, structure, appearance & texture of different wines.  Coded scales provide some ability (depending on how well the scale is constructed and understood by the participants and the researchers) to compare responses from many people in a study and to compare responses from the same individuals over time.  Qualitative Data collected in scientific studies are essentially controlled testimonials.

The scientific community has developed two research processes (Experiments and Surveys) that enable controlled, interactive data collection by researchers from subjects on whom products are tested.  There are several important characteristics of experiments (and to some extent, surveys) that help minimize and control for possible sources of bias.  These include:

• Use of Control Groups and Experimental Groups:  The goal of an experiment is to isolate and identify those effects in a group of subjects that are actually caused by the product that's being tested.  To help identify experimental observations that are the result of the action of the product, another group of subjects that have similar traits is recruited.  Subjects in the Control Group undergo exactly the same routines as subjects in the Experimental Group, and they use a product as nearly identical to the test product as possible - except that it has no biological action (the Placebo).  Results are collected from both the experimental and control groups.  Similar observations from both groups are assumed to be the result of placebo activity, biases or chance and not the result any action caused by the product under investigation.
• Blinding:  This technique prevents the researchers and/or subjects from knowing whether a specific subject is using the real product that's being tested or an inactive placebo.  In the case of product comparisons the identity of all products that are being compared are hidden from the researchers and/or subjects.  Blinding reduces bias introduced by expectations of how a product 'should' work.
• Understanding and Managing Experimental Subjects: 
  • Collect more data than is strictly necessary to evaluate the product.  In other words, collect data like age, gender, race, health status, activity levels, political preferences and any other attributes that might have an impact on how the product that's being tested functions or is perceived.  This is important because during the analysis of the data the results are adjusted to account for any effects that might be caused by these other attributes instead of (or in addition to) the specific contribution of the product.  A product, for example, might work only on children and not adults.  That important detail would be missed and the results might be interpreted incorrectly if age was not collected.
  • Select subjects that have similar traits.  Whether a study is testing the effects of a product on the subject's health, or a group of subjects is comparing different products, the analysis is often easier if the subject attributes are similar - there are fewer possible cause and effect relationship to try and untangle.  If a study population ranges from 10 to 90 years old and half of them have serious health issues, it would be very difficult to determine whether any observed results were cause by the product or by the extreme differences in the subjects.
  • Randomize subjects into the study groups.  This is another way to minimize bias.  If someone is allowed to decide which individuals are in the experimental group and which are assigned to the control group they may consciously or unconsciously assign subjects with specific characteristics (they appear sicker, perhaps) into one or the other group.

I am going to be honest here.  These examples are brutally detailed.  However, the detail is critical to provide enough information for a non-scientist to gain some understanding of how one of the most important scientific tools, controlled experiments works, and how controlled experiments differ from testimonial 'evidence' provided to support pseudoscientific products and services.

A. Description/Definition of Experiments
      Open
      Single Blinded
      Double Blinded
B. Examples of Double Blind Experiments
      Effectiveness Experiment (Double Blinded)
      Product Comparison (Double Blinded)
C. Summary

• Description:  An experiment is an orderly procedure carried out with the goal of verifying, refuting, or establishing the validity of a hypothesis, an expectation about how a particular process or phenomenon works. Experiments provide insight into cause-and-effect by demonstrating what outcome occurs when a particular factor is manipulated.  An experiment usually tests a hypothesis. However, an experiment may also aim to answer a "what-if" question, without a specific expectation about what the experiment will reveal, or to confirm prior results. If an experiment is carefully conducted, the results usually either support or disprove the hypothesis. (Wikipedia)
  Scientific experiments can be extremely complex and costly. However, it is fairly easy to illustrate important characteristics of experiments that allow collection of controlled data that can provide useful information about whether a product is effective at treating a health issue - or whether one type of wine, water, beer, soft drink, coffee, etc. really tastes better than another.

  Experiments can be designed to collect two kinds of information about products:
  ■ Product Comparison - Attributes (taste and aroma for example) of two or more products are compared.
  ■ Product Effectiveness - Claims that a product has a real, beneficial effect on a subject's health are tested.

  There are three recognized types of experiments that I will describe below in increasing order of complexity and with an increasing opportunity to collect reliable data.

  1. Open Experiment:  No attempt is made to hide the identity of the product(s) or the expected effects.
  • Product Comparison - All products to be compared are fully identified to those conducting the experiment and those making the comparison.  Limitations: There is no way to minimize researcher or subject biases.
  • Product Effectiveness - Subjects are randomly assigned to control and experimental groups.  Both the researchers and the experimental subjects know which group everyone is in and who will be taking the placebo or the test product.  Limitations: There is no attempt to minimize researcher or subject biases - little better than collecting uncontrolled testimonials, except that negative data is not excluded.
    
  2. Single Blinded Experiment: A better way than the open experiment to collect information about the effectiveness of a product or how different products compare to each other is to conduct a blinded study where the researchers know the identity of the product(s), but the subjects do not.
     
     • Product Comparison - The researchers conducting the comparison know the identity of the products that are being compared.  The subjects who are comparing the products and providing information about the product are blinded - they are unable to see the labels, and in some cases, where appearance might influence taste perception, they might not even be able to see the product.  Limitations: If the researchers can be seen by, or are interacting with, the subjects they may unconsciously pass along biases that could influence how the subjects evaluate the products.  Researcher bias might also influence how the data are analyzed.
     • Product Effectiveness - Subjects are randomly assigned to control and experimental groups by the researchers.  The subjects do not know which group they are in or whether they will take the test product or the placebo.  The researchers know which are the experimental and the control groups and thus which subjects will be taking the placebo and the test product.  Limitations: There is no way to minimize researcher bias in the collection and analysis of data, and it is possible the researchers might unconsciously communicate information that would enable the subjects to figure out whether they were in the experimental or control group.
       
  3. Double Blinded Experiment:  The most effective way to collect unbiased information about the effectiveness of a product or how different products compare to each other is to conduct a double blinded study where neither the researchers nor the subjects know whether a given individual is in the experimental group and using the product or in the control group and using a placebo.  When products are being compared, neither the researchers nor the subjects know the identity of the product(s). 
     • Product Comparison - The researchers in charge of the comparison do not know the identity of the products that are being compared (they are blinded) - someone who will not interact with the researchers or the subjects sets up the products for comparison, de-identifies them and labels them with a code.  The subjects who are making the comparisons and providing information about the product are also blinded - they are unable to see the labels, and in some cases, where appearance might influence taste perception, they might not even be able to see the product.  
     • Product Effectiveness - Researchers are blinded.  Subjects are randomly assigned to control and experimental groups by assistants.  The assistants will also create and blind doses of the product or the placebo for distribution to the appropriate study groups.  The assistants will not communicate information about which was the experimental group and which was the control to either the researchers, the analysts or the subjects until after the results have been analyzed and evaluated. 
       

Specific Examples - a Double Blind Experiment will be described below.  The intent of providing the examples is that you can use the outlines, and substitute other products that have some measurable claims and actually test the claims you might be interested in evaluating.  These examples will test the claims of bottled OmyGod Oxygen Water that is marketed as containing eight times the normal amount of dissolved oxygen and has just become the rage on college campuses.  Two testable claims are made for OmyGod Oxygen Water (OmGOBW):
  1)  Better taste
  2)  Extra energy and vitality

Participants in a typical research experiment or survey might include:

• The Researcher designs and manages the experiment/survey.
• Assistants help conduct the experiment or administer the survey.
• Subjects are the active participants who sample the experimental product and record their experiences or provide responses to survey questions.
• Analysts analyze the data to see if the results are statistically significant or whether they might have been This is just an example of those who might participate in many research projects. 
• The Researcher might actually conduct the experiment and analyze the data in an open trial. Researchers can analyze the data in a double blind trial if they are competent statisticians.  There may also be more than one researcher.  There may be groups that provide funding and administration, and so on.

• Experiment - Double Blind Examples:  
  These fairly detailed examples will describe steps of of double blinded experiments for a product comparison and product effectiveness.  Hopefully it will be obvious how each of the steps helps preserve the integrity and reliability of the data and reduce possible biases that can influence the data collection and ultimately the reported outcomes of the experiment.
  
  Experimental details described below are just examples of basic experimental designs: These experiments have not been evaluated by the FDA. These experiments are not intended to diagnose, treat, cure or prevent any disease.
   
  1. Double Blind, Product Effectiveness Experiment:
     This experiment will test the claim that drinking OmyGod Oxygen Bottled Water (OmGOBW) causes extra energy and vitality over drinking regular water.  Extra energy is a subjective claim, but that extra energy and vitality would have to be caused by something, and the only cause based on the claims would be the extra dissolved oxygen.  The only way the dissolved oxygen could possibly actually increase energy and vitality would be if it was absorbed into the blood stream and transported to the brain, muscles and other organs.  Increased oxygen in the bloodstream is a real biological change that will produce objective data that can be measured with a pulse oximeter.  The hypothesis that will be tested is "there will be no difference in reported energy or vitality or in oxygen saturation levels between the group that is drinking OmGOBW or filtered tap water" - if subjects who drink oxygenated water actually have higher recorded energy and blood oxygen levels than the controls, the hypothesis will be disproved, and scientific evidence for the effectiveness of the oxygenated water will have been obtained.  The process is as follows:
     
     1. The Researcher will recruit one or more Assistants who will then recruit 16 test subjects who have similar traits - 8 men and 8 women, all enrolled in the same college biology class; ages from 19 to 20; moderately active; healthy with no history of chronic diseases and no active illnesses.  None of the subjects will have used Oxygenated water before.  The researcher will not participate in the process below and will have no knowledge of which group is which or which subjects have been assigned to each group.
     2. Subjects will be assigned to one of two groups the 'A' group or the 'B' group.  Each group of test subjects will consist of 4 men and 4 women.  The men and women will draw straws or have some other random assignment into their groups.  The subjects will not be told what the types of water the experiment will be testing, so there can be no product-based expectations.
     3. All subjects in each group will be introduced to the "Energy/Vitality Scale" - a self-rating of how much energy the person is experiencing at the time the measurement is made.  The Energy Scale will have ten levels - from 1 = extremely low energy and minimal vitality (I'm barely able to move) to level 10 = I have so much energy and vitality that I can't sit still (I'm all wound up and feel like I have to "run it off).  Level 5 would indicate moderate levels of energy & vitality.  Subjects will also be instructed not to talk about the experiment to others to avoid potential bias.  Subjects cannot wear fingernail polish during the experiment since that can interfere with the Pulse Oximeter readings.
     4. All subjects will be instructed to spend three days recording baseline information.  They will drink a 12 oz. bottle of normal filtered water at 8:00 am, at noon, at 4:00 pm, and at 8:00 pm.  Five minutes before they drink the water and 20 minutes after drinking the water they will record their energy levels.  Five minutes after drinking the water their blood oxygen levels will be measured and recorded with a pulse oximeter.  The subjects will also record activities they were participating in when they drank the water, record the amount and time they drank any other water during the day, and record any other unusual occurrences that might effect their oxygen or energy levels.
     5. The assistants will manage the blinding and distribution of the water samples:
        • They will obtain 64, 12 oz. bottles of OmGOBW for each day of the experiment (each of the 16 subjects will drink four bottles of water per day).
        • The bottle labels will be removed, then 32 of the bottles will be re-labeled 'A' and 32 will be re-labeled 'B'.
        • The water in all 'A' bottles will be dumped out and replaced with filtered tap water - the controls.
        •  If the OmGOBW has obvious bubbles on opening, and it is easy to tell which bottles contain the oxygen, a small quantity of pressurized nitrogen or argon (odorless, tasteless gases without any biological activity) would need to be added to the control bottles to create bubbles when opened.
        • The lid seals of all 'B' bottles will be cracked and lids retightened so it will not be possible to
        determine by sealed lids which bottle contents were altered.
        • Water bottles will be prepared any time before early morning of the day they will be used.
     6. At 6:45 am on each day of the experiment the bottle managers will set all 64 labeled bottles for the day on a table and leave the room without communicating with the subjects.
     7. At 7:00 am the subjects will enter the room and each take four bottles of water according to their group and leave for their normal daily activities. 
        This is one key to a blinded experiment - the subjects do not (and cannot) know whether they are drinking the treated water or the untreated water so their expectations and beliefs can't bias their behaviors or observations.  The other key is that the researchers and analysts do not have knowledge of which group is which until after the analysis step.
     8. The subjects will drink one 12 oz. bottle of their 'A' or 'B'  water at 8:00 am, noon, 4:00 pm, and 8:00 pm.  Five minutes before they drink the water and 20 minutes after drinking the water they will record their energy levels.  Five minutes after drinking the water their blood oxygen levels will be taken and recorded with a pulse oximeter.  The subjects will also record activities they were participating in when they drank the water, record the amount and time they drank any other water during the day, and record any other unusual occurrences that might effect their oxygen or energy levels.
     9. Steps 6 through 9 will be repeated for each day of the experiment - perhaps a week - to accumulate sufficient data to draw some conclusions.
     10. Crossover double blinded study: There is an enhancement to the double blind experiment that allows even better collection of reliable, unbiased data.  After a period of time where the subjects in group 'A' drink the filtered tap water and those in group 'B' drink the OmGOBW as described above, the type of water assigned to each group is switched, so group 'B' drinks filtered water and group 'A' drinks OmGOBW.  The subjects are unaware of the switch.  Steps 6 through 9 are repeated for the remainder of the experiment. 
         Crossover studies provide the opportunity for the experiences of each person to be recorded for both the experimental product and the control placebo.  That has the tremendous opportunity to reduce biases introduced into a study by differences in how subjects might respond to the experimental product or to the experimental process. 
     11. After the data collection period has ended an Analyst who has no knowledge of which group drank the oxygenated water and which group drank the regular, filtered water (in a double blinded experiment, the researcher could analyze the data) will compare all results from all subjects in both groups and determine if there were differences in the reported energy levels or the blood oxygen levels after drinking oxygenated water, and if those who drank the control water.  Statistical tests would be used to explore the possibility that some small observed differences could be caused by chance rather than the experimental product.
     12. After the results have been analyzed the groups will be unblinded and the results presented and explained.
          
  2. Double Blind, Product Comparison Experiment:
  
  Since it would be fairly boring to only compare the taste of one other kind of water to the OmyGod Oxygen Bottled Water (OmGOBW), four other kinds of water will be used in the comparison - tap water, tap water filtered with a high-end activated carbon filter and two commercial bottled water brands, Fiji and Evian.  The hypothesis that would be tested by this experiment is that there will be no clear 'winner' in the taste test.  The process is as follows: 
  1. The Researcher will recruit an Assistant.
  2. The assistant will recruit 16 Subjects, 8 men and 8 women in this example, ages between 19 and 20 who are all enrolled in the same college chemistry class.  None of the subjects will have tried Oxygenated water before.
  3. The assistant will manage the blinding and distribution of the water samples and instruction of the subjects. 
  4. The researcher will not have any communication with the assistant or the subjects.
  5. The assistant will obtain 64 unlabeled plastic bottles that are identical to the OmGOBW bottles.  16 of the bottles will be labeled 'A', 16 will be labeled 'B', 16 will be labeled 'C' and 16 will be labeled 'E'.  The labels of 16 bottles of OmGOBW will be removed and the bottles relabeled 'D'.
  6. 16 stations will be set up on tables with five bottles (one each 'A', 'B', 'C, 'D' and 'E') at each station.
  7. The assistant will fill and cap all 'A' bottles with the Evian spring water, all 'B' bottles with tap water, all 'C' bottles with filtered tap water and all 'E' bottles with Fiji artesian water.  The cap seals for all 'D' bottles will be broken but the caps will be retightened so the extra oxygen will not escape.  As in the double blinded effectiveness experiment described above, if the OmGOBW has obvious bubbles on opening, and it is easy to tell which bottles contain the oxygen, a small quantity of pressurized nitrogen or argon (odorless, tasteless gases without any biological activity) would need to be added to at least one of the the control bottles (in this case 'C') to create bubbles when opened.
     
     All bottles should look identical.  In most beverage taste tests there is no need to go to the fairly elaborate precautions of keeping the liquid in bottles, the different samples could just be poured out into identical glasses.  In this experiment, however, the beverage being tested for taste contains pressurized oxygen.  If the oxygen or the bubbles actually contribute to the perception of taste, it would be unfair to pour the water into a glass where the bubbles might be gone before the water is sampled.  Blinding an experiment can be complicated, and sometimes it is impossible.
  8. The subjects will only know that they will be tasting up to five different water samples, but not what specific types of water they will be testing.  The subjects will not observe the bottle filling process.  Subjects will be instructed to go to one of the 16 tasting stations when the samples are ready, taste each of the five types of water in whichever order they wish and as many times as they wish.  As they sample the water they will be asked to make notes (numeric scales could be devised to provide more precise data) about taste, odor, feel of the water in their mouth, etc. for each water sample and complete the test by ranking the five water samples from the best (=1) tasting to the worst (=5) tasting.  Subjects will also be instructed not to make any comments as they taste the samples or otherwise indicate how they feel about any of the water samples.
  9. After all samples have been tasted, the researcher or analyst would compare all results, and among other analyses, would average the ranks for each of the five water samples to determine which, if any, were consistently ranked best (or worst).
  10. After analysis, the samples 'A' - 'E' would be unblinded and the results presented and explained.

These examples are fairly simple and could be adapted for a high school science fair project, but they provide a basic outline of the experimental processes that are similar whether the experiment is small and trivial or a multi-million dollar, multi-year experimental study.

I hope it is obvious how evidence collected in a controlled environment is more trustworthy than the uncontrolled testimonials used by those trying to peddle a product for which no objective controlled supporting evidence can be obtained because their product only works in the imagination and not in the real world.  It should be obvious too that surveys can't provide the experimental controls that are possible with blinded experiments.

Additional Resources:
  Experimental Design for Advanced Science Projects
  The Basics of Experimental Design

Disclaimer: The above discussion is not an exhaustive description of how scientific experiments work nor an outline all of the specific details, controls and statistical analyses that must be managed in order to design and execute a high-quality, reproducible experiment that stands a chance at publication in a high-quality journal.  Nor are all of the potential pitfalls that can sabotage even the best-intentioned experiment mentioned.  The discussion is provided to highlight some of the more important characteristics of scientific experiments, illustrate how they can provide useful information about whether or not claims for a product (or process or procedure) are true, and provide instructions so you can conduct your own blinded tests.

Surveys are not much different from uncontrolled testimonials in that researchers collect and record information about their subjects without manipulating the study environment.  The survey participants are simply asked questions about the topics of interest without asking the subjects to make any changes in their behavior.  Surveys are a good way to gather data to generate ideas and starting points for conducting experimental studies. 

Because surveys can quickly, and relatively inexpensively reach and collect information on a large number of people, they are often the only practical way a representative picture of the attitudes, beliefs and other characteristics of a large population can be developed.

Surveys are also able to collect data on a broad range of subjects that are not easily studied by scientific experiments, for example; attitudes, opinions, beliefs and values.  Ironically, these are exactly the human characteristics that can cause some of the unreliability of individual testimonials.  These are also the traits that can contribute to the failure of science.  Strongly held attitudes, opinions, beliefs and values can result in experiments and observational studies that are poorly designed and carried out only to provide 'evidence' to confirm a specific bias - not to truly understand a situation.  Surveys also provide information about relationships between people, places, and things as they exist in the real world that are difficult to study in laboratory experiments.

Surveys have several advantages over experimental studies:

1. They are much easier to develop, less expensive and faster to conduct than experimental studies.
2. Information can be gathered on thousands of people just by asking questions.
3. They can be conducted remotely by phone calls, online or by mail.
4. Information can be gathered about non-scientific topics like attitudes, opinions, beliefs and values.
5. Like experiments, surveys can be replicated by others to check and validate results.

Surveys also have significant disadvantages over experimental studies:

Measurement Errors Associated With Surveys Question Wording: Does the question have a consistent meaning to respondents? Problems can occur with Lengthy wording Words are unnecessarily long and complicated. Length of question Question is unnecessarily long. Lack of specificity Question does not specify the desired information. Lack of frame of reference Question does not specify what reference comparisons should be made to. Vague language Words and phrases can have different meanings to respondents. Double negatives Question uses two or more negative phrases. Double barreled Question actually asks two or more questions. Using jargon and initials Phrasing uses professional or academic discipline-specific terms. Leading questions Question uses phrasing meant to bias the response. Cultural differences in meaning Phrases or words have different meanings to different population subgroups. Respondent Characteristics: Characteristics of respondents may produce inaccurate answers. These include Memory recall Problems remembering events or details about events. Telescoping Remembering events as happening more recently than when they really occurred. Agreement or acquiescence bias Tendency for respondents to “agree.” Social desirability Tendency to want to appear in a positive light and therefore providing the desirable response. Floaters Respondents who choose a substantive answer when they really do not know. Fence-sitters People who see themselves as being neutral so as not to give the wrong answer. Sensitive questions Questions deemed too personal. Presentation of Questions: The structure of questions and the survey instrument may produce errors including: Open-ended questions Response categories are not provided, left to respondent to provide. Closed-ended questions Possible response categories are provided. Agree-disagree Tendency to agree when only two choices are offered. Question order The context or order of questions can affect subsequent responses as respondents try to remain consistent. Response set Giving the same response to a series of questions. Filter questions Questions used to determine if other questions are relevant. Interviewer: The use of an interviewer may produce error. Mismatch of interviewer-interviewee demographic characteristics. Unconscious judgmental actions to responses. Fundamentals of Social Work Research, Chapter 8; Rafael J. Engel and Russell K. Schutt

1. Like testimonials, surveys cannot provide strong, reliable evidence of cause and effect relationships.
2. Like testimonials, for various reasons survey respondents may not provide accurate, honest answers.
   •  they might want to present themselves in a favorable (or fabricated) manner
   •  they might have faulty memories of the subject matter
   • none of the answer options provided fit the respondent, or different responses fit at different times
   • they might have an agenda to either support or compromise the survey goals
3. Like testimonials, respondents may provide accurate answers but have confused correlation with causation or forgotten key information.
4. Survey questions and answers may be interpreted differently by different respondents and researchers.
5. Open-ended survey questions,  where respondents can answer a question any way they like, must be interpreted and coded by the researcher.  This can lead to a variety of problems like biased interpretation and too many response types to analyze.
6. Closed-ended survey questions, which provide the respondents with pre-selected choices, may not allow an accurate answer to be entered.
7. Even slight variations in the wording or order of questions and answers can produce different results.
8. It can be difficult to survey a representative sample of the population, so survey results may be biased
   • the surveyed population may contain answers from individuals who should have been excluded
   • the  surveyed population may not contain answers from individuals who should have been included
   •  people with certain strong views may choose not to participate in a specific survey
   •  people with certain strong views may be more likely to participate in a specific survey
9. Survey results can only represent a picture of those who chose to respond to the survey

Cross-Sectional Survey - These studies collect information at a single point in time.  A number of individuals are contacted and asked the questions of interest.  The individuals are not followed and asked the same set of questions later to see what might occur over time.  This example will collect data on the energy benefits of OmyGGod Oxygen Water (OmGOW):

• A caveat here - in order for this specific survey to work, a sizable portion of the population contacted actually would have had to tried the oxygenated water -- probably an unrealistic assumption, unless the survey population was restricted to members of health clubs or college campuses where fads can develop -- but the process is the same whether the survey is about experiences with oxygenated water or is sampling residents of Colorado about experiences with medical marijuana use.

1. The researcher will develop a set of questions to ask the subjects - in this survey, there will be eight questions,
      (1) have you ever tried OmGOW; If the answer is yes, continue with:
      (2) how long have you been drinking OmGOW, and how often do you drink it? 
      (3) what is your gender?
      (4) what is your age?
      (5) on a scale of 1-10, how active are you (1=not active)?
      (6) on a scale of 1-10 how would you rate your overall health (1=poor)?
      (7) does OmGOW taste better than tap water or worse?
      (8) after you drink OmGOW do you believe your energy and vitality levels increased or decreased?
2. The researcher or assistants will call or contact in person as many individuals as is feasible within the constraints of available resources, ask people the eight questions and record the answers.
3. After enough people have been contacted to provide usable information - perhaps several hundred people who have experienced OmGOW - the results will be analyzed to see what percent of the respondents thought their energy and vitality levels increased after drinking the oxygenated water.  Other results could be reported as well like the percent of users who thought OmGOW tasted better than tap water, whether females experienced the product differently than males, whether the length of time using the oxygenated water might have influenced their answers, etc.
   

Longitudinal Survey - These studies collect information at several points over time.  A number of individuals are contacted and asked the questions of interest.  The individuals are contacted again at one or more time intervals to see what might have occured over time.  This example describes a survey on the energy benefits of drinking OmyGod Oxygen Water (OmGOW) over time:

1. The researcher will develop a set of questions to ask the subjects - in this survey, seven questions that will be asked at the beginning of the survey to collect basic data on the subjects,
      (1) have you ever tried OmGOW; If the answer is yes, continue with:
      (2) how long have you been drinking OmGOW, and how often do you drink it? 
      (3) does OmGOW taste better or worse than tap water?
      (4) what is your gender?;
      (5) what is your age?;
      (6) on a scale of 1-10, how active are you?
            1 = not active 'couch potato' | 10 = regularly work out and run marathons)
      (7) on a scale of 1-10 how would you rate your overall health
            1 = poor, lots of problems | 10 = extremely good health
   Three questions will be asked during the course of the survey.
      (1) does OmGOW taste better than tap water or worse?
            (who knows, the extra oxygen might fry the subject's taste buds)
      (2) after you drink OmGOW do you believe your energy and vitality levels increased or decreased?
            -10 = less energy | 0 = no change in energy | 10 = experience higher energy
      (3) have there been any significant changes in your daily routine and activity levels?
            -10 = less activity | 0 = no change in activity | 10 = higher activity levels
2. The researcher or assistant willcontact as many people as it takes to find 20 individuals who drink OmGOW daily and who are willing to provide information about their experiences.  Answers to the initial six questions plus the three ongoing questions will be recorded.
3. The researcher or assistant will call each subject daily  between noon and 1:00 pm and record the answers to the questions about taste and energy/vitallity and routine changes.
4. After a week or two of collecting data on OmGOW - the results will be analyzed to see if the subjects experienced any changes in how they experienced OmGOW or in their energy levels during the experimental period.

Surveys provide somewhat more control over the collected data than uncontrolled testimonials and have legitimate uses.  In my opinion, though, the inherent methodological limitations and opportunities to influence results make surveys less than ideal for gathering data to make important decisions

Every time I see results of a survey presented as providing an accurate representation of reality I just remember my own experiences with surveys I have been asked to take.  If I choose to participate, there are almost always questions that I simply can't answer within the constraints of the question - or my answer might be variable depending circumstances not addressed in the survey - or, if I feel the survey topic, questions or interviewer are 'out of bounds', I may decide to answer randomly or provide answers that will confuse interpretation.  Usually, I choose not to participate, and my experiences and thoughts are not incorporated into the results.  If a significant fraction of survey potential respondents treat surveys the way I do, there is little hope for their accuracy.

In the context of this discussion, another method in which valid scientific evidence can be obtained needs to be mentioned - Scientific Observation.  Those who try to justify the use of uncontrolled testimonials to validate product claims might point out that science has always depended on observing the world and the universe to understand how the natural universe behaves - and that is just what their testimonials are, observations of what happened when someone tried their product.  What they apparently don't understand is that Observations are generally just the first step in the development of all scientific hypotheses and theories - but science never stops with just observations. 

The curiosity of scientists who observe natural phenomena can be engaged, and they come up with tentative explanations (or hypotheses) to explain what causes them.  At that point the explanations are just educated guesses, but these guesses must lead to predictions that can be validated.

• Many of these hypotheses can be tested experimentally as described on this page.  The experimental process will test the predictions made by the hypothesis and either validate or invalidate it.
• Sometimes it is impossible to conduct experiments to test a prediction.  Many hypotheses in astronomy, evolutionary biology and geology are founded on observations of natural phenomena from beyond the confines of our solar system or created in the distant past.  But you can't currently travel to another solar system or galaxy or travel back in time to conduct scientific experiments.  So validation of these hypotheses depends on devising predictions that can either be tested experimentally or tested by making additional observations.

In both cases, however, the processes of science require the hypotheses based on observations to be tested and confirmed.  In many cases the predictions based on theories developed using careful observations are validated experimentally when science progresses to the point where evolving technology finally enables experiments to be conducted. 

The Theory of Evolution, for example, was formulated by Charles Darwin in the mid 1800s and was based largely on his observations.  Since that time a number of predictions about evolution and natural selection formulated on his and other observations have been experimentally validated.  Similarly, observations of geological features by several scientists in the early 1900s lead to the development of the Theory of Plate Tectonics. 

Another key characteristic of legitimate Scientific Observation is that, like well designed Scientific Experiments, they deal with natural events that are subject to validation by anyone else who wishes to make their own observations. 

The 'observations' of pseudo-scientists and those collecting uncontrolled testimonials have two characteristics that quickly separate them from scientific observations. 

1. Only those who believe in the alleged phenomena are actually able to make the observations.  Whenever a non-believer tries to make an observation something always 'goes wrong' to cause failure.  Examples include: structured water, ghosts, 'miracle' products & cures, UFOs, psychic auras, etc.
2. These 'observations' never lead to the development of any testable theories or useful predictions - they are only used to market phony, products, services or ideas.

The Bottom Line:  The process of Observational Science and the validity of theories based on the observations are completely different from information collected by and conclusions based on Uncontrolled Testimonials.  Testimonials used to promote pseudoscientific products, services and ideas certainly make claims, but the claims are either not testable or they have never been tested by legitimate scientific processes.