“Of Course it’s True I Saw it on the Internet!” Critical Thinking in the internet Era

Critical Thinking in the internet Era

Leah Graham and P. Takis Metaxas

The internet is revolutionizing research methods at colleges and universities aroundthe world. Though the internet can be extremely useful to researchers, it presents a significantchallenge as it is quite different from traditional sources. The lack of uniform standards andthe ease of access have made the internet a powerful but uncertain medium. Substantial effortis required to adequately evaluate information provided on the internet, and this may notalways be apparent to users. [5] This is particularly challenging for students as many havecome to rely on the internet as a primary source of information without formal instructionabout the difficulties involved. The internet has gained a primary place in research methods,and it is vital that students become able to critically evaluate information on the internet.

Several solutions have been suggested to facilitate accuracy determination in internetresearch. In Libraries and the Academy, Jerry Campbell argues in support of the Associationof Research Libraries’ plan to develop an internet portal to “trustworthy” information. [1]This portal would “promote the development of and provide access to the highest qualitycontent on the Web.” Many colleges have also adopted this approach by providing lists of“approved” online sources to students. While it appears to provide a practical alternative to“information.coms” that focus more on advertising than accuracy, this approach suffers fromseveral drawbacks. First, it is impossible to continually monitor all of the content found usingthese portals. Websites change overnight and expand at exponential rates, and attempting toverify every page of each linked site every day would be an incredibly time-consuming task.Clearly, this is not feasible, but it would be necessary to ensure the accuracy and timelinessexpected of information found using a “scholars’ portal.” Additionally, this approach places ∗

Corresponding author’s address: Department of Computer Science, Wellesley College, Wellesley, MA 02481

1

the responsibility of evaluation on the webmasters of these portals. A more interactiveapproach that encourages users to develop critical thinking skills would provide lasting value,while preventing them from becoming dependent on these portals for the “right answers.”

Developing other approaches requires a firm understanding of how students currentlyuse the internet for research. Consider the results of an informal questionnaire distributed atSUNY College of Agriculture and Technology in Morrisville, New York, by Angela Weilerin 1999. In response to a question asking how students would ascertain if online sources wereaccurate enough to be considered “a good source of information,” 29% said they acceptedinternet information regardless, with only 34% considering additional verification important.[5] These startling results confirm the importance of further study to provide specificinformation about students’ online research practices. To address this, we developed a six-question survey that was administered to 180 Wellesley College students during the 2000-2001 academic year. Students’ responses to this survey helped explain how college students,from different backgrounds, class years, and majors, react to information on the internet.

Research Methods

The students participating in this study were in Computer Science 110 (“Computersand the Internet”), and this survey was their first assignment. Students were told the purposeof the survey was to understand how students conducted searches. The survey was dividedinto seven emails. The first explained the process of responding to the survey and included apersonal information questionnaire. The following six emails each contained one question andasked students to report their answer and search strategies.

The survey was designed to answer three research questions:

• First, how strongly do students rely on the internet for information?• Second, what claims are students more likely to believe?• And finally, who is most susceptible to misleading claims?

To identify students’ reliance on the internet, they were told to answer the questions inwhatever way they wished. They were free to use any resource available, and they were asked

2

to report which search methods were used for each question.

The six survey questions were used to determine students’ ability to evaluateinformation, as well as their inclination to verify their responses. Four questions testedparticular areas of misinformation: advertising claims, government misinformation, lobbygroup propaganda, and ‘scams.’ Preliminary research indicated these areas could present asignificant challenge to students. Two additional questions—one very easy and one verydifficult—were used to determine if students were more diligent about accuracy andverification when the information was easy to find.

Each response was given a score from 0-3, with 3 being the highest score. The scoringsystem placed equal weight on accuracy and the students’ efforts to double-check responses.An optimal answer was therefore defined as a correct response confirmed in at least twosources. Other scores were categorized as follows:

A 0 indicates no response, a 1 an incorrectresponse that was not double-checked, and a 2either a correct answer that was not double-checked or an incorrect response that wasdouble-checked. The 2 category contains bothtypes of responses, as dividing the categorywould require placing more importance on

- Table 1 -accuracy or verification. Neither of these

attributes, when considered individually, wholly constitutes adequate research practices. Assuch, the 2 category remains the “middle” category for responses that are not entirelyacceptable due to a lack of accuracy or verification.

Finally, to evaluate which groups of students are in greater need of assistance, studentswere asked to fill out a questionnaire asking for age, class year, and other factors. These datawere matched with their responses to the survey questions.

3

Results

The conclusions to these research questions were remarkable. Regarding students’reliance on the internet, it became apparent that students are very eager to use theinternet—and only the internet—in conducting research. Though the survey was not in anyway limited to internet resources, less than 2% of students’ responses to all questions includednon-internet sources. Many of these responses also quoted online sources at some point. Thisfinding emphasizes the importance of teaching good internet research skills, as students relyso heavily on the internet.

This survey also revealed the extraordinary confidence students have in searchengines. If the question did not mention a particular website, almost all students immediatelyturned to a search engine. Many remained faithful to one search engine throughout the survey,even if it did not immediately provide the answer sought. This is particularly interesting asexperts believe that no single search engine captures more than 16% of the entire internet.With all search engines combined, this only increases to 42%. [2] Additionally, students wereasked a question in the personal information questionnaire to determine the extent of theirunderstanding of search engines. Few students responded with any degree of awareness of theprocess by which search engines post results. This is distressing as the reliability of searchengines to faithfully and selflessly guide users to appropriate materials has often beenquestioned. [8]

The second research question about the types of information that are most problematicto students yielded disheartening results. Students were overwhelmingly susceptible to threetypes of misinformation—advertising claims, government misinformation, andpropaganda—and somewhat susceptible to scam sites.

The two most successful misleading claims were advertising and governmentmisinformation. To study the impact of advertising claims, students were asked this question:“List three major innovations developed by Microsoft over the past ten years.” The term“major innovation” was left vague, as Microsoft’s innovative history is a widely debated

4

issue. There are many opinions on the topic, and we expected students overall to discuss atleast several.

However, 63% of students respondedthat Microsoft was responsible for many majorinnovations based on information from onlyone source. Almost all of these studentsimmediately went to the Microsoft website andused the Microsoft Museum Timeline thatdetails Microsoft’s achievements—or at least,what Microsoft claims to be its achievements.Only 12% checked several sources and made

- Figure 1 -some more complete argument about this. 22%

fell in between these two groups, receiving a score of 2. These results are intriguing in view ofrecent litigation against Microsoft that drew worldwide attention to its business practices andinnovative efforts. Yet almost two-thirds of students responded without a shadow of a doubtthat Microsoft was completely honest about its claims.

Government misinformationfollowed closely behind advertisingclaims. Students were asked: “Did the1999 Rambouillet Accords allowNATO to operate in all of Yugoslaviaor only in Kosovo?” The correctanswer—all of Yugoslavia—can befound in the actual document, though itis difficult to wade through the 82-pagepaper. The complete text can be found

- Figure 2 -online, but summaries and reviews are much more common. A frequently found summary isthe U.S. Department of State Bureau of European Affairs fact sheet released on March 1,1999, which implies that NATO presence is limited to Kosovo. [7]

5

62% of students said that NATO is limited to acting within Kosovo based on onesource, and many listed the State Department memo mentioned above as their only source.26% said the same thing but made some effort to double-check the information or happenedto find the right answer on the first try. Many students in this category stumbled on anti-NATO websites and reported that information without checking another less-biased source.Only 10% found the correct answer and verified it in two places.

Political lobby groups are another common source of misinformation or half-truths.Students were asked to evaluate a claim made by getoutraged.com. This website is the workof an anti-smoking lobby, though it is officially copyrighted by the Massachusetts Departmentof Public Health. Students were asked: “Getoutraged.com says that tobacco is responsiblefor 30% of all deaths in the 35-69 age group. Would you cite this information in aresearch paper?” This statistic, taken from a pamphlet called “Growing Up Tobacco Free,”is actually a projection made in 1992 on how many deaths tobacco will probably cause in the1990s, but getoutraged.com lists this as if it were proven fact. [3] The number of deaths isactually estimated to be closer to 20% by organizations such as the American Cancer Societyand the U.S. Center for Disease Control and Prevention. [6]

Despite this, 48% of students said that they notonly believed the statistic fromgetoutraged.com but that they wouldconfidently cite it in a research paper. They didnot attempt to find a corroborating source. Only21% expressed reluctance to use thisinformation after checking with additionalsources, with 30% falling in between with ascore of 2. What is most disturbing is that many

of the students who readily believed this- Figure 3 -statistic realized that the site was probably the product of an anti-smoking lobby, but the factthat it was sponsored by the Massachusetts Department of Public Health reassured them.

6

Students seemed to believe that because a source was cited and the Massachusetts governmentcopyrighted the website, the statistic would naturally be accurate.

Fortunately, the results are not entirely dim. Students were much less susceptible tothe scam website. They were asked to evaluate vespro.com’s ‘revolutionary’ product VesproGHS containing Human Growth Hormone (hGH), an emerging medical treatment to combatthe effects of aging. According to the website, this product will decrease body fat, reducewrinkles, restore lost hair, and normalize blood pressure, among a variety of otherbenefits—an absolute miracle drug. This website provides quotes from medical journals thatare generally taken out of context to support its claims. For instance, there is a quote from a19 article in the New England Journal of Medicine that seems to support the beneficialeffects of hGH, though the conclusion of this article simply states that further research isnecessary. [4]

Students were asked: “Wouldyou recommend Vespro Life Science’shGH product to a friend concernedabout getting older?” Only 13% ofstudents immediately agreed torecommend this product, withoutconsulting another source. 35% ofstudents conducted further research andreported that they would not recommendthis product without more information.

- Figure 4 -52% of students received a score of 2. Though these results are not overly encouraging, theydemonstrate that students remain skeptical of this type of information on the internet.

The remaining two questions were used to determine students’ inclination to verifyinformation. Students were asked one easy and one hard question. The first question askedstudents to report the creator of Linux. The answer is easily found in minutes online. Thesecond asked students to find the land area of Lisbon, Portugal. While this sounds elementary,

7

it can take a tremendous amount of time to locate any answer on the internet, and even longerto find a second source. For the easy Linux question, 78% of students reported the firstanswer they found, without verifying it in another source. For the hard Lisbon question, 75%of students reported the first answer they found without double-checking. It appears thatstudents are just as likely to avoid verifying an answer, regardless of the time or effort neededto do so.

Finally, to determine which groups of students are more susceptible to misleadingclaims, responses to the personal information questionnaire were matched with answers to thesix survey questions. Using class year, we hoped to see if students became better internetresearchers over the course of their years at Wellesley. The results indicate that there was nosignificant difference in performance based on class year.

- Table 2 -We then looked at students’ self-reported confidence in their internet searchingabilities to determine if students who were more “internet-savvy” were better able to criticallyevaluate information on the internet. The categories available were very confident, fairlyconfident, slightly confident and not very confident. The following chart indicates the totalnumber of scores (0-3) given to students in each confidence level.

8

- Table 3 -Notice that the distribution of scores for all questions is very similar for eachconfidence level. Only the not very confident group shows notable, though not overly large,differences. This suggests that the confidence a student has in her abilities to search theinternet effectively does not significantly affect her performance.

Conclusions

Clearly, students consider the internet a primary source of information. The resultspresented here suggest that many students have trouble recognizing trustworthy sources,though perhaps the underlying problem is a lack of understanding of the internet as anunmonitored well of information. All future educational ventures must focus on teachingusers that the internet is unmonitored method of sharing information. Specifically, thisinstruction should equip users to use search engines effectively, and this requires anawareness of their technological and financial constraints. This is not to recommend teachingstudents that all search engines are devoid of useful information, but rather to promote a betterunderstanding of the actual service provided by search engines.

Students are also not consistently able to differentiate between advertising and fact.Many responses to vespro.com mentioned that as the website was just trying to sell a product,its claims could not be readily believed. However, many of these same students immediately

9

believed claims made by Microsoft on its commercial website. Students must understand thatall information on the internet is there for a reason, and it is vital to determine the purpose ofthe information when evaluating its accuracy.

The very small amount of students who double-checked information is also ofconcern. It is commonly believed that the triangle method—locating three independentsources that point to the same answer—produces the most accurate information. Thisapproach does not differentiate a great deal between “good” and “bad” sites, but ratherencourages users to double-check information regardless of the source. Students in this studyseemed to have a great deal of confidence in their abilities to distinguish the good sites fromthe bad. Colleges themselves often encourage this attitude as they determine “good” or“trustworthy” websites to help students begin internet research. While it is certainly useful toprovide guidance, it is equally important to promote the development of critical thinking skillsthat will allow students to make use of the entire internet, rather than a few “approved” sites.

Our findings also suggest that students across the board have similar difficulties incarefully evaluating information found on the internet. Older students with stronger traditionalresearch skills performed no better than other students, which suggests that these skills aresimply not sufficient when evaluating information on the internet. In the past, the greatestproblem facing researchers was finding information; now, with the advent of the internet, thegreatest problem is evaluating the vast wealth of information available. Students in thissurvey placed greater emphasis on the process of finding an answer than on analyzing theactual information. The difficulties students encountered suggest that this practice is of littleuse in determining the accuracy of online information. It is therefore important to developspecific research practices for internet searches that take into account the structure andpurpose of the internet.

As students continue to view the internet as a primary source of information, without asignificant shift in training methods, this problem will only grow worse. It is vital that theybetter understand the nature of the internet and develop an instinctive inclination for verifying

10

all information. This will allow students to take advantage of the tremendous benefitsprovided by the internet without falling prey to the pitfalls of online research.

References

1. Campbell, Jerry. “The Case for Creating a Scholars Portal to the Web: A White Paper.”

Libraries and the Academy. 1.1, 2001.2. Introna, Lucas & Nissenbaum, Helen. “Defining the Web: The Politics of Search Engines.”

IEEE, 2000.3. Lynch, Barbara S. & Bonnie, Richard J., eds. “Growing Up Tobacco Free: Preventing

Nicotine Addiction in Children and Youths.” Washington, D.C.: National AcademyPress, 1994.

4. Salomon, F. et al. “The Effects of Treatment with Recombinant Human Growth Hormone

on Body Composition and Metabolism in Adults with Growth Hormone Deficiency.”New England Journal of Medicine. 321:26, 28 December 19.5. Weiler, Angela. “Two-Year College Freshmen and the Internet: Do they really ‘know allthat stuff?’” Libraries and the Academy. 1.2, 2001.6. “Cigarette Smoking Related Mortality.” Centers for Disease Control and Prevention.

United States, 1990. Morbidity and Mortality Weekly Report 1993; 42 (33): 5-8;http://www.cdc.gov/tobacco/research_data/health_consequences/mortali.htm7. “Understanding the Rambouillet Accords.” Fact sheet released by the Bureau of European

Affairs, U.S. Department of State, Washington, D.C., 1 March 1999;http://www.state.gov/www/regions/eur/fs_990301_rambouillet.html8. See, for example, “Defining the Web: The Politics of Search Engines” (Introna &

Nissenbaum, IEEE, 2000), “Information Retrieval on the World Wide Web”(Gudivada et al., IEEE, 1997), and “Searching the World Wide Web” (Knoblock,IEEE Expert, January-February 1997).11