Subject: File: "KIDPLAN RESRCH01" KIDCAFE: a Global Dialog in ACTION Copyright (c) 1993 Daniel D. Wheeler All rights reserved. This file contains a series of messages that I posted to KIDPLAN during 1993 May and June reporting on research studying KIDCAFE participants. << Daniel D. Wheeler Internet: Dan.Wheeler@UC.Edu >> << University of Cincinnati Bitnet: wheeler@ucbeh >> =========================< attachment >========================= Date: Wed, 19 May 1993 09:21:05 -0400 (EDT) Subject: KIDCAFE Research Part 1: Introduction I'm now teaching an introductory course on research methods for graduate students in the College of Education. I believe that the best way to teach research methods is to have the students actually do a simple research project. This quarter I have had the students study participation in KIDCAFE. I've divided the class into four groups. Each group selected their own topic for the research. But we drew only one sample; all the groups used this same sample. I also added some questions that I thought would be useful for us. This class meets once a week for the quarter. Since the first was disrupted by a strike, there are only nine meetings of the class this quarter. It is really pushing to get a project like this done in nine weeks. The students go from thinking up a research topic to a finished paper in the style required for submission to an academic journal in this short period of time. The results are now coming in. I'm planning to write a series of brief reports to KIDPLAN over the next couple weeks. This is the first of the series. ================================================================= Date: Wed, 19 May 1993 09:41:54 -0400 (EDT) Subject: KIDCAFE Research Part 2: The Sample I've had my research class study KIDCAFE messages in previous quarters. What we are doing this quarter is different in a very important way: we are focusing on a sample of *kids* rather than looking at the individual messages. The study covered the activity in KIDCAFE for the first three months of 1993. But the sample of kids was drawn from those who participated in the month of January. Thus we could follow their activity in KIDCAFE for two months after a time when they were definitely known to have sent a message to KIDCAFE. There were 700 messages in KIDCAFE during the month of January. A database of these messages was created with information about the sender and the intended recipient of the message. There were 449 messages that were addressed to specific recipients and 224 that were "to anyone." Only 27 messages did not fit in one of these two categories. Counting the number of different participants was complicated by the fact that in some sites kids worked together in pair or even larger groups. For the purposes of this study, it seemed reasonable to count a pair of students who worked together consistently as a single participant. But some kids sent individual messages *and* were part of a pair or small group. In the count of participants, the pair or small group was not considered to be a separate participant if the kids had sent individual messages. There were 438 different participants in KIDCAFE during January, 388 individuals, 45 pairs, 4 triples, and one group of four. A true random sample of 40 kids was drawn from the list of 388 individual participants. This is not a large group to study, but it is large enough to give us a general picture of what KIDCAFE looks like from the perspective of individual kids. The fact that this is a random sample makes it possible to generalize to the population from which it was drawn. In the next installment of this report, I will describe how we searched for all messages from these 40 kids and for all the messages that were written *to* these kids. ================================================================ Date: Wed, 19 May 1993 11:37:11 -0400 (EDT) Subject: KIDCAFE Research Part 3: The Search for Messages Once the sample of 40 children had been selected, the next step was to search for messages in KIDCAFE during the period January through March that were: (1) from one of the 40 kids, or (2) intended for one of the 40 kids. It was relatively easy to find all of the messages from a sample child. Most kids sign at least their first name and they usually spell it consistently. The combination of first name and email address was enough to unambiguously identify the messages from the child. Identifying messages to the child was much more difficult. Some messages in KIDCAFE carry no identification of the intended recipient. There are also a lot of kids with the same first name in KIDCAFE, so it was often difficult to tell whether a message was for a child in the sample or for another child with the same name. Kids are also not very consistent in their spelling of names. Forty LISTSERVer database searches were performed, one for each child in the sample. The search was made by looking for all messages in KIDCAFE for the three month period containing substrings from the child's first and last name. For instance, if "Daniel Wheeler" were a child in the sample, the search might have been made for all messages containing DAN or IEL or WHEEL. In addition to messages containing DANIEL, this would pick up messages with the misspelled forms DANIL, WHEELR, etc. Each student in my research class examined the output of two database searches. Each message was categorized as being (1) from the sample child, (2) intended for the sample child, or (3) not relevant--neither from nor to the sample child. Since the students only had two of these to do, they could work carefully. In cases where there were more than one child with the same name, they had all the other messages, so they were often able to use the context to tell which child of the same name a message was intended for. I was satisfied with the adequacy of the identification of the messages. It was not perfect, but the kids themselves are often confused about whether a message was intended for them or for another child with the same name. The end result of the search process was that we had forty message sets. Each set showed the activity of a single child in KIDCAFE for the three month period. The rest of the reseach was done by analyzing these message sets. ================================================================= Date: Thu, 20 May 1993 10:03:32 -0400 (EDT) Subject: KIDCAFE Research Part 4: Messages sent and received This is the first report of the results of the study of 40 randomly selected January KIDCAFE participants. There was a wide range in the number of messages sent by the sample kids during the three month period of January - March. The highest number was 36, more than twice the number sent by the second highest participant. The full distribution in shown in this table: Number Cumulative Cumulative Msg Sent Frequency Percent Frequency Percent ------------------------------------------------ 1 18 45.0 18 45.0 2 6 15.0 24 60.0 3 5 12.5 29 72.5 4 3 7.5 32 80.0 5 1 2.5 33 82.5 6 1 2.5 34 85.0 7 3 7.5 37 92.5 11 1 2.5 38 95.0 17 1 2.5 39 97.5 36 1 2.5 40 100.0 Eighteen of the students sent only a single message to KIDCAFE. There was no further correspondence from them during the three month period. This group represents 45 percent of the sample. The number of messages received shows a similar distribution: Number Cumulative Cumulative Msg Rcvd Frequency Percent Frequency Percent ------------------------------------------------ 0 15 37.5 15 37.5 1 10 25.0 25 62.5 2 3 7.5 28 70.0 3 1 2.5 29 72.5 4 5 12.5 34 85.0 5 2 5.0 36 90.0 6 1 2.5 37 92.5 7 1 2.5 38 95.0 8 1 2.5 39 97.5 19 1 2.5 40 100.0 The highest number of messages received was 19. Fifteen of the students never received any reply at all to the message(s) they sent. The table below shows the relationship between the number of messages sent and the number of messages received. The students who sent (or received) 4 or more messages have been combined into a single row (or column). Each cell in the table shows the number of students and the percentage of the total sample of 40. Messages | Messages Received Sent | 0 | 1 | 2 | 3 | >3 | Total ---------+--------+--------+--------+--------+--------+ 1 | 10 | 5 | 1 | 0 | 2 | 18 | 25.00 | 12.50 | 2.50 | 0.00 | 5.00 | 45.00 ---------+--------+--------+--------+--------+--------+ 2 | 2 | 2 | 0 | 1 | 1 | 6 | 5.00 | 5.00 | 0.00 | 2.50 | 2.50 | 15.00 ---------+--------+--------+--------+--------+--------+ 3 | 1 | 2 | 0 | 0 | 2 | 5 | 2.50 | 5.00 | 0.00 | 0.00 | 5.00 | 12.50 ---------+--------+--------+--------+--------+--------+ >3 | 2 | 1 | 2 | 0 | 6 | 11 | 5.00 | 2.50 | 5.00 | 0.00 | 15.00 | 27.50 ---------+--------+--------+--------+--------+--------+ Total 15 10 3 1 11 40 37.50 25.00 7.50 2.50 27.50 100.00 My gut feeling is that we cannot claim to have created a dialog unless students have sent and received at least one message a month. For this study covering three months, that means three or more message. In the table above, these are the rightmost two columns and the bottom two rows. By this standard, only 8 students in our sample (20 percent) are involved in a dialog. These figures do not take into account the messages that are sent by private email. We know that at least one of the students was receiving messages by private email because the messages sent by the child referred to the messages that had been received. I'd really like to get your reactions to these figures. ================================================================= Date: Sat, 12 Jun 1993 21:45:03 -0400 (EDT) Subject: KIDCAFE Research Part 5: Answers to 4 Questions About 3 weeks ago I posted a series of 4 messages reporting the research that my students have done on KIDCAFE participants. I'm sorry there has been a delay, but I am now continuing the series. The study is based on a true random sample of 40 participants in KIDCAFE during the month of January, 1993. All messages to and from this sample of 40 during the three month period of January - March were retrieved for the study. One of the requirements for all KIDCAFE participants is that they send their answers to the four KIDLINK questions to the RESPONSE list. We had hoped to relate the contents of these messages to the students participation in KIDCAFE. I made a thorough search of the RESPONSE archives going back to the very beginning of the list for the answers from the 40 students in the sample. Only 11 sets of answers were found. That is only 28 percent of the sample. We also examined the KIDCAFE messages to see if the students had sent their answers to KIDCAFE by mistake. We did find 3 additional sets of answers, bringing the total to 14. That is still only 35 percent of the sample. The KIDCAFE messages that we examined were only those that were sent during the study period of January through March. There may have been others that were sent to KIDCAFE by the sample children before the study period; but this seems unlikely to be a major factor. The set of 14 messages is not large enough for further study of the content of these messages. No attempt was made to relate the answers to any of the other aspects of the study. A compliance rate of under 50 percent would seem to indicate a fair amount of resistance to the requirement of answers to the four questions. I would appreciate your comments on this. What do you think is going on? ================================================================= Date: Sat, 12 Jun 1993 22:25:25 -0400 (EDT) Subject: KIDCAFE Research Part 6: Exchanges of Information I'm continuing the report of the research on 40 KIDCAFE participants done by students in my research methods class. The class was divided into four groups and each group studied a different aspect of the messages. This part of the report covers the work of two groups who were interested in the exchange of information among the KIDCAFE participants. Requests for Information ======================== Kids do use KIDCAFE to ask and answer questions. One group was interested in the extent to which kids ask questions and get them answered. They were also interested in whether asking questions made it more likely for their email message to be answered. First, it is necessary to make a distinction between requesting information and asking a question. The group was actually interested in requests for information, whether or not they were written in the syntactic form of questions. "I'd like to know more about your horse" is a request for information that is not in the form of a question. "Will you write to me?" is syntactically a question but serves semantically as an imperative requesting an action. There were a total of 153 messages sent by the 40 sample children during the study period. Each message was categorized by the number of requests for information it contained. The replies received by each child were also examined to see whether each message received a reply. Here are the results: Table 1. Messages categorized by number of requests Number Percent of Percent Requests Messages total msg Replies Replies ======== ======== ========== ======= ======= 0 37 24.2 5 13.5 1-2 80 52.3 25 31.3 3 or more 36 23.5 11 30.6 ---- ------ --- 153 100.0 41 26.8 It seems clear from this table that the students are making a lot of requests for information in KIDCAFE. About three-quarters of the messages contain at least one request for information. It is also apparent that messages containing requests for information were more likely to receive a reply. The percentage of replies to messages not containing requests for information was less than half of the others. Exchange of Geographic and Cultural Information =============================================== Many teachers may be interested in having their students participate in KIDCAFE because they hope that the students will learn about other regions of the world. Another of the groups in the research class investigated the extent to which kids exchanged geographic and cultural information. By geographic information they meant any information about the country or region in which the child lives. By cultural information they meant anything about the people living in that region. These can be distinguished from individual information. If a child writes "I like to eat pizza," that is information about that individual child. But if the child writes "Most kids here like pizza," it counts as cultural information about the people in that area. This group analyzed the messages on a per participant basis rather than on a per message basis. Of the 40 students in the sample, 25 (63%) either gave or asked for geographic or cultural information. More of them gave information than asked for it: 55% vs 30%. A more detailed analysis was performed by rating messages on a three-point scale: 0 = no information, 1 = one item of information, and 2 = more than one item. Ratings on this scale confirmed that students gave more information than they asked for. Gender differences are shown in this table: Table 2. Mean message ratings on 2-point scale Ask Info Give Info --------- --------- Male 0.092 0.375 Female 0.072 0.185 It does appear that males give more cultural/geographic information than females, but a significance test showed that this difference is not large enough to be considered reliable. The picture that emerges from this is that kids spontaneously give a fair amount of cultural and geographic information in their messages. They are less likely, however, to ask for specific information. We know from the earlier results that kids are asking a lot of questions in KIDCAFE, but these are primarily seeking individual information about the other kids. ================================================================= Date: Sun, 13 Jun 1993 11:02:40 -0400 (EDT) Subject: KIDCAFE Research Part 7: Content Analysis Two of the groups in my research methods class did content analyses of the KIDCAFE messages. One group focused on sports while the other looked at the presentation of information about physical appearance. Sports ====== Sports activities are certainly a large part of the lives of many kids, both as participants and as spectators. Each set of messages from a child in the sample of 40 was examined to see whether the child reported participating in sports or being a spectator. Over half of the kids (23 kids or 58%) mentioned participating in sports. For most of these kids, participating in sports seemed to be a regular part of their lives. There did seem to be a gender difference: 41% percent of the girls and 80% of the boys were sports participants. The most popular sport was basketball, followed by football, swimming, tennis, soccer, and volleyball. The popularity of basketball probably reflects the fact that the sample was collected during the basketball season. Many fewer of the kids reported being spectators: only 5 out of the sample of 40 (12.5%). This should *not* be taken as indicating that kids do not watch sports; it only means that they do not write about it in KIDCAFE. Physical Appearance =================== One of the characteristics of email is that it hides the physical appearance of the participants--information that is often important in establishing face-to-face relationships. Do kids provide some of this information in their email messages? Each set of messages was examined for the presence of information about physical appearance. Fewer than half (18/40 or 45%) gave any information in any of their messages. There was little difference between the genders except that more boys reported their height (7/20 or 35%) than girls (2/17 or 12%). Even among the messages that did mention physical appearance, the amount of information was small. Physical characteristics do not seem to be a large part of the presentation of self in KIDCAFE. ================================================================= Date: Sun, 13 Jun 1993 13:35:18 -0400 (EDT) Subject: KIDCAFE Research Part 8: Caveats and Conclusions This is the last message in the series reporting the research on KIDCAFE participants conducted by my research methods class. First, I'd like to say something about the strengths and limitations of this research. The major strengths of this study are that it focused on *participants* rather than on messages and that it was based on a true random sample. Successful participants in KIDCAFE send more messages than unsuccessful participants. Thus if we look at messages in KIDCAFE, most of them will be from successful participants. We will not be able to judge the extent to which kids are trying to join KIDCAFE and dropping out because they are not successful. Each of us (especially the teachers) is close to a particular group of students active in KIDCAFE. Our impression of how well KIDCAFE is functioning is based largely on that group of students. But I can see that there are big differences among the sites, both in their level of participation and in their expectations for how it will work. A true random sample of participants gives us a picture of how it is working overall, without the bias that comes from each of our own group of contacts. The sample size of 40 is a limitation; I wish it could have been larger. But I feel that I am already exceeding the norms for the amount of work expected of students in a 3-credit course; I did not feel comfortable asking the students to do more. How accurate are percentages based on a sample of 40? Obviously, a change of just one person in 40 changes the percentage by 2.5%. The standard deviation of a midrange percentage based on a sample of 40 is 8 percent. This means that you should consider the percentages I've reported as "probably" within 8 percent and "certainly" within 16 percent. I should also caution you against assuming causal relationships. For instance, I have reported that messages asking for information were more likely to get a reply. But we *cannot* assume that individual students can make it more likely that they will get answers by including questions. It may be that the kids in "exotic" countries are the ones who have been asking for information. They may get more replies, not because they ask questions, but because more kids are interested in writing to exotic countries. [By "exotic" countries, I mean those outside of the Italian/Scandinavian/US group of heavy participants.] Conclusions =========== I'm quite pleased by the overall sense I get of the activity in KIDCAFE. Kids are making friends and getting to know each other. Although I would be happier with more exchanges of cultural and geographic information, I do think the current level is fine. It is perfectly reasonable for kids to be more interested in finding out about their new friends as individuals than they are in the more general information. In addition, the dividing line between individual and cultural information is very fuzzy. Kids can learn a lot about the culture by getting to know some individuals within that culture. I do have two major concerns that have emerged from this research. These are (1) the low success rate of KIDCAFE participants, and (2) the failure of many participants to answer the four KIDLINK questions. In message 4 of this series, I said that I did not consider that we had created a dialog unless the person send and received at least 3 messages during the 3 month period. Only 20% of the sample (8 kids) are participating in a dialog according to this standard. Another way to look at it is that kids who send only one message or who never get any answers are clearly not successful. Over half of the students (23/40 or 58%) are not successful. There are two factors that may make these figures misleading. The first is that kids may switch to private email after the initial contact in KIDCAFE. More of the kids may be successful, but we just don't know about it. I know that this factor operates to some extent, and this reduces my concern. The other factor operates in the opposite direction. The successful kids are the same ones from month to month, but each month we get a new group of kids who try to make a go of it in KIDCAFE and fail. The success rate of kids trying to join KIDCAFE might be as low as 5 percent. Here is an example to show how that might be true: Suppose that each month 400 kids try to join KIDCAFE. Only 20 of them succeed (that is 5 percent). The rest (380 of them) are not successful and they quit trying--we don't hear from them again. If this happens each month starting in October, by January we'll have a group of about 80 active participants. We will also have the 380 kids who attempt to join KIDCAFE in January but who fail. These hypothetical figures are not far from what we found. After I posted the preliminary report on these findings, there were reports in KIDPLAN from teachers confirming both that private email is a significant factor and that the success rate for their students had been disappointingly low. My own conclusion is that we need to do something to make KIDCAFE function better. Your suggestions would be appreciated. The second concern that emerged from this study was the low rate of answering the 4 KIDLINK questions. I have some ideas about what we could do about this, but they go beyond the scope of this report on KIDCAFE messages. I'll write about them in a separate message. =========================< end of file >=========================