Improvement of panish Language kills and Intercultural Competence During tudy Abroad Assessment Projects in Longwood University s General Education ummer Abroad Dr. Lily Anne Goetz, Professor of panish Longwood Assessment Conference 2014 tudy Abroad The number of U students who study abroad has more than tripled in the past 20 years 60% study abroad for 8 weeks or less ( shortterm ) There has been little formal research regarding learning outcomes in short-term study abroad 1
Longwood University: General Education Abroad Program Four week interdisciplinary CLAC immersion program in Valencia, pain tudents choose two courses from: History, intermediate panish, or advanced writing seminar Development of language proficiency and intercultural competency Coordinated syllabi, visits to local sites, guest lectures by experts in Valencia Reading, speaking, listening, and writing in panish required in History and writing courses (as well as in panish courses) Goals of the General Education ummer Abroad Program Meet General Education Goals through Developing intercultural competency by sensitizing students to cultural differences through real-world interaction with paniards Developing critical thinking skills Engaging students in historical inquiry Engaging students as active learners through experiential learning in an immersion setting Increase numbers of students studying abroad 2
Interdisciplinary Assignments Meaningful use of language: Daily journal: Report, React, Analyze Question of the day: Ask at least 2 paniards tudent research & reporting Individually Chosen Cultural Activities and Reports Conversation partners with locals Interaction with locals in volunteer organizations, schools, community groups Required reading: newspapers, signs, web materials Inquiry into issues of importance to locals Intercultural Competence A learning process in which students gain awareness of a different culture and become sensitive to that culture; The ability to understand how people from another culture think, and to communicate and work effectively with them. 3
Assessing Oral Proficiency Development in a hort-term Program Abroad Our Research Question: Do (our) students who take panish 201 abroad for 4 weeks in an interdisciplinary immersion setting show greater improvement in interpersonal speaking skills than students taking the same course for 4 weeks on campus? Methods Pre- and post-program surveys of studentreported abilities: functions and tasks Pre- and post-program assessments of oral proficiency: Vocabulary Grammar Meaningful communication 4
urvey of General Language Functions Defend opinion; hypothesize (superior) Narrate (advanced) Ask & answer questions (intermediate) List (novice) General Language Functions urveys pecific Language Tasks Given your current level of panish proficiency, which functions below can you perform when communicating in panish (please check all that apply): List things (novice level) Ask and answer questions (intermediate level) Make simple statements (intermediate level) Narrate an event in the present or past (advanced level) Defend an opinion; argue a point (superior level) Hypothesize (if this were the case, I would ) (superior level) When you are in pain, it is very likely that your host family will not speak English. In conversation with them, will you be able to (check all that apply): Tell them about my family (novice: lists) Tell them about what I eat at home for dinner (novice level) Ask them about how to get to a place where I need to go (intermediate level) Answer questions about what I did today (novice and intermediate level) Tell them about my happiest moment or other experience in the past (advanced level) Tell them about my future plans, such as what I plan to do when I graduate (advanced level) Explain my views on a political or social issue (advanced/superior levels) Explain under what circumstances I would do something (superior level) Convince them that the death penalty (or some other important issue) should or should not be legal in pain (superior level) 5
urveys of General Language Functions Given your current level of panish proficiency, which functions below can you perform when communicating in panish (check all that apply): Function Pre-Program % (N=24) List things 95.8 75 Ask & answer questions 83.3 94.4 Make simple statements 91.7 94.4 Narrate an event, pres. or past 66.7 55.6 Defend an opinion; argue a point 25 38.9 Hypothesize (if this were the case, I would ) 12.5 22.2 Post-Program % (N=36) urveys of pecific Language Tasks When you are in pain it is very likely that your host family will not speak English. In conversation with them, will you be able to (check all that apply): Function/Task Pre-Program % (N=24) Tell about my family (List things) 91.7 92.7 Tell them what I eat at home for dinner (List things) 91.7 75 Ask how to get to a place I need to go (Ask & answer) 83.3 94.4 Answer questions about what I did today (imple 87.5 94.4 statements) Tell about my happiest moment or other experience 50 38.9 (Narrate an event, present or past) Tell about future plans (Narrate an event in the future) 54.2 55.5 Post-Program % (N=36) 6
% of N (Respondents) urveys of pecific Language Tasks (continued) When you are in pain it is very likely that your host family will not speak English. In conversation with them, will you be able to (check all that apply): Function/Task Pre-Program % (N=24) Post-Program % (N=36) Explain my views on a political or social issue (Defend an opinion; argue a point) Convince them that the death penalty (*or some other important issue) should or should not be legal in pain (Defend an opinion; argue a point) 20.8 30.6 16.7 2.8 Explain under what circumstances I would do something unexpected (Hypothesize) 16.7 19.4 tudent Perceptions My foreign language skills were useful in completing the assignments in: 50 45 40 35 30 25 20 trongly Agree Agree Disagree trongly Disagree 15 10 5 0 History (N = 30) English (N = 19) 7
Perception vs. Reality: Oral Proficiency Assessments Pre-program assessments given to students before going abroad and to students about to take an on-campus summer class Post-program assessments given to students during last week of study abroad and to on-campus students before final exam Both classes took place during summer of 2013 for 4 weeks Oral Proficiency Assessment Prompt You are a strong proponent of protecting the environment, and you are trying to convince your friend to become more conscious of the effects of his/her lifestyle on the environment. You walk everywhere, use no perfumed products, have 5 recycling bins in your room, attend protest marches at local factories, and do a lot of other things to help conserve the environment. Your friend is entirely oblivious to the importance of preserving the environment, and commits all kinds of atrocities. Explain what you used to do (imperfect) before you became conscious of the need to be more friendly to the environment, cite any really memorable events that you participated in (pretérito) and tell what you do now. Try to convince your friend that he/she can be more helpful. uggest, insist, advise, be emotional (subjunctive). Vocabulario útil: Environment: medio ambiente Factory: fábrica. Cognados: Productos perfumados / Reciclar / Protestar / Proteger / Preservar / Cometer/ Conservar 8
Oral Proficiency Assessments Included in the prompt: You will be given about 4 minutes to read the prompt and think about what you want to say. Think about the vocabulary you will need, verb tenses, and other grammar. You will NOT write anything, but rather brainstorm ideas for the topic. When speaking, the main "rule" is that you must not use any English in your recording. Your panish can have mistakes, and you can stick to basic vocabulary and grammar that you already know, but just do not use English. If you can't think of a word, explain around it in panish (circumlocute). Do not worry about being perfect, but try to include as much information as possible. Try to use good pronunciation, vocabulary, and grammar. Oral Proficiency Assessment Prompt: You are a strong proponent of protecting the environment, and you are trying to convince your friend to become more conscious of the effects of his/her lifestyle on the environment. You walk everywhere, use no perfumed products, have 5 recycling bins in your room, attend protest marches at local factories, and do a lot of other things to help conserve the environment. Your friend is entirely oblivious to the importance of preserving the environment, and commits all kinds of atrocities. Explain what you used to do (imperfect) before you became conscious of the need to be more friendly to the environment, cite any really memorable events that you participated in (pretérito) and tell what you do now. Try to convince your friend that he/she can be more helpful. uggest, insist, advise, be emotional (subjunctive). Vocabulario útil: Environment: medio ambiente Factory: fábrica. Cognados: Productos perfumados / Reciclar / Protestar / Proteger / Preservar / Cometer/ Conservar 9
Rubric: Fluency Fluency (20%) 4 3 2 peech is natural and continuous; no unnatural pauses. peech seems spontaneous, flows naturally and easily. ome definite stumbling, but manages to rephrase or continue. peech halting and fragmentary; long, unnatural pauses. tudent is obviously reading prepared material. Rubric: Vocabulary Vocabulary (20%) 4 Rich and extensive vocabulary, appropriate to situation; very accurate usage; good circumlocution. 3 Often lacks needed words; some inaccurate usage; some use of English. 2 Lacks basic words and expressions; inadequate or inaccurate usage hampers communication. 10
Rubric: Pronunciation Pronunciation (20%) 4 Near-native pronunciation; little or no interference of English sounds. 3 Interference of English pronunciation is pervasive, but utterances are mostly comprehensible; communication may be impeded to some extent. 2 Entirely or almost entirely incomprehensible to a native speaker of panish who is not used to interacting with non-native speakers. Communication is impeded or impossible. (More practice is needed on circled sounds): [p] [t] [k] [β] [ð] [r] [г] Rubric: tructure/grammar tructure/grammar (20%) 4 Targeted grammatical structures almost always correct. A few insignificant errors. 3 Frequent errors in targeted structures. Communication is impeded but not severely. 2 Grammatical errors render speech incomprehensible. Communication is severely impeded. (More practice is needed on circled items): Verb tenses: ser-estar / por-para Conjug. of Preterite Preterite-Imperfect Conjug. of ubjunctive direct, ind. obj. pron Use of ubjunctive: Present tense adj. agreement influencing emotion ubject-verb agreement impersonal expr. adjective clauses doubt/disbelief 11
Rubric: Meaningful Communication Meaningful Communication (20%) (length of recording: mins., secs.) 4 ubstance of conversation is of good quality, addresses the required task with an abundance of information, arouses interest in the listener, and may provoke response. hows evidence of a command of the topics and contexts involved. Meaningful communication takes place. 3 Communication may be misunderstood at times; contains some evidence of contextual knowledge but information is lacking or incomplete, or task is not adequately addressed; negotiation of meaning is not always successful. 2 hows little or no evidence of contextual knowledge and/or task is not addressed. Negotiation of meaning does not occur and communication is non-existent; meaningful conversation does not take place. coring corers: Two panish instructors not teaching the on-campus class or the abroad group Norming process: Author of the study and the two other panish instructors met to score several of the recordings using the rubric developed by the author. Before listening to recordings, the group discussed scoring criteria in each category to arrive at consistent ideas. Each instructor then scored every student in both groups, and both scores for each student were recorded. cores in the following charts are shown for both instructors; example: 2/3 means that one instructor scored the student s skill as a 2 and the other gave a score of 3. 12
Fluency Vocabulary Pronunciation Grammar Communicatio n Total core Length Pre-Assessment Avg Fluency Vocabulary Pronunciation Grammar Communicatio n Total core Length Post-Assessment Avg Fluency Vocabulary Pronunciation Grammar Communication Total core Length Pre-Assessment Avg Fluency Vocabulary Pronunciation Grammar Communication Total core Length Post-Assessment Avg Assessments for tudents Abroad in panish 201 Pre-Program Assessment Post-Program Assessment tudent 201 tudent A 2/3 2/2 3/2 2/2 3/3 12/12 1-40s 2.4 3/3 3/3 3/3 3/3 3/3 15/15 2-24s 3 tudent B 2/2 2/2 3/2 2/3 2/2 11/11 2-30s 2.2 4/3 4/3 3/3 3/3 4/3 18/15 1-53s 3.3 tudent C 3/3 3/2 3/3 3/2 2/2 14/12 0-19s 2.6 3/3 3/3 3/3 3/3 3/3 15/15 2-06s 3 tudent D 2/2 2/2 2/2 3/2 2/2 11/10 1-18* 2.1 3 3 3 3 3 15 2-01s 3 tudent E 2/2 3/3 3/2 2/2 3/2 13/11 1-22s 2.4 3/2 3/2.5 3/2 2/2 3/3 14/12 1-18s 2.55 tudent F 2/2 2/2 2/2 2/3 2/3 10/12 2-50s 2.2 3 3 3 3 3 15 1-3s 3 tudent G 2/2 2/2 2/2 2/2 2/2 10/10 2-10* 2 3/2 3/2 2/2 2/3 3/3 13/12 2-07s 2.5 tudent H 2/2 2/2 2/2 2/2 2/2 10/10 1-37s 2 2/2 2/2.5 2/2 2/2.5 2/3 10/12 2-10s 2.2 tudent I 2/2 2/3 2/2 2/2 2/3 10/12 2-58s 2.2 2/3 3/3 2/2 2/3 2/3 11/14 1-26s 2.5 tudent J 3/3 3/3 3/2 3/3 3/2 15/13 0-36s 2.8 3/2 3/2 2/2 2/2 2/3 12/11 1-0s 2.3 Avg for 201 2.25 2.3 2.3 2.3 2.3 2.29 2.72 2.8 2.5 2.58 2.88 2.7 +.4 diff Assessments for tudents On Campus in panish 201 Pre-Program Assessment Post-Program Assessment tudent 201 tudent 1 2/2 2/2 2/2 2/2 2/2 10/10 4-12s 2 3/2 3/2.5 2/3 2/2.5 2/3 12/13 3-31s 2.5 tudent 2 /2 /2 /2 /2 /2.5 /10.5 1-54s 2.1 2/2 2/2 2/2 2/2.5 2/3 10/11.5 2-7s 2.15 tudent 3 2/2 2/2 2/2 2/2 2/2 10/10 5-37s 2 2/2 2/2 2/2 2/2 2/2.5 10/10.5 4-18s 2.05 tudent 4 3/2.5 2/2.5 3/2 3/2.5 3/3 16/12.5 3-57s 2.65 3/3 4/2.5 3/2.5 3/3 4/3 17/14 3-15s 3.1 tudent 5 2/2 2/2 2/2 2/2 2/2 10/10 4-9s 2 2/2 2/2 2/2 2/2 2/2.5 10/10.5 4-10s 2.05 tudent 6 3/2.5 3/2.5 3/3 3/2.5 3/3 15/13.5 2-6s 2.85 3/3 3/3.5 3/3 3/3 3/3.5 15/16 2-18s 3.1 tudent 7 2/2 3/2 2/2.5 2/2 2/2 11/10.5 2-41s 2.15 2/3 2/3 2/2.5 3/3 3/3 12/14.5 3-40s 2.65 tudent 8 2/2 2/2 2/3 2/3 2/2.5 10/12.5 3-42s 2.25 2/3 2.5/3 3/2.5 3/2.5 2.5/3 13/14 2-38s 2.7 tudent 9 3/2 3/2.5 3/3 3/3 3/2.5 15/13 1-17s 2.8 2/2 2/2 3/2.5 3/3 2/2.5 12/12 1-11s 2.4 tudent 10 3/2 3/2.5 3/2.5 3/2 3/3 15/12 1-45s 2.7 2/2.5 2/3 3/2 3/3 2.5/3 12.5/13.5 3-3s 2.6 tudent 11 2/2.5 3/2.5 2/2 2/2 2/2.5 11/11.5 4-16 2.25 3/2.5 3/3 3/2 3/3 3/3 15/13.5 3-34s 2.85 tudent 12 3/2.5 2/2.5 3/3 2/2.5 2/3 12/13.5 3-50s 2.55 /2 /2.5 /2.5 /2 /2.5 /11.5 2-23s 2.3 tudent 13 2/3 2/3 2/3.5 3/3 3/3 12/15.5 2-10s 2.75 3/3 3/3 3/3 3/3 3/3 15/15 4-40s 3 Avg for 201 2.31 2.35 2.44 2.37 2.48 2.39 2.39 2.55 2.48 2.64 2.75 2.57 +.18 diff 13
Comparisons of Assessments for tudents Abroad and On Campus Abroad group improved more: +.4 diff compared to +.18 diff. Abroad group ended up scoring slightly higher in overall speaking skills even though they began with a slightly lower average of speaking skills. Implications for Future tudy Abroad Include assessment of writing skills Increase the focus on pronunciation in abroad program tudy the effect of student GPAs on development of language skills Consider modification of rubrics and scoring procedure Develop pre- and post-program assessments for intercultural competence 14
Benefits of an Interdisciplinary tudy Abroad Program Outcomes: Positive affective and cognitive engagement Development of self confidence and independence Development of language proficiency Development of intercultural competence Longwood University Academic trategic Plan http://www.longwood.edu/staff/goetzla/clac/clacacadtratplan.pdf Increase number of students in experiential learning initiatives Investigate a cultural competency requirement Facilitate development of interdisciplinary courses and programs Evaluate the current status of study away and study abroad experiences; make improvements Expand the development of faculty and staff international exchange programs 15
Thank you! Gracias! Questions? Comments? Have you conducted a similar study? I would love to hear from you! Dr. Lily Anne Goetz goetzla@longwood.edu 16