An overview of Triangulation and its effect in Language Assessment

 Dr. Farnaz Sahebkheir

Department of English Language Teaching, Tabriz Branch, Islamic Azad University, Tabriz, Iran

Abstract

Triangulation is defined as using more than one method of data collection and analysis when studying a social phenomenon so as to seek convergence and corroboration between the results obtained from different methods, therefore, eliminating the bias inherent in the use of a single method (Denzin, 2012). Moreover, he believes triangulation originally refers to the use of multiple forms of qualitative research methods, not the combination of quantitative and qualitative methods. Triangulation is a tool or a strategy for validation; triangulation is an alternative to validation, reflecting and it is an attempt to secure understanding of the phenomenon which was under study. Through triangulation of qualitative and quantitative research methods one can reach a complementarity. Since different data types and analysis are appropriate for different research questions and processes. In this way, quantitative and qualitative results may be used to interpret different aspects of the phenomenon. The basic logic for complementarity relies on viewing social phenomena as multi-layered. This complementarity is best achieved by performing each method interactively/interdependently and concurrently, to focus on all possible complexity of the phenomena under study. Furthermore, it seems that teachers for achieving a correct criterion for assessing students’ performance should use mixed approaches method or triangulation of qualitative and quantitative data assessment for reaching a correct answer in assessing and strengthening the research.

Keywords: Language Assessment, Quantitative Methods, Qualitative Methods, Triangulation

(To Download PDF File of This Full Article in full Details, Click Here)

1. Introduction

Appropriate standard for assessment is very essential for analyzing students’ performance. Sometimes some assessments show high scores for students, on the other hand, the same students may fail in another standard assessment. So, it seems that teachers for achieving a correct criteria for assessing students’ performance should use mixed approaches method or triangulation of qualitative and quantitative data assessment for reaching a correct answer in assessing and strengthening the research.

1.1Classroom Assessment

Assessment involves gathering information about students’ performance to be sure of what they have learned. It’s done to gain evidence on student learning to be sure of improvement in students’ performance. (Smith, Teemant, & Pinnegar, 2004). There are two kinds of assessment: qualitative and quantitate assessments

1. Qualitative Methods

Qualitative methods are a good starting point when you begin your assessment. These methods can be useful for describing a situation and can provide insight for your intervention approach. It can include observation, management meeting for discussion, focus group, one and one interview, and expert consultations.

b. Quantitative Methods

Quantitative methods are useful in that they often take less time to administer than qualitative methods. They are also easier to evaluate and may produce clearer, more objective results. It includes Pre-existing Records, Self-report Surveys, and treatment.

1.2Multiple modes of assessment

Wiggins and McTighe (2005) mention that students ought to be assessed using three modes of assessment:

1. Performance task
2. Knowledge and skill
3. Criteria referenced assessment

In this view, students represent their learning by performing a task which show that they have mastered the content under study. They may also represent their learning through explanation, writing, or discussion to show their knowledge and skill of the content. Finally, the students may also demonstrate their learning through criterion-referenced assessments such as standardized assessments.

Smith, Teemant, and Pinnegar (2004) believe that teachers should gather data regarding student learning through classroom observation by observing what students do. They should also listen to what students say or write, so orally or in writing, in order to know what content or skill students have mastered. Finally, teachers need to examine students’ production that show what skills, knowledge, and understandings they have learned.

2. Literature review

2.1Triangulation

It’s a common teaching method in qualitative research which relies on multiple sources of data (Cresswell, 1998). According to Denzin (1979) there are four types of triangulation techniques for strengthening the qualitative research.

1. Data triangulation, in which the researcher uses a variety of sources to collect data.
2. Investigator triangulation, in which more than one researcher is investigating the phenomenon.
3. Theory triangulation, in which the researcher applies multiple theories, perspectives for interpreting the data.
4. Methodological triangulation, in which the researcher uses multiple methods to study the problem.

Triangulation is suggested in qualitative research to increase the trustworthiness of analysis for inclusive and complete narratives (Mason, 1994). It also reduces the bias and limitations of one methodology compensating with the strengths of other methods. In addition, using multiple methods, sources of data, theories or researchers and multiple methods which lead to the same results, increase the validity of the interpretation of the data. Moreover, such multiple perspectives add richness and new perspectives to the data collection (Brannon, 1992). In addition, not only triangulation increases the validity and data interpretation but also it is required in the study of language culture and complex phenomena like age, gender, etc.(Perlesz & Lindsay, 2003 ). Besides, combination of both qualitative and quantitative methods of inquiry can lead to the confirmation of the argument either through divergence(especially when multiple methods lead to similar conclusion, also when the results are not the same, it can open a pathway to a new theory and a new area for further exploration and research(Perlesz & Lindsay, 2003)

 The concept of triangulation was first initiated in the social sciences field when Campbell and Fiske published a paper in 1952 that discussed the application of a multi-method matrix procedure to assess the validity of measures and traits in the psychological repertoire (Campbell & Fiske, 1959). However, the use of this method as an assessment methodology is relatively recent and, therefore, has needed various types of scientific research and literature evaluation to affirm its validity.

The main features of triangulation in the assessment process are its utility as a method or tool to enhance the credibility of research work, eliminate bias, and to illustrate the differences between results to establish a valid well-reasoned proposition. As detailed by Mathison, the concept of triangulation is carried out through four different processes: a) data triangulation including time, space, and person; b) investigator triangulation; c) theory triangulation; and d) methodological triangulation (Mathison, 1988).

First, the data triangulation method consists of using several data sources to evaluate the same outcomes. Obviously, convergence to same outcomes establishes the validity of those outcomes. The data acquisition process may involve multiple human interventions in addition to spatial and temporal factors. This aspect addresses the multi sources conditions and effects of a single phenomenon. An example discussed by Mathison to illustrate this evaluation approach is if observations were made at different times of the day and at different times of the year to evaluate the learning outcomes in a school classroom setting (Mathison, 1988). If the outcomes were similar irrespective of the time and day, it could be concluded that the learning outcomes were valid.

In addition, the other form of triangulation used in research is investigator triangulation, which requires more than one investigator to be involved in the research process and fulfill the requirements of adequate data collection. This type of triangulation is subject to many questions regarding the choice of people designated to accomplish the investigation task and their assigned roles. In addition, it is essential to question the investigation process and to determine how much hands-on data collecting the principal investigator needs to do in order to analyze the data, and how much data analysis is relegated to field workers because much of the analysis occurs as data are collected (Mathison, 1988). On the other hand, the concept of triangulation theory refers to a simple, yet essential, component of studies and research assessment. The theoretical triangulation is nothing more than the statement of the necessary presence of theory perspectives in any performed study or research work (Turner & Turner, 2011).

Finally, methodological triangulation remains as one of the most prominent forms of triangulation and its value resides in the fact that it utilizes different methods in the evaluation of scientific statements, research and proposals. Several research articles and publications have emphasized and supported the effectiveness of this approach in the assessment process and the establishment of valid and accurate statements and results. Denzin, in his book the Research Act: A Theoretical Introduction to Sociological Methods highlights the benefits of using multi methodological triangulation by stating that The rationale for this strategy is that the flaws of one method are often the strengths of another: and by combining methods, observers can achieve the best of each while overcoming their unique deficiencies (Denzin ,1978).  The research substantiates that triangulation is a pertinent tool and strategy in the assessment and evaluation of research work. The value of triangulation resides in its effective methodologies, which permit the use of multiple data sources, measures and investigations throughout to cancel out the inherent bias and establish a convergent proposition (Mathison, 1988).

Under the strict ABET (Accreditation Board for Engineering and Technology) accreditation criteria; many engineering colleges and programs are seeking pertinent methods and tools to assess specific engineering disciplines outcomes continuously. Triangulation can be used as one of the best and most effective approaches in engineering programs and curriculum evaluation to provide multiple measures for a particular program and establish valid and reliable outcomes (Brent & Felder, 2003). For instance, the assessment of an engineering education learning application such as the ability to work in multi-disciplinary teams can be evaluated using the methodological triangulation approach through: 1) the student’s self-assessment of their enjoyment of working on teams via closed-form questionnaires; 2) ratings by a student’s peers on the team; or 3) the direct observation of a team by a trained evaluator. In this case, the triangulation process would enable the assessor to evaluate the accuracy of the methodologies chosen and the validity and accuracy of the outcome using that methodology. Once the results are obtained from the triangulation process, statistical methods may be used to investigate the relationship and patterns that exist among the measurements. Furthermore, after carrying out the statistical analysis and in the presence of strong correlation between variables and outcomes, the accuracy of the results can then be verified easily. Contrary to some beliefs, over-assessment using multiple triangulation measures is deemed unnecessary (Besterfield-Sacre, Shuman, & Wolfe, 2000). Engineering departments usually rely on at least two assessment tools (such as multi-source feedback systems and closed from surveys), to investigate the quality and outcomes of a specific program or curriculum. In engineering curriculum assessment, the multisource feedback experience is implemented in a classroom setting using students to provide evaluation of peers for team-based projects, in addition to inputs from faculty members to determine the students’ overall learning experience and interpersonal performance (Ghrayeb, Damodaran & Vohra, 2011). From historical trends, this approach has proven to be both an effective feedback tool and it has helped to improve engineering learning outcomes to a greater extent (Besterfield-Sacre, Shuman, & Wolfe, 2000). Besides, it is considered that closed form questionnaires serve as a classical method for obtaining feedback from individuals. This type of assessment source is used in engineering courses to evaluate the student’s perspective and attitude toward the various engineering educational aspects, as well as obtaining a self-assessment of individual abilities and competencies (Besterfield-Sacre, Shuman, & Wolfe, 2000).  In the above mentioned engineering program assessment, triangulation can be used to evaluate and compare the results of the two traits used in the process to find the method that gains more accurate results and consequently can be used to improve the engineering program outcomes by completing the feedback loop.

A non-traditional application of triangulation as an assessment tool may be used in some capstone design courses where unlike the conventional classroom environment, students are exposed to hands-on team work, through design projects in their specific fields of study assigned by academic and industrial partnerships.

In this case, the assessment process relies on the effective use of triangulation to evaluate the feedback and results obtained from three major sources (Turner & Turner, 2011).  First, from the industrial mentors assigned to each team to serve as an external source of evaluation of technical and engineering skills from a potential employer professional perspective; second, from the faculty advisors designated to work closely with small teams on sequential semesters of design project courses, and third, from the self-rating of the students. All three sources are critical in assessing the learning process and are used to determine student competence. In order to sustain coherence and validity of the results, the triangulation process may be used to assess outcomes from the several rating sources mentioned above to determine the learning outcomes associated with the capstone design course. Assessment of the capstone course is important as it establishes the claim that engineering students are exiting the curriculum with the skills that match program objectives (Knight, Kotys-Schwartz, & Pawlas, 2010). The data from the three sources may be used to converge to the final determination using the triangulation process.  In an implementation of the method, the students participating in the evaluation process were the ones enrolled in the course and were rated on technical knowledge, oral writing communicational skills, team work and project management skills. The analysis of the triangulation outcomes was processed in three different phases.

The first step involved the investigation of commonalities among the three different raters. The second phase of the triangulation results assessment resided in finding similarities between the open comments and additional explanations provided by raters from the surveys. Finally, the third step consisted of finding the inter-rating disagreements by looking up the differences in ratings. Eventually, statistical analysis was performed on the data using the ANOVA Procedure, which could test the ratings of one specific aspect of the design project and evaluate if the rating diverged significantly among feedback providers (Knight, Kotys-Schwartz, & Pawlas, 2010). The results from the triangulation method were found to be an efficient way to determine the deficiencies in the course layout and illustrate the specific areas where personal and technical skills for students were lacking. In addition, they provided valid and reliable proposals and recommendations for rebuilding the course in order to assure improvement in students’ skills and abilities, and achievement of academic objectives.

In most publications and articles, using triangulation as an assessment tool was considered a concept or tool to eliminate bias and improve the convergence of outcomes (Miles & Huberrnan, 1984). However, this approach is criticized by Mathison and categorized as being far from realistic. On a more practical stand as suggested by Mathison, triangulation provides evidence for the researcher to make sense of some social phenomenon, but that the triangulation strategy does not, in and of itself, do this (Mathison, 1988). The conventional approaches claim that the outcomes of a triangulation assessment would result solely in the convergence of results. In addition, metrics investigated through several methods and sources would all support one proposal. On the other hand, a new practical perspective of triangulation as an assessment strategy would expect there to be inconsistency and contradiction between results. The presence of inconsistency in outcomes, while using different methods for the purpose of triangulation assessment, is a somewhat naturally expected result. It is possible that different approaches will lead to inconsistent and ambiguous perspectives instead of supporting a single source of evidence. The third possible outcome of triangulation is to end up with contradictory data. Not only can inconsistency exist in results and provide ambiguous perspectives, but serious contradictions can also arise from using different methods. In some cases, triangulation may lead to contradictory statements, which may require further investigation to reach a reasonable conclusion. Inconsistencies and contradictions in the results should not affect the validity of the research or theory under question.  Restricting the outcomes of triangulation to the convergent outcomes approach lessens the potential and effectiveness of the research in progress. It is, therefore, deemed necessary to expect inconsistencies and contradictions to be present in the results of a triangulation assessment strategy. After all, triangulation is a tool to provide evidence and better explanations of social phenomena and not a mathematical process that allows room for only one solution as the ultimate truth (Turner & Turner, 2011).

2.2. Different types of Triangulation

The term triangulation itself is infrequently used in presence research, instances of explicit use predominantly occurring in studies with an emphasis on social presence. Most approaches to triangulation, whether or not using the specific term, appear to have the “soft” intent of providing as complete a picture as possible, or to better understand data obtained from different sources, this latter point being a common justification for the use of qualitative methods. However “harder” exemplars can be found. We present a sample of both styles of triangulation below. Illustrations are drawn from across the body of presence literature, but predominantly from later sources in order to focus our discussion on the current state-­‐of the-­‐art (Turner & Turner, 2011).

a. Data triangulation

Data triangulation entails obtaining data from different sources, or at different times or under different conditions, but would not include studies where these comprise the independent variables in an experiment. That being said, data triangulation is commonplace in presence research although rarely explicitly commented upon. For example, both Bailenson and Yee (2006) and Mark and Kobsa (2005), discussed below in the context of investigator triangulation, use multiple groups of participants as do very many other studies. To take just one illustrative example of triangulation of data sources, the analysis of social presence in a pedagogic computer conferencing application discussed in Rourke et al. (2001) takes data from two different graduate-­‐level conferences. While the main thrust of their argument is methodological, the authors identify differences in the degree of social presence between the conferences, leading them to suggest that an unexpectedly low density of social presence indicators may relate to a high degree of familiarity among participants. Further, the sensitivity to such differences is taken to be an indicator of the robustness of the coding instrument. In an instance of temporal data collaboration, a technique which is less widely adopted outside explicitly longitudinal studies, Bouchard and colleagues (Bouchard et al., 2007) examined data at different stages of the therapeutic process in their investigation of the comparative efficacy of therapy administered face-­‐to-­‐face or by video link.

b. Methodological Triangulation

Methodological triangulation which involves using more than one method to gather data is ready-­‐to-­‐hand in the literature. Perhaps the most common approach is to combine qualitative and quantitative measures. Edmondson (2007) is typical here, and states an explicit aim of triangulating qualitative and quantitative data in a multi-­‐methods approach exploring the potential of tele-­‐presence technologies in of teacher professional development. Groups of teachers undertook training in traditional and online training.

Quantitative methods employed comprised the collection of data from a “concerns based” measure of how far teachers had adopted the instructional strategies which were the subject of the training -­‐ and the results of a mathematics test which again assessed aspects of training content. Qualitative data was obtained from a grounded, thematic analysis of video of the online training sessions and interviews which formed part of the concerns based assessment. It is observed that, taking into account practical limitations, the triangulation produced “corroborating evidence” for the conclusions drawn about the effectiveness of the training. A particularly comprehensive application of multiple methods is described in Di Bias and Poggi (2007), who report a large scale collaborative learning project mediated through virtual reality learning spaces and other collaborative spaces. “Social virtual presence” was identified as the key factor in the project’s success. Data was gathered through a combination of surveys, interviews, focus groups, chat logs, video of class interactions, written reports from tutors, student-­‐produced artifacts, forum posts, capture of tutors’ screens in online sessions and expert review. Analysis methods included quantitative analysis of closed questions, thematic coding and classification of open-­‐ended responses, identification of explanatory or illustrative examples, and visualization using graphs. As stated, the aim and subsequent results of this panoply of methods was “to provide a picture as complete as possible of the learning experience.” rather than to challenge or contest findings.

From the early years of the field, presence researchers have advocated the corroboration of subjective reports by physiological data (IJsselsteijn, et al. 2000). Among the more recent and rigorously reported examples of this genre, Garau et al. (2004) and later Slater et al. (2006) present results from a study conducted using a ‘CAVE-­‐like environment’ – a virtual bar -­‐ to investigate the relationship between physiological responses, breaks in presence and the behavior of virtual characters towards the participants. The Garau report focuses on a range of qualitative measures – an immediate post-­‐experience question, a longer semi-­‐structured interview, subjected to thematic analysis, and a graphical representation by participants of temporal variations in sense of presence. Discussion of the results notes that subjective responses mirrored the experimentally-­‐induced break in presence, while in overall consideration of the methods employed the authors note that the qualitative work provided insights regarding temporal variations and produced the unexpected finding of spatial variations in presence within the same environment. Slater et al. consider the results of the physiological measures -­‐ galvanic skin response, heart rate, heart rate variability, and event-­‐related heart rate changes. Changes in heart rate, heart rate variability and galvanic skin response were shown to be responsive to induce breaks in presence and utterances by virtual characters. Many other presence studies combine physiological and subjective, self-­‐report measures of presence. While in some cases these are simply reported as complementary measures, in others discrepancies between the results of different measures are carefully interrogated. Callan and Ando (2007), for example, in an investigation of sound, imagery and presence, used discrepancies in correlations between fMRI measures of neural activity and subjective estimates of strength of imagery as a basis for a discussion of underlying mechanisms and relationships.

c.     Investigator triangulation

Most instances of investigator triangulation lie in studies using qualitative methods, usually where coding of data is required, and are broadly self-­‐similar. Triangulation here is confirmatory in nature and a means of demonstrating the reliability of the coding instrument rather than challenging conclusions. (Indeed, it is difficult to imagine how this might be otherwise unless the aim of the work is to highlight differences in researchers’ interpretive frames.) Two or more coders/raters categorize the data and a reliability index is calculated and reported. Among the sizeable body of studies relating to social presence in collaborative learning media, for example, Rourke et al. (2001) provide a detailed description of the development of a scheme for coding social presence in computer conferencing transcripts together with inter-­‐rater reliability in the scheme’s application. Three researchers initially worked together to establish coding procedures which were then applied in coding the transcripts by two coders. Reliability was, as expected, higher for ‘manifest’ indicators such as addressing by name than for ‘latent’ indicators such as humor.

Another typical use of multiple raters is in scoring task performance. Mark and Kobsa (2005), for example, investigated the effects of system transparency and differing modes of collaboration in a collaborative information visualization environment, using two coders to score the quality of responses to collaborative tasks. Coders first coded a sample of results as a means of calibration, and then worked independently, achieving a high degree of reliability. Discrepancies were discussed and resolved between the coders. Similarly, in their longitudinal study of collaboration in an immersive CVE (collaborative virtual environment), Bailenson and Yee (2006) had two raters score collaborative verbal tasks requiring creativity with acceptable levels of reliability. (The study is also an instance of method triangulation, combining analysis of non-­‐verbal behavior and subjective ratings of presence, co-­‐presence, simulator sickness and– cohesion-­‐ with task performance).Triangulation of coder results is not limited to verbal media: Patel et al. (2006) report on the relative efficacy of learning tai chi moves in 2D video and 3D immersive applications. Participants’ moves, knowledge of tai chi and overall performance were blind-­‐coded by two coders, albeit with relatively modest degrees of reliability.

d. Theory triangulation

Theory triangulation, which involves using more than one theoretical framework in the interpretation of the data is relatively infrequently encountered in presence research. However, the results of studies adopting this mode of triangulation are generally rigorously discussed and produce rewarding conclusions.A strong element of theoretical triangulation can be found in the development of questionnaire instruments. Most components closely paralleled each other, while apparent divergencies could be attributed to the scope of the questions included in the different instruments. A further hard instance of theoretical triangulation can be found in de Kort et al. (2007), who report the development of a Social Presence in Gaming questionnaire. Using focus group data, the scale was developed through factor analysis and the results discussed in the light of the Biocca et al. (2001) conceptualization of social presence as operationalized in the Networked Minds instrument. The authors note, in contrast to Biocca and colleagues, the absence of co-­‐presence as a distinct dimension in the gaming scale, while the Psychological Involvement dimension only partially coincides. The differences are discussed and attributed to the varying degrees of interdependence engendered by the application domains of gaming and telecommunication.

Work in social presence provides a number of further instances of theoretical triangulation. These include Hwang and Lombard (2006), whose study used both social presence theory and uses and gratifications theory to explore predictors of instant messaging use. Their analysis provides suggestions for the further refinement of both theories. Taking a similarly robust approach, Abeele et al. (2007) foreground triangulation in their investigation of the relationship between social presence, connectedness and perceptual awareness. Social facilitation (presence) theory (Zajonc, 1965) is invoked, the authors arguing that “If the social facilitation framework can be successfully applied, this would provide us with extra evidence that perceptual presence is a prerequisite property of social presence.” (p.217) Participants completed tasks in the real presence or ‘symbolic’ presence (in the form of an image) of either friends or strangers. Only a partial correspondence between measures of social presence and mere presence was established, leading the authors to question aspects of task and experimental design. It is argued that this theory is a valuable tool in the triangulation of social presence data.

2.3The reasons for using Triangulation

Standardized assessments as one source of data teachers may use in assessing student learning, have several shortcomings. Since standardized tests demonstrate only general estimates of learners’ abilities. They also have some problems of validity and reliability in outcome assessment of different skills. The reliability of standardized tests is under question for several reasons:

1. the health, mood, motivation, test-taking skills, or general abilities of students,
2. the quality of directions and the ambiguities of language,
3. distracting conditions in the environment and interruptions during test administration,
4. biases of the observer, errors on the scoring sheet, or even bad luck
5. variations in the way tests are designed, rated, administered, and conditions of testing that may influence outcomes (e.g., interlocutor behavior may influence outcomes of task assessments)
6. speech rate, length of passage, syntactic complexity, vocabulary, noise level, accent, register, amount of redundancy, amount of context provided, clarity of instructions, response format, availability of question preview, listener memory, listener interest, prior background knowledge of listener, motivation of listener may all influence outcomes of assessments (Brindley & Slatyer,2002).

Furthermore, English language learning is a complex process and using one method for its assessing can’t be useful. For learning English one should master oral speaking, listening, comprehension, reading comprehension, and writing.

Triangulated data approach is used to assess learners knowledge based on information gathered from multiple sources to be sure of  the  proficiency level of the English language learners. This approach includes:

1. An in-classroom observational protocol to assess the speaking and listening proficiency of the students,
2. Analysis of the students’ standardized test scores in English, and

3.      Formal test(s) of English proficiency (Switzer, 2006)

In this study a Case study by the writer of this article is considered.

This study focuses on the results of previous studies done by the same researcher. For conducting this study two assessments were used to be sure of the results.

(To Download PDF File of This Full Article in full Details, Click Here)

3. Method

3.1. Participants

The participants of this study were 40 EFL students from Islamic Azad University-Tabriz Branch in Tabriz, Iran. They were chosen after assigning a writing task for having homogeneous groups. Students were all at intermediate level. The researcher randomly assigned them as the experimental and the control group. Teacher as the researcher completed a critical thinking questionnaire for two groups in the pre-test and the post-test.

3.2. Instrumentation

Different instruments used in the present study involve two argumentative writing tasks in the pre-test and the post-test. Also, there was a critical thinking questionnaire.

3.3. Procedure

The purpose of this study was to examine the effects of instructing critical reading and critical writing on developing writing skill. The researcher used a quasi-experimental research design with a sample of forty Iranian college learners from Tabriz Azad university-Tabriz Branch, who were passing General English courses. After assessing the groups’ homogeneity by using a writing task as a pre-test, the researcher randomly assigned them as the experimental and control groups. The control group was supposed to write about different topics at home and they got teacher feedback in class for their writing. However, in the experimental group, teacher in class explained about critical reading and critical writing. They read the text in class critically with students and they wrote about the same topics as control group wrote. Teacher helped the learners in every stage both in reading critically and writing critically. Learners wrote their essays in class while they received feedback from teacher and peers at the same time. The critical reading and writing process that teacher explained and conducted it in class was as follows:

The researcher used critical thinking questionnaire from www.criticalthinkin.org. The questionnaire was scored from 1 to 4 for each item. The researcher added up all the items score.

The results showed that students in the experimental group who received critical reading and critical writing instruction, their writing improved better than students in the control group. The researcher scored students writing based on Hughesʼs(2003) holistic approach.

4. Results and Discussion

         After assessing the homogeneity of students through a proficiency test, Students first writing task was considered as the pre-test and their last writing task was considered as the post-test. In the pre-test, there was not any significant difference between two groups. However, the results of the post-test revealed that there is a significant difference between two groups in the post-test. In the post-test, the experimental group outperforms the control group in using critical thinking and improving writing skill (sahebkheir, 2016). Teacher first considered the whole writing score through Hughes (2005) scoring system and also tried to assess the different items of writing separately. Through using two writing assessment complete view of students’ improvement was revealed.

5. Conclusion and implication(s)

More reliable and valid assessment of ELL proficiency will have a positive effect on the placement and annual monitoring of students. A careful monitoring of student progress during the course of a semester or a year can yield valuable data regarding the student’s English proficiency.

The techniques of triangulating data in qualitative research are appropriate to assessing progress of learners because English language learning is a complex and multi-varied process that require multiple perspectives to fully comprehend the process.

Moreover, multiple methods of assessment allow students the opportunity to represent their knowledge in ways that any one single assessment, a standardized test, might not capture. Furthermore, the use of multiple modes of assessment/triangulation allows students to show learning and to receive comprehensive feedback to improve the quality of their learning as part of the process of assessment. It also promotes equal opportunities for learners to progress and develop their language production. Finally, the “triangulated data” approach encourages improvements in teaching to support each student’s learning. (Smith, Teemant, & Pinnegar, 2004)

There has been some new research about the positive effect of triangulation on improving reliability and validity of tests (e.g., Aydin, 2016; Riazi & Candlin, 2014). It seems that teachers should try to use triangulation for getting a correct score in assessing students’ language production.

References

Abeele, M.V., Roe. K., & Pandelaere, M. (2007). Construct Validation of the Concepts Social Presence, Emotional Presence and Connectedness and an Application of Zajonc’s Socia Facilitation Theory to Social Presence Research, Proc. 10th International Workshop on Presence, 215-­‐224.

Altrichter, H., Posch, P., & Somekh, B. (1996). Teachers Investigate Their Work: An Introduction To The Methods Of Action Research. London: Routledge.

Bailenson, J.E. and Yee, N. (2006). A Longitudinal Study of Task Performance, Head Movements, Subjective Report, Simulator Sickness, and Transformed Social Interaction in Collaborative Virtual Environments, Presence: Teleoperators and Virtual Environments, 15(6), 699 –716.

Baños, R.M., Botella C., Alcañiz M., & Liaño V. (2004). Immersion and Emotion: Their Impact on the Sense of Presence 2004. Cyberpsychology & Behaviour, 7(6), 734 -­‐741.

Besterfield-Sacre, M., Shuman, L., & Wolfe, H. (2000). Triangulating Assessments: Multi-Source Feedback Systems and Closed Form Surveys. University of Pittsburgh (me.pitt.edu).

Biocca, F. (2001). Inserting the Presence of Mind into a Philosophy of Presence: A Response to Sheridan and Mantovani and Riva. Presence: Teleoperators and Virtual Environments, 10(5), 546 –556.

Biocca, F., Harms, C. and Gregg, J. (2001). The Networked Minds Measure of Social Presence: Pilot Test of the Factor Structure and Concurrent Validity. E. Lansing, MI: Media Interface and Network Design (M.I.N.D.) Lab.

Blaikie N.W.H. (1991). A critique of the use of triangulation in social research. Quality and Quantity, 25,115-­‐136.

Bouchard, S., Robillard, G., Marchand, A., Renaud, P., & Riva, G.(2010). Presence and the Bond Between Patients and their Psychotherapists in the Cognitive-­‐Behavior Therapy of Panic Disorder with Agoraphobia Delivered in Videoconference. Proc. 10th International Workshop on Presence, 265-­‐276.

Bryman, A. (1988). Quantity and Quality in Social Research.London: Unwin Hyman.

Boyd, C.O. (2000). Combining Qualitative And Quantitative Approaches. In P.L. Munhall and C.O. Boyd(Eds.) Nursing Research: A Qualitative Perspective. Boston: Jones & Bartlett, 454-­‐475.

Bracken, C. C. & Pettey, G. (2007). It is REALLY a Smaller (and Smaller) World: Presence and Small Screens. Proc. 10th International Workshop on Presence, 283-­‐290.

Brannon, J. (Ed.). (1992). Mixing methods: qualitative and quantitative research. Alershot: Avebury.

Brent, R. & Felder, R. (2003). Designing and teaching courses to satisfy the ABET Engineering Criteria. J. of Engng. Educ., 92(1), 7-25.

Brindley, G., & Slatyer, H. (2002). Exploring task difficulty in ESL listening assessment. Language Testing, 19, 369-394.

Callan, A. & Ando, H. (2007). Neural corre ates of imagery induced by the ambient sound, Proc. 10^th International Workshop on Presence, 73-­‐77.

Campbell, D.T. & Fiske, D.W. (1959). Convergent and discriminant validation by the multitrait- multimethod matrix. Psychological Bulletin, 56(2), 81-105.

Casey, E.S. (1997). The Fate of Place. Univ. of California Press, Berkeley & Los Angeles.

Cohen, L. & Manion, L. (1986). Research Methods In Education. London: Croom Helm.

Cresswell, J.W. (1998). Qualitative inquiry and research design: Choosing among five traditions. Thousand Oaks, CA: Sage.

Davis, M. (2003). Theoretical Foundations for Experiential Systems Design. Proc.  ETP’03, 45-­‐52.

Denzin, N. (1978). Sociological Methods: A Sourcebook. NY: McGraw Hill.

Denzin, N.K. (1978). The Research Act: A Theoretical Introduction to Sociological  Methods. New York: McGraw-Hill.

Denzin, N.K. (1979). The Research Act: A theoretical introduction to sociological methods (2^nd ed.). New York: McGraw Hill.

Denzin, N. K. (2012). Triangulation. Journal of Mixed Methods Research, 6(2), 80–88.

Di Bias, N. and Poggi, C. (2007). European virtual classrooms: building effective  ‘‘virtual’’ educational experiences. Virtual Reality, 11, 129-­‐143

Dzurec, L.C., and Abraham, I.L. (1993). The nature of inquiry: Linking quantitative and qualitative research. Advances in Nursing Science, 16(1), 73-­‐79.

Edmondson, R.S. (2007). Investigating the Effectiveness of a Telepresence-­‐Enabled Cognitive Apprenticeship Model of Teacher Professional Development. Proc. 10th International Workshop on Presence, 265-­‐276.

Fielding, N.G. & Fielding, J.L. (1986). Linking Data. Beverley Hills: Sage.

Flyvbjerg, B. (2001). Making Social Science Matter. Cambridge: Cambridge University Press.

Freeman, J., Lessiter, J., Keogh, E., Bond, F.W., & Chapman, K. (2004). Relaxation Island: Virtual, and Really Relaxing. Proc. 7th International Workshop on Presence, 67-­‐72.

Garau, M. Ritter-­‐Widenfeld, H., Antley, A., Friedman, D., Brogni, A. & M. Slater.

(2004). Temporal and spatial variations in presence: A Proc. 7th International Workshop on Presence, 232—239.

Ghrayeb, O.,  Damodaran, P., & Vohra, P. (2011). Art of triangulation: an effective assessment validation strategy. Global Journal of Engineering Education, 13( 3), 96-108.

Gaver, W. W. (1991). Technological Affordances. Proc. CHI ’91. NY: ACM, 79-­‐84.

Gaver, W. (1992). The Affordances Of Media Space For Collaboration. Proc. CSC  ’92. NY: ACM Press,17-­‐24.

Giles, D.C. (2002). Advanced Research Methods In Psychology. Routledge, 167-­‐180.

Gray, W.D. & Salzman, M.C. (1998). Damaged Merchandise? A Review of Experiments That Compare Usability Evaluation Methods. Human-­‐Computer Interaction, 13, pp. 203-­‐261.

Gribbin, J. (2008). The Universe: A Biography. Penguin Harrison, S. & Dourish, P. (1996). Re-­‐Place-­‐ing Space:The Roles of Place and Space in Collaborative Systems. Proc. CSCW’96. NY: ACM Press, 67-­‐76.

Heidegger, M. (1927/1962). Being and Time. (Translated by J. Macquarrie and E.  Robinson) New York: Harper Collins.

Heidegger, M. (1971). Building Dwelling Thinking. In Basic Writings. London: Routledge.

Hindmarsh, J., Fraser, M., Heath, C., Benford, S.& Greenhalgh, C. (1998). Fragmented Interaction: establishing mutual orientation in virtual environments. Pro.c CSCW’98. NY: ACM Press, 217-­‐226.

Hwang, H.S. & Lombard. M. (2006). Understanding Instant Mend Social Presence,  Proc. 10th International Workshop on Presence, 50-­‐56.

IJsselsteijn, W., Freeman, J., de Ridder, H., Avons, S.E. & Pearson, D. (2000).

Towards an Objective Corroborative Measure of Presence: Postural Responses to Moving Video, Proc. 3rd International Workshop on Presence.

Jones, M. (2007). Presence as External Versus Internal Experience: How Form, User, Style, and Content Factors Produce Presence from the Inside. Proc. 10th International Workshop on Presence, 115-­‐126.

Kallinen, K., Salminen, M., Ravaja, N., Kedzior, R. & Sääksjärvi, M. (2007).

Presence and Emotion in Computer Game Players during 1st Person vs. 3rd Person Playing View: Evidence from Self-­‐Report, Eye-­‐ Tracking, and Facial Muscle Activity Data. Proc. 10th International Workshop on Presence, 187-­‐190.

Kim, T. & Biocca, F. (1997). Telepresence via television: Two dimensions of telepresence may have different connections to memory and persuasion. Journal of Computer Mediated Communication, 3(2) http://www.ascusc.org/jcmc , no page numbers.

Knight, D., Kotys-Schwartz, D. & Pawlas, G. ( 2010). Triangulation: an effective assessment tool for capstone design program evaluation. Proc. Capstone Conference (2010).

Larsson, P., Västfjäll, D., & Kleiner, M. (2001). The actor-­‐observer effect in virtual reality presentations. CyberPsychology and Behavior, 4, 239-­‐246.

Lauria, R. (2001). In Answer to a Quasi-­‐Ontological Argument: On Sheridan’s “Toward an Eclectic Ontology of Presence” and Mantovani and Riva’s “Building a Bridge between Different Scientific Communities”. Presence: Teleoperators and Virtual Environments, 10(5), 557 –563.

Lessiter, J., Freeman, J., Keogh, E. & Davidoff, J. (2001). A cross-­‐media presence questionnaire: The ITC-­‐ Sense of Presence Inventory. Presence: Teleoperators & Virtual Environments, 10, 282-­‐298.

Lincoln, Y.S. & Guba, E.G. (2000). Paradigmatic controversies, contradictions, and emerging confluences. In N.K. Denzin and Y.S. Lincoln (Eds.), Handbook of qualitative research. Thousand Oaks, CA: Sage.

Lombard, M. & Ditton, T. (1997). At the heart of it all: The concept of presence. Journal of Computer Mediated Communication, 3(2), http://www.ascusc.org/jcmc , no page numbers.

Lomba d, M. & Jones, M. (2004). Presence and Sexuality. Proc. 7th International  Workshop on Presence, 28-­‐35.

Mantovani, G. (2001). Building a Bridge between Different Scientific Communities: On Sheridan’s Eclectic Ontology of Presence. Presence: Teleoperators and Virtual Environments, 10(5), 537 –543.

Mark. G. & Kobsa, A. (2005). The Effects of Collaboration and System Transparency on CIVE Usage: An Empirical Study and Model, Presence: Teleoperators and Virtual Environments, 14(1), 60 – 80.

Mason, J. (1994). Linking qualitative and quantitative data analysis. In A. Berman, & R. G.Burgess (Eds.), Analyzing qualitative data (pp. 89-110). London: Routledge.

Mathison, S. (1988). Why triangulate? Educational Researcher, 17(2), 13-17.

Miles, M.B. & Huberrnan, A.M. (1984). Qualitative Data Analysis. Beverly Hills: Sage.

Nunez, D. (2007). Effects of Non-­‐Diegetic Information on Presence: A Content Manipulation Experiment, Proc. 10th International Workshop on Presence, Barcelona, 19-­‐26.

Nunez, D. & Blake, E.H. (2006). Learning, experience and cognitive factors in the presence experiences of gamers: An exploratory relational study. Presence: Teleoperators and Virtual Environments, 15(4), 373-­380.

Patel,  K.,  Bailenson,  J.N.,  Jung,  S.-­‐H.,  Diankov,  R.  &  Bajcsy,  R.  (2006).  The  Effects  of  Fully  Immersive Virtual Reality on the Learning of Physical Tasks, Proc.9th International Workshop on Presence, 129-­‐138.

Perlesz, A. & Lindsay, J. (2003). Methodological triangulation in researching families: Making sense of dissonant data. International Journal of Social Research Methodology, 6, 25- 40. Provenzo, E. (1991). Video Kids: Making Sense of Nintendo. Cambridge, MA:    Harvard University Press.

Riazi, M. & Candlin, C. (2014). Mixed-methods research in language teaching and learning: Opportunities, issues and challenges. Language Teaching, 47(2), 135 – 173.

Schubert, T.F. &Thomas, F.R. (2001). The Experience of Presence: Factor Analytic Insights. Presence: Teleoperators and Virtual Environments, 10(3), 266-­‐281.

Schubert, T.W., Friedmann, F. & Regenbrecht, H.T. (1999). Decomposing the sense of presence: Factor analytic insights. Proc. 2nd International Workshop on Presence, Essex, 1999.

Selami, A.(2016). A Qualitative Research on Foreign Language Teaching Anxiety. The qualitative report, 21(4).630-542.

Sheriden, T. B. (2001). Response to “Building a Bridge between Different Scientific Communities: On Sheridan’s Eclectic Ontology of Presence”. Presence: Teleoperators and Virtual Environments, 10(5), 544 –545.

Silverman, D. (1993). Interpreting Qualitative Data. Methods for Analysing Talk, Text and Interaction, Thousand Oaks: Sage. Silverman, D. (2000) Doing Qualitative Research, London: Sage.

Smith, M. E., Teemant, A., & Pinnegar, S. (2004). Principles and practices of sociocultural assessment: Foundations for effective strategies for linguistically diverse classrooms. Multicultural Perspectives, 6, 38-46.

Turner, P., & Turner, S. (2010). Triangulation In Practice. Retrieved fromhttps://www.researchgate.net/publication/220530134_Triangulation_in_practice

Wiggins, G. & McTighe, J. (2005). Understanding by Design (2^nd Ed.). Washington, D.C.: ASCD.

Wirth, W., Hartmann, T., Böcking, S., Vorderer, P., Klimmt, C., Schramm, H., Saari, T., Laarni, J., Ravaja, N., Gouveia, F. R., Biocca, F., Sacau, A., Jäncke, L., Baumgartner, T. & Jäncke, P. (2007). A Process Model of the Formation of Spatial Presence Experiences. Media Psychology, 9(4), 493-­‐925

Witmer, B.G., & Singer, M.J. (1998). Measuring Presence In Virtual Environments: A Presence Questionnaire. Presence: Teleoperators and Virtual Environments, 7(3), 225– 240.

Witmer, B.G., Jerome, C.J. & Singer, M.J. (2005). The factor structure of the presence questionnaire. Presence: Teleoperators and Virtual Environments, 14(3), 298-­‐312.

 

(To Download PDF File of This Full Article in full Details, Click Here)