S. Mubaraz, J. Heikkilä, J. Luomakoski, R. Khan Haaga-Helia University of Applied Sciences (FINLAND)
The aim of this study is to empirically explore the student peer review practices as part of project work assessment process. Due to COVID-19 pandemic, a hybrid entrepreneurial hackathon (combination of face-to-face and virtual learning environment) was arranged, where the students from a Finnish higher education institute participated in the process of business idea generation and created numerous innovative business ideas while working in respective teams. The ideas were presented to an audience at the end of the event. Based on the literature review, eight variables pertaining to teamwork performance were selected to be included in this study. Students were divided into teams. Each team member was asked to conduct a self and peer assessment of performance of the other team members as well as to give a score out of 5 points. This study follows a quantitative research methodology. Empirical data was collected at the end of the hybrid hackathon. IBM’s SPSS analytical tool was used for the analysis. The result shows that students assessed the performance of each team member in comparison to the responsibilities that were given to them while working in teams. The results highlight that the students, showed a tendency of assessing themselves higher than their peers. This difference was consistent but statistically insignificant in all eight selected variables related to the teamwork performance and contribution. The inter-item correlation matrix confirms high internal consistency with high correlations between the variables. All correlations are statistically significant even when responsibilities were divided differently. Students’ self-assessments were quite high, ranging between 4,30 and 4,70 in a 1-5 scale. The highest means in self-assessment were “ability to meet deadlines” and “willingness to take responsibility”. Additionally, the “ability to meet deadlines” was also ranked highest in peer assessment. The findings of the study reveal insight into the self and peer review assessment and students’ approach to factors that influence peer review of student performance in a project-based hybrid learning environment. Keywords: Peer assessment, hybrid hackathon, COVID-19, student practices, entrepreneurial education.
The self and peer assessment activities in educational institutes may seem relatively recent, however the literature on this subject shows that this type of assessment has been taken up in some fields of education since 1774 in University of Glasgow, Scotland . Today, the Higher Education Funding Council for England recognizes such activities to help students to become lifelong learners and to progress with confidence . Andrade & Du [3 p.160] defines self-assessment as a process of ‘formative assessment during which students reflect on and evaluate the quality of their work and their learning, judge the degree to which they reflect explicitly stated goals or criteria, identify strength and weaknesses in their work, and revise accordingly’. On the other hand, the peer assessment can be seen as an arrangement, where equal-status learners judge the quality of performance and skills of other equal-status learners . In self-assessment activities, students judge the quality of their own work , whereas in peer assessment ‘students provide a grade (or feedback or both) to peer learners on performance, which is based on a criteria of excellence for that event in which the students may have been involved in determining’ [6 p. 132]. Assessment in principle encapsulates ‘making decisions about the standards of performance and making judgments about the quality of the performance in relation to these standards’ ( in [8 p. 3]). At present, it can be agreed that both types of assessment initiatives are grounded in the active learning philosophies (see for example , , ). The summative peer assessment is aimed at providing a grade or score whereas the formative peer assessment, also known as heuristics assessment, mainly provides feedback and helps students to drive their own learning and develop professional skills . The importance of self-assessment in the higher education institutes (HEIs) has been emphasized by many researchers claiming that it can enable students to become self-evaluating lifelong learners . Proceedings of INTED2021 Conference 8th-9th March 2021 ISBN: 978-84-09-27666-0 4619 Similarly, peer assessment in HEIs is considered to promote learning . Several studies advocate the compelling and potential effectiveness of self and peer assessment, however, Adachi et al.  claim that there is still room for further application of self and peer assessment in the HEIs. One particular study by Hounsell and McCulloch  revealed that more than twenty five percent of student assessment initiatives at HEIs in Scotland used self and/or peer assessment . At present, the team-based learning has become a common method to promote student learning in the HEIs, receiving considerable attention . Because learners while working in teams take ownership of their learning process, hence show increased cooperation, problem-solving capabilities, negotiation skills, and interpersonal communication . Besides, due to numerous advantages, this method has been applied in traditional course implementation as well as short-term event-like learning settings . Research shows that, on the one hand peer assessment increases motivation among team members; improving overall team performance and on the other hand, it highlights certain personal attitudes and behaviours, which are not visible to outside people . Researchers argue that team members interact and observe each other during the course of learning, thus become an essential source of feedback related to individual performance and contribution towards team’s overall goal reducing the chances of free riding . With such proliferation of team-based learning activities in events like hackathons, teachers and instructors tend to avail the possible methods of self and peer assessment of performance and quality of participation within a team context. Availability of new user-friendly information technologies has afforded numerous forms of blended-learning approaches including face-to-face settings combined with on-line virtual participant presence . Several studies which investigated peer assessment in the Massive Open Online Courses (see for example  and ) posit that due to students’ virtual interactivity, the peer review and feedback on the performance becomes highly relevant during online learning environment. This study presents a case of a hybrid entrepreneurial hackathon in which international students from a Finnish higher education institute participated. Students’ self and peer review practices were explored within the student teams during this hybrid event. Such exploration is intended to become a means of improvement in the quality of learning process as well as to increase student engagement in the hybrid event. The assessment was carried out using variables including ‘attendance and promptness’ of a team member, ‘quality of participation’ extended by a member, ‘degree of cooperation’ offered by a member, ‘communication skills’, ‘ability of a member to meet deadlines’, a member’s ‘ability to carry out own workload’, a member’s ‘willingness to accept responsibility’, and ‘leadership contribution’ offered by a member within a team. Each member of the team performed self and peer assessment at the end of the hybrid event. The research design and context of the hybrid event are provided in the next section. The results of the analysis are presented in the conclusion of the study. Research limitations are provided with recommendations of the possible future direction of the research in this topic area.
The description of the context of the study and research design are provided as under: 2.1 Hybrid Hackathon This study focusses on students’ peer review practices as a part of project work assessment process. The data for this study is derived from a hybrid hackathon, which was arranged during COVID-19 pandemic. The hackathon was a hybrid event, which had both online and onsite parts. It was organized as a three-day event where students collaborated to generate innovative business ideas while working in teams. This event was arranged in a hybrid format as opposed to a pure online event to improve the level of student engagement and for better interaction between the participants and mentors. The aim of the hybrid hackathon was to enable multidisciplinary student-teams to generate innovative business ideas, which were to be presented at the end of the event. The first day was arranged faceto-face at a physical location in Helsinki, Finland and the following two days were managed online through virtual learning platforms. The event was held in September, 2020. The event saw the enrolment of ninety students and eight mentors. During the first day, seven mentors and thirty-one students gathered physically event onsite while sixteen students and one mentor participated in the event remotely as the event was streamed live. During the next two days, thirty-seven students participated virtually while working in their respective teams. All eight coaches were also mentoring the students 4620 virtually. A total of thirty-seven students completed the course out of which twenty-seven male and female undergraduate students formed the sample of this study. The student teams collaborated to innovate business ideas and pitched their specific business ideas to mentors at the end of the event. These teams were continuously mentored by eight mentors during the three-day event. The mentors supported the student learning during the event by continuously guiding the students. A total of twenty-eight students submitted the form including their own self-assessment and their team members’ peer assessments. One form was excluded from the sample as it did not follow the instructions of filling in the information. The overall number of peer assessments is ninety-four. The students were given two days to submit the self- and peer assessment forms and the last student submitted their forms five days after the deadline. Authors agreed to include all the completed forms in the analysis. 2.2 Quantitative study This study follows a quantitative research methodology that utilizes empirical data collected during the hybrid hackathon to study students’ peer review practices. The questionnaire items were based on the literature review and eight variables pertaining to students’ peer review process were included in the questionnaire. These variables are attendance, participation, cooperation, communication, meeting deadlines, carrying out own workload, responsibility and leadership. Each team member had to conduct self and peer review assessment of the performance of other team members and give a score out of 5 points. The questions were based on a Likert scale from the “poor” (1) to the “excellent” (5). Nominal scales were also used to study other attributes like the background information of the respondents. Participants were also asked to describe in their own words, what were the responsibilities of each team member in creating a novel business idea during the hybrid event. The questionnaire was reviewed and revised by the researchers several times. Each item was critically analyzed, which resulted in minor changes to the presentation of the questionnaire. The students were informed about the voluntary nature of their participation and the confidentiality of their responses. The researchers analyzed the data with the help of SPSS and Excel. The empirical data was entered into IBM SPSS statistical tool for the analysis. The analysis methods applied were Cronbach’s alpha, intercorrelation coefficient (version 1, k) and inter-item correlations.
The results show that the students’ self and peer assessments are in-line with each other (Table 1). There were no statistical differences between the peer and self-assessments of the participants. However, the students’ self-assessments are consistently somewhat higher that their peer assessments. Table 1. Means and standard deviations of the self and peer assessments. Peer assessment Self-assessment Mean SD Mean SD Attendance 4,13 1,18 4,59 0,69 Quality of participation 4,33 1,09 4,56 0,64 Cooperation 4,46 1,06 4,63 0,63 Communication 4,38 1,10 4,63 0,63 Meeting deadlines 4,52 1,10 4,70 0,61 Carrying out own workload 4,39 1,24 4,59 0,69 Responsibility 4,37 1,15 4,67 0,62 Leadership 3,96 1,10 4,30 0,87 Overall grade 4,28 1,09 4,52 0,64 For the peer assessments, the Cronbach’s alpha was calculated to assess the reliability of the assessments. The internal consistency was found to be high (Cronbach’s alpha = 0.959). The inter-item correlation matrix confirms this finding with high correlations between the variables (Table 2). All correlations are statistically significant (p < 0.01). 4621 Table 2. Correlation coefficients of the variables. 1 2 3 4 5 6 7 8 1. Attendance - 2. Quality of participation 0,77 - 3. Cooperation 0,76 0,79 - 4. Communication 0,68 0,76 0,83 - 5. Meeting deadlines 0,70 0,73 0,82 0,78 - 6. Carrying out own workload 0,72 0,74 0,82 0,76 0,83 - 7. Responsibility 0,69 0,77 0,81 0,74 0,76 0,82 - 8. Leadership 0,63 0,72 0,77 0,76 0,69 0,68 0,62 - The intercorrelation coefficient (ICC) which measures the inter-rated reliability was found to be very high 0.956 (p < 0.01). The 95 percent confidence interval for the ICC is from 0.940 to 0.968. These results indicate that the students’ assessments are consistent. (Table 3.) Table 3. Intercorrelation coefficient. Intra-class Correlation 95% Confidence Interval F Test with True Value 0 Lower Bound Upper Bound Value df1 df2 Sig 0,955 0,940 0,968 22,330 93 658 0,000 The results of the analysis are discussed and summarized in the following section. 4 CONCLUSIONS The aim of this study is to explore the student self and peer assessment practices while working in their respective teams as part of project work assessment process. The context of this study is a hybrid entrepreneurial hackathon, due to COVID-19 pandemic situation, where the students from a Finnish higher education institute participated in the process of business idea generation. The assessment was carried out using variables including ‘attendance and promptness’ of a team member, ‘quality of participation’ extended by a member, ‘degree of cooperation’ offered by a member, ‘communication skills’, ‘ability of a member to meet deadlines’, a member’s ‘ability to carry out own workload’, a member’s ‘willingness to accept responsibility’, and ‘leadership contribution’ offered by a member within a team. Each team member was asked to conduct a self and peer assessment of performance of the other team members as well as to give a score out of 5 points. The analysis of the data sheds light on the students’ peer review practices, while comparing the performance against responsibility and delivery of tasks. There were a few things to be noted about the differences between self and peer assessment. Firstly, the students, while working in teams, showed a tendency of assessing themselves higher than their peers. This difference was consistent but statistically insignificant in all eight selected variables related to the teamwork performance and contribution. The analysis of the data revealed high internal consistency. The inter-item correlation matrix confirms this finding with high correlations between the variables. All correlations are statistically significant. Secondly, there were no statistically significant differences between different variables that were analysed, even when responsibilities were divided differently. Thirdly, less than half of the participants filled the assessment questionnaire and they did it within 7 days after the assessment form was published for the students to be filled. Fourthly, it can be noted, that the students’ self-assessments were quite high, namely the means were ranging in a scale of 1-5 between 4,30 and 4,70. The highest means in self-assessment were “ability to meet deadlines” and “willingness to take responsibility”. Additionally, the “ability to meet deadlines” was also ranked highest in peer assessment. The study presents international students’ self and peer review practices in a hybrid hackathon where students from a Finnish higher education institute participated. The small data sample of the study is considered as a research limitation, hence care should be taken in generalizing the finding of this study. It is recommended that a larger data should be used to further explore the self and peer review practices of the students. Similarly, a comparative research can be conducted using data sample from event-based learning environment and the traditional classroom setting in HEIs.
The authors would like to express their gratitude to EU Interreg Central Baltic funded project NOCCA – Novel opportunities for new company creation and accelerated growth for providing the valuable research material and opportunity to conduct the research.
REFERENCES  K.J. Topping, “Self and peer assessment in school and university: Reliability, validity and utility” in Optimising new modes of assessment: In search of qualities and standards (M. Segers, F. Dochy & E. Cascallar, eds.), pp. 55-87, Dordrecht: Springer, 2003.  B.A. Simonsmeier, H. Peiffer, M. Flaig, and M. Schneider, “Peer Feedback Improves Students’ Academic Self-Concept in Higher Education,” Research in Higher Education, vol. 61, pp. 706-724, 2020. https://doi.org/10.1007/s11162-020-09591-y  H. Andrade, and Y. Du, “Student responses to criteria-referenced self-assessment, ” Assessment and evaluation in higher education, vol. 32, no. 2, pp. 159-181, 2007.  K.J. Topping, “Peer assessment,” Theory into practice, vol. 48, no. 1, pp. 20-27, 2009.  N. Falchikov, and J. Goldfinch, “Student peer assessment in higher education: A meta-analysis comparing peer and teacher marks,” Review of educational research, vol. 70, no. 3, pp. 287-322, 2000.  N. Falchikov, “The place of peers in learning and assessment” in Rethinking assessment in higher education: Learning for the longer term (D. Boud and N. Falchikov, eds.), pp. 128-143, Abingdon, Oxon: Routledge, 2007.  D. Boud, Enhancing Learning through Self-assessment. London: Kogan Page, 1995.  D. Spiller, “Assessment matters: Self-assessment and peer assessment,” The University of Waikato, vol. 13, 2012.  L. Vygotsky, Thought and language. Cambridge, MA: MIT Press. 1962.  J. Piaget. Science of education and the psychology of the child. Longman: London, translated by D. Coltman, 1971.  K.P. Cross, Adults and learners. San Francisco: Jossey-Bass, 1981.  K. Tan. “Conceptions of self-assessment” in Rethinking assessment in higher education: Learning for the longer term (D. Boud and N. Falchikov, eds.), pp. 114-127, Abingdon, Oxon: Routledge, 2007.  C. Adachi, J.H.M. Tai, and P. Dawson, “Academics’ perceptions of the benefits and challenges of self and peer assessment in higher education,” Assessment & Evaluation in Higher Education, vol. 43, no. 2, pp. 294-306, 2018.  D. Hounsell, and M. McCulloch, “Assessing skills in Scottish higher education” in The Learning Society: International Perspectives on Core Skills in Higher Education (E. Dunne eds.), pp. 149- 158, London: Kogan Page, 1999.  T.N. Crutchfield, and K. Klamon, “Assessing the dimensions and outcomes of an effective teammate,” Journal of Education for Business, vol. 89, no. 6, pp. 285-291, 2014.  R.J. Almond, “Group assessment: Comparing group and individual undergraduate module marks,” Assessment & Evaluation in Higher Education, vol. 34, no. 2, pp. 141-148, 2009.  C.O. Mayfield, and R.T. Jay, “Why peer evaluations in student teams don’t tell us what we think they do,” Journal of Education for Business, vol. 94, no. 2, pp. 125-138, 2019.  T. O’Neill, N. Larson, J. Smith, M. Donia, C. Deng, W. Rosehart, and R. Brennan, “Introducing a scalable peer feedback system for learning teams,” Assessment & Evaluation in Higher Education, vol. 44, no. 6, pp. 848-862, 2019.  H.K. Suen, “Peer assessment for massive open online courses (MOOCs),” International Review of Research in Open and Distributed Learning, vol. 15, no. 3, pp. 312-327, 2014.  J. Costello, and D. Crane, “Technologies for learner-centered feedback,” Open Praxis, vol. 5, no. 3, pp. 217-225, 2013.