DOI: 10.3897/jucs.137103

Authors: Hendrik Hendrik, Silmi Fauziati, Adhistya Erna Permanasari

Abstract: Knowledge graphs are a powerful way to represent and organize complex knowledge. They are used in many fields, like healthcare and finance. They allow for more insightful decision-making and discoveries. However, the quality of knowledge graphs depends heavily on their sources. Current methods for evaluating these sources are often slow and not scalable. They struggle to keep up with the large amount of online information. We created a new tool to address this problem. Our tool uses Large Language Models (LLMs) to assess online sources quickly. It evaluates websites based on credibility, relevance, content quality, coverage, comprehensiveness, and accessibility. We tested our tool on Halal tourism websites in Japan. We compared LLM evaluations with human expert judgments. Our comprehensive analysis revealed that certain LLM models, particularly GPT-3.5-turbo, GPT-4, and Mixtral-8x7B-Instruct-v0.1, showed strong correlation with human evaluations. Using a temperature setting of 0.4, these models demonstrated consistent and reliable performance across multiple evaluation runs. Our structured evaluation framework, incorporating weighted criteria validated through both expert input and statistical analysis, provides a robust foundation for automated source assessment. While some models showed varying performance across different criteria, our findings suggest that careful model selection and potential ensemble approaches could optimize evaluation accuracy. Our work contributes significantly to improving knowledge graph construction by demonstrating the viability of LLM-based source evaluation, while also identifying key areas for future research in scalability, cross-domain validation, and automated optimization.

HTML

PDF

DOI: 10.3897/jucs.116830

Authors: Alessander Osorio, Paulo Roberto Ferreira Jr., Gerson Geraldo H. Cavalheiro

Abstract: Researchers have access to a vast repository of scientific papers reporting research results through the Internet. In this context, fostering collective knowledge growth, researchers may encounter situations where the content of a paper is not effectively utilized due to flaws in the presentation of conducted research. Various fields of knowledge have developed research protocols to assist in the production of scientific reports. In the field of Computer Science, proposals for such protocols are still in development. This paper describes a case study conducted with undergraduate and graduate students in Computer Science, involving guided reading of scientific papers, and recording students’ perceptions regarding adherence to a set of guidelines for the presentation of scientific results. The case study focused on papers presenting research results with a quantitative focus, such as performance evaluations. The exercise took place over two academic semesters, where reading instructions were provided, and impressions about the observed content were collected every two or three weeks. The analysis results indicated that, despite being well-written, the papers had gaps in relation to the reading guidelines. The study’s conclusions suggest that authors underestimate the importance of providing a comprehensive account of their experi-ence, highlighting the need for protocols to report scientific results in the field of Computer Science.

HTML

PDF

DOI: 10.3897/jucs.103264

Authors: Ildevana Poltronieri, Avelino Francisco Zorzo, Maicon Bernardino, Edson OliveiraJr

Abstract: Software architects and developers often use Domain-Specific Languages (DSLs) to model or code applications. However, designing a DSL that effectively represents its domain can be a challenge, potentially contributing to poor uptake and usage. To the best of our knowledge, one issue is that DSL designers may evaluate their language’s usability using ad hoc processes, due to a lack of expertise in usability evaluation. Additionally, current approaches lack well-defined processes and may not yield the desired results for DSL designers. Therefore, DSL designers require a well-defined usability evaluation process to assess how architects, developers, and end users perceive their DSL. This paper introduces Usa-DSL, a Usability Evaluation Process for Domain-Specific Languages. Usa-DSL aims to assist DSL designers in evaluating their languages in terms of ease and quality of use, without requiring deep knowledge of usability evaluation. We analyze the feasibility of Usa-DSL and show that it is a useful and user-friendly tool for evaluating DSLs.

HTML

PDF

DOI: 10.3897/jucs.96347

Authors: Ricardo Caceffo, Jacques Wainer, Guilherme Gama, Islene Garcia, Rodolfo Azevedo

Abstract: Perceptual Learning Modules (PLMs) is a variation of Perceptual Learning based on multiple-choice questionnaires. There exists successful research of the use of PLMs in math and flight training. The possibility of designing and adopting PLMs in Introductory Programming Courses (CS1) is still an open area of study. The goal of this study is to test whether students that received a PLM training on recognising segments of programs will perform better at writing programs. Two PLM interventions were administered to students. The first intervention was a nonrandom controlled experiment, in which students opted to answer the PLM questionnaire (N=40), while the control group consisted of students that did not answer it (N=629). The second intervention was a randomized controlled experiment with a placebo, in which students were randomly assigned to perform either the PLM questionnaire (N=51) or another a placebo activity (N=51). The different forms of analysis of the first experiment results yielded Cohen’s d ranging from 0.23 to 0.34 in favor of the PLM intervention. For the second experiment, the effect size was d = -0.11 against the PLM intervention, but the two results were significant. We believe that the cautious conclusion is that there is a null effect in using a PLM activity as part of a CS1 course. The paper is also of interest because of the methodological decisions and techniques used.

HTML

XML

PDF

DOI: 10.3897/jucs.95718

Authors: Gerardo Matturro

Abstract: The purpose of this paper is to present an undergraduate research experience process model and the evaluation of seven years of its application in an undergraduate research program in software engineering. Undergraduate students who participated in research projects between 2015 and 2022 were surveyed to find out a) their motivations for participating in research projects in software engineering, b) the skills they consider they have acquired or improved by participating in those projects, and c) their perception of benefits and utility for their future work and professional activities. Results reveal that participation in real research projects in software engineering is highly valued by undergraduate students, who perceive benefits in the development of research and soft skills, and for their future professional activity. In addition, these undergraduate research projects and the process followed show that it is feasible to make original contributions to the body of knowledge of software engineering.

HTML

XML

PDF

DOI: 10.3897/jucs.2020.063

Authors: José Martín-Núñez, Susana Sastre, José Peiró, José Hilera

Abstract: The use of mobile devices in the classroom is increasingly frequent. However, the LMS are still not completely adapted to this format, preventing students from using all the LMS web-functionalities in their mobiles. Hence, we present and evaluate the use of a new mobile application fully integrated with Learning Management Systems (LMS). We examined access to LMS by 95 postgraduate university students, differentiating between the services accessed and the means used. Students belonged to four consecutive promotions. In the first two, access to the system was through the web, while in the third and fourth, an app fully integrated with the LMS was available. The results showed an overall increase in access to LMS, with a considerable reduction in access via the web in favor of access via the application. Significant differences were found in the access patterns to communication and assessment services depending on the students' age, gender, academic major and previous m-learning experience. Satisfaction with the LMS rose when the app was available, with greater growth within the academic major on IT and previous m-learning experience group. Finally, students with high performance accessed the system significantly more than those with low performance. In conclusion, the integration of the app with the system showed useful and efficient results. The app eased the use of the system, increased student satisfaction with LMS, and student performance improved with increased access.

HTML

XML

PDF

DOI: 10.3897/jucs.2020.053

Authors: Daphne Economou, Melissa Russi, Ioannis Doumanis, Markos Mentzelopoulos, Vassiliki Bouki, Jeffery Ferguson

Abstract: One in every six persons in the UK suffers a hearing loss, either as a condition they have been born with, or they developed during their life. Nine hundred thousand people in the UK are severely or profoundly deaf. Based on a study by Action on Hearing Loss UK in 2013 only 17 percent of this population, can use the British Sign Language (BSL). That leaves a massive proportion of people with a hearing impediment who do not use sign language struggling in social interaction and suffering from emotional distress. It also leaves even a larger proportion of Hearing people who cannot communicate with those of the deaf community. This paper presents a Serious Game (SG) that aims to close the communication gap between able hearing people and people with hearing impairment by providing a tool that facilitates BSL learning targeting the adult population. The paper presents the theoretical framework supporting adult learning based on which a SG game using Virtual Reality (VR) technology has been developed. The paper explains the experimental framework of the study. It presents the creation of the research instruments to facilitate the study comprising of a SG that integrates video and conventional video-based educational material. It reports and analyses the study results that demonstrate the advantage of the SG in effectively supporting users learning a set of BSL signs. It also presents qualitative outcomes that inform the further development of the game to serve learning needs. The paper closes with conclusions, directions for further development of this educational resource, and future studies.

HTML

XML

PDF

DOI: 10.3897/jucs.2020.049

Authors: Gareth Young, Sam Stehle, Burcin Walsh, Egess Tiri

Abstract: irtual reality (VR), as an informative medium, possesses the potential to engage students with immersive, interactive, and informative experiences. When presented in VR, immersive virtual environments (IVEs) can provide three-dimensional visual simulations that can be used to inform students about concepts in specific contexts that would be near impossible to achieve with more traditional teaching methodologies. It is proposed that existing learning frameworks can benefit from exploring the modalities of interaction that are presently afforded via VR from the experiential perspectives of the students. An evaluation is presented that focused on the appraisal of student experiences of immersive technologies as applied in a higher education context, specifically in the use of VR for the exploration of geomorphology theory by physical geography students. This research supports further development of the immersive learning discipline from three different perspectives. First, an empathy mapping method was applied to visualize student experiences and externalize our observed knowledge of student users for creating a shared understanding of their needs and to aid in lesson planning decision making when using VR in the classroom. Second, student experiences were captured using a technology-focused user experience questionnaire to obtain student attitudes immediately post-task. Finally, to assist teachers with the creation of a student-centered lesson plans that incorporate VR in the classroom, eight heuristic guidelines (focus, provocation, stimulation, collaboration, control, digital life, learner skills, multimodal experience) were developed. It is proposed that these findings can be used to provide support for the use of mixed reality and immersive virtual environments in learning that encompass the challenges faced by students and the interdisciplinary education community at large.

HTML

XML

PDF

DOI: 10.3897/jucs.2020.047

Authors: Louis Nisiotis, Styliani Kleanthous

Abstract: Using technology in education is crucial to support learning, and Virtual Worlds (VWs) are one of the technologies used by many educators to support their teaching objectives. VWs enable students to connect, synchronously interact, and participate in immersive learning activities. Such VW has been developed at Sheffield Hallam University (UK), and is used to support the teaching of a specific module, as well as for conducting empirical research around the topics of Transactive Memory Systems (TMS) and Students Engagement. TMS is a phenomenon representing the collective awareness of a group's specialisation, coordination, and credibility with interesting results. This paper presents the lessons learned while using the VW over the past few years at a higher education institution to support collaborative learning within working groups. A review of these empirical findings is presented, together with the results of a follow up study conducted to further investigate TMS and student Engagement, as well as students perceived Motivation to use a VW for learning, and their Learning Outcomes. The findings of this study are corroborating and contributing to previous results, suggesting that a VW is an effective tool to support collaborative learning activities, allowing students to engage in the learning process, motivate them to participate in activities, and contribute to their overall learning experience.

HTML

XML

PDF

DOI: 10.3217/jucs-025-12-1589

Authors: Zuzana Kubincová, Dana Šuníková, Martin Homola

Abstract: Team projects and group work have proven to be useful and rewarding educational activities that help students develop important abilities and skills. On the other hand, they have also proven to be difficult to grade in a balanced way which would be perceived as fair by the students. Numerical ratings such as those delivered by the Fink method provide ways to determine the shares of contributions of each individual team member. But such bare numerical evaluation needs also some secondary justification that helps team members to understand why their rating is high or low. Methods to collect such additional feedback often rely on surveys; however, extensive surveys may be perceived as tedious by the students, especially as they need to be filled out for each other team member participating in the team. This may have negative effect on the quality of feedback collected in form of surveys. We present a novel approach to peer assessment of a team-based project exploiting badges that represent individual contributions to the task and teamwork related traits.We show that this approach has positive influence on engagement in peer assessment compared to free-text open questions. We also study if the feedback obtained in this way is informative and to which extent it can serve as replacement for open free-text questions.

HTML

XML

PDF

DOI: 10.3217/jucs-025-04-0310

Authors: Vladimir Ciric, Ivan Milentijevic, Oliver Vojinovic, Nemanja Manic

Abstract: There is a lot of educational institutions that own blended learning platforms, and even more that do not. The era of cloud computing offers a possibility for those who do not own such platforms to utilize public cloud resources for courses delivery. The goal of this paper is design and cost estimation of flexible blended learning platform suitable for cloud implementation. In this paper we present technical requirements and create a mathematical model which allows one to estimate the cost of course delivery through public cloud. As a basis for the cost estimation, a model of blended learning course and a platform that supports the model will be proposed. The platform will be based on open source software, and it will be designed to be suitable and simple for cloud implementation. The proposed blended learning model will be applied and the platform will be implemented on the public cloud on the example of computer networking course. In order to estimate the cost of public cloud hosting, a stress test will be performed varying both virtual hardware and the number of students. Three different virtual hardware configurations will be evaluated with up to 400 students. The analysis of the stress test results will be given, along with the recommendations regarding the virtual hardware in the respect to the audience. The cost function in the respect to the number of enroled students will be given, too. It will be shown that the proposed course delivery model is low-cost, while it is based on the state of the art learning tools. Moreover, the students' achievements will be discussed in the example of two different cohorts of students.

HTML

XML

PDF

DOI: 10.3217/jucs-024-10-1424

Authors: Giani Petri, Alejandro Calderón, Christiane Gresse von Wangenheim, Adriano Borgatto, Mercedes Ruiz

Abstract: Driven by the need to provide more hands-on opportunities to computing students, educational digital and non-digital games have been used as an instructional strategy for teaching Software Project Management (SPM). However, a question that arises is to which regard the expected benefits of these games are real. Thus, the objective of this study is to analyse the benefits of digital and non-digital games used for SPM education in order to evaluate their quality in terms of player experience and perceived learning adopting the MEEGA+ evaluation model. The analysis is based on data collected from 27 case studies, evaluating 11 different SPM games, involving a population of 562 students. Results provide evidence that both digital and non-digital games contribute positively to the students' perceived learning, as well as to social interaction, being considered relevant to students' learning, and promoting fun. A slightly higher usability degree was observed with respect to digital games due the easiness to learn how to play the games. On the other hand, a slightly more positive evaluation of the achievement of the learning objectives of non-digital games has been observed. The results of this study may guide instructors in the selection of games as an instructional strategy for SPM education and/or game creators with respect to aspects to be considered in the development of new educational games in order to maximize their benefits and continuous improvement.

HTML

XML

PDF

DOI: 10.3217/jucs-024-02-0171

Authors: Samah Felemban, Michael Gardner, Victor Callaghan, Anasol Pena-Rios

Abstract: 3-D virtual worlds and other immersive environments offer features that other learning systems cannot easily replicate. As such, they have the potential to revolutionise the way in which people learn. They are well suited to visualise 3-D objects and their relations to explain complex phenomena. In addition, they enable practical experiments to be performed that are difficult to conduct in the real world. They can also help to facilitate collaborative learning in real-time and enable students to become fully engaged in what they are doing. However, these environments require further exploration to improve their learning affordances. For instance, assessing students' performance and collecting learning evidence is still in its early stages. This paper is primarily devoted to furthering our understanding of observation and assessment. In so doing, a virtual observation model has been developed to effectively map classroom-based observations with how people can be evaluated in virtual 3-D environments. The observation model has been applied to a multi-user virtual environment (MUVE) and examples that illustrate its potential effectiveness are provided. In essence, our research aims to support and enhance the learning experience by demonstrating the advantages of 3-D virtual worlds as a means for advancing learning processes.

HTML

XML

PDF

DOI: 10.3217/jucs-024-02-0149

Authors: Roberto Munoz, Rodolfo Villarroel, Thiago Barcelos, Alexandra Souza, Erick Merino, Rodolfo Guiñez, Leandro Silva

Abstract: Learning Analytics is the intelligent use of data generated from students with the objective of understanding and improving the teaching and learning process. Currently, there is a lack of tools to measure the development of complex skills in real classroom environments that are flexible enough to add and process data from different sensors and oriented towards a massive public. Based on this finding, we developed a free software system that permits to capture and to visualize a set of 10 body postures using the Microsoft Kinect sensor, along with the ability to track custom body postures and data from other sensors. The developed tool was validated by means of precision and usability tests. Furthermore, with the goal of demonstrating the potential of incorporating this type of software into the classroom, the software was used as a tool to give feedback to the teacher and to the students at the moment of giving and evaluating oral presentations. Also, a clustering analysis of data gathered from 45 student presentations indicate that presentations on similar topics with also similar complexity levels can be successfully discriminated.

HTML

XML

PDF

DOI: 10.3217/jucs-023-12-1238

Authors: Eser Çeker, Hüseyin Uzunboylu

Abstract: The aim of this study was to examine the research of open access studies conducted and published in 'Web of Science' indexed electronic journals and publications, as well as Master & Doctorate level open thesis studies from PQDT Open, OADT Org., EthOS and Council of Higher Education Thesis Center-Turkey on "web based/mobile teacher assessment". The total number of articles reached was 1330. According to the results depending on selected variables, there is a clear, steady increase in the number of publications on this subject throughout the years, and researchers from USA are the clear leaders in the order list; followed by UK, China, Australia and Taiwan. The leading universities are from USA, Taiwan and UK as well. "Articles", "Proceeding Papers" and "Meeting Abstracts" are the most preferred document types used. Following the number of articles on "computer science" and "engineering" research areas, articles classified to "Educational Research" areas are popular as well. Interestingly, "teacher assessment" oriented articles mostly were more focused to "assessing students by teachers", but very few examples exists for "assessing teachers by students". After this research and the results achieved, the authors of this article has conclusions that sufficient information exists to further study the title subject towards a new PhD study.

HTML

XML

PDF

DOI: 10.3217/jucs-023-12-1200

Authors: Teresa Magal-Royo, Jesus Garcia Laborda, Sara Price

Abstract: Computer adaptive language testing offers the possibility to research and practice m-learning using ubiquitous technology. Virtual education in m-learning uses conventional Learning Objects (LO) to enable the possibility of development of several tasks oriented towards language learning including the assessment and verification of skill improvements in second language acquisition. So far, there are few research papers on the impact of using new multimodal digital resources as LO in the design process of foreign language assessment tests through mobile devices because many English certification tests still continue to use traditional testing techniques [face-to-face and pen-and-paper assessments] combined with conventional digital environments oriented to virtual education. Learning languages requires not only new m-learning scenarios for assessment but also multimodal interactive environments to improve the user's experience during proficiency tests or language certification. Multimodal object learning such as augmented environments, learning games, spatial sound, etc. can be integrated into an assessment process to enhance the user´s experience by simulating natural communicative scenarios. The present article defines an innovative new m-learning scenario for listening comprehension assessment in an on-line test by implementing a multimodal audio learning source named binaural sound. Use of this technology will enable demonstration of other possibilities of human interaction to improve the user's experience in language learning through sound perception and its cognition from the user in an especific learning task.

HTML

XML

PDF

DOI: 10.3217/jucs-023-09-0845

Authors: Julia Kasch, Peter Van Rosmalen, Marco Kalz

Abstract: Although the terms scale and scalable are often used in the context of Open Online Education (OOE), there is no clear definition about these concepts from an educational perspective on the course level. This paper critically discusses the origins of these concepts and provides a working definition for educational scalability. A heuristic framework, which integrates four common educational design principles, is introduced, in order to study support and formative assessment and feedback at large scale. The proposed framework is presented, discussed and applied to five case studies. First qualitative results of the case studies show that the designs are relatively similar. The detailed study of their units of learning, however, indicates practices which can potentially be interesting for other MOOC developers to enhance their design and their scalability. Further research will apply the framework to zoom in on scalable best practices in MOOCs with a focus on scalable practices of formative assessment and feedback.

HTML

XML

PDF

DOI: 10.3217/jucs-022-07-0992

Authors: Giani Petri, Christiane Gresse von Wangenheim

Abstract: Educational games have been used as an innovative instructional strategy in order to achieve learning more effectively. However, it is essential to systematically evaluate such games in order to obtain sound evidence on their impact. Thus, the objective of this article is to present the state of the art on how to systematically evaluate educational games. Therefore, we performed a systematic literature review with an initial sample of 21,291 articles from which 11 relevant articles have been identified, describing 7 approaches to systematically evaluate educational games. Based on these studies we analyze how the approaches are defined (quality factors, theoretical constructs), operationalized (research designs, data collection instruments, data analysis methods), how they have been developed (development methodology) and evaluated (evaluated aspects, number of applications & data points and data analysis methods). As a result, we can confirm that exist few approaches to systematically evaluate educational games. The majority of the approaches are developed in an ad-hoc manner, not providing an explicit definition of the study, its execution and data analysis. We also observed that among the few encountered approaches no clear pattern emerges on which quality factors to evaluate. This shows that there exists a need for research on the definition and operationalization of educational game evaluations in order to obtain more valid and uniform results.

HTML

XML

PDF

DOI: 10.3217/jucs-022-01-0146

Authors: Zehra Ozcinar, Nihat Ekizoglu, Sezer Kanbul

Abstract: The aim of this study was to develop a scale to determine the educational usage of mobile communication apps among teacher candidates. 52 draft statements were prepared by researchers through literature review and opinions of teacher candidates. Statements were reduced to 40 statements based on expert opinions. The draft scale was administered to 412 teacher candidates who applied to the Pedagogical Formation Certificate Program at Near East University during the academic year of 2014-2015. Factor analysis was applied for the structural validity of the scale and Cronbach Alpha coefficients were calculated. Exploratory factor analysis was applied for the construct validity and Cronbach Alpha internal consistency coefficients were calculated for the reliability of the scale. 4-factor scale consisting of 33 statements was obtained based on the factor analysis. Cronbach Alpha internal consistency coefficient of the scale was calculated as 0.94. According to these results, it is revealed that the scale is highly reliable and this scale is considered to be a guide for the researchers who would like to use mobile communication technologies for educational purposes.

HTML

XML

PDF

DOI: 10.3217/jucs-021-13-1708

Authors: Enrique Flores, Alberto Barrón-Cedeño, Lidia Moreno, Paolo Rosso

Abstract: Nowadays, Internet is the main source to get information from blogs, encyclopedias, discussion forums, source code repositories, and more resources which are available just one click away. The temptation to re-use these materials is very high. Even source codes are easily available through a simple search on the Web. There is a need of detecting potential instances of source code re-use. Source code re-use detection has usually been approached comparing source codes in their compiled version. When dealing with cross-language source code re-use, traditional approaches can deal only with the programming languages supported by the compiler. We assume that a source code is a piece of text ,with its syntax and structure, so we aim at applying models for free text re-use detection to source code. In this paper we compare a Latent Semantic Analysis (LSA) approach with previously used text re-use detection models for measuring cross-language similarity in source code. The LSA-based approach shows slightly better results than the other models, being able to distinguish between re-used and related source codes with a high performance.

HTML

XML

PDF

DOI: 10.3217/jucs-021-12-1577

Authors: Rosario Ortego, Juan Antonio Gil Pascual

Abstract: In this research, a new methodology for the conceptual evaluation of students of a Massive Open Online Course (MOOC) is addressed. The electronic circuits MOOC course "Bases de Circuitos y Electrónica Práctica" (Fundamentals of Circuits and Practical Electronics) was used as a model. The conceptual evaluation of the course was measured by the semantic proximity between ten relevant concepts of the course. This semantic structure involves conversion and reduction processes through Pathfinder associative networks and the minimum spanning tree. The gain between the start and the end of the course is calculated by statistical tests considering the similarities between the students' networks and the teacher's network, as a reference network. The results show significant gains, particularly when evaluating courses with badges. We propose this evaluative model as a recommended generic model that can be integrated into MOOCs due to its immediate and online response.

HTML

XML

PDF

DOI: 10.3217/jucs-021-08-1086

Authors: Lluís Safont, Sergi Villagrasa, David Fonseca, Ernest Redondo

Abstract: Using enhanced learning technologies (TEL) including immersive virtual reality environments, we are seeking to achieve a new way of assessing subjects of 3D arts. We have developed a project based on Scenario Centered Curriculum (SCC), where the students have to think, design, convey, validate, and build a civil project using new technologies that help in the assessment process. We have used gamification techniques and game engines to evaluate planned tasks in which students can demonstrate the skills they developed in the scenarios. The assessment is integrated in the creation of a 3D complex model focused on the construction of a building in a virtual space. This whole process will be carried out by gamification techniques to embed the assessment of the 3D models with the objective of improving students learning.

HTML

XML

PDF

DOI: 10.3217/jucs-021-08-1022

Authors: Angélica Lozano-Alvarez, Juan Asensio-Pérez, Guillermo Vega-Gorgojo, Alejandra Martínez-Monés

Abstract: ePortfolios are recognized as effective tools for formative assessment. Learning outcomes are demonstrated by a number of work samples, which provide a good overview, not only of the acquired competences, but also of the path followed to get there. However, ePortfolios do no provide means for their automatic integration in Distributed Learning Environments, where the work samples (learning evidence) are scattered across a number of systems, such as Virtual Learning Environments and Web 2.0 tools. This paper proposes a scalable approach to tackle evidence collection in DLEs, by aligning the generated artefacts with the pedagogical purpose the teacher aims at. The proposal takes advantage of Learning Design practices, making alignment explicit to guide the technological infrastructure for the automatic collection and organization of pieces of evidence in an ePortfolio and thus reducing the associated burden on teachers and students. The feasibility of the proposal was assessed by means of an evaluation study, in which a University teacher designed and enacted an authentic collaborative learning situation in the context of a master's degree course.

HTML

XML

PDF

DOI: 10.3217/jucs-021-08-1001

Authors: Enosha Hettiarachchi, Enric Mor, M. Huertas, Ana-Elena Guerrero-Roldán

Abstract: E-assessment is the process where technologies are used for the management of the end-to-end assessment process. It involves the use of any technology-based method that allows systematic inferences and judgments to be made about the student's skills, knowledge and capabilities acquisition. In online learning, technology is used to facilitate the implementation of the appropriate models that can be used for both teaching and learning. In addition, e-assessment technologies allow creating tools able to improve student engagement and performance, offer possibilities to practice competences and skills, provide personalized feedback and improve student motivation. This paper presents a general e-assessment tool with a formative e-assessment model. The e-assessment tool under a formative assessment model is integrated into the learning process and provides aligned and authentic learning experience, focused on skill and knowledge acquisition. Results show that the e-assessment tool and its model had a positive effect on students' performance and learning. The tool itself is capable of providing students with personalized feedback, guidance and marks as well as automatically storing and tracking students' progress, competences and outcomes, going beyond the e-assessment approach based mainly on quizzes. The e-assessment tool and model offer innovative assessment approaches, mainly focusing on skill assessment where dynamic and interactive questions were provided with the guidance of personalized feedback and hints.

HTML

XML

PDF

DOI: 10.3217/jucs-021-01-0066

Authors: Sean Goggins, Wanli Xing, Xin Chen, Bodong Chen, Bob Wadholm

Abstract: This study proposes a process-oriented, automatic, formative assessment model for small group learning based on complex systems theory using a small dataset from a technology-mediated, synchronous mathematics learning environment. We first conceptualize small group learning as a complex system and explain how group dynamics and interaction can be modeled via theoretically grounded, simple rules. These rules are then operationalized to build temporally-embodied measures, where varying weights are assigned to the same measures according to their significance during different time stages based on the golden ratio concept. This theory-based measure construction method in combination with a correlation-based feature subset selection algorithm reduces data dimensionality, making a complex system more understandable for people. Further, because the discipline of education often generates small datasets, a Tree-Augmented Naïve Bayes classifier was coded to develop an assessment model, which achieves the highest accuracy (95.8%) as compared to baseline models. Finally, we describe a web-based tool that visualizes time-series activities, assesses small group learning automatically, and also offers actionable intelligence for teachers to provide real-time support and intervention to students. The fundamental contribution of this paper is that it makes complex, small group behavior visible to teachers in a learning context quickly. Theoretical and methodological implications for technology mediated small group learning and learning analytics as a whole are then discussed.

HTML

XML

PDF

DOI: 10.3217/jucs-020-09-1304

Authors: Luis Llana, Enrique Martin-Martin, Cristóbal Pareja-Flores, J. Velázquez-Iturbide

Abstract: Currently, automated systems for program submission and assessment play a central role in the teaching of programming. A number of such systems have been developed in the last two decades. However, their adoption in regular programming teaching presents an obstacle: the overhead work required for the design of each problem, for compilation of problem collections, and for mundane management tasks. An open challenge is to make these systems easier to use and to reduce to a minimum the number of management tasks. In this article we present the FLOP system, which was developed to satisfy this goal. The contribution of the article is twofold. On the one hand, we present the FLOP system itself and its user-friendly features. On the other hand, we present in detail the user-centered design process used to design and enhance the ease of use of FLOP. Several actions were undertaken to inquire users’ concerns and needs, with a usability evaluation of FLOP conducted with students being the most fruitful action.

HTML

XML

PDF

DOI: 10.3217/jucs-020-01-0039

Authors: Christian Haintz, Karin Pichler, Martin Ebner

Abstract: Question-driven Audience Response Systems (ARSs) are in the focus of research since the 1960s. Since then, the technology has changed and therefore systems have evolved too. This work is about conception and implementation of the web-based ARS RealFeedback which uses the principle of bring your own device (BYOD). A state-of-the-art analysis compares the features of existing web-based ARSs. The most important findings are used for the conception and the implementation of the system. Thinking-aloud tests, and the first usages during lectures confirm that the chosen requirements are very significant and valuable for lecturers.

HTML

XML

PDF

DOI: 10.3217/jucs-019-18-2698

Authors: Katerina Kalyvioti, Tassos Mikropoulos

Abstract: Research suggests that Virtual Reality has a key role in the development of new diagnostic tools in neuropsychology and shows great rehabilitative potentials for individuals with specific neurological, intellectual and cognitive disabilities. In the case of dyslexia, a neurodevelopmental reading disorder, the use of Virtual Reality technologies has only been recently documented in a handful of studies. The main focus of these studies has been the identification of visuospatial strengths, the exploration of nonverbal problem solving treatment and the increase of awareness in educators and parents with children with dyslexia. Even fewer are the studies of Virtual Reality and the lifelong memory difficulties of adult individuals with dyslexia. With a more clinical, rather than technological, perspective the goal of this paper was to design specialized tasks in virtual environments to be part of a screening process/assessment of characteristic memory difficulties for undergraduate students diagnosed with dyslexia. Results showed that there were no statistically significant differences in the performance of students with dyslexia and students without dyslexia, a finding which highlights the development and successful use of compensatory memory strategies by the participants with dyslexia. Taking into consideration the real life representations, the multisensory approach, the increased sense of presence, the well-designed tasks and the recorded positive attitude of all participants, the study concludes that the use of Virtual Reality in neurological and neurodevelopmental memory disorders will be innovative and suggests that hands on Virtual Reality applications, become an indispensable part of these deficits' cognitive assessment and rehabilitation.

HTML

XML

PDF

DOI: 10.3217/jucs-019-07-0932

Authors: Loc Hoang, Ngamnij Arch-Int

Abstract: Currently, the assessment of learners in conventional e-learning systems is only one dimension in which learners are required to produce answers, for example, by selecting multiple-choice, true/false, or matching answers or by giving short answers. This type of assessment still lacks interactions among the learners, and thus, it might not fully support learning. Many researchers have endeavored to propose an open-ended question method for evaluation, but their methods still focus on content assessment rather than learners' activities, which again lacks interactions among the learners. This paper concentrates on creating a new assessment method using open-ended questions with the aim of enhancing collaborations, activities and interactions of learners at the same time. The objectives are as follows: 1) to develop a process model for multidimensional assessment (M-DA) to enable effective learning; 2) to develop free-text answer assessments using a vector space model and a semantic extraction model; and 3) to develop an algorithm for evaluating learners based on a M-DA to encourage learners' activities. In addition, we created an environment for learners to be actively assessed and to interact with others when studying online. Two groups of parallel learners taking an e-course were tested on the two systems in a virtual learning environment. The results of the experiment noted that the system with multidimensional assessment showed a better outcome than the system without M-DA.

HTML

XML

PDF

DOI: 10.3217/jucs-019-05-0672

Authors: Amandeep Dhir, Khalid Buragga, Abeer Boreqqah

Abstract: Twitter is a well-known Web 2.0 microblogging social networking site that is quite popular for organizing events and sharing updates. It provides just in time communication, social connectivity and immediate feedback through Web, smartphones, tablet PCs, etc. The use of Twitter has also attracted educators and researchers due to its growing popularity among students, teachers, and academic communities as a whole. This study provides a critical review of Twitter use in educational settings. By practicing a systematic research methodology in the selection and review of literature, different pedagogical and instructional benefits and drawbacks of Twitter use in education were discussed. Based on these discussions, it was discovered that Twitter has positive impact on informal learning, class dynamics, motivation, as well as the academic and psychological development of young students. However, the potential long-term impact of Twitter on academic performance of students and its long-term effect on learning is still worth investigating. 

HTML

XML

PDF

DOI: 10.3217/jucs-018-10-1279

Authors: Rafael Duque, Lars Bollen, Anjo Anjewierden, Crescencio Bravo

Abstract: The analysis of problem-solving activities carried out by students in learning settings involves studying the students' actions and assessing the solutions they have created. This analysis constitutes an ideal starting point to support an automatic intervention in the student activity by means of feedback or other means to help students build their own knowledge. In this paper, we present a model-driven framework to facilitate the automation of this problem-solving analysis and of providing feedback. This framework includes a set of authoring tools that enable software developers to specify the analysis process and its intervention mechanisms by means of visual languages. The models specified in this way are computed by the framework in order to create technological support to automate the problem-solving analysis. The use of the framework is illustrated thanks to a case study in the field of System Dynamics where problem-solving practices are analysed.

HTML

XML

PDF

DOI: 10.3217/jucs-018-03-0410

Authors: Isabella Rega, Francesca Fanni

Abstract: The case study proposed in this article is the MELISSA project - Measuring E-Learning Impact in primary Schools in South African disadvantaged areas. MELISSA measures the impact of exposure to ICTs in teacher training/learning applying the Self-Efficacy construct. The intention is here to understand and analyse changes in attitudes to and uses of ICTs in term of Computer and Teacher Self-Efficacy. To accomplish this goal, the MELISSA team applied a mixed investigative method, merging quantitative and qualitative methodologies.

HTML

XML

PDF

DOI: 10.3217/jucs-017-10-1474

Authors: Ján Paralič, Christoph Richter, František Babič, Jozef Wagner, Michal Raček

Abstract: Knowledge practices mirroring is one of the central elements of the Trialogical Learning Approach as it is supposed to be a driving force in processes of practice transformation and knowledge creation. The exploitation of historical logging data holds promise to provide a great deal of information about group activities without requiring additional efforts for recording of events by the users. Building on the Trialogical Learning Approach as well as related work in the fields of Data Mining and Knowledge Discovery, Computer-Supported Collaborative Learning, and Information Visualization, this paper suggests high-level requirements for mirroring tools in support of practice transformation and introduces a software tool called Timeline-Based Analyzer (TLBA) that was designed and developed in response to these requirements. One of the main TLBA features is the possibility to define patterns as sequences of relevant actions that resulted into critical moments in investigated practices. Such kind of patterns might also represent conceptually interesting practices that emerged within a particular context - either being positive (a sort of good practices), or negative (bad practices). The usability of the whole analytical solution has been tested through the first iteration of several practical experiments and case studies. One of them is described in this paper and illustrates how TLBA can be used to support collaborative analysis and mirroring. The results of these evaluations have been used for continued improvement of the TLBA in order to provide a stable and intuitive tool not only for researchers, but for daily use of teachers or other users.

HTML

XML

PDF

DOI: 10.3217/jucs-016-15-1992

Authors: Kris Mcglinn, Eleanor Neill, Alan Gibney, Declan Sullivan, Dave Lewis

Abstract: The promise of smart buildings (SBs) is a safer more productive environment for users and a more operationally efficient building for owners. The automation of building function is highly dependent on sensing devices and Smart Building Applications(SBAs), which are often only evaluated in situ post deployment, making re-development costly. In this paper we explore our experiences developing a Simulated Context (SimCon) Model which currently supports taking information from a Virtual Reality (VR) SB and converting it into three types of location context to conduct early rapid evaluation of location based SBAs. This model is expressed using the Sensor Modelling Language (sensorML). It also explores the integration of this model into the Industry Foundation Classes (IFC) for modelling and simulating SBs. It also details usability evaluations of the SimConfig and SimConViz Tool for improving evaluation during the design phase of smart building development life cycle.

HTML

XML

PDF

DOI: 10.3217/jucs-016-08-1075

Authors: Jeroen Stragier, Pieter Verdegem, Gino Verleye

Abstract: As ICT provide a lot of possibilities, high expectancies exist towards the electronic public service provision. All governments are increasingly establishing their e-strategies. Nevertheless, both in research as among practitioners, some questions are formulated towards the current approaches. E-government policy has to deal with two main challenges: how to offer new and better services for the same (or lower) budget and how to increase user uptake? The challenges relate to both the efficiency of e-government, as its effectiveness. In order to further develop e-government strategies, governments need a thorough knowledge base to (a) to evaluate their current ways of working and (b) to found their activities of the future. In this paper we consider trends in e-government measurement and we discuss ongoing research towards the development of an e-government monitor for the Belgian government. This monitor will consist of different types of information from various sources, providing the necessary knowledge for all stakeholders involved to underpin their current and future e-strategies.

HTML

XML

PDF

DOI: 10.3217/jucs-015-07-1472

Authors: Francisco Jurado, Ana Molina, Miguel Redondo, Manuel Ortega Cantero, Adam Giemza, Lars Bollen, H. Hoppe

Abstract: Learning to program is an important subject for students of Computer Science. Mentoring these students is a time-consuming and complex task. In this paper, we present a learning and tutoring environment that integrates task/solution delivery, assessment support and tutor’s annotations, by extending Eclipse to a "Real World Integrated Development Environment". We will present a distributed system that uses Tuple Space architecture to integrate Eclipse with an evaluation module and a hand-writing annotation feature.

HTML

XML

PDF