Game Learning Analytcs for Evidence=Based Serious Games

Summary of a talk by Baltasar Fernandex Manjon at CELDA 2019

ie.ft.com/uber-game

Serious games

• -         Have been used successfully in many domains – medicine, military
• -         But low adoption in mainstream education
• -         So we say we’re working in ‘game-like simulation’

http://getbadnews.com

-         Fake news, trolls, e-influencers

http://play.centerforgamescience.com

http://centerforgamescience.org

http://www.re-mission2.org

• -       Has been formally evaluated

Citizen science

• -         Uses games for crowdsourcing
• Also:
• -         Educational versions of commercial games

Do serious games actually work?

• -         Very few sg have been formally evaluated
• -         Evaluation could be as expensive as producing the game  - difficult to get funding
• -         It is difficult to deploy the game in the classroom

Learning analytics

• -         Long and Siemens

Game Analytics

• -         Application of analytics to game dev and research
• -         Telemetry – info obtained at a distance
• -         Game metrics – interpretable measures of data related to games
• -         Mostly used for commercial purposes; proprietary

Business analytics

• -         From what happened, to why it happened, to what will happen, to how I can make it happen
• -         Ie., hindsight – insight – foresight
• -         Needs all the dat
• -         Now being used in MOOCs, because they have so much data

Game Learning Analytics (GLA)

• -         Learning analytics applied to serious games
• -         Collect, analyze and visualize

Uses of GLA

• -         Game testing – eg., how many finish, avg. time to completion
• -         Game deployment in class – tools for teachers, eg. ‘stealth’ student evaluation
• -         Formal game evaluation

RAGE – game analytics (using xAPI)

Beaconing – game deployment

GLA or Informagic?

• -         Informagic - false expectations of gaining full insight on the game educational experience based on shallow data
• -         Need to set realistic expectations – most of the games are black boxes

Minimum Requirements for GLA

• -         Need access to what’s going on during the game
• -         Need access to the game ‘guts’, or the game must communicate
• -         Need to understand the meaning of the data – access to developers
• -         Also must consider ethics of data collection
• o   Are user informed?
• o   Is data anonymized
• o   Note: GDPR – creates an overhead load

GLA structure

• -         Need to be based on learning objective
• -         Based on traces + analysis
• -         Different levels of design – LAM

Experience API

• -         New defacto standard, becoming an IEEE standard
• -         e-UCM group in collaboration with ADL for profile for serious games (xAPI-SG)
• -         xAPI-SG defines a set of verbs, activity types, and extensions

Game trackers / Analytics frameworks as open code

-          http://github.com/e-ucm

Systematization of Analytics Dashboards

• -         Provided analytics uses xAPI-SG, dashboards do not require additionalconfiguration
• -         You can also do real-time analytics and warnings – more complex to do
• -         We were surprised to find how hard it is to make a visualization understandable by the average teacher – eg. Teacher interprets difficulty as ‘you are in Facebook’

uAdventure

• -         uAdventure tool (on top of Unity)
• -         game development platform
• -         includes analytics

Overview of research – 87 papers

• -         GLA purposes – mostly assessment, n-game behavious; little on interventions\techniques: mostly classical linear analytics, clusters; neural nets not broadly applied
• -         Stakeholders – teachers came third; not widely deployed
• -         Focus – to teach, most domains math and science, small sample sizes
• -         Assessment – mostly pre-post assessments
• -         Method – 2 steps – game validation phases, game deployment phase

Research questions

• -         Can we predict student knowledge after playing the game
• o   With/without pretest
• o   Can we use for evaluation?
• -         Need to have greater student numbers for analysis to be useful
• -         Result – using naïve bayes – yes, we can predict student outcomes
• -         Not sure about use for evaluation

Case Study

• -         Game on Madrid Metro used with Down Syndrome students

Case

• -         Connectado – high school cyberbullying
• -         Some minigames you can never win
• -         Result – increase in cyberbullying perception

Simva

• -         Tool used for scientific validation of serious games
• -         Goal: to simplify the validation and deployment