Ready for Digital Learning? A Mixed-Methods Exploration of Surveyed Technology Competencies and Authentic Performance Activity

Introduction

Driven by innovation in digital technologies and educational praxis, institutions of higher learning around the globe seek to implement successful programs of digital learning, including forms of online, distance, blended and mobile education (Aparicio et al. 2016; Crompton et al. 2016; Siemens et al. 2015). Readiness for digital learning is an international research domain addressing factors influencing successful technologyenriched education. More specifically, this domain produces frameworks, instruments and empirical studies addressing e-learning readiness/preparedness (Leigh and Watkins 2005; Mosa et al. 2016; Parkes et al. 2015), online-learning readiness (Farid 2014; Horzum et al. 2015), and mobile-learning readiness (Lin et al. 2015).

Researchers in this field generally conduct either macro-level investigations of organizations, regions and countries (Beetham and Sharpe 2007; Bui et al. 2003), or micro-level studies of students (Dray et al. 2011; Parkes et al. 2015) and teachers (Gay 2016; Hung 2016). At the micro level, digital competencies, defined as knowledge, skills and attitudes supporting purposeful and effective use of technology (Ala-Mutka 2011), figure as highly significant readiness factors within conceptual frameworks (AlAraibi et al. 2016; Demir and Yurdugül 2015) and instruments (Aydın and Tasci 2005; Hung 2016; Hung et al. 2010; Lin et al. 2015; Parasuraman 2000; Watkins et al. 2004). However, readiness instruments tend to adopt unidimensional and inconsistent conceptualizations of digital abilities, thus lacking the fine-tuned, operational approaches of digital-competency frameworks (Blayone et al. 2017a, b).

To address this shortcoming, researchers at the EILAB, University of Ontario Institute of Technology, Canada, are exploring the General Technology Competency and Use (GTCU) framework (Desjardins 2005; Desjardins et al. 2001; Desjardins and Peters 2007; Desjardins and vanOostveen 2015) as a readiness apparatus. In this capacity, the GTCU has several strengths. First, it offers a multi-dimensional model of digital competency and use, built on the IEEE’s definition of computer hardware: Bphysical equipment used to process, store, or transmit computer programs or data^ (IEEE 1990). Second, it has been used successfully for a decade to conceptualize and measure the digital competencies of students and teachers for fully online learning (Desjardins and vanOostveen 2015; Desjardins et al. 2010; DiGiuseppe et al. 2013, b). Finally, it integrates the Digital Competency Profiler (DCP), an online application incorporating a validated instrument, and offering profile-generation and comparison functionality (Fig. 1). Importantly, owing to growing international adoption, the DCP is being translated and adapted for non-Western contexts of use, such as Eastern Europe (Blayone et al. 2017a, b).

One limitation of the DCP as a readiness instrument, however, is that researchers have not yet gathered data relating self-reported digital-competences to observed performance. This is a significant gap because self-reports are sometimes unreliable predictors of performance. Combining instrument development with observational research, although onerous, can help a research team improve survey indicators, and draw more reliable inferences from self-reports (Bradlow et al. 2002; Hargittai and Shafer 2006; Litt 2013; van Deursen et al. 2015).

Fig. 1 Individual profile visualization and comparison in the Digital Competency Profiler

This study adopts a mixed-methodology, aligned with the target educational environment of fully online, social-constructivist learning (Blayone et al. 2017a, b; vanOostveen et al. 2016), for comparing reported digital competencies and authentic performance activity. The literature review organizes observational practices of leading digital-competency researchers, providing a theoretical context for the research design developed for this study. Exploratory correlational findings from 15 pilot observations are reviewed, and participant case-studies, based on recorded activity and interviews, are reported. Based on the full set of data and findings, a threshold approach to using the DCP as a readiness tool is proposed for further study. Finally, limitations are addressed and key research contributions identified.

Literature review

To build a methodology that leveraged the EILAB’s digital affordances, extensive technical configuration and pilot-testing were coupled with a purposive literature review organizing the observational-design choices made by other digital-competency researchers. These choices are addressed thematically below under the following headings: (a) context of performance activity; (b) computer devices used; (c) observer positioning, and data streams of interest; (d) activity-design orientations; and (e) criteria for analysing performance.

Context of performance activity

Two major observational studies, addressing digital skills and involving hundreds of participants, were instigated in controlled university settings (Hargittai and

Shafer 2006; van Deursen and van Dijk 2010). Others have sacrificed sample size to capture activity in Breal world^ contexts. For example, Asselin and Moayeri (2010) leveraged usability software to capture the routine computer activity of two students in their homes. With respect to observing mobile-device interaction, Esbjörnsson et al. (2006) argued that a laboratory may not adequately simulate the context where mobile devices are typically used. Some have responded to this concern by introducing peripheral sounds and movement into a lab to simulate Breal world^ contexts that split an individual’s attention (Sun and May 2013).

Computer devices used

Not all researchers report what devices were used for performance observation (e.g., Eshet-Alkalai and Amichai-Hamburger 2004; Greene et al. 2014). However, when reported, desktop computers tend to dominate (Hargittai and Shafer 2006; van Deursen 2010). To account for their growing popularity, Litt (2013) and Park (2015) called for greater inclusion of mobile devices in digital-competency measurement studies. To date, aside from small-sample studies (Jayroe and Wolfram 2012), large-scale observational research, like that carried out with desktop computers (Hargittai and Shafer 2006; van Deursen and van Dijk 2010), has not incorporated mobile devices.

Observer positioning, and data streams of interest

Conducting observations in a university setting, Hargittai (2002) and Hargittai and Shaffer (2006), seated the researcher behind participants to help maintain levels of motivation and flow during multi-task activities. Screen activity and spoken thoughts were captured by system-recording software on the participant desktop. Van Deursen and van Dijk (2010) adapted Hargittai’s approach. Rather than using recording software, the researcher was present to time task completion and note correct/incorrect responses (van Deursen and van Dijk 2010). Eshet-Alkalai and Amichai-Hamburger (2004) neither used recording software nor were conspicuous during activity. They focused their involvement on designing activities, and assessing outcomes after the fact.

Activity-design orientations

Focused on digital inclusion, Van Deursen and Van Dijk (2010) designed Bfactbased tasks^ with a specific correct action/answer. BOpen-ended tasks were avoided because of the ambiguity of interpretation of the many potential answers^ (van Deursen and van Dijk 2010, p. 901). Addressing higher-order thinking skills, Eshet-Alkalai and Amichai-Hamburger (2004) designed observations using Bauthentic tasks,^ which are aligned with professional practice, offer high complexity, favour ill-defined procedures, and allow for competing solutions (Herrington et al. 2006). Ding and Ma (2013) help conceptualize these different approaches by calling attention to three key facets of activity design: (a) level of procedural structure, (b) degree of goal definition, and (c) activity context (e.g., academic or daily life). From a Problem-based Learning (PBL) perspective, Savin-

Baden (2000), similarly proposes five activity orientations focusing largely on the degree to which goals and procedures are defined.

Criteria for analyzing performance

Several patterns emerge in relation to analysis methodology. First, researchers tend to focus on either processes or outcomes. Second, they tend to measure the speed and correctness of task completion, or use a rubric to assess the quality of open-ended activities. Finally, they tend to score the activity in real time, or conduct a delayed analysis of recorded data streams. To review some examples, Hargittai and Shafer (2006) recorded task processes (i.e., screen activity and voiced thoughts), but focused their analysis on outcomes (without consideration of time), reporting percentage of tasks completed successfully and the perceived difficulty of each task. Similarly focused on well-defined tasks, Van Deursen and van Dijk (2010) scored all participants on successful completion and time required, reporting average percent of tasks completed correctly and average time spent. This outcomes focus was somewhat balanced by also noting common procedural errors. Eshet-Alkalai and Amichai-Hamburger (2004) assessed the outcomes of more open-ended activities using several rubrics. For example, a photo-visual activity, which involved modelling a theatre stage in a multimedia application, was assessed in relation to completeness, number of elements and complexity.

It is noteworthy that among these researchers, digital recording and analysis of participant data-streams (e.g., hand-device interactions, facial expressions, physiological responses, eye-movements, etc.) do not figure prominently. Yet, new methodologies incorporating a variety of digital-research affordances are making inroads into the domain (Bhatt and de Roock 2014; Knoblauch 2012).

Research questions

Integrating insights from the reviewed practices, and seeking to incorporate emerging methodologies supported by the EILAB’s research affordances, the twin purposes of this study were defined as: (a) proposing a digitally rich, observational-research design, and (2) deepening the empirical foundations for DCP as a readiness instrument by exploring relationships between self-reports and authentic digital-learning performances. This small-sample study was intended neither to validate the DCP statistically nor achieve broadly generalizable results. The research questions are:

1. How do self-reported digital competencies, collected via DCP indicators, relate to performance, as instigated, recorded and analysed through the proposed, digitalresearch design?

2. What problems and situational influencers of performance are observed through dual-perspective, video-analysis, case-studies that are beyond the scope of the DCP’s digital-competency, self-report indicators?

3. Based on the full body of findings, how does the DCP function most effectively as a readiness-assessment application?

Data collection

A mix of qualitative and quantitative data were generated to address the research questions. These included survey responses, audio-visual recordings of participant activities and post-activity interviews, and performance artefacts. In addition, selected baseline statistics were calculated from the accumulated DCP database of almost 700 profiles, at the time of writing.

Self-report instrument

Self-reported digital competencies were collected online using DCP items. These consisted of: (a) seven socio-demographic items, (b) combined, device-ownership and purpose-of-use items, and (c) 21 (device + action) indicator groups, covering the three primary GTCU dimensions (seven groups for each of the epistemological, informational and social competency dimensions). These groups presented three identical action-items each coupled with a different device type, following a consistent structure: BI use a specific hardware device to perform a software-level action.^ The devices included computer/laptop (as a single type), smartphone and tablet. Two (indirect competency) measures were attached to each device-action item, using 5-point Likert scales. The frequency with which an individual performs a device-specific action was measured from (1) never to (5) daily. Frequency of action is an important indicator of competency because transferable procedural knowledge is reinforced through repeated activity (i.e., practice leads to acquired ability). The confidence with which an individual performs a devicespecific action was measured from (1) not confident to (5) very confident. Deviceaction confidence, a key motivational facet of competency, addresses an individual’s willingness to explore novel situations (Bandura 1993) with a particular set of tools. It is expected that individuals are able to differentiate (a) the general frequency with which they perform certain actions, and (b) their relative levels of comfort performing an action on a particular type of device (Desjardins et al. 2010; DiGiuseppe et al. 2013).

Performance activities design

As outlined in Table 1, three scenarios were designed to: (a) incorporate a mix of interaction types, (b) provide a significant degree of challenge, and (c) adopt the characteristics of an authentic activity (Herrington et al. 2006). Each scenario was comprised of six (goal-driven and procedurally flexible) tasks aligned to a DCP item. For example, Scenario 1, Task 1, asked the participant to find a journal article on a research topic of interest. This task was mapped to DCP item I16, which related to searching journals online. (See Appendix 1 for the full list of DCP indicators and scenario mappings.)

Taken together, these scenarios incorporated 15 DCP activity items drawn from three dimensions of competency defined by the GTCU: social (S), informational (I) and epistemological (E) (Desjardins 2005). All scenarios shared one common item (S11:

Table 1 Authentic performance scenarios and task-to-DCP mappings

|Scenario|Premise|Task mappings| |---|---|---| |1: Presentation|Reflecting on a research topic from your own educational|I16, I17, E22, E23, S12,S11| |preparation|experience, this six-step activity asks you to prepare|| ||and share materials for a fictitious presentation (for a|| ||general academic audience).|| |2: Democracy|The World Values Survey (WVS) is a global network|I15, E25,S13, E26, S8,S11| |data analysis|of social scientists studying changing values and their|| ||impact on social and political life. In preparation for|| ||a symposium on education and democracy, this six-step|| ||activity asks you to select and manipulate data, and|| ||share this data with a collaborator via text messaging|| ||and email.|| |3: Conceptual|The Community of Inquiry (CoI) framework is a model|S13, I18, E24, I20, S14,S11| |model drawing|of collaborative learning. The Fully Online Learning|| ||Community (FOLC) is another model derived from|| ||the CoI. One difference between these models is that|| ||CoI consists of three dimensions (Social, Cognitive|| ||and Teaching Presence), but FOLC consists of two|| ||dimensions (Social and Cognitive Presence) situated|| ||within a fullyBdigital space.^ This six-step activity|| ||asks you to develop and share conceptual materials|| ||for a FOLC research project.||

The bolded items are those used in each of the three activities. The italicized items are those used in two of the three activities

communicating via email). A second item (S13: using an online collaboration platform) was shared between Scenario 2 and 3 only. This minimal repetition was required to maintain scenario authenticity, and facilitate the collection of activity artefacts.

Post-activity interviews

Post-activity interviews were conducted and recorded. They were guided by five questions addressing scenario difficulty, levels of comfort, and readiness to engage in fully-online learning. They also included open discussion, prompted by the researcher with a request for feedback.

Participants

Following approval from UOIT’s Research Ethics Board, participants were recruited from the student population at University of Ontario of Technology (UOIT), Canada through an email recruitment process facilitated by departmental administrators. A target sample size was initially set at 30 following a rule-of-thumb in qualitative research related to expected saturation points (Mason 2010). However, owing to an aggressive time frame for data collection, the researcher adopted 15 as the minimum sample for achieving saturation following Bertaux (1981). Participants were randomly assigned to one of three scenario groups. As shown in Table 2, participants were 73% female, and primarily undergraduate students. The average age was 29, with six participants over 30. Most were studying education, with three indicating another specialty. Most (80%) used a laptop computer, smartphone, and tablet regularly.

Table 2 Participants grouped by activity

|||Participant|Gender|Age|Educ. level|Domain|Devices used|Principal use| |---|---|---|---|---|---|---|---|---| |Activity|1|P1|Male|40|Bachelors|Education|Laptop|Study| ||||||||Smartphone|Social| ||||||||Tablet|Social| |||P3|Male|26|Bachelors|Education|Laptop|Study| ||||||||Smartphone|Social| ||||||||Tablet|Study| |||P6|Female|23|Graduate|Education|Laptop|Study| ||||||||Smartphone|Social| ||||||||Tablet|Entertainment| |||P8|Female|34|Bachelors|Education|Laptop|Social| ||||||||Smartphone|Creative| ||||||||Tablet|Study| |||P12|Female|24|Bachelors|Education|Laptop|Work| ||||||||Smartphone|Entertainment| ||||||||Tablet|Entertainment| |||P13|Female|46|Bachelors|Education|Laptop|Study| ||||||||Smartphone|Social| ||||||||Tablet|Entertainment| |Activity|2|P2|Female|24|Bachelors|Education|Laptop|Study| ||||||||Smartphone|Social| ||||||||Tablet|Entertainment| |||P4|Male|41|Bachelors|Education|Laptop|Study| |||P5|Female|22|Bachelors|Biology/Educ.|Laptop|Work| ||||||||Smartphone|Entertainment| ||||||||Tablet|(Not reported)| |||P10|Female|22|Bachelors|Science/Educ.|Laptop|Study| ||||||||Smartphone|Other| |Activity|3|P7|Female|23|Bachelors|Education|Laptop|Study| ||||||||Smartphone|Social| ||||||||Tablet|Teaching| |||P9|Male|22|Bachelors|Education|Laptop|Study| ||||||||Smartphone|Work| |||P11|Female|30|Bachelors|Education|Laptop|Study| ||||||||Smartphone|Entertainment| ||||||||Tablet|Entertainment| |||P14|Female|22|Bachelors|Commerce|Laptop|Entertainment| ||||||||Smartphone|Work| ||||||||Tablet|Entertainment| |||P15|Female|51|Post-Doctoral|Economics|Laptop|Work| ||||||||Smartphone|Social| ||||||||Tablet|Entertainment|

The bolded items are the reported primary uses of tablet devices, the device type used by participants during their performance activity

Sequenced procedures and lab setup

The online survey was completed by participants before visiting the EILAB. As shown in Fig. 2, a custom, activity-station setup was wired, designed (and extensively tested) by the lead researcher to support HD video recordings of the participant’s hand-device interactions and facial expressions while interacting with a tablet. To maintain line-of-sight and image quality, camera and tablet positioning, participant seating, and lighting conditions were carefully configured. A studio-quality, ambient microphone was installed to facilitate audio recording of interaction sounds and voiced expressions/thoughts.

Upon arrival at the EILAB, the participant was greeted and seated at the activity station. Two tablet devices were presented for selection. These were the Apple iPad Air and Samsung Galaxy Table 2, both wirelessly connected to the university network, configured with a variety of Google and social-media applications, and made accessible with shared EILAB accounts. (Participants were permitted to install additional applications using the shared account.) The researcher, using a script, reviewed procedures and optimal body/device positioning, and then exited to the observation room. When the participant was ready, the activity description was displayed on a front-facing display (set to control head tilt), and the activity/ recording started. The researcher, making occasional notes, maintained optimal positioning of both cameras, and monitored audio-video streams (using AXIS Camera Station and Noldus Media Recorder on a work-station with twin, 4 k– displays and a soundboard). If the participant was still active after 50 min, a verbal prompt was given to finish. After each session, a post-activity interview was recorded, and digital assets produced during the activity were located, and aggregated on secure storage.

Fig. 2 Performance activity recording station in EILAB

Analysis

Analysis was conducted in several phases beginning with the self-report data, moving to the recorded audio-video data, and then to a comparison of these data sets. Next, case studies were produced based on the broader qualitative data set, and the entire body of findings were reviewed to position the DCP as a readiness tool.

Self-reported competency data reduction and scoring

Self-report data for each participant were exported for initial analysis in SPSS. To reduce the data set, a total-reported-competency score was calculated by summing six reported values for each of six mapped activity items, addressing frequency and confidence of activity using a desktop/laptop, smartphone and tablet. (This produced a maximum total score of 180.) The rationale for summing these values was rooted in the GTCU’s operational logic, which considers: (a) the frequency with which an individual performs an action, and (b) the confidence felt towards this action, as twin, synergistic indicators of competence (Desjardins 2005). The rationale for including values from three device types (computer/laptop, tablet and smartphone) was rooted in the logic that one’s capacity to perform an action on a computer and/or smartphone will transfer to a tablet, even if one does not generally use a tablet for that action. Moreover, differences in device-use preferences (e.g., using a laptop for productivity, a tablet for entertainment and a smartphone for communication) were such that the same scores were not typically multiplied by three.

Performance audio-video reviewing and coding

The audio-video data were imported into Noldus The Observer XT for coding and annotation. The coding scheme adapted insights from the literature, and extended these through an emergent process. The final scheme addressed:

- & Task and scenario, execution strategy/sequence and durations

- & Problems or breaks in performance flow related to navigation, software-application procedures, and strategy

- & Technical failures

- & Competency demonstrations, including Bskillfulness^ (related to the application of prior knowledge), and Badaptiveness^ (related to on-the-fly, problem-solving)

- & Exploratory activity (focused on the digital device itself)

- & Frustration and fatigue (two situational, psycho-physiological influencers)

Using this scheme, a coded and annotated activity-timeline was produced for each participant.

Performance quality assessment and scoring

A second-pass review of the audio-video data was conducted to ensure consistency of coding across participants, and assign a process-quality score using a three-point scale (Blow quality^ to Bhigh quality^). The artefacts produced for each scenario task were

also scored using the same three-point scale. These two scores were summed to produce a total performance score with a maximum of 36.

Comparison of survey and performance data

A data matrix was created in Excel 2016. Statistical correlations, which can be useful with sample sizes as low as n = 3 (Wilhelm 2016), were conducted for participants at the task and scenario levels using Microsoft’s Analysis ToolPak. Owing to the exploratory nature of this study, tests for statistical significance were not conducted. Task performance scores were correlated to the reported DCP competency scores for each of the mapped self-report items. At the aggregate level, a scenario performance score was calculated by summing the six task-performance scores, and correlated to the total reported competency score, calculated by summing the values for the six aligned DCP items.

Case-study analyses of situational influencers of performance

In order to explore beyond the statistical correlations to situational factors influencing performance, the self-reports, coded video-timelines, and post-activity interviews of Group 3 participants were analysed to produce case-study reports (Merriam 1998). These participants were selected as a manageable set, and owing to the fact that they demonstrated the weakest correlations between self-reported competencies and performance.

Interpreting findings in relation to readiness

Finally, the findings were reviewed and interpreted to explore how the DCP might be used most effectively as a readiness instrument. The starting point was the participants with the highest and lowest reported digital competencies. Having explored these Bends,^ attention was given to proposing a self-report score that might function as a threshold, under which individuals who struggled to perform their scenario could be located.

Relating self-reports and performance quality

## 4.1.1 Scenario 1: BPresentation planning^

In their post-activity interviews, participants of this scenario reported a medium-level of perceived difficulty. The average reported competency score was 119 for the mapped — DCP self-report items well above the average, for these same items, calculated from the full DCP database. Higher self-report item scores were correlated with higher task performance scores for each of the six tasks (Appendix 2, Table 6). Four of the tasks produced correlation coefficients greater than .5, indicating a large effect (Task 2: r = .59; Task 3: r = .65; Task 5: r = .8; Task 6: r = .79). Task 1, which related to finding a journal article on Google Scholar, produced a medium-range coefficient (r = .31). Task 5, which related to preparing a basic concept map, produced a lowrange coefficient (r = .07). However, in both cases, performance scores were

consistently high, and the participants with the lowest ranked, self-reported competency achieved scores lower than the majority. Correlations at the scenario level produced a coefficient of r. = .83, indicating a large effect. Therefore, overall, this scenario produced results in which reported competencies show a strong positive relationship with performance quality.

It is noteworthy that the three participants with the highest performance scores (P1, P8 and P6), completed the scenario in under 40 min. The two participants with the lowest performance score (P3 and P13) took somewhat longer. Participant 12 had the fastest completion time at 23 min, and functioned as an operational virtuoso on the iPad. However, this participant, who reported the activity as Brelatively easy,^ displayed signs of hasty execution, and consequently, the quality of task outcomes suffered. These findings highlight the precarious nature of time-on-task as a criterion for competency assessment.

## 4.1.2 Scenario 2: BDemocracy data analysis^

Participants of this scenario reported a high level of perceived difficulty. The four participants had an average, total reported competency score of 86—again, above the average, for these same items, calculated from the full DTP database. However, among the participants, P4 reported a competency score of only 62 on mapped items—the — lowest reported competency score in the study and he failed to complete a single task in 50 min. In general, performance scores and completion times generated from Scenario 2 showed a greater diversity than those of Scenario 1.

Four of the tasks produced correlation coefficients greater than .5 (Task 1: r = .66; Task 2: r = .77; Task 3: r = .69; Task 4: r = .9), indicating a large effect (Appendix 2, Table 7). Task 5, related to text-messaging on Facebook Messenger, produced a lowrange coefficient (r = .19). Task 6, which required participants to send an email with attachments, produced a medium-range coefficient (r = .43). As shown in Appendix 1, these two tasks were mapped to self-report items (S12 and S11) having the highest general competency scores in the full DCP database, and therefore, of relatively low difficulty. Correlations at the scenario level, produced a coefficient of r. = .76, indicating a large effect. Therefore, this scenario also produced results in which reported competencies showed a strong positive relationship with performance quality.

## 4.1.3 Scenario 3: BModel drawing^

Participants of this scenario reported a high level of perceived difficulty. Performance scores in this group were diverse, with only five perfect scores. The average, total — reported competency for the mapped items was 116 also well above the average, for these same items, calculated from the full DTP database. The expectedly difficult Task 3, produced no perfect scores and was observed to present a significant challenge. Surprisingly, Task 6 based on item S11 (Bto communicate with others using email^), an expectedly easy task, produced no perfect scores. This finding suggests that regardless of a participant’s competency in a certain sphere of activity, situational variables may function to heighten the level of challenge. The recorded scenario times ranged from 22 min (P7) to 57 min (P9), with the latter participant achieving only a modestly higher performance score. Overall, higher performance scores showed a positive correlation with higher completion times, with a medium effect (r = .2).

For the five participants performing this scenario, correlations between self-report scores and performance scores were inconsistent (Appendix 2, Table 8). The first four tasks, generated correlations in a positive direction. Task 1 produced a correlation coefficient less than .3 indicating a small effect (r. = .14). Task 2 produced a correlation coefficient greater than .3, indicating a medium effect (r. = .43). Task 3 produced a correlation coefficient – greater than .5, indicating a large effect (r. = .56). However, Tasks 4 6 produced negative correlation coefficients, all lower than .3 (Task 3: r. = −.05; Task 4: r. = −.28; Task 5: r = −.06). In short, for half the scenario, differences in self-reported competencies were not statistically related to task performance in the expected direction. However, correlations at the scenario level, produced a coefficient of r. = .09, indicating a small effect. Therefore, this scenario, on the whole, also produced results in which reported competencies show a positive relationship with performance quality.

Aggregate-level results

Looking at all 15 participants across the three activities (Table 3), higher self-report item scores were correlated with higher scenario performance scores, producing a correlation coefficient greater than .5, indicating a large effect size (r. = .66). This finding suggests DCP potential to function effectively as a predictor of successful digital learning.

Exploring deeper with timeline case studies

By combining analyses of interview data, self-reported device preferences, and coded performance timelines, detailed, case-study reports were prepared for participants of Scenario 3 (Table 2, page 8). Two were selected and condensed for reporting here.

Table 3 Aggregate level reported competency and performance scores

|Person number|Activity number|Perceived difficulty|Total reported score|Total performed score| |---|---|---|---|---| |P1|1|2|168|36| |P14|3|2|137|27| |P8|1|2|135|36| |P7|3|2|120|30| |P6|1|1|119|36| |P12|1|1|113|34| |P15|3|2.5|112|28| |P9|3|2.5|110|33| |P11|3|2|101|24| |P5|2|2|99|33| |P3|1|2|92|30| |P2|2|3|92|16| |P10|2|2|92|27| |P13|1|2|87|30| |P4|2|3|62|13| ||||Correlation|coefficient: .66 (high+)|

Case-study 1: Speedy operational virtuoso

Participant 7 was a twenty-something female, undergraduate in education with a self— reported competency score of 120 well above the 64-average calculated from the full DCP database. She owned a laptop, smartphone and tablet. The primary uses of these devices were reported as studying (laptop), social (smartphone) and teaching (tablet). That the tablet was used primarily for teaching appears significant because seven of 12 participants owning a tablet, reported entertainment as the primary use.

She appeared comfortable throughout the activity, particularly with the operational features of Apple’s iOS, completing the activity in 22 min with a performance score of 30 (out of 36). As shown in the performance timeline (Fig. 3), she displayed keen operational skills by changing the device settings to give Google Drive access to a local photo (3:26[1] ), and by using an iOS key combination to take a screenshot and circumvent the need for an application’s difficult, file-export function (11:20). (Other participants struggled with these items.) A high level of confidence was conveyed through facial expressions and focused activity, and no incidents of frustration were coded. She maintained a strategic flow as evidenced by the structure of task completion, and only one period of device exploration.

Some process-related problems were coded. Affordance-alignment markers appear at 6:05 and 7:16 as the participant sought to draw a conceptual model. Prior to this, she scanned the device quickly for installed applications. Rather than download a drawing application, she opted for an installed, mind-mapping tool. During this period of activity, she looked unsettled—aware that the mind-mapping application did not suit the task. She called up the reference model and studied it. She explored the mindmapping program, and her facial expressions suggested disapproval. She reviewed the task description and tried to find a template that provided the elements required. At 5:41, she smiled sheepishly, and once again reviewed the installed affordances. At 5:52, she used the Apple Store to search for a more suitable tool (of which there are many). Strangely, she searched for Bconcept map,^ saw more of the same, and then returned to Mindomo to construct a model dissimilar to the exemplar provided.

This episode had little to do with the digital competences measured by the DCP. Rather, it related to the participant’s level of task commitment. Indeed, this episode contrasts to next case, where we encounter a highly committed performer, refusing to take steps towards a goal until the most suitable tool is found.

Case-study 2: Patient, steady and resourceful, wins the prize

Participant 9, was a twenty-something male, undergraduate student in education, with a self-reported competency score of 110—also well above the 64-average calculated — from the full DCP database. He owned a laptop and smartphone the primary uses of which were reported as studying (laptop) and working (smartphone). He reported moderate to high confidence for performing all six tasks of the scenario on a tablet, apparently expecting his laptop and smartphone experience would transfer to this device. Indeed, he appeared comfortable and engaged throughout the activity,

> 1 Time markers in this section are provided in minutes and seconds from the beginning of the synchronized audio-video recordings.

Fig. 3 Performance timeline of Participant 7

particularly during his execution of the expectedly difficult, Task 3. As evidence, he reported that Bthis was definitely a fun experience…really fun.^ In the end, he achieved the highest performance score in the Scenario 3 group.

As shown in Fig. 4, his skillfulness and patience was marked near the beginning of Task 3 (7:45) as he began a careful search for a task-appropriate drawing application. He searched for applications on the Apple Store, and explored features of four apps. Eventually, he turned to Google (16:38), entered Bdiagram creator^ as a search phrase, selected the cloud-based Draw.io, and swiftly completed the Google Drive integration steps to manage his work. Using browser-based application was not considered by other participants. At 19:06, he again used Google to find procedures for producing transparent shapes in Draw.io. Through this entire sequence, which runs from 7:14 to 31:44 on the timeline and ends with a smile of accomplishment, he displayed patient persistence, focused on achieving a high-quality product. His persistence continued when, at the 32-min mark, and having reviewed the activity description, he realized he had drawn the wrong model. He quickly returned to the task, spending only 12 min to apply his new-found skills to the correct model.

Visible confidence and keen focus, punctuated by brief smiles of understanding and accomplishment, continued throughout the activity. No incidents of frustration were observed. He maintained a well-managed flow of activity as evidenced by the structure of task completion, and only one period of exploratory activity (50:49), when having been advised that 50 min had elapsed, he doubled-checked his interpretation of Btwo dimensions^ as presented in Task 3.

The problems coded for this participant were minor. During Task 2, there appeared to be some hesitation when trying to use Google Drive, and this brief episode was punctuated by a technical issue in which a file, stored on the device, did not appear in

Fig. 4 Performance timeline of Participant 9

the Google Drive dialog box. However, this problem was handled without frustration by repeating the process. At 13:42, during the early portion of Task 3, he experienced the task-comprehension problem noted above, which was later remedied. Owing to the – almost 30 min allotted to Task 3, Tasks 4 6 were completed swiftly. In the end, for this participant, confidence to try new things, and the dexterity to solve problems on-the-fly, led to high-quality results.

Consolidating insights from all video case studies

The full set of five case studies (from which two examples were reported) highlighted several performance influencers beyond the purview of the DCP, including: (a) task difficulty; (b) general comfort with the tablet device; (c) levels of participant engagement, comfort, persistence, frustration and fatigue; (d) task comprehension; and (e) scenario completion strategy. In some cases, these influencers—for example, the frustration expressed by Participant 15 and fatigue shown by Participants 14—decreased performance quality despite high, self-reported competencies. In other cases—for example, the engagement shown by Participant 9—elevated the performance (and led to a lengthier overall completion time). Although completing the same scenario three times faster, Participant 7, sacrificed the quality of outcomes. Here, the issue of a participant’s motivational orientation comes to the fore. Is a learner most interested in achieving efficient completion of an instigated activity (a posture weighted toward extrinsic motivation), or does one take on the activity as their own, enjoying the challenge and ensuring the highest quality outcomes (a posture weighted towards intrinsic motivation)? A third response was reflected by Participant 11, who adopted an amotivated orientation (Deci and Ryan 2000), resulting in several task-comprehension problems that negatively impacted performance.

Interpreting the data in relation to readiness

Incorporating data from all 15 participants and three scenario activities, higher selfreport scores showed a strong correlation with higher performance scores (see 4.1.4). In addition, via case-study analyses, situational variables not measured by the DCP instrument, but influencing the quality of human performance in digital-learning contexts, were identified. The remaining challenge is to address how the DCP might be deployed as a readiness application.

Readiness at the high and low ends

Beginning at the low end, Participant 4 failed to complete the first task of Scenario 2, with generally low difficulty, even after 50 min. In addition, this participant was hesitant and focused on exploring the affordances. Although signs of frustration were largely absent, lengthy recurring pauses in device interaction, facial expressions of uncertainty, and recurring navigational and input

problems were observed. Eventually, he adopted a posture of quiet resignation, and during his post-activity interview, he reported that Btechnology is not enhancing my learning—it’s hindering it.^

This performance stands in stark contrast to that of Participant 1, the individual with the highest self-reported competency in this study. Although P1 lacked the smooth operational style displayed by some participants, he exhibited confidence and strategic purpose, completing all tasks with a high level of quality, despite initial moments of nervousness. In fact, during the execution of Task 1, it took this participant 13 min to export a journal reference in the required APA format. This was four minutes above the average, and P12 completed the same task correctly in under 5 min. Yet, after this inauspicious start, he quickly settled his nerves and completed the remaining tasks with precision. In his post-activity interview, he offered feedback that demonstrated enthusiasm and highlighted his experience using digital technologies in educational contexts.

Therefore, at the high and the low end of our participant list, tremendous differences were both reported and observed. Moreover, because both general comfort and the ability to complete educational tasks effectively with digital devices are essential in digital-learning environments, it is defensible to position these individuals at the opposite ends of a readiness spectrum as shown in Fig. 5. Having done so, what patterns can be detected among the middle participants?

Readiness in the middle

At the group-level, an overall pattern of positive correlations between reported competencies and performance was established. In order to compare scores at the individual level easily, the total reported scores were adjusted as presented in Table 4. Focusing on differentials between reported competencies and performance scores (columns four and five), a consistent pattern of sub-7 differentials emerge among the middle participants from highest to lowest reported competencies (P14:0; P7:6; P15:6; P11:4; P2:2). Moreover, where there are more pronounced differentials (P8:9; P6:12; P12:11; P9:11; P5:13; P3:12; P13:13) the performance score exceeds the reported competency score, and thus, overestimation of one’s abilities is not an issue in this context.

Having reviewed the coded, audio-video data, the performances of P2 and P11, which produced scores of 24 and 16, were marked with signs of struggle. Both participants experienced numerous problems, low-to-moderate levels of engagement, task-completion strategy issues (evidenced by many starts and stops in the timeline), and visible moments of confused activity. In short, neither P2 nor P11 appeared ready

Fig. 5 P1 and P4 positioned on a readiness spectrum

Table 4 Aggregate level reported competency and performance scores

|Person number|Activity number|Total reported score|Adjusted total reported score|Total performed score| |---|---|---|---|---| |P1|1|168|34|36| |P14|3|137|27|27| |P8|1|135|27|36| |P7|3|120|24|30| |P6|1|119|24|36| |P12|1|113|23|34| |P15|3|112|22|28| |P9|3|110|22|33| |P11|3|101|20|24| |P5|2|99|20|33| |P3|1|92|18|30| |P2|2|92|18|16| |P10|2|92|18|27| |P13|1|87|17|30| |P4|2|62|12|13|

to engage successfully in digital learning without significant support. All those positioned above P11’s adjusted reported score of 20 displayed far fewer signs of struggle. Therefore, in this context, P11’s adjusted reported score of 20 functions as a threshold for identifying a segment with individuals who struggled during performance (Fig. 6). Here, the key finding is not the specific threshold score, but the DCP instrument’s potential to identify an Bat risk^ segment above the lowest performer.

Discussion

This study explored the self-reported digital competencies of 15 participants, using DCP indicators, in relation to their performance on authentic digital-learning scenarios. Tablets were selected as activity devices owing to the increasing role of mobile technology in education. A scientific, video-analysis application (The Observer XT 13) was used for reviewing and coding the extensive audio-video data, resulting in the production of coded, activity-performance timelines. Both these timelines and task artefacts were assessed and scored to facilitate exploratory, correlational analyses. A general pattern of positive correlations was found at the aggregate level, but some negative correlations were found at the task level. By means of participant case-studies incorporating the full mixed-methods data set, several situational variables beyond the purview of the DCP were addressed as influencers of performance.

Fig. 6 Contextual readiness threshold

By reviewing the findings, the DCP was positioned as an instrument for identifying (a) learners with high and low levels of digital-learning readiness, and (b) a lower segment with members who may struggle in rich contexts of digital learning. Through further research, if patterns of self-reported digital competency and performance follow those found in this study, contextual thresholds might be similarly established to identify Bat-risk^ segments. Furthermore, self-administered performance activities could be provided to these segments for self-assessment, with remedial opportunities offered for those requiring support.

Limitations

As an exploratory, Bfirst-step^ investigation, several limitations must be noted. First, digital competency was conceptualized and measured as an individual-level construct. In contexts of collaborative learning, effective use of digital affordances involves both individual and group activities that build social and cognitive presence (Blayone et al. 2017a, b; vanOostveen et al. 2016). Therefore, digitalcompetency research, conducted at the group level, should be incorporated into conceptualizations of readiness for digital learning. Second, the sample was limited to 15 students drawn from a single Canadian technological university. It was noted that, for the most part, the reported digital competencies of these participants were positioned above the average calculated from the full DCP database. (Importantly, however, this sample did provide participants who struggled to use digital technologies effectively to complete their assigned scenario.) Third, inter-rater reliability procedures were not implemented in this study. This was owing to an aggressive timeframe for completion, and a decision that, given the nature of this small-sample study, little benefit would be gained by preparing multiple researchers to conduct analyses in The Observer XT on a single-license, work-station. Indeed, digitallearning researchers have highlighted the difficulty of achieving high levels of interrater reliability when independently coding digital-learning interactions, without incorporating additional processes of negotiation (Garrison et al. 2006; Rourke and Anderson 2004; Rourke et al. 2001).

Conclusion

In our view, this techno-methodological, pilot study makes several contributions to research. First, it offers a digital-research design for assessing authentic performances by: (a) leveraging multi-stream, audio-visual recordings in a digitalobservatory setting; and (b) using The Observer XT to produce coded, timelinevisualizations of such data, showing task durations, completion strategies, competence displays, problems (or disruptions) in performance flow, and various situational influencers. Second, it demonstrates the usefulness of the secondgeneration, DCP application for separating high and low performers, and identifying segments who may struggle as digital learners. Third, it introduces tablet devices as the basis for performance activity to address a gap in the digitalcompetency observation research (Litt 2013). Finally, it highlights the ongoing potential of the GTCU framework and the DCP application as an effective apparatus for probing digital-learning readiness.

## Appendix 1: DCP activity item list

Table 5 DCP activity items with scenario mappings and ranked average competency scores

||Item #|Item description|Scenario|Avg.<br>scorea|Rankb| |---|---|---|---|---|---| |Social|S8|To communicate with others using text messaging|2|16.4|3| |||(SMS, Facebook Messenger, etc.).|||| ||S9|To communicate with others using audio (e.g., phone,|Not used.|14.7|5| |||Skype, Viber, etc.).|||| ||S10|To communicate with others using video (e.g., Skype,|Not used.|12.6|7| |||Google Hangouts, Adobe Connect, etc.).|||| ||S11|To communicate with others using email.|1,2,3|16.8|1| ||S12|To use online social-networking systems (e.g., Facebook,|1|16.5|2| |||Twitter, LinkedIn, etc.).|||| ||S13|To use an online data/document sharing platform for|2,3|11.9|9| |||collaboration (Google Apps, Dropbox, etc.).|||| ||S14|To share my own ideas online with my network or the|3|8|14| |||public (using blog, photo or video sites, etc.).|||| |Informational|I15|To access digital maps online (e.g., MapQuest, Google|2|12.6|8| |||Maps) or use GPS to find my way or to get directions.|||| ||I16|To search and access journal and/or news articles online.|1|12.9|6| ||I17|To search and watch video online (e.g., YouTube).|1|15.4|4| ||I18|To search and access images, photos or infographics|3|9.7|13| |||online.|||| ||I19|To search, access and/or download music online.|Not used.|11.5|10| ||I20|To search and download books (e.g., PDF, eBooks,|3|10.7|11| |||audio) or purchase printed books online.|||| ||I21|To use an online application to collect and organize|Not used.|5.6|20| |||information automatically.|||| |Epistemological|E22|To use and share a calendar/personal agenda (e.g.,|1|10.4|12| |||Google Calendar, Outlook, etc.).|||| ||E23|To generate concept maps (e.g, Cmap), mind maps|1|6.3|19| |||(e.g., xMind) or flowcharts (e.g., Visio).|||| ||E24|To create, modify and use conceptual diagrams, models|3|6.9|18| |||or technical drawings.|||| ||E25|To sort large amounts of data (e.g., in a spreadsheet,|2|7.4|16| |||online application, database, etc.).|||| ||E26|To produce graphs and data visualizations automatically|2|7.1|17| |||from numerical data.|||| ||E27|To perform complex calculations (e.g., in a spreadsheet,|Not used.|7.6|15| |||statistics application, etc.)|||| ||E28|To do some form of programming, coding, scripting|Not used.|4.8|21| |||or markup to automate processes.||||

a Calculated from the aggregate DCP database, higher numbers indicate that individuals, who have contributed data to date, generally report greater competency in this area of activity b Items are ranked based on average competency scores, with the least difficult item ranked 1

|Person Device<br>selected<br>Total<br>reported<br>Total performed Time in minutes Task 1 DCP I16 Task 2 DCP I17 Task 3 DCP E22 Task 4 DCP E23 Task 5 DCP S12 Task 6 DCP S11|P1<br>iPad<br>168<br>36<br>32<br>6<br>30<br>6<br>30<br>6<br>30<br>6<br>18<br>6<br>30<br>6<br>30<br>P8<br>iPad<br>135<br>36<br>39<br>6<br>21<br>6<br>26<br>6<br>24<br>6<br>15<br>6<br>23<br>6<br>26<br>P6<br>iPad<br>119<br>36<br>29<br>6<br>14<br>6<br>27<br>6<br>12<br>6<br>9<br>6<br>29<br>6<br>28<br>P12<br>iPad<br>113<br>34<br>23<br>6<br>20<br>6<br>21<br>5<br>16<br>6<br>6<br>6<br>23<br>5<br>27<br>P3<br>Sam<br>92<br>30<br>49<br>4<br>13<br>5<br>25<br>5<br>13<br>6<br>6<br>6<br>16<br>4<br>19<br>P13<br>Sam<br>87<br>30<br>40<br>4<br>22<br>5<br>15<br>5<br>7<br>5<br>10<br>5<br>8<br>6<br>25<br>Correlations<br>.83 (high + overall)<br>.31 (med+)<br>.59 (high+)<br>.65 (high+)<br>.07 (low+)<br>.8 (high+)<br>.79 (high+)|Maximum total reported score is 180. Maximum performance score is 36. DCP activity items are listed in Appendix1| |---|---|---|

|Person Device<br>selected<br>Total<br>reported<br>Total performed Time in minutes Task 1 DCP I15 Task 2 DCP E25 Task 3 DCP I13 Task 4 DCP E26 Task 5 DCP S8 Task 6 DCP S11|P5<br>iPad<br>99<br>33<br>29<br>6<br>18<br>5<br>10<br>6<br>21<br>4<br>9<br>6<br>19<br>6<br>22<br>P2<br>iPad<br>92<br>16<br>50<br>4<br>19<br>2<br>6<br>2<br>12<br>2<br>6<br>3<br>27<br>3<br>22<br>P10<br>iPad<br>92<br>27<br>50<br>4<br>14<br>5<br>18<br>6<br>13<br>4<br>12<br>4<br>18<br>6<br>17<br>P4<br>iPad<br>62<br>13<br>50<br>3<br>12<br>2<br>8<br>2<br>11<br>2<br>6<br>2<br>12<br>2<br>13<br>Correlations<br>.76 (high + overall)<br>.66 (high+)<br>.77 (high+)<br>.69 (high+)<br>.9 (high+)<br>.19 (low+)<br>.43 (med+)| |---|---|

|Person Device<br>selected<br>Total<br>reported<br>Total performed Time in minutes Task 1 DCP S13 Task 2 DCP I18 Task 3 DCP E24 Task 4 DCP I20 Task 5 DCP S14 Task 6 DCP S11|P14<br>Sam<br>137<br>27<br>50<br>5<br>27<br>5<br>28<br>4<br>9<br>4<br>20<br>4<br>27<br>5<br>26<br>P7<br>iPad<br>120<br>30<br>22<br>5<br>21<br>6<br>25<br>3<br>6<br>5<br>20<br>6<br>22<br>5<br>26<br>P15<br>Sam<br>112<br>28<br>42<br>5<br>13<br>5<br>21<br>5<br>9<br>3<br>15<br>5<br>24<br>5<br>30<br>P9<br>iPad<br>110<br>33<br>57<br>6<br>19<br>6<br>24<br>5<br>14<br>6<br>16<br>5<br>12<br>5<br>25<br>P11<br>Sam<br>101<br>24<br>41<br>4<br>17<br>4<br>21<br>2<br>9<br>5<br>12<br>5<br>15<br>4<br>27<br>Correlations<br>.09 (low + overall)<br>.14 (low+)<br>.43 (med+)<br>.56 (high+)<br>−.05 (low-)<br>−.28 (low-)<br>−.06 (low-)| |---|---|

References

- Ala-Mutka, K. (2011). Mapping digital competence: Towards a conceptual understanding. In Seville: Institute for Prospective Technological Studies (IPTS). European Commission: Joint Research Centre Retrieved from http://ftp.jrc.es/EURdoc/JRC67075_TN.pdf.

- Al-Araibi, A. A. M., Mahrin, M., & Mohd, R. C. (2016). A systematic literature review of technological factors for e-learning readiness in higher education. Journal of Theoretical and Applied Information Technology, 93(2), 500–521.

- Aparicio, M., Bacao, F., & Oliveira, T. (2016). An e-learning theoretical framework. Journal of Educational Technology & Society, 19(1), 292–307. http://www.jstor.org/stable/10.2307/jeductechsoci.19.1.292

- Asselin, M., & Moayeri, M. (2010). New tools for new literacies research: An exploration of usability testing software. International Journal of Research & Method in Education, 33(1), 41–53. https://doi. org/10.1080/17437271003597923.

- Aydın, C. H., & Tasci, D. (2005). Measuring readiness for e-learning: Reflections from an emerging country. Educational Technology and Society, 8(4), 244–257. http://www.jstor.org/stable/jeductechsoci.8.4.244

- Bandura, A. (1993). Perceived self-efficacy in cognitive development and functioning. Educational Psychologist, 28(2), 117–148. https://doi.org/10.1207/s15326985ep2802_3.

- Beetham, H., & Sharpe, R. (2007). Rethinking pedagogy for a digital age: Designing for 21st century learning. New York: Routledge.

- Bertaux, D. (1981). From the life-history approach to the transformation of sociological practice. In D. Bertaux –

- (Ed.), Biography and society: The life history approach in Social Sciences (pp. 29 45). London: Sage.

- Bhatt, I., & de Roock, R. (2014). Capturing the sociomateriality of digital literacy events. Research in learning technology, 21, 21281. https://doi.org/10.3402/rlt.v21.21281.

- Blayone, T., Mykhailenko, O., vanOostveen, R., Grebeshkov, O., Hrebeshkova, O., & Vostryakov, O. (2017a). Surveying digital competencies of university students and professors in Ukraine for fully online collaborative learning. Technology, Pedagogy and Education. https://doi.org/10.1080/1475939X.2017.1391871.

- Blayone, T., vanOostveen, R., Barber, W., DiGiuseppe, M., & Childs, E. (2017b). Democratizing digital learning: Theorizing the fully online learning community model. International Journal of Educational Technology in Higher Education, 14(1), 13. https://doi.org/10.1186/s41239-017-0051-4.

- Bradlow, E. T., Hoch, S. J., & Hutchinson, J. W. (2002). An assessment of basic computer proficiency among active internet users: Test construction, calibration, antecedents and consequences. Journal of Educational and Behavioral Statistics, 27(3), 237–253. https://doi.org/10.3102/10769986027003237.

- Bui, T. X., Sankaran, S., & Sebastian, I. M. (2003). A framework for measuring national e-readiness. International Journal of Electronic Business, 1(1), 3–22. https://doi.org/10.1504/ijeb.2003.002162.

- Crompton, H., Burke, D., Gregory, K. H., & Gräbe, C. (2016). The use of mobile learning in science: A systematic review. Journal of Science Education and Technology, 1–12. https://doi.org/10.1007/s10956015-9597-x.

- Deci, E. L., & Ryan, R. M. (2000). The" what" and" why" of goal pursuits: Human needs and the selfdetermination of behavior. Psychological Inquiry, 11(4), 227–268. https://doi.org/10.1207/s15327965 pli1104_01.

- Demir, Ö., & Yurdugül, H. (2015). The exploration of models regarding e-learning readiness: Reference model suggestions. International Journal of Progressive Education, 11(1), 173–194.

- Desjardins, F. J. (2005). Information and communication technology in education: A competency profile of francophone secondary school teachers in Ontario. Canadian Journal of Learning and Technology/La revue canadienne de l’apprentissage et de la technologie, 31(1), 1–14. https://doi.org/10.21432/T2PG69.

- Desjardins, F. J., & Peters, M. (2007). Single-course approach versus a program approach to develop technological competencies in pre-service language teaching. In M.-A. Kassen, L. Lavine, K. Murphy–

- Judy, & M. Peters (Eds.), Preparing and developing technology proficient L2 teachers (pp. 3 21). Texas: Texas State University.

- Desjardins, F. J., & vanOostveen, R. (2015). Faculty and student use of digital technology in a "laptop" university. In S. Carliner, C. Fulford, & N. Ostashewski (Eds.), EdMedia: World Conference on Educational Media and Technology 2015 (pp. 990-996). Montreal: Association for the Advancement of Computing in Education (AACE).

- Desjardins, F. J., Lacasse, R., & Belair, L. M. (2001). Toward a definition of four orders of competency for the use of information and communication technology (ICT) in education. Paper presented at the computers and advanced Technology in Education. Canada: Banff http://eilab.ca/wp-content/uploads/2013/04/2001 CATE.pdf.

- Desjardins, F. J., vanOostveen, R., Bullock, S., DiGiuseppe, M., & Robertson, L. (2010). Exploring graduate student’s use of computer-based technologies for online learning. In J. Herrington & C. Montgomerie (Eds.), EdMedia: World Conference on Educational Media and Technology 2010 (pp. 440-444). Norfolk: Association for the Advancement of Computing in Education (AACE).

- DiGiuseppe, M., Partosoedarso, E., vanOostveen, R., & Desjardins, F. J. (2013). Exploring competency development with mobile devices. In M. B. Nunes & M. McPherson (Eds.), International Association for Development of the information society (IADIS) international conference on e-learning (pp. 384–388). Prague: International Association for Development of the Information Society.

- Ding, R., & Ma, F. (2013). Assessment of university student web searching competency by a task-based online test: A case study at Wuhan University, China. The Electronic Library, 31(3), 359–375. https://doi. org/10.1108/EL-03-2011-0044.

- Dray, B. J., Lowenthal, P. R., Miszkiewicz, M. J., Ruiz-Primo, M. A., & Marczynski, K. (2011). Developing an instrument to assess student readiness for online learning: A validation study. Distance Education, 32(1), 29–47. https://doi.org/10.1080/01587919.2011.565496.

- Esbjörnsson, M., Brown, B., Juhlin, O., Normark, D., Östergren, M., & Laurier, E. (2006). Watching the cars go round and round: Designing for active spectating. In. In R. Grinter, T. Rodden, P. Aoki, E. Cutrell, R. Jeffries, & G. Olson (Eds.), Proceedings of the SIGCHI conference on human factors in computing –

- systems (pp. 1221 1224). New York: ACM.

- Eshet-Alkalai, Y., & Amichai-Hamburger, Y. (2004). Experiments in digital literacy. Cyberpsychology & Behavior, 7(4), 421–429.

- Farid, A. (2014). Student online readiness assessment tools: A systematic review approach. Electronic Journal of e-Learning, 12(4), 375–382.

- Garrison, D. R., Cleveland-Innes, M., Koole, M., & Kappelman, J. (2006). Revisiting methodological issues in transcript analysis: Negotiated coding and reliability. The Internet and Higher Education, 9(1), 1–8. https://doi.org/10.1016/j.iheduc.2005.11.001.

- Gay, G. (2016). An assessment of online instructor e-learning readiness before, during, and after course delivery. Journal of Computing in Higher Education, 28(2), 199–220. https://doi.org/10.1007/s12528016-9115-z.

- Greene, J. A., Seung, B. Y., & Copeland, D. Z. (2014). Measuring critical components of digital literacy and their relationships with learning. Computers & Education, 76, 55–69. https://doi.org/10.1016/j. compedu.2014.03.008.

- Hargittai, E. (2002). Beyond logs and surveys: In-depth measures of people's web use skills. Journal of the American Society for Information Science and Technology, 53(14), 1239–1244. https://doi.org/10.1002 /asi.10166.

- Hargittai, E., & Shafer, S. (2006). Differences in actual and perceived online skills: The role of gender. Social Science Quarterly, 87(2), 432–448. https://doi.org/10.1111/j.1540-6237.2006.00389.x.

- Herrington, J., Reeves, T. C., & Oliver, R. (2006). Authentic tasks online: A synergy among learner, task, and technology. Distance Education, 27(2), 233–247. https://doi.org/10.1080/01587910600789639.

- Horzum, M. B., Kaymak, Z. D., & Gungoren, O. C. (2015). Structural equation modeling towards online learning readiness, academic motivations, and perceived learning. Educational Sciences: Theory and Practice, 15(3), 759–770. 10.12738/estp.2015.3.2410.

- Hung, M.-L. (2016). Teacher readiness for online learning: Scale development and teacher perceptions. Computers & Education, 94, 120–133. https://doi.org/10.1016/j.compedu.2015.11.012.

- Hung, M.-L., Chou, C., & Chen, C.-H. (2010). Learner readiness for online learning: Scale development and student perceptions. Computers & Education, 55(3), 1080–1090. https://doi.org/10.1016/j. compedu.2010.05.004.

- IEEE. (1990). IEEE standard computer dictionary: A compilation of IEEE standard computer glossaries. In (pp. 218). New York: The Institute of Electrical and Electronics Engineers.

- Jayroe, T. J., & Wolfram, D. (2012). Internet searching, tablet technology and older adults. Proceedings of the American Society for Information Science and Technology, 49(1), 1–3. https://doi.org/10.1002 /meet.14504901236.

- Knoblauch, H. (2012). Introduction to the special issue of qualitative research: Video-analysis and videography. Qualitative Research, 12(3), 251–254. https://doi.org/10.1177/1468794111436144.

- Leigh, D., & Watkins, R. (2005). E-learner success: Validating a self-assessment of learner readiness for online –

- training. In. In ASTD 2005 research-to-practice conference proceedings (pp. 121 131). Alexandria: ATD.

- Lin, H.-H., Lin, S., Yeh, C.-H., Wang, Y.-S., & Jansen, J. (2015). Measuring mobile learning readiness: Scale development and validation. Internet Research, 26(1), 265–287. https://doi.org/10.1108/IntR-10-2014-0241.

- Litt, E. (2013). Measuring users’ internet skills: A review of past assessments and a look toward the future. New Media & Society, 15(4), 612–630. https://doi.org/10.1177/1461444813475424.

- Mason, M. (2010). Sample size and saturation in PhD studies using qualitative interviews. Forum qualitative Sozialforschung/Forum: Qualitative Social Research, 11(3), 1–5.

- Merriam, S. B. (1998). Qualitative research and case study applications in education. Revised and expanded from case study research in education. San Francisco: Josey-Bass Publishers.

- Mosa, A. A., Naz’ri bin Mahrin, M., & Ibrrahim, R. (2016). Technological aspects of e-learning readiness in higher education: A review of the literature. Computer and Information Science, 9(1), 113–127. https://doi.org/10.5539/cis.v9n1p113.

- Parasuraman, A. (2000). Technology readiness index (TRI) a multiple-item scale to measure readiness to embrace new technologies. Journal of Service Research, 2(4), 307–320. https://doi.org/10.1177 /109467050024001.

- Park, Y. J. (2015). My whole world’s in my palm! The second-level divide of teenagers’ mobile use and skill. New Media & Society, 17(6), 977–995. https://doi.org/10.1177/1461444813520302.

- Parkes, M., Stein, S., & Reading, C. (2015). Student preparedness for university e-learning environments. The Internet and Higher Education, 25, 1–10. https://doi.org/10.1016/j.iheduc.2014.10.002.

- Rourke, L., & Anderson, T. (2004). Validity in quantitative content analysis. Educational Technology Research and Development, 52(1), 5–18. https://doi.org/10.1007/bf02504769.

- Rourke, L., Anderson, T., Garrison, D. R., & Archer, W. (2001). Methodological issues in the content analysis of computer conference transcripts. International Journal of Artificial Intelligence in Education, 12, 8–22.

- Savin-Baden, M. (2000). Problem-based learning in higher education: Untold stories. Philadelphia: Open University Press.

- Siemens, G., Gašević, D., & Dawson, S. (2015). Preparing for the digital university: A review of the history and current state of distance, blended, and online Learning Retrieved from http://linkresearchlab. org/PreparingDigitalUniversity.pdf.

- Sun, X., & May, A. (2013). A comparison of field-based and lab-based experiments to evaluate user experience of personalised mobile devices. Advances in Human-Computer Interaction, 2013, 2. https://doi.org/10.1155/2013/619767.

- van Deursen, A. J. A. M. (2010). Internet skills: Vital assets in an information society. (Ph.D. Thesis), University of Twente, Enschede, the Netherlands. Retrieved from http://doc.utwente.nl/75133/1/thesis_ van_Deursen.pdf.

- van Deursen, A. J. A. M., & van Dijk, J. A. G. M. (2010). Measuring internet skills. International Journal of Human-Computer Interaction, 26(10), 891–916. https://doi.org/10.1080/10447318.2010.496338.

- van Deursen, A. J. A. M., Helsper, E. J., & Eynon, R. (2015). Development and validation of the internet skills scale (ISS). Information, Communication & Society, 1–20. https://doi.org/10.1080/1369118 X.2015.1078834.

- vanOostveen, R., DiGiuseppe, M., Barber, W., Blayone, T., & Childs, E. (2016). New conceptions for digital technology sandboxes: Developing a fully online learning communities (FOLC) model. In G. Veletsianos –

- (Ed.), EdMedia 2016: World conference on educational media and technology (pp. 665 673). Vancouver: Association for the Advancement of Computing in Education (AACE).

- Watkins, R., Leigh, D., & Triner, D. (2004). Assessing readiness for e-learning. Performance Improvement Quarterly, 17(4), 66–79. https://doi.org/10.1111/j.1937-8327.2004.tb00321.x.

- Wilhelm, J. (2016). What is the minimum sample size to run Pearsons R? (Online Expert Database). Retrieved June 7, 2017, from ResearchGate: https://www.researchgate.net/post/What_is_the_minimum_sample_ size_to_run_Pearsons_R.