Dimension-reduction procedures

As explained in the main text, we used principal component analysis (PCA) in conjunction with factor analysis (FA) for testing the dimensionality of the purported scales. For each scale in question, we began the analysis by checking three crucial prerequisites for both PCA and FA: First, we applied the Kaiser-Meyer-Olkin (KMO) measure of sampling adequacy in order to test whether, given a particular set of items to be factorized, our sample of observations (n_max=149) was sufficient to apply factor analysis in the first place. ^[24] Following recommendations by Kaiser (1974), we took KMO values>0.5 to be acceptable; lower values indicated that no PCA/FA could be performed (without, for example, removing any of the items in question, cf. also Backhaus et al. 2016: 398). Secondly, the correlational structure of the respective items was tested, in two complementary ways: Bartlett’s test of sphericity (Bartlett 1951), implemented in the R function cortest.bartlett() in the ‘psych’ package, determines whether there is a minimal correlational structure in the data (by comparing the data to a so-called identity matrix without any correlations). The opposite scenario, i. e. that the items correlate too highly, is as problematic for FA as such extreme multicollinearity is for multiple regression models, because it becomes impossible to gauge the unique contributions of highly correlated items to a given factor. This scenario was tested by calculating the so-called determinant, using the det() function from the R base distribution; the determinant value should be substantially greater than 0.00001 (since perfect correlation, so-called singularity, is indicated by a determinant of zero, cf. Field et al. 2012: 771).

For the PCAs, we used the PCA()function from the ‘FactoMineR’ package. Missing values in our data were imputed by the function imputePCA() from the ‘missMDA’ package (cf. Josse and Husson 2012 for technical details). For determining the optimal number of principal components, we inspected the eigenvalues of each component, using scree plots (cf. Catell 1966) in conjunction with Kaiser’s (1960) criterion of choosing components with eigenvalues greater than 1. The ideal scenario in our case is, of course, one in which a single principal component emerges and where, following Stevens’ (2002) guidelines for our sample size of 100<n<200, each item has a loading of>0.4 onto the component in question.

For the factor analysis, we used the function factanal() from the R base distribution. In each case, we took the smallest number of components suggested as adequate by the PCA (i. e. ideally just one) and tested whether this yields a satisfactory factor solution in FA. In order to improve interpretability with multi-component solutions, we employed ‘promax’ as an oblique method for factor rotation. ^[25] To inspect the factor loadings, we examined the pattern matrix and again took Stevens’>0.4 criterion as the benchmark for the contribution of an item to a factor. Finally, we also used the maximum-likelihood residual analysis implemented in factanal(), which indicates whether the number of factors specified is sufficient in order to account for the variation in the participating items. ^[26]

The last step of the analysis was to assess the reliability of the resulting scales, applying Cronbach’s alpha (Cronbach 1951). Following standard recommendations in the literature (e. g. also Dörnyei 2010b: 95 on questionnaire data), we used α ≥ 0.7 as the critical threshold for the reliability of a scale. In Appendix 1, where we provide the full questionnaire, we also indicate which items were reversed for the reliability analysis.

In the following overview, we turn to each scale from the paper individually again and provide the central results from the PCA, FA and reliability analysis. Unless specified otherwise, it can be taken for granted that both the Bartlett criterion and the determinant yielded acceptable results.

1. Integrativeness (items 1–3): KMO=0.57. PCA suggests that a single-component solution may be acceptable, but this is not corroborated by FA (item 3 showed a loading<0.4 and a single-factor solution is not judged sufficient). α_(1–3)=0.48, α_(1–2)=0.42. We thus obtain a heterogeneous and unreliable scale, even if shorted to the more homogeneous items 1 and 2.

2. Attitudes towards the L2 speakers and the community (items 4–8): KMO=0.7. PCA suggests that a single-component solution is perfectly adequate. In FA, a single-factor solution is not judged sufficient by the significance test, but all items show loadings>0.4 on the factor. The variance accounted for (and the solution as a whole) can be improved if item 8 is removed, and the final scale reported in the paper is reliable at α_(4–7)=0.71.

3. Instrumentality (items 9–12): An initial analysis showed that item 12, which (in contrast to Kormos and Csizér 2008) we had reallocated to this scale because of its instrumental flavour, introduced severe heterogeneity, as it did not correlate with the other items on the scale. Consequently, we abandoned it from the scale and operated with items 9–11 only. KMO=0.64. PCA then suggested a satisfactory one-component solution. FA’s significance test did not consider a single factor to be sufficient, but again all items had acceptable loadings of>0.4 on the factor. The scale is borderline reliable (α_(9–11)=0.68) and we hence retained it in further analyses.

4. Cultural interest (items 13–15): KMO=0.57. PCA suggests that a single-component solution may be acceptable, but this is not corroborated by FA (item 3 showed a loading<0.4 and a single-factor solution is not judged sufficient). α_(13–15)=0.57. The scale is thus both heterogeneous and unreliable.

5. Vitality of the L2 community (items 16–19): KMO=0.72. PCA suggests a clearly one-dimensional structure with high loadings of all items. FA with one factor confirms loadings>0.4, but fails the significance test. In view of the strong signals from all other criteria, however, the scale was included in its original composition. This is confirmed by reliability analysis (α_(16–19)=0.78).

6. Linguistic self-confidence (items 20–22): In this case, the Bartlett test yields a non-significant result (p=0.08), and KMO=0.44. Therefore, the correlational structure of these items is so weak that not even the basic prerequisites for PCA/FA are fulfilled.

7. Language use anxiety (items 23–26): KMO=0.73. Both PCA and FA suggest a one-dimensional structure with high loadings of all items onto the single factor. This monofactorial solution also passes FA’s significance test and is reliable at α_(23–26)=0.74.

8. Classroom anxiety (items 27–30): KMO=0.65. An initial analysis showed that item 30 makes the scale inconsistent and leads to an unacceptable alpha value. Removing this item returns a single-component structure in PCA and a monofactorial solution in FA with all loadings>0.4. This scale is reliable at α_(27–30)=0.73.

9. Milieu (items 31–34): This scale violates some crucial prerequisites for PCA/FA, i. e. Bartlett’s test is non-significant (p=0.08) and KMO=0.49. Even if we remove item 32 because of its large amount of NAs, the results do not improve significantly. Not surprisingly, then, the scale is also completely unreliable (α_(31–34)=0.09).

10. Parental encouragement (items 35–38): KMO=0.77. Although both PCA and FA suggest a satisfactory one-dimensional solution, some improvement in the factor loadings and in the variance explained by a single factor can be achieved if item 36 is removed. The resulting scale is reliable at α_(35,37,38)=0.74.

11. Language-learning attitudes (items 39–42): KMO=0.51. Both PCA and FA suggest a two-component solution with complex cross-loadings and an overall unreliable composition (α_(39–42)=0.51). As explained in the paper, a subscale of items 39 and 42 achieves a much higher degree of reliability (α_(39,42)=0.7) and was hence retained in further analyses.

12. International posture (items 43–47): Item 45 had to be removed due to a large number of NAs. For the remaining items, KMO=0.53. PCA suggests that a single-component solution might be possible (though certainly not optimal), and all signals of a FA clearly speak against this. Removing individual items does not improve this situation, and the scale is not reliable (α_(43–47)=0.55).

13. Ideal L2 Self (items 48–54): KMO=0.76. Both PCA and FA suggest a two-dimensional construct, but since an outstanding first component emerges in PCA, with high loadings of all items, and since a monofactorial solution is still reliable (α_(48–54)=0.71), we had sufficient reason to believe that this construct is valid enough (and certainly more homogeneous than truly diverse scales). Note that removing items with the lowest loadings does not improve the residual diagnostics in FA, nor does it lead to a more reliable alpha value.

14. Ought-to L2 Self (items 55–59): KMO=0.59. Both PCA and FA suggest a complex (at least two-factor) structure with many cross-loadings, and an insufficient α_(55–59) of −0.52.

15. Self-efficacy (items 60–67): KMO=0.54. Both PCA and FA suggest a very heterogeneous composite of three factors. As far as reliability is concerned, neither the scale in its original composition (α_(60–67)=0.06) nor in a version reduced to the most strongly correlating items (α_{(60, 64, 67)}=0.52) reaches the criterion level.

16. Perceived L2 difficulty (items 68–71): KMO=0.69. Both PCA and FA suggest a satisfactory one-component solution, as all items show high loadings on the first dimension and even the residual analysis shows that a monofactorial analysis is sufficient. The scale is reliable at α_(68–71)=0.7.

17. Motivated learning behaviour (items 72–82): A first round of analysis showed that the two items we added to the original scale (i. e. 81 and 82) introduced heterogeneity. Without them, KMO=0.83. PCA suggests that a one-dimensional solution may be warranted, with high loadings of all items on the first dimension (similarly to the Ideal L2 Self above). A similarly good loading structure emerges in a monofactorial FA (even though this does not pass the residual test). The scale is reliable at α_(72–80)=0.83.

Statistical parameters of reliable scales

This part of the appendix relates to Section 4 of the paper. The relevant data can be found in Table 3 on the next page, which has to be read as follows: For each of the reliable scales listed in the first column, we provide comparative data on three different subsamples, i. e. female/male, remote/urban and pupil/student. We first list the mean and standard deviation for each part of the respective subsample (e. g. female versus male) and then compare the difference in their average scores by means of non-parametric Wilcoxon rank-sum tests (excluding missing values on a pairwise basis).

In the right-hand part of the table, we then display the (non-parametric) correlation coefficient Spearman’s ρ (rho) for the correlation of the scale in question with the scale of motivated learning behaviour, i. e. the mean value of items 72–80. In testing these correlations for significance, we generally employed one-tailed tests because our hypotheses were clearly directional (e. g. we supposed that an Ideal L2 Self would correlate positively with motivated learning behaviour). However, for testing such correlations in the subsamples (e. g. differences between male and female participants), two-tailed tests were used.

The last two columns of the table feature a test statistic z and its associated p-value. z was obtained by applying Fisher’s r-to-z transformation, which allows comparing whether two correlation coefficients (e. g. that of the rural and the urban subsample) differ significantly from one another. ^[27]

Regression analyses

As is explained in the main text, our specific data set (with many predictors relative to the number of observations, heavy deviations from normality, potentially correlated predictors, etc.) suggests employing a specific non-parametric regression procedure known as conditional inference trees (CIT) and random forests (RF). The gist of these techniques is described in the main text, and we elaborate on them here and add some specifications of the software and algorithms we drew on.

Individual CITs were produced by using the ctree() function in the ‘party’ package (cf. Strobl et al. 2009a), which has an inbuilt ‘stop criterion’ when no significant association between a predictor and the response variable can be detected anymore, thus preventing further splits of the tree. The function works on the basis of an unbiased algorithm, which (in contrast to other implementations like rpart()) does not favour variables with many distinct levels (such as continuous or multilevel categorical predictors over binary ones). The function is thus particularly suitable to our specific mix of predictors.

RFs are so-called ensemble methods in which many individual trees are grown from random subsets of the data. For each of the trees involved, the whole sample is randomly divided into a learning set (the so-called ‘in-bag’ observations) and a test set (the so-called ‘out-of-bag’ observations), and only a random subset of the predictor variables is used to generate the tree. In this way, a truly diverse set of trees is grown, and by averaging over these trees, the relative importance of each predictor can be assessed more reliably than in a single tree model for the data (since a whole ensemble of trees is less sensitive to idiosyncratic properties of the specific sample at hand). The relative weight of a predictor is gauged by a rather complex procedure of permutation (cf. Breiman 2001). The basic idea is that the values of the predictor variable in question are shuffled so that there is no association with the response variable anymore; if this permuted version of the predictor weakens the performance of the model as compared to using the original, unpermuted predictor, one can conclude that the predictor in question is essential for modelling the response variable (cf. also Tagliamonte and Baayen 2012: 160–161 for a digestible summary of the technicalities).

For growing random forests, we used the cforest() function of the ‘party’ package. ^[28] For the reasons mentioned above, we used the default option controls=cforest_unbiased to ensure unbiased treatment of different types of variables in each tree. Furthermore, in assessing the relative impact of the predictors, we extracted the conditional permutation variable importance, using the varimp(obj, conditional=T) routine. This procedure, proposed in Strobl et al. (2008), specifically ensures that the importance of correlated predictors is not overestimated and thus reduces the potential problem of collinearity. In growing the random forest, we used the default mtry=5 setting (i. e. with five randomly preselected predictor variables at each split of each tree) but with a considerably larger number of trees than specified in the default settings (3,000 instead of 500), in order to do justice to the rather high number of predictor variables and to make the results more reliable (cf. Strobl et al. 2009b: 343). Following recommended practices, we tried several different parameter settings (including varying random seeds), but obtained stable results as far as the relative variable importance of the strongest predictors was concerned (while the relative ranking of less influential predictors changed across the various runs).

For determining the statistical significance of a predictor in the random forest solution, we followed Strobl, Malley and Tutz (2009: 343), who suggest that “all variables with importance that is negative, zero, or positive but with a value that lies in the same range as the negative values can be excluded from further exploration. The rationale for this rule of thumb is that the importance of irrelevant variables varies randomly around zero. Therefore, positive variation of an amplitude comparable to that of negative variation does not indicate an informative predictor variable, whereas positive values that exceed this range may indicate that a predictor variable is informative.” More specifically, we took the absolute value of the most negative predictor as a threshold for the significance of the positive predictors.

The goodness-of-fit of CITs and RFs can be obtained in a similar way as for multiple linear regression models, i. e. by using the regular formula for the coefficient of determination: R²=1 – SS_res/SS_tot, where SS_res denotes the residual sum of squares and SS_tot the total sum of squares (cf. Field et al. 2012: 250). This procedure capitalizes on the fact that, in both CITs and RFs, one can compare the predicted and the observed values of the response variable (based on a single model in CITs and the average of many models in RFs). What we observe in our data with regard to R² (compare Figures 2 and 3, or 4 and 5) is rather typical: The goodness-of-fit of a single CIT is usually slightly worse than the predictive accuracy of a whole ensemble of trees, i. e. of a RF, especially if the latter is grown from a very large number of trees.