PLSpredict
PLSpredict, developed by Shmueli et al. (2016), uses training and holdout samples in a k-fold cross-validation procedure to generate and evaluate out-of-sample predictions from a PLS path model. It gives researchers a systematic way to assess a model's predictive power beyond in-sample fit, and to diagnose possible overfitting to the training data.
How PLSpredict Works
The procedures proposed by Shmueli et al. (2016) are combined in the PLSpredict package https://github.com/ISS-Analytics/pls-predict for the statistical software R. They generate different out-of-sample and in-sample predictions (e.g., case-wise and average predictions), which facilitate evaluating predictive performance on new data that was not used to estimate the PLS path model.
Based on the procedures suggested by Shmueli et al. (2016), the current PLSpredict algorithm implementation in the SmartPLS software allows researchers to obtain k-fold cross-validated prediction errors and prediction error summaries statistics such as the root mean square error (RMSE), the mean absolute error (MAE), and the mean absolute percentage error (MAPE) to assess the predictive performance of their PLS path model for the manifest variables (MV or indicators) and the latent variables (LV or constructs). Note that all three criteria are available for the MV results, while it is only possible to compute the RMSE and MAE for the LV results. These criteria allow to compare the predictive performance of alternative PLS path models.
Sharma et al.’s (2019) Monte Carlo simulation shows that the RMSE and mean absolute deviation MAE are particularly suitable when the aim is to select the best predictive model among a set of competing models. Researchers need to compare RMSE and MAD values for alternative model set-ups and select the model, which minimizes RMSE and MAD values in the latent variable scores.
In addition, to assess the results of a specific PLS path model, its predictive performance can be compared against two naïve benchmarks (Shmueli et al., 2019):
(1) The Q²predict value in PLSpredict compares the prediction errors of the PLS path model against simple mean predictions. For this purpose, it uses the mean value of the training sample to predict the outcomes of the holdout sample. The Q²predict value results interpretation is similar to the assessment of Q²predict values obtained by the blindfolding procedure in PLS-SEM. If the Q²predict value is positive, the prediction error of the PLS-SEM results is smaller than the prediction error of simply using the mean values. In that case, the PLS-SEM models offers better predictive performance.
(2) The linear regression model (LM) offers prediction errors and summary statistics that ignore the specified PLS path model. Instead, the LM approach regresses all exogenous indicator variables on each endogenous indicator variable to generate predictions. Thereby, a comparison with the PLS-SEM results offers information whether using a theoretically established path model improves (or at least does not worsen) the predictive performance of the available indicator data. In comparison with the LM outcomes, the PLS-SEM results should have a lower prediction error (e.g., in terms of RMSE or MAE) than the LM. Note that the LM prediction error is only available for the manifest variables and not the latent variables.
The out-of-sample predictions used in PLSpredict assist researchers in evaluating the predictive capabilities of their model. Therefore, PLSpredict should be included in the evaluation of PLS-SEM results (Hair et al., 2019, 2022).
Additional procedures and extensions are under development and may become part of future SmartPLS versions. The most recent extension is the cross-validated predictive ability test (CVPAT), which can be used to test the predictive ability of the model (Liengaard et al., 2021; Sharma et al., 2023). CVPAT results are available in the PLSpredict results report in SmartPLS.
The difference between Q²predict and PLS-SEM and IA results comparisons for the RMSE and MAE
Q²predict is a summary statistic that evaluates whether the PLS-SEM prediction outperforms the naïve IA benchmark. It is calculated as one minus the ratio of the sum of squared prediction errors (SSE) of the PLS-SEM predictions to the SSE of the IA benchmark predictions. A value greater than zero indicates that the PLS-SEM predictions yield lower overall error than the benchmark. By contrast, the RMSE and MAE comparisons provide a direct, metric-specific assessment of predictive accuracy. For each indicator, the RMSE and MAE from the PLS-SEM predictions are compared with those from the IA (or LM) prediction benchmark. PLS-SEM demonstrates better predictive performance if its RMSE or MAE values are smaller than the corresponding benchmark values.
PLSpredict Settings in SmartPLS
| Setting | Default | What it controls |
|---|---|---|
| Number of Folds | 10 | In k-fold cross-validation, the algorithm splits the full dataset into k equally sized subsets. It then predicts each fold (holdout sample) using the remaining k-1 subsets as the training sample. For example, with k = 10, a dataset of 200 observations is split into 10 subsets of 20 observations, and the algorithm predicts each fold ten times using the nine remaining subsets. |
| Number of Repetitions | 10 | How often PLSpredict repeats the k-fold cross-validation on random splits of the dataset. A single random split can make predictions strongly dependent on that particular assignment of observations to folds, so results (e.g., RMSE, MAPE) may vary between runs. Repeating the cross-validation with different random partitions and averaging across repetitions gives a more stable estimate of predictive performance. |
Frequently Asked Questions
What does PLSpredict actually tell me that R² does not?
R² reflects in-sample explanatory power, computed on the same data used to estimate the model. PLSpredict instead evaluates out-of-sample predictive power, using holdout samples the model was not trained on — this is a more direct diagnostic for overfitting and generalizability.
How do I interpret the Q²predict value?
A positive Q²predict means the PLS-SEM model's prediction error is smaller than that of a naïve mean-based benchmark, indicating the model has predictive relevance for that indicator. A negative value means the naïve benchmark predicts better than the model.
Why compare against both the IA (mean) benchmark and the LM benchmark?
The two benchmarks test different things. The Q²predict/IA comparison checks whether the model beats simply predicting the mean. The RMSE/MAE comparison against a linear regression model (LM) checks whether the theoretically specified path model outperforms a model-free regression of exogenous indicators on each endogenous indicator.
Which error metric should I use, RMSE or MAE?
Sharma et al. (2019) show via Monte Carlo simulation that RMSE and MAE are particularly suitable for selecting the best predictive model among competing models; researchers should compare these values across candidate model setups and prefer the model that minimizes them for the latent variable scores.
How does PLSpredict relate to CVPAT?
CVPAT (cross-validated predictive ability test) is a more recent extension that formally tests the predictive ability of a model rather than only comparing prediction errors descriptively. CVPAT results appear in the PLSpredict results report in SmartPLS; see CVPAT for details.
Related SmartPLS Methods
- Cross-Validated Predictive Ability Test (CVPAT)
- Blindfolding
- Finite Mixture PLS-SEM (FIMIX-PLS)
- PLS Prediction-oriented Segmentation (PLS-POS)
References
- Hair, J. F., Hult, G. T. M., Ringle, C. M., & Sarstedt, M. (2027). A primer on partial least squares structural equation modeling (PLS-SEM) (4th ed.). Sage.
- Hair, J. F., Risher, J. J., Sarstedt, M., & Ringle, C. M. (2019). When to use and how to report the results of PLS-SEM. European Business Review, 31(1), 2–24.
- Liengaard, B. D., Sharma, P. N., Hult, G. T. M., Jensen, M. B., Sarstedt, M., Hair, J. F., & Ringle, C. M. (2021). Prediction: Coveted, yet forsaken? Introducing a cross-validated predictive ability test in partial least squares path modeling. Decision Sciences, 52(2), 362–392.
- Sharma, P. N., Liengaard, B. D., Hair, J. F., Sarstedt, M., & Ringle, C. M. (2023). Predictive model assessment and selection in composite-based modeling using PLS-SEM: Extensions and guidelines for using CVPAT. European Journal of Marketing, 57(6), 1662–1677.
- Sharma, P. N., Sarstedt, M., Shmueli, G., Kim, K. H., & Thiele, K. O. (2019). PLS-based model selection: The role of alternative explanations in information systems research. Journal of the Association for Information Systems, 20(4), 346–397.
- Shmueli, G., Ray, S., Estrada, J. M. V., & Chatla, S. B. (2016). The elephant in the room: Predictive performance of PLS models. Journal of Business Research, 69(10), 4552–4564.
- Shmueli, G., Sarstedt, M., Hair, J. F., Cheah, J. H., Ting, H., Vaithilingam, S., & Ringle, C. M. (2019). Predictive model assessment in PLS-SEM: Guidelines for using PLSpredict. European Journal of Marketing, 53(11), 2322–2347.
- More literature ...
Cite correctly
Please always cite the use of SmartPLS!
Ringle, Christian M., Wende, Sven, & Becker, Jan-Michael. (2024). SmartPLS 4. Bönningstedt: SmartPLS. Retrieved from https://www.smartpls.com
