Is ChatGPT detrimental to innovation? A field experiment among university students
This article has been Reviewed by the following groups
Listed in
- Evaluated articles (PREreview)
Abstract
ChatGPT represents a momentous technological breakthrough whose implications – along with other AI innovations – are yet to fully materialize. This paper is among the first attempts to experimentally test the effect of AI applications (in the form of ChatGPT) on three dependent variables usually assumed to be AI-collaterals: innovation, readiness to exert effort, and risk behaviour. We took advantage of the delayed introduction of ChatGPT in Egypt and conducted a pre-registered field experiment with nearly 100 senior university students at a public university. Over one month during term time, participants were asked to submit three graded essay assignments. In the treatment group, students were asked to write the essays using ChatGPT whereas in the control group, such option was neither mentioned nor allowed (the experiment was fielded before ChatGPT was legally operable in Egypt). One week after all assignments were submitted, the two groups were invited to the lab to play an innovation game (deploying multiple strategies to increase the sales of a hypothetical lemonade stand), a risk game (bomb risk elicitation task), and do a real effort task. The ChatGPT group was significantly less innovative, significantly less risk averse, and exerted less effort (however not statistically significant). Our results point to possible negative effects of AI applications but need further testing and larger samples to be confirmed.
Article activity feed
-
This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/13138508.
Summary and Strengths
ChatGPT, among other automated or artificial intelligence, is of concern – especially to educators – because of its impact on human behavior. This study aimed to test the effect of continuous usage of ChatGPT in a classroom setting on three behaviors related to economic growth: innovation, effort, and risk. Students submitted essays using ChatGPT (treatment group) or not (control group); the authors subsequently measured the focal behaviors through an innovation and risk game and a real-effort task. The authors highlighted three key findings: students who used ChatGPT 1) were significantly less innovative, 2) were significantly less risk averse, and 3) exerted less …
This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/13138508.
Summary and Strengths
ChatGPT, among other automated or artificial intelligence, is of concern – especially to educators – because of its impact on human behavior. This study aimed to test the effect of continuous usage of ChatGPT in a classroom setting on three behaviors related to economic growth: innovation, effort, and risk. Students submitted essays using ChatGPT (treatment group) or not (control group); the authors subsequently measured the focal behaviors through an innovation and risk game and a real-effort task. The authors highlighted three key findings: students who used ChatGPT 1) were significantly less innovative, 2) were significantly less risk averse, and 3) exerted less effort compared to their non-AI using counterparts.
An obvious strength of this manuscript is the quality of the argument: supporting literature was provided, the objectives were clear, the experimental designs were appropriate, the conclusions were supported with evidence, and several limitations and rebuttals were addressed. Additionally, the study is both novel and interesting. We commend the authors for using this unique opportunity to examine the impact of AI prior to its spread and extensive use. We believe the results of this study could open opportunities to look further into the negative effects of AI applications in educational contexts.
We would recommend this article for publication following minor revisions (as outlined below) - primarily to the experimental design.
Major Issues
The reviewers have not identified any major issues with the manuscript.
Minor Issues
Methods
The reviewers would find it helpful if the authors provided more detail on the topic and guidelines for the assigned essays in each course.
Did the essays require any innovation on the part of the students?
Were students in the treatment group required to submit an exact output of their ChatGPT query or were they allowed to edit it at all?
Supplemental materials and an appendix were mentioned in the Methods and Results but were not attached to the preprint. We recommend attaching these so that all materials can be reviewed.
The reviewers would find it helpful if the authors provided justification or more content for why the specific methods (i.e., lemonade and bomb game) were used.
Results
The reviewers would find it helpful if the authors included indicators of statistical significance on the relevant plots.
Discussion
The reviewers recommend including comments on how their results can be used to inform use of AI in the classroom – how can educators work with the existence of ChatGPT to still promote innovation in students?
Competing interests
The authors declare that they have no competing interests.
-
