Is ChatGPT detrimental to innovation? A field experiment among university students

Abstract

This paper investigates the potential collateral effects of A.I innovations, specifically ChatGPT, on three key variables: innovation, readiness to exert effort and risk behavior.

Design/methodology/approach

A field experiment was conducted involving nearly 100 senior university students at a public university in Egypt, at a time when ChatGPT had not yet been legally operational. Over a one-month period, participants submitted three graded essay assignments. The treatment group utilized ChatGPT to write the essays, while the control group completed the assignments without such assistance. After submission, both groups participated in a lab-based innovation game, a risk game and a real effort task to measure their respective innovation, risk aversion and effort exertion.

Findings

The results reveals that students who used ChatGPT demonstrated significantly lower levels of innovation (ChatGPT usage is associated with a decrease in innovation scores by approximately 0.6–0.72 standard deviation points, at the 95% confidence level) and risk aversion (individuals in the ChatGPT group are more likely to become risk lovers, at the 90% confidence level) compared to the Non-ChatGPT group. Although the reduction in effort exerted by the ChatGPT group was not statistically significant, the overall trends suggest a potential decrease in effort related to the use of A.I. applications.

Research limitations/implications

On avenues for future research, although field experiments will always have the advantage of high ecological validity, testing the effect of AITGs on behavior could also benefit from the controlled environment of lab experiments. In such designs, spill-over worries would be minimal and internal validity would be high. To address the issue of external validity however, new experimental designs could be thought of to test the generalizability of the findings; longitudinal studies that trace the effect of technology across time, expanding the participant pool across multiple institutions that vary in terms of type (public/private), age of students (schools/universities), academic background and majors … etc.

Practical implications

On practical and policy implications – and in line with economic theories on innovation and economic growth and development (Schumpeter, 1942; Romer, 1990; Acemoglu and Robinson, 2013) – our humble findings point to an urgent need to augment existing education with concrete, innovation-based practices. These could include embedding design thinking and problem-based learning modules directly into curricula, as well as choice architecture that alters people’s behavior without restricting options. In addition, designing student innovation competitions and startup incubators that reward novelty and impact would incentivize the value of innovation.

Originality/value

This study is among the first to empirically test the impact of ChatGPT on innovation, effort, and risk behavior in a real-world academic setting. It provides preliminary evidence of the potential negative effects of A.I. applications on these variables, offering valuable insights for further research into the broader implications of A.I. on human behavior.

Article activity feed

This Zenodo record is a permanently preserved version of a PREreview. You can view the complete PREreview at https://prereview.org/reviews/13138508.

Summary and Strengths

ChatGPT, among other automated or artificial intelligence, is of concern – especially to educators – because of its impact on human behavior. This study aimed to test the effect of continuous usage of ChatGPT in a classroom setting on three behaviors related to economic growth: innovation, effort, and risk. Students submitted essays using ChatGPT (treatment group) or not (control group); the authors subsequently measured the focal behaviors through an innovation and risk game and a real-effort task. The authors highlighted three key findings: students who used ChatGPT 1) were significantly less innovative, 2) were significantly less risk averse, and 3) exerted less effort compared to their non-AI using counterparts.

An obvious strength of this manuscript is the quality of the argument: supporting literature was provided, the objectives were clear, the experimental designs were appropriate, the conclusions were supported with evidence, and several limitations and rebuttals were addressed. Additionally, the study is both novel and interesting. We commend the authors for using this unique opportunity to examine the impact of AI prior to its spread and extensive use. We believe the results of this study could open opportunities to look further into the negative effects of AI applications in educational contexts.

We would recommend this article for publication following minor revisions (as outlined below) - primarily to the experimental design.

Major Issues

The reviewers have not identified any major issues with the manuscript.

Minor Issues

Methods

The reviewers would find it helpful if the authors provided more detail on the topic and guidelines for the assigned essays in each course.

Did the essays require any innovation on the part of the students?

Were students in the treatment group required to submit an exact output of their ChatGPT query or were they allowed to edit it at all?

Supplemental materials and an appendix were mentioned in the Methods and Results but were not attached to the preprint. We recommend attaching these so that all materials can be reviewed.

The reviewers would find it helpful if the authors provided justification or more content for why the specific methods (i.e., lemonade and bomb game) were used.

Results

The reviewers would find it helpful if the authors included indicators of statistical significance on the relevant plots.

Discussion

The reviewers recommend including comments on how their results can be used to inform use of AI in the classroom – how can educators work with the existence of ChatGPT to still promote innovation in students?

Competing interests

The authors declare that they have no competing interests.

Read the original source