Can Researchers Agree on Levels of Resilience?

Resilience has been challenging to define, though it is intuitively apparent in many ways as the capacity to bounce back from adversity. We set out to determine if we could agree on ratings resilience based on our collected life story interviews. After reviewing our life stories, we settled on five ratings of resilience based upon a water/swimming metaphor. At level 1, the person sinks to the bottom and remains there. At level 2, the person is not sitting at the bottom but has yet to reach the surface. At level 3, the person is treading water. They are working hard to stay afloat but not making significant progress in altering their overall situation. At level 4, they’re swimming toward shore, toward a more favorable environment where they can thrive. At level 5, they've managed to climb out of the water and change their circumstances. Their resilience allows them to overcome challenges and seek better surroundings. We reviewed the life story interviews of thirty-five participants with an eye to these categories. We also used ChatGPT to determine if it could perform better than us. For 35 participants with five ratings and three raters, we had a percent overall agreement of 84.00% with a free-marginal kappa of 0.80 (95% CI for free-marginal kappa; 0.66 to 0.94). The fixed-marginal kappa was 0.78, with a 95% CI of 0.62 to 0.93. ChatGPT had a percent overall agreement of 71.33% with a free-marginal kappa of 0.65 (95% CI for free-marginal kappa was 0.50 to 0.79: fixed-marginal kappa of 0.61 (95% CI for fixed-marginal kappa of 0.51 to 0.71. Cohen suggested the Kappa result be interpreted as follows: values ≤ 0 as indicating no agreement and 0.01–0.20 as none to slight, 0.21–0.40 as fair, 0.41– 0.60 as moderate, 0.61–0.80 as substantial, and 0.81–1.00 as almost perfect agreement (Cohen, 1960). All results were statistically significant at p < 0.01. The Ohio State Resilience Scale did not correlate with our ratings, suggesting that people’s assessments of their level of resilience do not match those of clinically trained observers.