Mitigating Persistence Effects in LSTM-Based Stock Prediction: Portfolio Optimization with Sentiment Integration

Stock price prediction and portfolio optimization remain challenging due to market efficiency and the persistence problem in recurrent neural networks, where models achieve misleading accuracy by tracking recent prices rather than forecasting genuine movements. This study integrates Long Short-Term Memory (LSTM) networks with financial sentiment analysis for return prediction, combined with mean-variance portfolio optimization. We evaluate the framework on 50 S\&P 500 stocks spanning 2016--2024, with rigorous out-of-sample testing from January 2022 to December 2024. We explicitly address the persistence problem through five complementary strategies: log return prediction, temporal feature engineering, sentiment integration, dropout regularization, and limited input windows. This approach reduces lag-1 autocorrelation from 0.89 to 0.23, enabling the model to capture genuine predictive patterns rather than trivial extrapolation. The LSTM achieves modest directional accuracy (59.3%) with explained variance of \((R^2 = 0.16)\). When integrated into monthly portfolio rebalancing, the strategy delivers a cumulative three-year return of 28.7%, modestly exceeding equal-weight benchmarks (22.4%) with marginal statistical significance (\((p = 0.036)\)). The Sharpe ratio improves by 18% over equal-weight portfolios, though the strategy underperforms the S\&P 500 index over the same period. Ablation studies demonstrate that sentiment features contribute meaningfully beyond price-based features alone. A critical limitation is the absence of transaction cost modeling. Preliminary estimates suggest moderate trading frictions could reduce returns by approximately one percentage point annually. Despite modest individual prediction accuracy, results demonstrate that carefully designed machine learning predictions can generate measurable portfolio-level improvements over naive strategies when subjected to rigorous validation.