Abstract
The self-evolution of large language models refers to the process where models iteratively improve their capabilities using self-generated data or feedback without relying on external human-annotated data. This paradigm has the potential to break through the bottleneck of human supervision, enabling LLMs to continuously and autonomously advance toward higher levels of intelligence. This paper systematically reviews the core concepts, mainstream technical approaches (including self-training, self-rewarding, self-reflection, and curriculum learning), representative works with their strengths and limitations, and finally discusses key challenges and future directions.
1. Introduction
As the capabilities of large language models such as GPT-4, Claude, and Llama rapidly increase, how to push models beyond the boundaries of existing training data has become a research focus. Traditional fine-tuning heavily relies on high-quality human-annotated data, which is costly and difficult to scale. Self-evolution emerges as a response—models leverage their own generation, reasoning, and evaluation abilities to build a closed-loop learning cycle, achieving “learning to become stronger.” Inspired by AlphaGo Zero’s self-play, this idea is rapidly developing in the natural language domain.
2. Core Framework of Self-Evolution
Self-evolution typically consists of four basic stages:
- Data Generation: The model generates training samples (e.g., question-answer pairs, chain-of-thought reasoning paths) based on prompts or seed tasks.
- Quality Filtering: Low-quality data is filtered out via rules, model self-scoring, or external validators.
- Training Update: The filtered data is used for supervised fine-tuning (SFT) or reinforcement learning (RLHF/DPO).
- Iterative Evaluation: The above process is repeated in subsequent rounds, forming an upward spiral.
3. Main Technical Approaches
3.1 Self-Training
The most classic method. The model first generates pseudo-labels from a small set of seed data, then selects high-confidence samples to add to the training set, and retrains. Representative work includes STaR (Self-Taught Reasoner): the model attempts to reason through problems; if it obtains the correct answer, the reasoning chain is retained as training data. Repeated iterations significantly improve mathematical reasoning ability. Limitations include susceptibility to amplifying initial biases and sensitivity to noise.
3.2 Self-Rewarding
To address the reliance on human-annotated reward models in RLHF, researchers propose letting the LLM act as both “actor” and “critic.” For example, Self-Rewarding LM: at each iteration, the model generates answers and scores its own responses according to predefined criteria, then uses these scores as reward signals for DPO training. Experiments on AlpacaEval show that alignment capability improves steadily over multiple rounds of self-rewarding iteration. However, self-scoring may suffer from preference collapse.
3.3 Self-Reflection
The model reviews its erroneous outputs and produces improved versions. A typical example is Reflexion: after failing a task, the agent converts error information and environmental feedback into textual memory, references these reflections in subsequent attempts, and gradually increases success rates. This approach performs well in interactive tasks such as programming and decision-making. A limitation is that reflection quality depends on the model’s original understanding ability.
3.4 Curriculum Learning and Progressive Evolution
Inspired by human learning, models start self-evolution from simple tasks and gradually progress to complex ones. For example, Self-Play Fine-Tuning (SPIN): the model plays against its older version, generating increasingly difficult adversarial examples, forcing itself to learn finer discrimination capabilities. This method yields significant gains in mathematical reasoning and instruction following.
4. Overview of Representative Works
| Method | Core Mechanism | Applicable Scenario | Key Results |
|---|---|---|---|
| STaR | Self-generated reasoning chains + answer verification | Mathematical reasoning | Significant accuracy improvement on GSM8K |
| Self-Rewarding LM | Self-scoring + DPO | Alignment optimization | Iterative growth on AlpacaEval scores |
| Reflexion | Error reflection + memory replay | Agent tasks | Improved success rates in coding, QA |
| SPIN | Self-play + adversarial training | General capabilities | Continuous score increase on MT-Bench |
Additionally, OpenAI’s o1 model internally employs some form of self-search and reflection mechanism, further confirming the potential of the self-evolution approach.
5. Key Challenges
Despite promising prospects, self-evolution currently faces the following difficulties:
- Mode Collapse: The model may fall into homogeneity during iterations; generated data lacks diversity, causing performance stagnation or even degradation.
- Reward Hacking: Under self-rewarding mechanisms, the model learns to exploit scoring loopholes rather than genuinely improving capabilities.
- Computational Cost: Each iteration requires full-scale training, resulting in huge resource consumption.
- Evaluation Dilemma: Lack of independent, uncontaminated external benchmarks to measure real progress, as test sets may also be covered by model-generated data.
6. Future Directions
- Hybrid Evolution: Combining limited human feedback with large-scale self-evolution to balance efficiency and stability.
- Multimodal Self-Evolution: Incorporating visual, audio, and other signals into the closed loop to achieve cross-modal capability growth.
- Open-Source Ecosystem Synergy: Community-driven self-evolution platforms allowing different models to distill and compete with each other.
- Theoretical Understanding: Establishing convergence theories for self-evolution to guide algorithm design.
7. Conclusion
Self-evolution of LLMs is moving from concept to practice, demonstrating potential beyond static training in areas such as mathematical reasoning, code generation, and dialogue alignment. Although we are still far from fully “self-aware” evolution, this paradigm offers a feasible technical path to break through the data ceiling. Future research needs to find better trade-offs among stability, efficiency, and generalization to truly realize continuous autonomous growth of large models.
Note: This article is based on publicly available research up to 2025. For the latest advances, please refer to the most recent official publications.
