Continual learning remains a critical challenge in AI, as models must adapt to new tasks without erasing previously acquired knowledge. The Octopus framework, presented by vivo's internet technology team, tackles this by introducing a history-free gradient orthogonalization (HiFGO) method. Unlike traditional approaches that rely on storing past data or complex regularization, HiFGO applies orthogonal constraints directly at the gradient level, effectively minimizing interference between tasks. The framework employs a two-stage training strategy that balances plasticity and stability, allowing the model to learn new tasks while retaining old ones. On the UCIT benchmark, Octopus achieves state-of-the-art results, with Avg and Last metrics surpassing existing methods by 2.14%. This improvement is particularly notable because it does not require historical data, making it more practical for real-world applications where data storage is limited or privacy concerns exist. For AI engineers and researchers, Octopus offers a promising direction for building more robust and adaptive systems. The approach could be especially valuable in domains like robotics, personalized recommendations, and autonomous systems, where models must continuously evolve. While the paper is from a corporate research team, the methodology is generalizable and could inspire further innovations in the field.
Octopus introduces a two-stage continual learning framework using history-free gradient orthogonalization (HiFGO) to reduce task interference. It achieves SOTA performance on the UCIT benchmark, improving Avg and Last metrics by 2.14% over prior methods. This is significant for AI systems that need to learn sequentially without catastrophic forgetting.