Cross-block synchronization in CUDA is a well-known challenge, especially on GPUs with compute capability below 9.0 (e.g., RTX 4090, V100) that lack hardware cluster sync. A recent Chinese blog post tackles this head-on by reverse-engineering the grid.sync() cooperative group mechanism. The author reveals that grid.sync() relies on a software barrier using atomic increments and sign-bit flipping on a shared variable. Building on this insight, they implement a custom sync_ctas function that achieves cross-block synchronization without hardware support. The post includes detailed code examples and performance considerations, making it a valuable resource for GPU programmers working on complex parallel algorithms. This is not a beginner tutorial but a deep technical analysis that fills a gap in official documentation. For developers building custom CUDA kernels that require global synchronization, this approach offers a practical, if not officially supported, solution. The novelty lies in the reverse-engineering effort and the clear explanation of the underlying mechanism, which is rarely documented elsewhere.
This article explores software-based cross-block synchronization in CUDA for devices with compute capability below 9.0, where hardware cluster sync is unavailable. The author reverse-engineers the grid.sync() cooperative group mechanism and implements a custom sync_ctas function using barrier variables and atomic operations. This is a rare, practical resource for GPU programmers facing synchronization challenges.