2D-RoPE Enables Models to Copy Long Text More Accurately
State-of-the-art large language models, while solving advanced reasoning tasks, still cannot accurately copy long repeated strings, and the longer the input or the more repeated patterns, the less reliable the copying. Research finds that standard 1D RoPE positional encoding is partly responsible, while 2D-RoPE treats the source text and output as independent rows, aligning corresponding tokens in the same column. In synthetic tests, a single-layer model trained only on lengths 1-100 perfectly copied inputs up to 1000 times longer; this advantage also appeared in a 1.4B parameter pretrained model, but the current design still relies on newline characters and is not a general solution.
Read the original (opens in a new tab)
News stream data aggregated by AI HOT