Grokked Transformers are Implicit Reasoners_ A Mechanistic Journey to the Edge of Generalization

1. Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization

2. Phased Consistency Model The Consistency Model (CM) has advanced diffusion model generation, yet its adaptation for high-resolution, text-conditioned image generation in latent space (LCM) has been suboptimal. This paper identifies three critical flaws in LCM and introduces the Phased Consistency Model (PCM), which expands the design space and resolves these issues. Evaluations show that PCM significantly outperforms LCM in settings ranging from 1 to 16 generation steps. Notably, PCM is designed for multi-step refinement but also excels in 1-step generation, matching or surpassing the performance of state-of-the-art methods tailored for single-step processes. Moreover, PCM's approach proves versatile, extending to video generation and achieving leading results in few-step text-to-video generation.

3. An Introduction to Vision-Language Modeling The recent surge in LLMs has spurred efforts to adapt these models for visual applications, leading to the development of vision-language models (VLMs). VLMs, capable of tasks like navigating unfamiliar environments or generating images from text descriptions, are poised to significantly change our interaction with technology. However, the integration of discrete language with the high-dimensional, continuous nature of vision presents unique challenges. This paper serves as an introduction to VLMs, covering their fundamentals, operation, and training methodologies. It also explores evaluation techniques for VLMs and extends the discussion to video applications, aiming to clarify the complexities of bridging vision with language for newcomers to the field.

4. GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning Knowledge Graphs (KGs), which represent factual knowledge as a graph of triplets (head, relation, tail), facilitate Question Answering over KGs (KGQA) by grounding reasoning in provided information. While LLMs excel in natural language understanding and are thus dominant in QA tasks, Graph Neural Networks (GNNs) are effective in handling the complex graph structure of KGs. This paper introduces GNN-RAG, a novel method that merges the language understanding capabilities of LLMs with the reasoning power of GNNs in a retrieval-augmented generation (RAG) approach. The process involves using a GNN to reason over a dense KG subgraph to retrieve answer candidates, then extracting and verbalizing the shortest paths between question entities and these candidates for LLM processing. Additionally, a retrieval augmentation technique is developed to enhance KGQA performance. GNN-RAG has shown to surpass or match GPT-4 in widely recognized KGQA benchmarks like WebQSP and CWQ, particularly excelling in multi-hop and multi-entity question scenarios, improving answer F1 scores by 8.9--15.5 percentage points.

5. Transformers Can Do Arithmetic with the Right Embeddings The limited capability of transformers in arithmetic tasks is primarily due to their inability to precisely track digit positions in large numerical spans. This issue is addressed by introducing a position-encoding embedding for each digit, enhancing the transformer's performance in arithmetic operations. Further architectural enhancements like input injection and recurrent layers amplify this effect. With improved position tracking, the study explores whether transformers can tackle arithmetic problems that surpass the complexity and size encountered during training. Results show that with training on only 20-digit numbers using a single GPU for one day, the enhanced model reaches up to 99% accuracy on 100-digit addition problems. These advancements in numeracy also lead to performance improvements in other complex reasoning tasks such as sorting and multiplication.

6. MAP-Neo: Highly Capable and Transparent Bilingual Large Language Model Series LLMs have achieved notable success across various tasks, yet leading models like GPT, Gemini, and Claude remain proprietary, often without detailed public insights into their training. In contrast, open-source initiatives have released models such as LLaMA-3, although these typically lack comprehensive disclosure, such as intermediate checkpoints and training codes. To enhance transparency in the field, the research community has introduced fully open LLMs like Pythia, Amber, and OLMo, which provide extensive details including pre-training corpora and training methodologies. Despite these efforts, these fully open models still lag behind the performance of top proprietary LLMs in reasoning, knowledge, and coding tasks. Addressing this gap, MAP-Neo, a transparent, bilingual 7B parameter LLM trained on 4.5T high-quality tokens, is introduced as the first fully open-sourced bilingual LLM matching the performance of leading LLMs.

7. Attention as an RNN The introduction of Transformers has been a significant advancement in sequence modeling, capitalizing on GPU parallelism to enhance performance. Yet, their high computational cost at inference limits their use in resource-constrained environments, such as mobile and embedded devices. This paper presents a novel perspective where attention mechanisms are interpreted as a type of Recurrent Neural Network (RNN) that can efficiently produce a many-to-one RNN output. It further posits that Transformers are akin to RNN variants but lack efficient token updating capabilities crucial for sequence modeling. To address this, a new method leveraging the parallel prefix scan algorithm is introduced to compute attention's many-to-many RNN output more efficiently.

8. Meteor: Mamba-based Traversal of Rationale for Large Language and Vision Models The rapid advancement of large language and vision models (LLVMs) has significantly benefited from visual instruction tuning, particularly through the use of open-source datasets and enhanced vision encoders to compete with sophisticated proprietary LLVMs. These improvements stem from the complex information demands of tasks requiring deep image understanding, common-sense knowledge, and procedural reasoning for complex problem-solving. This paper introduces Meteor, a new efficient LLVM that utilizes a multifaceted rationale to boost its understanding and response capabilities. Meteor employs the Mamba architecture, which processes sequential data with linear time complexity and introduces a novel concept for efficiently embedding lengthy rationales.