Tokyo-based Sakana AI has introduced two groundbreaking approaches—Doc-to-LoRA (D2L) and Text-to-LoRA (T2L)—that promise to solve one of the most persistent bottlenecks in LLM customization: the trade-off between flexibility and computational efficiency. ## The Adaptation Dilemma Currently, customizing LLMs involves difficult trade-offs. In-Context Learning (ICL) offers convenience but suffers from quadratic attention costs as prompts grow. Context Distillation transfers information into model parameters but requires expensive per-prompt training. Supervised Fine-Tuning needs task-specific datasets and re-training when information changes. Sakana AI’s solution: amortize these costs through lightweight hypernetworks that generate Low-Rank Adaptation (LoRA) matrices in a single forward pass. ## Text-to-LoRA: Task Adaptation via Natural Language Text-to-LoRA uses a task encoder to extract vector representations from natural language task descriptions. Combined with learnable module and layer embeddings, MLP blocks generate the A and B matrices for LoRA adaptation. The system can be trained via LoRA Reconstruction (distilling existing adapters) or Supervised Fine-Tuning on multi-task datasets. SFT-trained T2L generalizes better to unseen tasks by clustering related functionalities in weight space. In benchmarks, T2L matched or outperformed task-specific adapters while reducing adaptation costs by over 4x compared to 3-shot ICL. ## Doc-to-LoRA: Internalizing Long Contexts Doc-to-LoRA extends this to document internalization, enabling LLMs to answer queries about documents without re-consuming the original context—effectively removing the document from the active context window. D2L uses a Perceiver-style cross-attention architecture that maps variable-length token activations into a fixed-shape LoRA adapter. For documents exceeding training length, a chunking mechanism partitions long contexts into K contiguous chunks, each processed independently to produce per-chunk adapters concatenated along the rank dimension. ## Dramatic Efficiency Gains The results are striking. For a 128K-token document, a base model requires over 12 GB of VRAM for the KV cache. D2L handles the same document using less than 50 MB—a reduction of over 99%. Update latency drops from minutes to sub-second regimes. Perhaps most surprising: D2L can even internalize visual information from a Vision-Language Model into a text-only LLM, allowing the text model to classify images with 75% accuracy despite never seeing pixel data during its primary training. ## Sources - [Sakana AI Doc-to-LoRA Paperhttps://pub.sakana.ai/doc-to-lora/){rel=“nofollow”} - [Doc-to-LoRA Codehttps://github.com/SakanaAI/Doc-to-LoRA){rel=“nofollow”} - [Text-to-LoRA Paperhttps://arxiv.org/pdf/2506.06105){rel=“nofollow”} - [Text-to-LoRA Codehttps://github.com/SakanaAI/Text-to-LoRA){rel=“nofollow”}