We demonstrate stable training dynamics for trillion-parameter sparse MoE architectures with novel load-balancing loss and expert-choice routing, achieving 3.2x throughput over dense baselines.

A scalable alignment framework that trains models to follow constitutional principles through iterative self-critique and refinement, reducing harmful outputs by 94% without human annotation overhead.

We introduce HATS — a hierarchical agent-task system that decomposes complex multi-step objectives into executable sub-goals, achieving 78% success rate on WebArena benchmark with 4x fewer steps than flat baselines.

Ring attention with blockwise computation enables training on 1M+ token contexts at near-linear scaling efficiency across 512 GPUs, unlocking new capabilities in document understanding and code synthesis.

A unified cross-attention architecture that grounds language in visual, audio, and sensor modalities simultaneously, surpassing single-modality fine-tuned models on 12 of 15 multimodal benchmarks.

Dynamic draft tree construction adapts speculative decoding depth to token-level uncertainty, achieving 2.8x inference speedup while preserving exact output distribution from the target model.