Notatka · · 1 min czytania
Mapa pojęć LLM
Hasła słownika ułożone tematycznie. W każdej grupie pojęcia podstawowe stoją przed tymi, które z nich korzystają. Teksty o AI zbiera strona AI.
Podstawy modelu językowego
- Token
- Typy tokenów LLM
- Byte Pair Encoding
- Token Embedding
- Autoregressive Language Modeling
- Masked Language Modeling
- Softmax
- LM head
- Cross-entropy
- Perplexity
- KL Divergence
Transformer i attention
- Transformer
- Self-attention
- Scaled Dot-Product Attention
- Multi-Head Attention
- Multi-Query Attention
- Grouped-Query Attention
- Cross-attention
- Causal Masking
- Positional Encoding
- Rotary Position Embedding
- Position-wise Feed-Forward Network
- Residual connection
- Layer Normalization
- RMSNorm
- Attention sink
Trening: dane i etapy
- Pre-training
- Fine-tuning
- LoRA
- Knowledge Distillation
- Overfitting
- Validation Set
- Xavier (Glorot) Initialization
- Sequence Packing
- Loss masking
Optymalizacja
- Stochastic Gradient Descent
- Momentum
- Nesterov Momentum
- Adam
- AdamW
- Optimizer State
- Weight decay
- Learning Rate Warmup
- Gradient Clipping
- Gradient Accumulation
- Learning Rate - Batch Size Scaling Rules
- Critical Batch Size
- Scale invariance
- Norma Frobeniusa
- Hyperball
- Delayed acceleration
Teoria uczenia
- Scaling Laws
- Edge Of Stability
- Implicit regularization przez hyperparametry
- µP i transfer hyperparametrów
- Grokking
- Neural Collapse
- Neural Feature Ansatz
- Universal representations i universal phenomena
Trening rozproszony i pamięć
- Data Parallelism
- Tensor Parallelism
- Pipeline Parallelism
- All-reduce
- All-gather
- Reduce-scatter
- ZeRO
- Activation Checkpointing
- Mixed Precision Training
- bfloat16
- Loss Scaling
Uczenie ciągłe
- Continual Learning
- Catastrophic Forgetting
- Experience Replay
- Reservoir Sampling
- Elastic Weight Consolidation
- Meta-learning
Uczenie ze wzmocnieniem i post-training
- Reinforcement Learning
- Reinforcement Learning from Human Feedback
- Direct Preference Optimization
- GRPO
- Reward hacking
- Sharpening Tax
- Pseudo-label
- On-policy self-distillation
- Test-time training
- Order-consistency SFT
Dekodowanie i próbkowanie
Inferencja i serwowanie
- Prefill
- Decode
- KV Cache
- Prefix Caching
- Chunked Prefill
- Continuous Batching
- PagedAttention
- FlashAttention
- Disaggregated Serving
- Time to First Token
- Inter-token Latency
- Goodput
- Post-training quantization
- Sliding window
- Test-time Compute
- Test-time scaling
Ewaluacja i wyszukiwanie
Interpretowalność
Agenci i bezpieczeństwo
- AI Harness
- Self-evolving agent
- CREATE-path
- Signed quarantine
- Corrigibility
- Self-Correction [English]
- AGI vs TAI [English]