Blog
Self-published2022Fine-tuning a pre-trained network works better when different layers move at different speeds. We set up discriminative learning rates with PyTorch parameter groups, so early layers retain what they learned while later layers adapt faster.
Self-published2021Sometimes the useful signal sits in a hidden layer rather than the output. We compare three ways to read those activations in PyTorch, from forward hooks to slicing the module list, and use them to pull out embeddings.
Self-published2021We compute channel-wise mean and standard deviation across an image dataset for normalization. The implementation batches through the data in PyTorch, so it never holds the full dataset in memory.
Self-published2021Large models often will not fit a useful batch size into GPU memory. We use gradient accumulation to reach a large effective batch anyway, walking through the modified training loop and its interaction with learning rate scaling and batch norm.
Self-published2020This post covers training PyTorch models on Cloud TPUs with PyTorch/XLA, on both Kaggle and Colab. We go through the XLA device model, multi-core distributed training, and the changes needed to port an existing GPU pipeline.