Blog
Self-published2022Fine-tuning a pre-trained network works better when different layers move at different speeds. We set up discriminative learning rates with PyTorch parameter groups, so early layers retain what they learned while later layers adapt faster.
Self-published2021Sometimes the useful signal sits in a hidden layer rather than the output. We compare three ways to read those activations in PyTorch, from forward hooks to slicing the module list, and use them to pull out embeddings.
Self-published2021We compute channel-wise mean and standard deviation across an image dataset for normalization. The implementation batches through the data in PyTorch, so it never holds the full dataset in memory.
Self-published2021Large models often will not fit a useful batch size into GPU memory. We use gradient accumulation to reach a large effective batch anyway, walking through the modified training loop and its interaction with learning rate scaling and batch norm.
Self-published2020This post covers training PyTorch models on Cloud TPUs with PyTorch/XLA, on both Kaggle and Colab. We go through the XLA device model, multi-core distributed training, and the changes needed to port an existing GPU pipeline.
Self-published2020When labeled training data is scarce but unlabeled test data is plentiful, the test set is still useful. We pre-train on surrogate labels derived from it, then fine-tune on the real labels to lift network performance.
Self-published2020A convolutional network that grades diabetic retinopathy severity from retina photographs, built for a Kaggle competition and the Udacity ML Engineer program. We cover retina-specific preprocessing, augmentation, and the ordinal grading setup.