The author shares insights into PyTorch internals, aiming to demystify its complex C++ codebase for potential contributors. The essay delves into the tensor data structure, emphasizing the importance of metadata like dtype, device, and strides. A unique feature highlighted is the concept of tensor views, allowing manipulation without creating new tensors. The essay also touches on automatic differentiation, an essential feature of PyTorch. The author discusses extensions to PyTorch tensors, focusing on device, layout, and dtype parameters. The essay delves into PyTorch’s code structure, highlighting key directories like torch/ for Python modules, torch/csrc/ for C++ backend, aten/ for tensor operations, and c10/ for a punny nod to Caffe2.
https://blog.ezyang.com/2019/05/pytorch-internals/