TraceOpt blog
Notes on PyTorch training performance
TraceML vs torch.profiler vs cProfile
The same input-bound ResNet-18 run, found three ways. TraceML runs live at 2% overhead and named the bottleneck in one line. torch.profiler wrote a 70.7 MB trace you read yourself, and cProfile named the JPEG decode but could not see the idle GPU.
Read the post
Same BERT run, three GPUs
I moved the same BERT-large training recipe across an L40S, A100 SXM, and H100 SXM on RunPod. The raw throughput ranking was expected. The useful part was the TraceML fingerprint: input wait and H2D copy were negligible, all three runs were compute-bound, and the H100 result pointed to a simple batch-size follow-up.
Read the post
Run it on one real training script.
The fastest way to understand TraceML is to run it on a job you already know and check whether the fingerprint matches your intuition.