What a minibatch approximates
Minibatch training estimates a gradient from part of the training set rather than recomputing the full-data average at every update. It reduces per-update work but introduces variation that depends on how batches are sampled. Larger batches can improve hardware utilization while changing memory demand and time between updates. Summing a batch loss instead of averaging it also changes gradient scale. A batch-size comparison is therefore incomplete without the loss reduction, learning rate and number of updates. Numerical throughput and the number of steps needed for a useful model are related but different quantities.
Separate update rules from regularization
SGD follows a gradient estimate, while momentum incorporates a history of update directions. Methods such as Adam use running statistics of gradients and squared gradients to adjust component-wise updates. None removes the need for a suitable learning rate. Regularization imposes preferences or constraints on a model rather than merely accelerating the same unconstrained fit. Weight decay and an L2 loss penalty are also not interchangeable under every optimizer. Inspect where each term enters the update equation and which parameters it affects before assuming that two similarly named configuration options implement the same mathematical procedure.
Look at what an architecture shares
Convolutional networks use local operations and shared weights, applying similar processing at different positions. Recurrent networks update a state sequentially to summarize past inputs. Transformers use attention to mix information across positions with input-dependent weights. These choices differ in computation, parallelism and the representation of dependencies. They are useful inductive biases, not rules that every image must use convolution or every sequence must use recurrence. Match the dependency structure and resource budget of a task to the assumptions built into the architecture, then test the match with an appropriately controlled evaluation.
Check training and evaluation behavior
Dropout behaves differently during training and evaluation, and batch normalization may use different statistics in those modes. Switching a module to evaluation mode is distinct from disabling gradient recording; understand both controls rather than using their names as synonyms. Compare candidates on the same splits and evaluation conditions, recording randomness and relevant software settings. One favorable run can reflect variation rather than a reliable improvement. Stable optimization, good held-out performance and acceptable deployment cost are three separate outcomes. An architecture or optimizer should be selected through those checks rather than its reputation alone.