These 51 practice questions cover PyTorch fundamentals, model training, data pipelines, saving and inference, plus advanced topics that may matter for specific machine-learning roles. They are study prompts—not predictions about what any particular employer will ask. Build your answers around the task, explain trade-offs, and check version-sensitive API details in the current PyTorch documentation.
Tensors and basic operations
1. What is PyTorch?
PyTorch is a machine-learning library centered on tensors and operations for CPU and GPU computation. It provides tools for building neural-network modules, computing gradients, and optimizing models. The official documentation distinguishes stable APIs from areas that may be less stable, so verify details for the version used in a role: PyTorch documentation.
As an Amazon Associate I earn from qualifying purchases.
2. What is a tensor?
A tensor is an n-dimensional array that supports PyTorch operations. A scalar is zero-dimensional, a vector is one-dimensional, a matrix is two-dimensional, and higher-dimensional tensors can represent batches, images, or other structured data. Tensors can participate in gradient tracking and run on a CPU or supported accelerator.
The Tool Desk
Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →3. How do you inspect a tensor’s shape, data type, and device?
Inspect its shape (or size()), dtype, and device attributes. For example, printing x.shape, x.dtype, and x.device helps catch mismatched dimensions, unintended precision, and CPU/GPU placement errors before a computation fails.
#1 Best Overall
4. What is the difference between a tensor’s shape and its number of elements?
Shape describes the size along each dimension; the number of elements is the product of those sizes. A tensor with shape (2, 3, 4) contains 24 elements. Two tensors can have the same number of elements but different shapes, and operations still depend on compatible dimensions.
5. How do you change a tensor’s shape?
Use operations such as reshape or view when the requested shape has the same number of elements. view requires compatible memory layout, while reshape may return a view or make a copy. Use permute or transpose to reorder dimensions rather than merely changing their interpretation.
6. What is broadcasting?
Broadcasting lets elementwise operations combine tensors with compatible shapes without manually copying data. Dimensions are compared from the right: their sizes must match or one must be 1 (with missing leading dimensions treated as 1). For instance, a tensor shaped (batch, features) can be combined with a vector shaped (features,). Check shapes explicitly to avoid silently broadcasting along the wrong axis.
7. What is the difference between indexing and slicing?
Indexing selects elements or dimensions, while slicing selects ranges within dimensions. For a batch-first tensor, x[0] selects the first item and x[:, :2] selects the first two entries along the second dimension for every batch item. Advanced indexing and basic slicing can differ in whether they return a view or a copy, so avoid assuming that edits to a selection always change the original tensor.
8. How do you change a tensor’s data type?
Use a conversion such as x.to(dtype=torch.float32) or a convenience method such as x.float(). Integer tensors are often appropriate for class indices, while floating-point tensors are commonly used for model inputs and parameters. Check that inputs, parameters, and loss computations use compatible types; conversions can affect memory use and numerical behavior.
9. How do you move a tensor between CPU and GPU?
Use x.to(device), where device is selected according to what is available, such as torch.device("cuda") when CUDA is available or torch.device("cpu"). Model parameters and input tensors generally need to be on compatible devices. Moving data repeatedly inside a hot training loop can add overhead, so plan placement deliberately.
Autograd and gradients
10. What does requires_grad do?
For a tensor, requires_grad=True tells autograd to track relevant operations involving it so gradients can be calculated. It is commonly enabled for learnable parameters. It does not mean every value or gradient is retained forever: graph lifetime and what is saved depend on the operations and whether the graph is retained.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
11. What is a computational graph in PyTorch?
As tracked operations execute, autograd records the relationships needed to differentiate the result with respect to inputs that require gradients. This graph represents the computation, not a manually specified symbolic formula. PyTorch can then use the recorded operations to propagate derivatives backward.
12. What does loss.backward() do?
It computes gradients of the loss with respect to eligible leaf tensors and accumulates them in their .grad fields. The usual training pattern clears old gradients, computes the loss, calls backward(), and then asks an optimizer to update parameters. A backward pass is needed for gradient-based updates, not for every tensor computation.
Rank #2
13. Why do gradients accumulate?
Autograd adds newly computed gradients to existing .grad values. This supports intentional gradient accumulation across multiple mini-batches, but ordinary training usually clears gradients before each update with optimizer.zero_grad() or an equivalent approach. If gradients are unexpectedly large or learning behaves oddly, check whether old gradients were cleared at the intended point.
14. What is a leaf tensor?
A leaf tensor is typically a tensor created directly by the user rather than as the result of a tracked operation. Learnable parameters are leaf tensors, and their gradients are normally available in .grad after backpropagation. Intermediate non-leaf tensors do not generally retain gradients unless explicitly instructed to do so.
Do these 3 things before closing this tab:
1Repair Windows errors before they cause bigger problems2Scan for outdated or missing drivers - takes under a minute3Clear out junk files and repair common Windows errors15. When would you use torch.no_grad()?
Use torch.no_grad() around computations that should not build an autograd graph, such as many evaluation or inference passes. It can reduce graph-related memory and computation overhead. Do not use it around a training forward pass when gradients are needed for the update.
16. What is the difference between model.eval() and torch.no_grad()?
model.eval() changes the behavior of modules that distinguish training from evaluation, such as dropout and batch normalization. It does not disable gradient tracking. torch.no_grad() disables gradient recording for the enclosed operations but does not switch those modules to evaluation behavior. In inference, applications often need both, then restore training mode if training will resume.
17. What is gradient descent, and how does backpropagation fit in?
Backpropagation computes derivatives of a loss with respect to model parameters by applying the chain rule through the computation. Gradient descent uses those derivatives to adjust parameters in a direction intended to reduce the loss. An optimizer implements an update rule; backpropagation supplies the gradients it uses.
18. How would you implement a custom gradient operation?
First check whether ordinary PyTorch operations already express the computation, since autograd can differentiate supported operations. For a custom operation with a special derivative, PyTorch supports custom autograd functions with forward and backward behavior. The backward implementation must return gradients consistent with the forward inputs and mathematical operation; validate it with small examples and gradient-checking tools where appropriate. See the PyTorch examples overview: Learning PyTorch with Examples.
Modules and model behavior
19. What is torch.nn.Module?
torch.nn.Module is the base class for neural-network modules. A model commonly subclasses it, defines layers or other submodules in __init__, and implements the computation in forward. The API reference describes how modules and registered submodules participate in module operations: torch.nn.Module.
20. Why define a model’s layers in __init__ and computation in forward?
Creating layers in __init__ registers them as part of the module, making their parameters discoverable by operations such as parameter iteration, device conversion, and state saving. forward describes how inputs flow through the model. Creating new learnable layers inside every forward call can leave their parameters unregistered or recreate them on each pass.
21. What is the difference between a parameter and a buffer?
A parameter is a learnable tensor registered with a module and ordinarily included among the values an optimizer can update. A buffer is registered module state that is not treated as a learnable parameter, but is included in module state handling; running statistics in batch normalization are a familiar example. Use buffers for persistent, non-optimized tensor state.
Rank #3
22. How does PyTorch register nested modules?
Assigning a child module as an attribute of a parent module registers it. Registered children are included in module traversal and operations such as moving the parent to a device or saving its state. Use module containers such as ModuleList when storing a variable collection of layers; an ordinary Python list alone does not register its contents as submodules.
23. What is the difference between nn.Sequential and a custom module?
nn.Sequential chains modules in a straightforward order, which suits simple feed-forward compositions. A custom Module gives explicit control over branching, multiple inputs, skip connections, conditional logic, or outputs that do not follow a single chain. Choose the simplest structure that clearly expresses the model.
24. What is the purpose of forward?
forward defines the computation from inputs to outputs. Call the module itself, such as model(x), rather than calling model.forward(x) directly; the module call path handles framework hooks and other module behavior around the forward computation.
25. What changes when a model switches between training and evaluation modes?
model.train() sets training mode, while model.eval() sets evaluation mode recursively for modules. This affects layers whose behavior depends on mode, including dropout and batch normalization, but does not itself enable or disable autograd. Set the intended mode explicitly at phase boundaries.
26. How would you add a new layer or learnable component to a model?
Define a standard layer as an attribute in __init__ and call it from forward. For a learnable tensor that is not represented by an existing layer, register it as a parameter. Then verify it appears in the model’s parameters and state, and that it is placed on the same device as the computation.
Recommended Free Tools
Losses, optimizers, and training loops
27. What is a loss function?
A loss function maps model outputs and target data to a value that training seeks to minimize. The choice depends on the task and expected output representation—for example, classification and regression use different objectives. Check the loss function’s input conventions, target shape, and reduction behavior to ensure it matches the model output.
28. What is an optimizer?
An optimizer updates parameters using their gradients and an update rule. PyTorch optimizers take the parameters to update and configuration such as a learning rate. The optimizer does not compute the loss or gradients; the model, loss function, and autograd provide those inputs to the update step.
29. What is the role of the learning rate?
The learning rate controls the scale of optimizer updates. If it is too large, training can become unstable or fail to converge; if too small, progress may be slow. Select it based on the model, data, optimizer, and training behavior rather than treating one value as universally correct.
30. What is the usual order of operations in a training step?
A typical step clears prior gradients, runs a forward pass, computes a loss, calls loss.backward(), and calls optimizer.step(). The order matters: stepping before gradients are calculated uses stale or missing gradients, and failing to clear gradients changes the update by accumulating them.
Rank #4
31. What does a complete PyTorch training loop include?
A practical loop connects batches from a data loader to model predictions, loss calculation, gradient computation, and optimizer updates. It also sets training mode and tracks metrics or validation results as needed. PyTorch’s beginner path covers tensors, data loading, autograd, optimization, and persistence as parts of an end-to-end workflow: Learn the Basics.
32. How should you evaluate a model during training?
Run validation data without optimizer updates, set the model to evaluation mode, and disable gradient recording when gradients are not needed. Compute metrics that match the task and keep the validation procedure separate from training updates. Return to training mode for the next training phase.
33. How can you diagnose a loss that is not decreasing?
Check, in order, that batches and labels are paired correctly, tensor shapes and types match the loss’s expectations, model outputs are plausible, gradients exist and are finite, and the optimizer includes the intended parameters. Then inspect the learning rate, preprocessing, and whether the model is actually in training mode. Compare a tiny batch or simple case to isolate pipeline errors before changing several settings at once.
Independent reader supportYour contribution helps us test, update, and keep practical guides available for everyone.Datasets and data loading
34. What is the difference between a Dataset and a DataLoader?
A dataset defines how individual examples and labels are retrieved. A data loader wraps a dataset to provide iteration over batches and options such as shuffling and worker processes. Separating these responsibilities keeps data access distinct from the training loop.
Free tools Windows power users keep installed
One-click scans. No signup required.
35. What should a custom dataset implement?
A map-style dataset typically implements __len__ to report its number of examples and __getitem__ to retrieve one example by index. The returned item should have a consistent structure that the training loop or a collation function can batch. For streaming or unknown-length data, an iterable-style dataset may be more suitable.
36. Why use mini-batches?
Mini-batches process multiple examples per model step. They make more efficient use of vectorized computation than processing examples individually, while requiring less memory than placing an entire dataset in one batch. Batch size affects memory use, update frequency, and training behavior.
37. When should you shuffle the data?
Shuffling training examples can reduce unintended dependence on their ordering and is commonly enabled for training loaders when the task permits it. Validation and test evaluation generally should not rely on shuffled order, although the computed metrics should be order-independent. For time-dependent or sequence data, preserve the ordering required by the problem.
38. What are transforms, and where should they be applied?
Transforms preprocess or augment examples, such as converting an image to a tensor or applying a training-time augmentation. Apply transforms consistently with the model’s input expectations, and avoid stochastic training augmentations in evaluation unless they are intentionally part of the method. Keep data leakage out of preprocessing—for example, fit data-dependent transformations using training data only.
Quick wins for a faster PC:
Fix the driver behind crashes, sound loss and screen glitchesFind Drivers →Repair Windows errors before they cause bigger problemsFix Now →39. What do data-loader workers and pinned memory do?
Worker processes can load or prepare batches in parallel, while pinned host memory can help CPU-to-GPU transfers in suitable setups. Neither option guarantees faster training: workload, storage, platform, and transfer patterns matter. Measure end-to-end throughput and watch for worker-related issues such as duplicated data in iterable pipelines.
Saving, loading, and reproducibility
40. What is a model state dictionary?
A state dictionary maps registered parameters and buffers to their values. Saving a model’s state dictionary is a common way to persist learned state separately from the Python class definition. The code that defines the architecture must still be available when reconstructing the model.
41. How do you save and load a trained model’s state?
Save the state dictionary to a file, instantiate the same model architecture when loading, and load the saved values into it. Choose the intended device when loading and set evaluation mode for inference. Follow the current official persistence guidance for the PyTorch version in use, especially when loading files from untrusted sources.
42. Why might you save optimizer state as well as model state?
For inference, the model state is usually the key artifact. To resume training faithfully, also save optimizer state, since it may contain update history such as momentum buffers, along with useful training progress information. A resumed run needs compatible model and optimizer definitions.
Windows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallOutdated Drivers Are Slowing You Down
One free scan finds every outdated or missing driver and matches the right update for your exact hardware.Free scan · exact hardware match43. How do you run inference without updating the model?
Set the model to evaluation mode, then run the forward computation with gradient tracking disabled when gradients are unnecessary. Do not call the optimizer step. If inference is followed by training in the same process, switch the model back to training mode before resuming updates.
44. How can you make a training run more reproducible?
Record the PyTorch version, model and data configuration, preprocessing, random seeds, and relevant environment details. Seed the random-number generators used by the program and data pipeline, while recognizing that exact determinism can depend on operations, hardware, and backend settings. Treat reproducibility as a documented configuration, not just one seed value.
Performance, deployment, and role-dependent topics
45. How do you check whether a model is using a GPU?
Inspect the device of model parameters and input tensors, and confirm both are placed on the intended accelerator. Check whether the device is available in the environment and monitor actual utilization during computation; selecting a device alone does not establish that the workload is running efficiently on it.
46. How would you investigate a slow training job?
First determine whether the bottleneck is data loading, host-to-device transfer, model computation, or synchronization. Measure representative portions of the workload with a profiler rather than relying on intuition, then optimize the dominant cost and measure again. PyTorch provides tutorial material on profiling: PyTorch tutorials.
47. What are common causes of GPU memory problems?
Large batches, large intermediate activations, retaining computation graphs, and keeping unnecessary tensors alive can all increase memory use. Check whether graph tracking is needed, reduce batch or activation demands when appropriate, and inspect tensor lifetimes. Clearing a cache does not free memory still referenced by live tensors.
48. What does model serving mean in a PyTorch workflow?
Serving makes a trained model available to handle inference requests in an application or service. It involves more than saving weights: the deployment needs compatible model code, input preprocessing, output handling, device and resource configuration, and operational monitoring. The right serving path depends on the application and current PyTorch support; use official serving tutorials as the version-specific starting point: PyTorch tutorials.
49. What should you consider before using compilation or other optimization features?
Establish a correct baseline, identify a measured bottleneck, and test the optimization on representative inputs. Compilation and related features can have version-, model-, and workload-dependent behavior, including setup costs or unsupported patterns. Compare end-to-end results and correctness against the unoptimized implementation, consulting the current documentation for the installed version.
50. What should you know about distributed training?
Distributed training spreads model computation or data across multiple devices or processes. Candidates should be ready to explain why scaling is needed, how each worker receives data, how gradients or model state are coordinated, and what communication overhead means. The particular APIs and setup are version- and environment-sensitive, so focus on the design and verify implementation details against current documentation.
Free tools Windows power users keep installed
One-click scans. No signup required.
51. How should you tailor PyTorch interview preparation to a role?
Start with tensors, autograd, modules, data loading, optimization, and persistence, then prioritize deeper topics according to the job. A modeling-focused role may emphasize architecture and training diagnosis; an infrastructure role may go further into profiling, memory, distributed execution, or serving. The official tutorial ecosystem includes fundamentals as well as profiling and serving material, making it a useful free baseline: PyTorch tutorials.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
