Scalars, Vectors, Matrices & Tensors Explained
From a single number to n-dimensional arrays — scalars, vectors, matrices, and tensors defined, notated, and compared in one read.
The Building Blocks of ML Math
Every number that flows through a neural network — inputs, weights, activations, gradients — lives in one of four structures. Understanding the difference between them is the prerequisite to reading any ML paper or debugging any PyTorch shape error.
0D Scalar
A scalar is just a single number. It has magnitude only — no direction, no index. Temperature, mass, learning rate, loss value — all scalars.
Properties: magnitude, arithmetic operations (+, −, ×, ÷).
1D Vector
A vector is an ordered array of numbers with both magnitude and direction. Where a scalar answers "how much?", a vector answers "how much, in which direction?". Word embeddings, feature rows, pixel intensities across a row — all vectors.
Operations: addition, subtraction, scaling, dot product (→ scalar), cross product (→ vector).
2D Matrix
A matrix is a 2D array identified by two indices: row i and column j. Written in uppercase bold (A), with element Aij at row i, column j. Weight matrices in fully connected layers, transformation matrices in graphics, covariance matrices in statistics — all 2D.
Key properties:
- Multiplication is non-commutative — AB ≠ BA in general
- Determinant and inverse defined for square matrices
- Transpose: swap rows and columns → Aᵀ
Applications: computer graphics transformations, solving linear systems (Ax = b), weight matrices in neural networks. To see the row-by-column mechanics in code, see matrix multiplication in Python.
nD Tensor
A tensor generalizes scalars, vectors, and matrices to any number of dimensions. A scalar is a 0D tensor. A vector is a 1D tensor. A matrix is a 2D tensor. Beyond 2D, tensors are the natural language of deep learning.
- 3D tensor: a single color image — (height, width, channels)
- 4D tensor: a batch of images — (batch, height, width, channels)
- 3D tensor: a tokenized sequence batch — (batch, sequence_length, embedding_dim)
How They Connect in Practice
In PyTorch, every data type is a torch.Tensor:
torch.tensor(3.14)→ scalar (0D)torch.tensor([1, 2, 3])→ vector (1D, shape [3])torch.zeros(3, 4)→ matrix (2D, shape [3, 4])torch.zeros(8, 224, 224, 3)→ 4D tensor (batch of 8 RGB images)
Every layer's forward pass is a sequence of tensor operations — matrix multiplications, element-wise activations, convolutions — all operating on these four structures. The dot product introduced above is the exact operation inside a transformer's attention mechanism, and the filters in a CNN are 4D tensors sliding across 3D image tensors.
Key Takeaways
- Scalar (0D): single number, magnitude only — loss, learning rate, temperature.
- Vector (1D): ordered array with magnitude + direction — embeddings, feature rows.
- Matrix (2D): rows × columns, non-commutative multiplication — weight matrices, transformations.
- Tensor (nD): generalization to n dimensions — everything in deep learning.
- PyTorch and TensorFlow treat all four as
Tensorobjects — shape tells you the dimension.
Frequently Asked Questions
What is the difference between a scalar, vector, matrix, and tensor?
A scalar is a single number (0D). A vector is a 1D ordered array with magnitude and direction. A matrix is a 2D array in rows and columns. A tensor is the n-dimensional generalization — scalar is 0D tensor, vector is 1D, matrix is 2D, higher-order arrays are 3D+.
What is a tensor in machine learning?
In ML, a tensor is a multi-dimensional array used to represent data and parameters. Images are 3D tensors (height × width × channels). Batches of images are 4D. PyTorch and TensorFlow use tensors as their fundamental data structure — all operations are tensor operations.
Why is matrix multiplication non-commutative?
Matrix multiplication AB ≠ BA in general because the order of row-column dot products changes. Reversing which matrix's rows pair with which matrix's columns produces a different result — and often a different output shape.
What is the difference between dot product and cross product?
The dot product (a · b) produces a scalar — it measures alignment between two vectors. The cross product (a × b) produces a new vector perpendicular to both — it measures the area of the parallelogram they span. Dot product is used in attention mechanisms; cross product is common in 3D graphics.
How are tensors used in deep learning?
Neural network weights, activations, gradients, and inputs are all tensors. A fully connected layer's weights are 2D tensors (matrices). Conv filters are 4D. Batched transformer inputs are 3D. Frameworks like PyTorch and TensorFlow optimize tensor operations for GPU parallelism.