Vector Calculus & Matrix

🔢 Vector Calculus & Matrix Operations

Vector calculus extends differential and integral operators to vector fields (functions that assign a vector to every point in space). Matrix calculus generalizes this to high-dimensional tensors.

---

🟢 1. Vector Differential Operators (Del $\nabla$)

The operator $\nabla$ (nabla) is a vector of partial derivative operators: $\nabla = \left[ \frac{\partial}{\partial x}, \frac{\partial}{\partial y}, \frac{\partial}{\partial z} \right]$.

1. The Gradient ($\nabla f$)

Applied to a scalar function $f$, result is a vector field.

• Points in direction of steepest ascent.

2. The Divergence ($\nabla \cdot \mathbf{F}$)

Applied to a vector field $\mathbf{F}$, result is a scalar.

• Measures the “outward flow” from a point.

3. The Curl ($\nabla \times \mathbf{F}$)

Applied to a vector field $\mathbf{F}$, result is a vector field.

• Measures the “rotation” or “swirl” at a point.

---

🟡 2. Matrix Calculus Rules

In machine learning and advanced engineering, we differentiate vectors with respect to other vectors.

1. The Jacobian Matrix ($\mathbf{J}$)

When differentiating a vector $\mathbf{f}(\mathbf{x})$ by another vector $\mathbf{x}$, the result is a matrix: $\mathbf{J} = \frac{\partial \mathbf{f}}{\partial \mathbf{x}} = \begin{bmatrix} \frac{\partial f_1}{\partial x_1} & \dots & \frac{\partial f_1}{\partial x_n} \\ \vdots & \ddots & \vdots \\ \frac{\partial f_m}{\partial x_1} & \dots & \frac{\partial f_m}{\partial x_n} \end{bmatrix}$

2. Essential Matrix Identities

• Derivative of Linear Form: $\frac{\partial (\mathbf{a}^T \mathbf{x})}{\partial \mathbf{x}} = \mathbf{a}^T$
• Derivative of Quadratic Form: $\frac{\partial (\mathbf{x}^T \mathbf{A} \mathbf{x})}{\partial \mathbf{x}} = \mathbf{x}^T (\mathbf{A} + \mathbf{A}^T)$
• Inverse Matrix: $\frac{\partial \mathbf{X}^{-1}}{\partial \mathbf{X}} = -(\mathbf{X}^{-1})^T \otimes \mathbf{X}^{-1}$ (using Kronecker product $\otimes$).

---

🔴 3. The Tensor Chain Rule

For a composition $\mathbf{z} = \mathbf{f}(\mathbf{g}(\mathbf{x}))$, the derivative is the product of the individual Jacobians: $\frac{\partial \mathbf{z}}{\partial \mathbf{x}} = \frac{\partial \mathbf{z}}{\partial \mathbf{g}} \frac{\partial \mathbf{g}}{\partial \mathbf{x}}$

Note

In Practice: In frameworks like PyTorch, we compute the Vector-Jacobian Product (VJP) because we usually only need to find the gradient of a scalar Loss function with respect to weights, rather than the full Jacobian.

---

🎯 4. Fundamental Theorems of Vector Calculus

1. Fundamental Theorem for Line Integrals: $\int_C \nabla f \cdot d\mathbf{r} = f(\mathbf{r}(b)) - f(\mathbf{r}(a))$.
2. Green’s Theorem: Relates line integrals around a closed curve to double integrals over the enclosed region.
3. Stoke’s Theorem: Relates the curl of a vector field over a surface to a line integral around the boundary.
4. Divergence Theorem (Gauss’s Theorem): Relates the divergence of a vector field in a volume to the flux through the surface.

---

💡 Practical Example: Jacobian calculation in Python

import torch

def f(x):
    # Vector-to-vector function
    # y1 = x1^2, y2 = x1*x2
    return torch.stack([x[0]**2, x[0]*x[1]])

x = torch.tensor([1.0, 2.0], requires_grad=True)
y = f(x)

# Calculating Jacobian manually in PyTorch
jacobian = torch.autograd.functional.jacobian(f, x)
print(f"Jacobian matrix:\n{jacobian}")

# Output should be:
# [[2*x1, 0],   => [[2, 0]
#  [x2,   x1]]  =>  [2, 1]]

---

🚀 Key Takeaways

• Vector operators (grad, div, curl) describe field properties.
• The Jacobian represents the “derivative” of a mapping.
• Fundamental theorems (Stoke’s, Divergence) link integration in different dimensions.