The training runs kept dying around epoch 20. After some debugging I pinned it to a single privacy_engine.get_epsilon() call and specifically to which privacy accountant Opacus was using under the hood. That sent me into understanding how privacy accountants actually work. Here’s what I found.

What differential privacy means

Differential privacy is a mathematical guarantee that your model’s behavior doesn’t change noticeably based on whether any single training example was included.

Two datasets are called adjacent if they differ by exactly one example — one has Alice’s data, the other doesn’t. An algorithm M is (ε, δ)-differentially private if, for any two adjacent datasets D and D’ and any set of possible outputs S:

$$P[M(D) \in S] \leq e^\varepsilon \cdot P[M(D’) \in S] + \delta$$

ε is the privacy budget. Smaller ε means the output distributions on adjacent datasets are closer together, which means an attacker watching the model has less power to infer who was in the training set. δ is a small probability that the guarantee fails entirely — in practice something like 10⁻⁵, a technical escape hatch the definition needs rather than a real risk in well-tuned settings.

The guarantee is about the worst case: over all possible outputs, over all possible adjacent datasets. It doesn’t promise any individual’s data is safe. It says the output distributions are close enough that statistical tests have limited power, regardless of who the attacker targets.

DP-SGD

DP-SGD (Abadi et al., 2016) is the standard way to train neural networks with a differential privacy guarantee. Normal SGD averages gradients over a batch and applies the update. DP-SGD changes two things.

Per-sample gradient clipping. Instead of averaging raw gradients, the gradient is computed separately for each training example, clipped to a maximum L2 norm C, then averaged. This bounds how much any single example can move the update.

Gaussian noise. After clipping, Gaussian noise calibrated to C and a noise multiplier σ is added to the averaged gradient before the weight update:

$$\tilde{g} = \frac{1}{B} \left[ \sum_i \text{clip}(g_i, C) \right] + \mathcal{N}!\left(0,; \frac{\sigma^2 C^2}{B^2} I\right)$$

Higher σ means more noise, stronger privacy, and a worse model. C controls sensitivity — how much a single gradient can shift the update.

Opacus also uses Poisson subsampling instead of fixed-size batches. Each training step includes each example independently with probability q = batch_size / dataset_size, rather than sampling a fixed batch. Poisson subsampling amplifies privacy (a subsampled mechanism is more private than the same mechanism on the full batch), and this gets built into the accounting.

Now the accounting question: after T optimizer steps with noise σ and clipping norm C, what is the actual (ε, δ)?

The composition problem

Each optimizer step is one application of the Gaussian mechanism. T steps compose T times. The naive bound, basic composition, says if each step costs ε₀ then T steps cost T·ε₀. That bound is correct but useless in practice. At any practical noise level, T·ε₀ blows past a useful privacy budget within a few epochs.

The better approach is to exploit the specific shape of the Gaussian mechanism’s privacy-loss distribution. Rather than bounding the worst case at each step independently, you track that distribution across compositions. RDP and PRV both do this, but through different methods with very different memory costs.

RDP accounting

Rényi DP (Mironov 2017) reframes the privacy guarantee using Rényi divergence. The Rényi divergence of order α between distributions P and Q measures how different they are, parameterized by how much weight you put on the tails. At α = 1 it recovers KL divergence; higher α emphasizes the heavy tail behavior. Formally:

$$D_\alpha(P | Q) = \frac{1}{\alpha - 1} \log \mathbb{E}_Q!\left[\left(\frac{P(x)}{Q(x)}\right)^\alpha\right]$$

A mechanism M is (α, ε)-RDP if D_α(M(D) || M(D’)) ≤ ε for all adjacent D, D’. Composition in Rényi terms is clean: T applications of an (α, ε)-RDP mechanism give (α, T·ε)-RDP. Still linear, but the per-step ε in Rényi terms is much smaller than in (ε, δ) terms for the Gaussian mechanism at practical noise levels, so the budget lasts.

To convert back to standard (ε, δ)-DP at the end:

$$\varepsilon(\delta) = \min_\alpha \left[ \mathrm{RDP}_\alpha - \frac{\log \delta}{\alpha - 1} \right]$$

The minimum over α matters because different orders are optimal at different step counts and δ targets. Opacus evaluates at 151 Rényi orders and takes the best.

Inside RDPAccountant.get_privacy_spent, compute_rdp is called once per segment in history, results are summed, then converted to (ε, δ)-DP:

# opacus/accountants/rdp.py
rdp = sum(
    compute_rdp(q=sample_rate, noise_multiplier=noise_multiplier,
                steps=num_steps, orders=alphas)
    for (noise_multiplier, sample_rate, num_steps) in self.history
)
eps, best_alpha = get_privacy_spent(orders=alphas, rdp=rdp, delta=delta)

The memory behavior becomes clear inside compute_rdp. It calls _compute_rdp once per Rényi order to get the per-step RDP value, then scales by steps:

# opacus/accountants/analysis/rdp.py

def _compute_rdp(q: float, sigma: float, alpha: float) -> float:
    if q == 1.0:
        return alpha / (2 * sigma**2)      # no subsampling: plain Gaussian
    return _compute_log_a(q, sigma, alpha) / (alpha - 1)

def compute_rdp(q, noise_multiplier, steps, orders):
    rdp = np.array([_compute_rdp(q, noise_multiplier, order) for order in orders])
    return rdp * steps  # steps is a scalar — output is always shape (151,)

steps never enters an array dimension. It scales the fixed-size output after the per-step computation is done. That’s why memory doesn’t grow with training length.

The per-step RDP at order α reduces to _compute_log_a(q, σ, α) / (α - 1). For integer α, _compute_log_a evaluates the binomial expansion from Mironov et al. 2019 §3.3 in log-space:

def _compute_log_a_for_int_alpha(q, sigma, alpha):
    log_a = -np.inf
    for i in range(alpha + 1):
        log_coef_i = (
            math.log(special.binom(alpha, i))
            + i * math.log(q)
            + (alpha - i) * math.log(1 - q)
        )
        s = log_coef_i + (i * i - i) / (2 * sigma**2)
        log_a = _log_add(log_a, s)  # numerically stable log-space add
    return log_a

The loop runs alpha + 1 times — bounded by the Rényi order, not the step count. Calling this every epoch adds no incremental cost.

The cost of this approach is accuracy. The RDP-to-(ε, δ) conversion is an upper bound on top of the RDP bound, so the reported ε is somewhat larger than the true value. For the same mechanism and the same T, RDP will always report a looser ε than a tighter accountant would.

PRV accounting

The Privacy Random Variable accountant (Gopi, Lee, Wutschitz, NeurIPS 2021) works with the privacy-loss distribution directly.

For adjacent datasets D and D’, the privacy-loss random variable Z captures how much a single output reveals about which dataset was used:

$$Z = \log \frac{P[M(D) = o]}{P[M(D’) = o]}$$

where o is drawn from M(D). The (ε, δ)-DP condition turns out to be equivalent to P[Z > ε] ≤ δ. Computing (ε, δ) from Z is then a tail probability question: find ε such that the probability mass above ε equals δ.

Composing T steps means summing T independent copies of Z (Z₁ + Z₂ + … + Z_T gives the total privacy loss across T steps). Summing independent random variables means convolving their distributions. Convolution in frequency space is pointwise multiplication. Hence the FFT:

composed_pmf = irfft(rfft(dprv.pmf) ** num_steps)

This is more accurate than RDP because it works with the actual composed distribution rather than a divergence bound.

But to do any of this numerically you have to discretize the distribution onto a finite grid, and that’s where the memory cost comes in.

The grid

The grid needs to cover the tails of the composed distribution. Gopi et al. derive the required domain half-width L_max from an RDP bound over all T compositions (Remark 5.6 of the paper). Because that RDP bound grows with T, L_max grows with T. The grid spacing dt is fixed by the requested error tolerances. So:

$$\text{grid_size} \approx \frac{2 \cdot L_{\max}}{dt}$$

Every call to get_epsilon at a larger T builds a bigger grid, runs a bigger FFT, and allocates more working memory. Here’s what that looks like with no GPU — just the accounting math:

import gc, psutil
from opacus.accountants import create_accountant

def watch(mechanism, steps_per_epoch=98, epochs=12):
    acc = create_accountant(mechanism=mechanism)
    acc.history = [(0.2, 256/25000, 0)]
    rss = lambda: psutil.Process().memory_info().rss / 1024**2
    for ep in range(1, epochs + 1):
        nm, sr, _ = acc.history[0]
        acc.history[0] = (nm, sr, ep * steps_per_epoch)
        acc.get_epsilon(1e-5)
        print(f"{mechanism} epoch {ep:2d}: steps={ep*steps_per_epoch:5d}  RSS={rss():.0f} MB")
    gc.collect()

watch("prv")
watch("rdp")

prv epoch  1: steps=196    RSS=552 MB
prv epoch  2: steps=392    RSS=755 MB
prv epoch  3: steps=588    RSS=1033 MB
prv epoch  4: steps=784    RSS=1685 MB
prv epoch  5: steps=980    RSS=2487 MB
prv epoch  6: steps=1176   RSS=3016 MB

rdp epoch  1: steps=196    RSS=401 MB
rdp epoch  6: steps=1176   RSS=402 MB

The gc.collect() after the PRV loop reclaims nothing. scipy.fft caches FFT plans keyed by transform size, and each new grid size adds a plan at the C level, below Python’s GC. Glibc also typically doesn’t return freed heap arenas to the OS when each successive allocation is larger than the previous one, since the old block can’t be reused. The resident set ratchets up and stays.

Tradeoffs

PRV gives a tighter ε. For the same mechanism and the same T, it reports a smaller ε than RDP. On long training runs this difference can be meaningful — PRV might report ε = 8 where RDP reports ε = 11. That accuracy comes from working with the actual privacy-loss distribution rather than a Rényi surrogate.

RDP stays at a fixed (151,) array throughout training. PRV’s working memory grows with the step count, and called every epoch on a long run it can reach gigabytes. That’s what killed my training runs.

For most experimental work, RDP is the right choice: the bound is correct, the memory is flat, and it’s what the LiRA and TF-Privacy reference implementations use, so the numbers are directly comparable. If you’re reporting ε in a paper where the tightness matters — comparing against PRV-reported numbers from another paper, or making a compliance argument — use PRV, but call it once at the end of training rather than every epoch:

for epoch in range(num_epochs):
    train_one_epoch(...)

epsilon = privacy_engine.get_epsilon(delta)  # one call, at the end

Worth knowing: that final-epoch PRV call is the most expensive one in the run, since the grid is at its largest. On a tight machine, using RDP throughout and doing one optional PRV call at the very end is the safest combination.

One last thing. Switching accountants does not change training at all. The noise multiplier, clipping norm, and gradient updates are identical between accountants. The accountant is purely a post-hoc measurement of the privacy cost. But ε values from RDP and PRV aren’t on the same scale, so don’t mix them in the same logs.

References

• M. Abadi et al., Deep Learning with Differential Privacy, CCS 2016.
• I. Mironov, Rényi Differential Privacy, CSF 2017.
• I. Mironov, K. Talwar, L. Zhang, Rényi Differential Privacy of the Sampled Gaussian Mechanism, 2019.
• S. Gopi, Y. T. Lee, L. Wutschitz, Numerical Composition of Differential Privacy, NeurIPS 2021.
• A. Yousefpour et al., Opacus: User-Friendly Differential Privacy Library in PyTorch, 2021.
• N. Carlini et al., Membership Inference Attacks From First Principles, IEEE S&P 2022.

The LiRA reproduction this came out of is at github.com/apzl/LiRA-pytorch, including shadow model training, DP-SGD grid, scoring pipeline, and a longer debugging writeup.

The bank will not be accepting cash on Saturdays.
The river overflowed the bank

We humans can understand the context and relate the word with relevant meaning. This is a very basic requirement in natural language understanding. Let’s see how BERT, or contextual embedddings in general solve this problem, commonly know as word sense disambigution.

Word Sense Disambigution

Word Sense Disambiguation (WSD) is the task to identify the correct sense of the usage of a word in a sentence. The solution to this problem impacts other computer-related writing, such as discourse, improving relevance of search engines etc. Various approaches, both supervised and semi-supervised methods, with the use of dictionaries have been introduced but the introduction of transformers made a significant impact like in almost all other NLP problems.

Deep Contextual Embeddings

Representing text as vectors is the core principle behind natural language processing. Earlier models like word2vec, GloVe, and FastText were used to used to train to get such representations. But the drawback of these representations is that they don’t aknowledge the disambiguity in meaning of words. Let’s go back to example mentioned above.

The bank will not be accepting cash on Saturdays.
The river overflowed the bank

In embeddings produced by word2vec like models, the word bank will be having same representation for both the sentences even though they have very different meaning. Also different words having different semantics will be having same representations.For example,in some sense, arrive and arrival, their semantics are almost the same, but they are different parts of speech: arrive is a verb and arrival is a noun, so they can appear in quite different places.Contextual embeddings solved these problems to an extent.

The concept of deep contextualized word embeddings was first introduced in ELMo.The main idea of the Embeddings from Language Models (ELMo) can be divided into two main tasks, first an LSTM-basedlanguage model is trained on a corpus and then the hidden embeddings are used to generate a vector representation for each word.

ELMos train a multi-layer, bi-directional, LSTM-based language model, and extract the hidden state of each layer for the input sequence of words. Then, they compute a weighted sum of those hidden states to obtain an embedding for each word. The weight of each hidden state is task-dependent and is learned during training of the end-task.

How does BERT do that?

Introduction of transformers revolutionised NLP domain. The Multi-layer bidirectional Transformer or Transformer was first introduced in the Attention is All You Need paper. It follows the encoder-decoder architecture of machine translation models, but it replaces the RNNs by a different network architecture. The key concept behind the effectiveness of transformers is Multi-Head Attention.

Several weighted sums are calculated for the same input using stacked multi-head attention models, with different linear transformations.
another factor is that, BERT is a non-directional model as it is trained by masking 15% of the words in a sentence and then making the model to learn by predicting the masked words using the information from the non-masked words.
All these factors help the contextual embeddings of bert to be efficient.

Principal Component Analysis (PCA) is a very famous technique in Machine Learning. It is an unsupervised algorithm to reduce the number of dimensions in data, without losing the meaning of data. Reducing the number of dimensions makes it easier to understand and visualise the data and also compresses data, which in turn helps in faster processing. It has got a wide variety of applications in many fields like data analysis, computer vision and NLP. In this blogpost I’d like to give a brief overview into PCA and the basic math in it.

PCA is actually just a transformation method in linear algebra, which transforms the given data into combinations of original data in such a way that the new combinations will capture maximum variance in data while reducing the correlation between variables. Data is at first centered by subtracting the mean from each dimensions and then the covariance between each dimensions are calculated and arranged into a matrix called covariance matrix. Eigen vectors and eigen values are found for the covariance matrix and which is then used to find the transformed data with requred number of dimensions. Ok, so now let’s break this down into steps.

First step is to center data, that is to make the mean of data to zero. PCA without centering data might result in the first component corresponding with the mean of data rather than variance. For this all the values have the mean of the values subtracted from it.

Once the data is centered, no we need to find the covariance matrix. For that we need covariance between each dimensions. So what is covariance? For that let’s see what variance is.

Variance is a measure of spread of data in a dataset, with equation as:
$$ s^{2}=\frac{ \sum_{i=1}^n \sqrt{\big( X_{i}- \overline{X} \big)^{2} } } { \big(n-1\big)} $$

Variance gives the spread of data in a single dimension, independent of other dimensions. Covariance gives how much the dimensions vary from the mean with respect to each other. The formula for covariance is very similar to the formula for variance. The formula for variance could be written like this: $$ var(X)=\frac{ \sum_{i=1}^n \sqrt{\big( X_{i}- \overline{X} \big)\big( X_{i}- \overline{X} \big)} } { \big(n-1\big)} $$ similarly, the covarience of two dimensions X and Y is calculated as, $$ covar(X,Y)=\frac{ \sum_{i=1}^n \sqrt{\big( X_{i}- \overline{X} \big)\big( Y_{i}- \overline{Y} \big)} } { \big(n-1\big)} $$ Covarience actually shows how the dimensions are related to each other. For example, considere a 2-dimensional data, with dimension H has the number of hours studied by a group of students and dimension M as the marks they recieved. A positive covarience value indicates that both marks and hous of study increases together, or in other words, as the hours of study increases, marks increases. Negative value of covarience indicates that the as the value of a dimension increases, value of other decreases. A zero value of covarience indicates that both the dimensions are independent of each other.

Next step is to make a covariance matrix out of the covariance values. Coaviance matrix will be a square, symmetric matrix of dimension nxn, where n is the number of dimensions. For a two dimensional data, the covariance matrix will be like: $$ \begin{bmatrix}covar(x,x) & covar(x,y) \\covar(y,x) & covar(y,y) \end{bmatrix} $$ Diagonal values of the matrix will be variance value of each dimension. Because they are square and symmetrical, covariance matrixes are diagonalizable, which means an eigen decomposition can be calculated on the matrix. Next step is to find the eigen vectors and eigen values of the covariance matrix.
Considere a square matrix A, v is the eigen vector of if A[v] is a scalar multiple of v. ie., $$ A.v = \lambda v $$ where λ is the eigen value.

In context of PCA, eigen vectors of covariance matrix are the axes of principal components in a dataset. Eigen vectors show the direction of principal components calculated by PCA, where eigen values show how stretched the axes are.

A covariance matrix of n dimensions will have n pairs of eigen vectors and eigen values.
First eigen vector will be along the axis having the largest variance in data and the next one with lower variance and so on.
The eigenvalues are important because they provide a ranking criterion for the newly found axes. The principal components (eigenvectors) are sorted by descending eigenvalue. The principal components with the highest eigenvalues are “picked first” as principal components because they account for the most variance in the data.

Next step is to select the required number of eigen vectors, typically n-1 vectors with the highest eigen value. The selected eigen vectors are arranged into a feature vector. Transpose of the feature vector is then multiplied with transpose of the original data to get the new transformed dataset.

Yeah, that’s it. we now have the new dataset with required number of dimensions. It is possible to get the old data back by just multiplying the the inverse of the feature vector with the new dataset, although there might a small loss in data.

PCA is a simple yet powerful tool. PCA is applied to compress images by reducing it’s dimensions. Most of the aappplication of PCA is around data visualisation as we are limited to represent data by 2 dimensions. It is effective at filtering noise and decreasing interdependency of variables. Many programming libraries have methods to apply PCA, so it’s not that important to know the programming side of PCA, but it is good to know theoretical side. There are many resources across internet, if you’d like to dive deeper into the maths of coding side of PCA.

Principal Component Analysis (PCA) from scratch in Python
A Tutorial on Principal Component Analysis has got detailed maths explanations.