Madiyar's Page https://maitbayev.github.io/ Madiyar's Page quarto-1.8.27 Sun, 11 May 2025 23:00:00 GMT Why are two random vectors near orthogonal in high dimensions? Madiyar Aitbayev https://maitbayev.github.io/posts/random-two-vectors/

Why are two random vectors near orthogonal in high dimensions?

Introduction

High-dimensional embedding vectors are fundamental building blocks in Machine Learning, particularly in transformers or word2vec. Typically, two vectors that are semantically similar point in roughly the same direction; if they are entirely dissimilar, they point in opposite directions; and if they’re nearly orthogonal, they are unrelated.

We usually think in two or three dimensions, but there are some unintuitive properties that only apply in higher dimensions. For example, two random vectors are expected to be near orthogonal in high dimensions. Intuitively, it makes sense for word2vec, as we expect that two words are unrelated in most instances.

In this post, I will explain why two random vectors are expected to be nearly orthogonal in high dimensions.

Subscribe to get a notification about future posts.

Two Dimensions

This problem is equivalent to selecting two random vectors u and v on a circle of radius 1 and computing their dot product.

As a reminder the dot product of u and v is:

The dot product is zero when u and v are orthogonal, and near zero when they are nearly orthogonal.

The dot product is invariant under rotations, which means that we can rotate both vectors in the same way such that the vector u is [0, 1]:

This simplifies the dot product to . The probability of being near orthogonal is pretty low in two dimensions, but this helps us in framing our problem for higher dimensions.

It turns out that the average value of or is exactly . There are numerous analytical proofs, but the simplest intuition is derived from . We have a total budget of 1, which we distribute between and ; thus, on average, receives half of the budget.

We only have two orthogonal vectors to u, and they are:

Three Dimensions

In three dimensions, we take two random vectors u and v on a unit sphere and rotate them together so that the vector u becomes the north pole ([0, 0, 1]):

Tip: The spheres in this post are interactive.

The dot product of and is:

In 2D, we only had two vectors that are orthogonal to u. In 3D, we have an entire subspace of vectors that are orthogonal to u:

In other words, any vector in the red circle above is orthogonal to u. Moreover, this circle is the largest one that can be found on the sphere. For example, compare with a smaller circle which spans non-orthogonal vectors:

The larger the circle from which we select the vector v is, the closer it is to being orthogonal:

For example, we can consider all vectors within the thin red stripe to be nearly orthogonal.

The average value of or is . Similar to the 2D case, this comes from the fact that . With a total budget of 1 split among , and , each gets, on average, one-third of it.

N-dimensions

Visualizing an N-dimensional sphere is challenging. However, we have all the tools needed to develop an intuition.

Similar to the 2D and 3D cases, we can fix the first vector as . The second vector is randomly chosen from an N-dimensional unit sphere. Their dot product is then given by:

The average value of or is . Since and the total value of 1 is divided equally among n components, giving a share of . As n becomes large enough, the average value of approaches zero.

In my opinion the above explanation is enough to understand the intuition. However, let’s also present a formal proof for the expected value of .

We’re looking for . And we also have:

Hence, . Here we used the property that all components of v have identical expected values and the linearity of expectation.

The End

I hope you enjoyed this post.

Subscribe to get a notification about future posts.

]]> math random vector https://maitbayev.github.io/posts/random-two-vectors/ Sun, 11 May 2025 23:00:00 GMT Implementation of ROC AUC Score Madiyar Aitbayev https://maitbayev.github.io/posts/roc-auc-implementation/

Implementation of ROC AUC Score

Introduction

This post is a continuation of the ROC and AUC Interpretation. Please make sure that you understand that post before reading this one.

In this post, we will implement a ROC AUC Score in Python with runtime complexity.

Subscribe to get a notification about future posts.

Explanation

Similar to the previous post we have:

  • A dataset with positive and negative items
  • An ML model that predicts a probability score from 0 to 1, representing the probability that the input belongs to the positive class

We want to compute the ROC AUC score of our model predictions. The algorithm that we are going to implement is explained more easily with a visualization (press the play button):

This is a slightly modified visualization from the other post. A few notes from the animation video:

  • The ROC score is the sum of the areas of trapezoids formed by two adjacent points on the ROC curve
  • Some trapezoids have zero area
  • We process the dataset items in order of their probability scores, from the highest to the lowest

Implementation

Let’s setup our environment:

import numpy as np

np.random.seed(0)
n = 100
target = np.random.randint(0, 2, n)
predicted = np.random.rand(n)

We randomly generated targets and predicted probability scores. Let’s check the result of sklearn.metrics.roc_auc_score:

import sklearn
sklearn.metrics.roc_auc_score(target, predicted)
np.float64(0.4277597402597403)

Our implementation should have the same score.

Trapezoid Area

First, let’s implement a helper function that finds the area of the trapezoid defined by two points and .

Area of trapezoid

To achieve this, we can add the area of the rectangle and the area of the right triangle, which is:

Let’s implement the formula in Python:

def trapezoid_area(p0, p1):
    return (p1[0] - p0[0]) * (p0[1] + p1[1]) / 2.0

ROC AUC Score

Now our main implementation:

def roc_auc_score(target, predicted):
    n = target.shape[0]
    num_positive = np.sum(target == 1)
    num_negative = n - num_positive 
    # argsort in reverse order
1    order = np.argsort(predicted)[::-1]
    last = [0, 0]
    num_true_positive = 0
    num_false_positive = 0
    score = 0
2    for index in range(n):
        # Make sure that the new threshold is unique
        if index == 0 or predicted[order[index]] != predicted[order[index - 1]]:
            # True positive rate
            tpr = num_true_positive / num_positive
            # False positive rate
            fpr = num_false_positive / num_negative
            # New point on the ROC curve
3            cur = [fpr, tpr]
            
4            score += trapezoid_area(last, cur)
            last = cur
        
        if target[order[index]] == 1:
5            num_true_positive += 1
        else:
6            num_false_positive += 1
    score += trapezoid_area(last, [1, 1])

    return score
1
Sort items by their predicted scores, from largest to smallest
2
Process the sorted items one by one in a loop:
3
Form the current point on the ROC curve by:
4
Add the trapezoid area formed by the previous point and the current one
5
If the current item is positive, then increase num_true_positive by one
6
If the current item is negative, then increase num_false_positive by one

Let’s verify the result:

roc_auc_score(target, predicted)
np.float64(0.4277597402597403)

Nice, we got exactly the same result as sklearn.

The End

I hope you enjoyed this post.

Subscribe to get a notification about future posts.

]]> loss code https://maitbayev.github.io/posts/roc-auc-implementation/ Mon, 03 Feb 2025 00:00:00 GMT ROC and AUC Interpretation Madiyar Aitbayev https://maitbayev.github.io/posts/roc-auc/

ROC and AUC Interpretation

Introduction

A binary classification is a machine learning model that classifies input data into two classes. We need different metrics to train or evaluate the performance of ML models. The Area Under the Receiver Operating Characteristic Curve (ROC AUC) score is a popular metric for evaluating binary classification models. In this post, we will try to understand the intuition behind the ROC AUC with simple and interactive visualizations.

Subscribe to get a notification about future posts.

Recap

Feel free to skip this section if you are already familiar with the confusion matrix, true positive rate and false positive rate. Next Section

I suggest you also checkout the following resources:

  1. evidentlyai.com/classification-metrics/… - quite detailed explaination
  2. developers.google.com/machine-learning/… - beginner friendly explanation
  3. madrury.github.io/jekyll/update/statistics/… - my favourite so far that gives a probabilistic intution of ROC AUC score
  4. www.alexejgossmann.com/auc/ - more advanced explanation

Let’s say we have two classes (“positive” and “negative”) and a machine learning model that predicts a probability score between 0 and 1. A probability of 0.9 signifes that the input is closer to “positive” than “negative”, a probability of 0.2 signifies that the input is closer to “negative”, and so on.

However, our task is to produce a binary output: “positive” or “negative”. To achieve this, we choose a threshold. Any input with a predicted probability score above the threshold is classified as positive, while inputs with lower scores are classified as negative.

Confusion Matrix

There are four outcomes of the predicted binary output, which can be nicely summarized with the following table, called a confusion matrix:

confusion matrix

The green row corresponds to the positive items in our dataset, while the red row corresponds to the negative items in the dataset. The columns correspond to the model predictions. The cells outlined with dark green are the items our model classified correctly, i.e., the accurate predictions of our model.

Now, we can give a few relevant metrics based on the confusion matrix.

Accuracy

Accuracy is the proportion of all accurate predictions among all items in the dataset. It is defined as:

Accuracy can be misleading, especially with imbalanced datasets. For example, if 99% of our dataset is positive, a model that always predicts positive will have an accuracy of 99%, but this doesn’t provide meaningful insight. Hence, we need other metrics.

True Positive Rate

The true positive rate or recall is the proportion of accurate predictions among positive items:

The recall only considers the green row (actual positives) from our confusion table, and completely ignores the red row.

True Negative Rate

The true negative rate or specificity is the proportion of accurate predictions among negative items:

The specificity only considers the red row (actual negatives) from our confusion table.

False Positive Rate

The false positive rate is the proportion of inaccurate predictions among negative items:

alternatively:

The false positive rate is related to the true negative rate. However, we will be using FPR more than TNR in the next sections.

Visualization

Let’s setup a visualization for better understanding. Assume that we have:

  • A dataset with positive and negative items
  • An ML model that predicts a probability score from 0 to 1, representing the probability that the input belongs to the positive class.

Then, we can visualize our dataset and their probability predictions in the same visualization as below:

Positive items in the dataset are shown in green, and negative items are shown in red. The sizes of circles represent the predicted probability scores, with smaller circles representing scores close to 0 and larger circles representing scores close to 1. The items are ordered according to their probability scores, from smallest to largest.

Next, we choose a threshold depending on the application of our ML model. But, for now, let’s visualize the threshold as well:

The circles with dark green outline represent items that are accurately classified, in other words, true positives and true negatives.

Why not calculate the true positive (TP), false positive (FP), false negative (FP) and true negative (TN) values from the confusion matrix:

Tip: You can adjust the threshold using the slider above (give it a try), and the tables above and below will update accordingly.

And the metrics as defined above:

Play with the threshold slider and make sure that you understand different metrics, especially the true positive rate and the false positive rate. Maybe try to reproduce the metrics youself first at different thresholds and compare with the table?

ROC and AUC

ROC Curve

Choosing a specific threshold can be difficult since it depends on the particular application. Luckily we have metrics that show the performance of an ML model at varying threshold values. One of them is a receiver operating characteristic curve or ROC curve.

The term “Receiver Operating Characteristic” originates from Word War II, where it was used in radar systems for detecting enemy objects.

The ROC curve is the plot of the true positive rate (TPR) against the false positive rate (FPR) at each threshold values.

Let’s bring back a slightly modified version of our visualization:

Earlier, we have been visualizing the true negative rate. Now, we are visualizing the false positive rate, outlined with dark red.

The true positive and the false positive rates accordingly:

Now, we need to plot FPR (x-axis) against TPR (y-axis).

Tip: You can adjust the slider above, then this visualization will update accordingly.

Play with the slider above and compare the values in the table and (x, y) values of the black point.

The new 2D visualization is similar to the previous 1D visualization, except the following differences:

  • The positives items (green) are visualized along the Y-axis
  • The negatives items (red) are visualized along the X-axis
  • The threshold is visualized by the black circle

Area Under the Curve

The area under the ROC curve is called ROC AUC score. ROC AUC stands for “Receiver Operating Characteristic Area Under the Curve”.

The ROC AUC score is a single number that summarizes the ML model’s performance across all threshold values. Note that the ROC curve summarizes with a visual plot, whereas the ROC AUC score summarizes with a single number.

The area of the light gray region in the above visualization is the ROC AUC Score.

The ROC AUC score is 0.5 for a classifier that performs no better than random guessing and approaches 1.0 for a classifier with perfect performance.

Probabilistic Interpretation

The ROC AUC score has a very nice probabilistic interpretation:

The ROC AUC is the probability that the model will predict a higher probability score to a randomly selected positive item than a randomly selected negative item.

It is nicely explained with the visualization that we already have:

Every point (e.g., the black dot above) under the ROC curve has a corresponding green circle (follow the dotted horizontal segment) and a corresponding red circle (follow the dotted vertical segment). Then, the mentioned green circle is always bigger or equal to the red circle.

Tip: Try the slider above and compare the matching green circle with the matching red circle.

This is the same interpretation as the probabilistic interpretation when the number of items in our dataset approaches infinity.

The End

I hope you enjoyed this post.

In the next post, we will work on an efficient Python implementation of the ROC AUC score based on probabilistic intuition.

Subscribe to get a notification about future posts.

]]> math cv loss https://maitbayev.github.io/posts/roc-auc/ Fri, 31 Jan 2025 00:00:00 GMT Linear Equation Intuition Madiyar Aitbayev https://maitbayev.github.io/posts/linear-equation/

Linear Equation Intuition

Introduction

A linear equation is an equation in the form . In two dimensions, it’s an equation of a line , while in three dimensions, it’s an equation of a plane .

A linear equation is the foundation of linear models in machine learning, such as linear regression and logistic regression.

In this post, let’s understand the intuitions behind the linear equation, such as:

  • Why is the distance between the point and the line?
  • What are the meanings of and ?
  • How are the dot product and linear equation related?

My previous post about the dot product is a required prerequisite to this post. You need to have a good intuition about the dot product.

Cheatsheet

A short summary for people in hurry.

The distance from the point (x, y) to the line is:

in n-dimensions:

The vector is the orthogonal vector to the line , and in n-dimensions.

The line equation can be expressed with the dot product as and in n-dimensions.

Equation of a line

ax+by

Let’s start with a simpler equation of a line in the form of , where we have . Does it remind you something ? Yes, the dot product between and . The dot product is zero when the two vectors are orthogonal. Oh nice, this means that (a, b) is an orthogonal vector to our line. An orthogonal vector is also known as a normal vector of a line. Also, notice that the simpler line contains the origin since holds true.

Let’s visualize our setup:

We have the following items visualized:

  • Our line is visualized with blue
    • It passes through the origin
  • A normal vector of the line is visualized with red
  • Note that multiple vectors can represent the same line

Feel free to play (by moving) with the point and notice how the line equation is changing with respect to the normal vector.

We can multiply or divide the line equation by any real number without changing the equation, in other words, and hold true to the same set of points. Let’s use this property and divide the equation by , since it gives us and .

We obtain a new normalized line equation with the coefficients:

Note that holds true.

From now on, we assume that . Let’s check what we have:

We introduced a slider that defines the angle of the normal vector . We also restricted so that . Now, our line is uniquely defined by , which was not case before.

Let’s introduce a point on the line:

Try moving the new green point! The point always lies on the line, in other words:

  • , where
  • Equivalently
  • The angle defined by the points is a right angle.

We have learned so far that and represent the normal vector , and we can express the line equation with the dot product .

What is the meaning of the line equation when the point is outside of the line, i.e., when ? Spoiler: the value is the signed distance from the point to our line. Let’s understand this intuitively.

The dot product is a non-zero value when the point is not on the line. From the previous post we know that the dot product is the projection of the vector to the vector . Imagine a 1D number line spanned by the vector like this:

The introduced 1D number line has ticks that represent the signed distances from our line. The projection of the point falls somewhere on that 1D number line (following the dotted line segment), which corresponds to the dot product. Remember that and are the same. Play with the point (and with the slider) to understand when is zero, positive and negative. Can you see that (and ) represents the signed distance from the point to our line?

ax+by+c

By this point, you need to understand the line equation of the form . Now, let’s explore the line equation of the form !

We represented with the dot product , similarly can be represented with . First, we land on the 1D number line with , then move steps forward on that 1D number line. It is easier explained with a visualization:

Play with the slider for , and notice the location of our line when and .

Can you see that is still the signed distance from the point to our line? What does represent?

First, we land on the 1D number line with . Then, let’s denote as . The is zero when ; when ; when ; and so on.

is the distance from the origin to our line, c is the signed distance. The signed distance is positive in the direction of the normal vector, and negative in the opposite direction.

The End

I hope you enjoyed this post. Subscribe to get a notification about future posts.

]]> math https://maitbayev.github.io/posts/linear-equation/ Mon, 20 Jan 2025 00:00:00 GMT Dot Product Intuition Madiyar Aitbayev https://maitbayev.github.io/posts/dot-product/

Dot Product Intuition

Introduction

The dot product (or scalar product) is a simple yet powerful operation that is used in many places in Machine Learning and other fields. In this post, I will explain the geometric intuition behind the dot product. You need to have a basic grasp of trigonometry and vector algebra to follow this post.

I will need this explanation for my future posts.

Feel free to ask questions on my telegram channel

Recap

More beginner friendly explanations are available in the following resources:

Feel free to checkout the above or other resources first.

Assume we have two vectors and , then there are two definitions of the dot product: algebraic and geometric.

Algebraic definition

The dot product of and is:

Geometric definition

The dot product of and is:

where is the angle between a and b and is the magnitude of a vector a.

The geometric definition gives us a few useful properties:

  • The dot product is zero when a and b are orthogonal, since
  • The dot product is positive for acute angles and negative for obtuse, e.g., or are positive but are negative.
  • We can find the angle between vectors by . A picture from Wikipedia:

Source Wikipedia

Equivalence?

Wait, how are the two dot product definitions the same? How is related to such a straightforward sum of components? A few questions I asked when I first encountered the dot product. I accepted the fact and moved on with my life until today. I will try to understand myself and also explain by interactive visualizations.

Explanation

Let’s first simplify the problem as much as possible. Once we build up an intuition for simpler cases, we can come back and try to understand the general cases.

Simplified version

First, we will only consider the case. Assume that we have 2D vectors: and with the dot product:

where is the length of the vector and is the angle between the vectors.

Second, scaling either or by a real number does not invalidate the dot product equivalence. Let’s say we scale by , then:

With the above, we can simplify further by constraining . If the equivalence holds for , then it holds for any .

A visualization for the setup we have so far:

We have the following items shown above:

  • The vector : it is interactive and visualized with blue; feel free to move the point around.
  • The vector : it is visualized with red and .
  • : the angle from to

The geometric definition implies that the dot product is invariant under rotations. So, if we rotate the both vectors and by the same angle , the dot product won’t change. It is not obvious from the algebraic definition, but a detailed proof is provided in the next section. Let’s trust this property for now until the next section, and simplify our problem for the third time.

We can rotate the both vectors such that is aligned with the x-axis. Click the button below for the illustration:

We arrived at a simplifed version of the problem: a vector and a fixed vector . Let’s check if the definitions agree now. The algebraic dot product is:

and the geometric is:

now, we have . This is correct due to the polar coordinates. Alternatively, we can easily derive from the cosine definition. The proof is left as an exercise to the reader. You can use the visualization below to convince yourself:

We introduced a new vector in the above visualization:

  • It is visualized in green and visually represents the dot product
  • It has coordinates :
    • Where is the dot product as we derived above
  • The vector is collinear with the vector
  • Since the vectors and lie along the same line:
    • We can express using vector arithmetic:
    • is a real number, which means how many `s are required to reach the vector
    • You can visualize a 1D number line spanned by the vector , then is the location of in that 1D coordinate system.
  • can be 0, positive or negative. When negative, the vector is opposite to the vector
  • The vector is actually the projection of the vector into .
  • Play with the point to gain more intuition!

Try different values of by moving it in the visualization, and verify the following questions:

  • When the dot product is zero?
  • When the dot product is negative ?
  • When the dot product is positive ?
  • What values of result in the same dot product?

The simplified form of the dot product is quite useful for intuition. I visualize this version in my mind when I forget some details of the dot product. At this point, you should be comfortable with the dot product of the simplified form: .

Rotational invariance

First, let’s visualize the rotation:

There is a slider which represents the angle (in degrees) by which both vectors are rotated. Feel free to play with the slider! You can also interact with the point .

It is easy to see why the dot product is invariant under rotations from the geometric definition. The geometric definition relies only on the lengths of the vectors, and the lengths don’t change when rotated.

However, it is not immediately clear from the algebraic definition. To prove it, we derive an alternative formula for the algebraic dot product:

which gives us:

The dot product above is derived purely from the lengths. Since the lengths remain unchanged under rotations, the algebraic dot product remains unchanged as well, which completes our proof.

Note that we derived the law of cosine, which is quite cool!

General version

It’s not hard to extend the 2D version of the proof to n-dimensional vectors, since we didn’t rely on any properties unique to 2D. The shortest way is:

  • Note that the dot product of n-dimensional vectors is invariant under rotations. The same proof as in the rotational invariance
  • Rotate the vectors such that they lie in the xy-plane
  • Now, we reduced n-dimensional vectors to 2D vectors, which we already know how to prove

I will provide more alternative proofs below, mostly for myself. They’re optional and collapsed, feel free to read them.

We only scaled the vector but we can also scale the vector , so that . Also switch to the polar coordinates: and .

Then, the geometric dot product is:

The algebraic is:

We arrive at the cosine subtraction rule:

We will extend the formula from the rotational invariance section further. We derived the following dot product formula:

From the law of cosine, we also have:

if we rearrange and combine the above two equations:

We can use the Pythagorean theorem:

This gives us the following:

if we substitute by :

which gives back the both geometric and algebraic dot product.

You can also find more proofs in proofwiki.

The End

I hope you enjoyed this post. You can ask further questions on my telegram channel

]]> math https://maitbayev.github.io/posts/dot-product/ Mon, 13 Jan 2025 00:00:00 GMT Geometric Intuition for Jensen’s Inequality Madiyar Aitbayev https://maitbayev.github.io/posts/jensens-inequality/

Geometric Intuition for Jensen’s Inequality

Introduction

Jensen’s inequality is fundamental in many fields, including machine learning and statistics. For example, it is useful in the diffusion models paper for understanding the variational lower bound. In this post, I will give a simple geometric intuition for Jensen’s inequality.

Subscribe to get a notification about future posts.

Feel free to leave feedback on my telegram channel.

Setup

The post contains collapsed code sections that are used to produce the visualizations. They’re optional, hence collapsed.

code for fig_jensen_inequality 
import itertools
from typing import Optional

import numpy as np
import plotly.graph_objects as go


def alpha_profiles(n: int):
    if n == 2:
        space = np.linspace(0.01, 0.99, 100)
        return np.column_stack((space, 1.0 - space))
    space = np.linspace(0.01, 0.99, 15 - max(0, (n - 3) * 5))
    space_prod = itertools.product(*[space for _ in range(n - 1)])
    profiles = np.array(list(space_prod))
    profiles = profiles[np.sum(profiles, axis=1) < 1.0]
    return np.concatenate([profiles, 1 - np.sum(profiles, axis=1).reshape(-1, 1)], axis=1)


def fig_jensen_inequality(f, x_range: list, x: np.array, y_range: Optional[list] = None):
    x_linspace = np.linspace(x_range[0], x_range[1], 100)
    points = np.column_stack([x, f(x)])
    n = len(points)
    steps = []
    hull_points = []
    titles = []
    for index, alphas in enumerate(alpha_profiles(n)):
        hp = np.average(points, weights=alphas, axis=0)
        hull_points.append(hp)
        title = ",".join(["\\lambda_" + f"{i + 1}={a:.2f}" for i, a in enumerate(alphas)])
        title = f"${title}$"
        titles.append(title)
        step = dict(name=index, label=index, method="update",
                    args=[{
                        "x": [[hp[0], hp[0]], [hp[0]], [hp[0]]],
                        "y": [[f(hp[0]), hp[1]], [hp[1]], [f(hp[0])]],
                    }, {"title": title}, [2, 3, 4]])
        steps.append(step)
    active_index = len(steps) // 2
    sliders = [dict(active=len(steps) // 2, steps=steps)]
    return go.Figure(data=[
        go.Scatter(
            name="f", x=x_linspace, y=f(x_linspace), hoverinfo="none"
        ),
        go.Scatter(
            name="Convex Hull", x=np.append(points[:, 0], points[0][0]),
            y=np.append(points[:, 1], points[0][1]),
            fillcolor="rgba(239, 85, 59, 0.2)", fill="toself", mode="lines",
            line=dict(width=3), hoverinfo="none",
            showlegend=False
        ),
        go.Scatter(
            x=[hull_points[active_index][0], hull_points[active_index][0]],
            y=[f(hull_points[active_index][0]), hull_points[active_index][1]],
            mode="lines",
            textposition="bottom center",
            hovertemplate="(%{x:.2f}, %{y:.2f})<extra></extra>",
            line={"color": "black", "dash": "dot", "width": 1},
            showlegend=False
        ),
        go.Scatter(
            name="A",
            x=[hull_points[active_index][0]],
            y=[hull_points[active_index][1]],
            mode=f"markers+text",
            text=["$(\\sum \\lambda_i x_i, \\sum \\lambda_i f(x_i))$"],
            textposition="top center",
            hovertemplate="(%{x:.2f}, %{y:.2f})<extra></extra>",
            marker={"size": 20, "color": "black"},
        ),
        go.Scatter(
            name="B",
            x=[hull_points[active_index][0]],
            y=[f(hull_points[active_index][0])],
            mode=f"markers",
            text=["B"],
            textposition="bottom center",
            hovertemplate="(%{x:.2f}, %{y:.2f})<extra></extra>",
            marker={"size": 20, "color": "#00CC96"},
        ),
        go.Scatter(
            name="$(x_i, f(x_i))$",
            x=points[:, 0], y=points[:, 1],
            mode="markers+text",
            marker={"size": 20, "color": "#ffa15a"},
            line={"color": "rgba(239, 85, 59, 0.2)"},
            text=[f"$(x_{i},f(x_{i}))$" for i in range(1, n + 1)],
            textposition="top center",
            hovertemplate="(%{x:.2f}, %{y:.2f})<extra></extra>",
            showlegend=False,
        ),
    ], layout=go.Layout(
        title=titles[active_index],
        xaxis=dict(fixedrange=True, range=x_range),
        yaxis=dict(fixedrange=True, scaleanchor="x", scaleratio=1, range=y_range),
        sliders=sliders,
        legend=dict(
            yanchor="top",
            xanchor="right",
            x=1,
            y=1
        ),
        margin=dict(l=5, r=5, t=60, b=20)
    ))


def sample_parabola(x):
    return 0.15 * (x - 15) ** 2 + 15
code for show_sample_jensen_inequality 
def show_sample_jensen_inequality(x: list):
    fig = fig_jensen_inequality(
        f=sample_parabola,
        x_range=[0, 30],
        x=np.array(x),
        y_range=[12, 48]
    )
    return fig.show(renderer="iframe")

Convex Function

A function is a convex function when the line segment joining any two points on the function graph lies above or on the graph. In the simplest term, a convex function is shaped like and a concave function is shaped like . If f is convex, then -f is concave.

A visualization from Wikipedia:

display image from Wikipedia 
from IPython.display import Image
Image(url='https://upload.wikimedia.org/wikipedia/commons/c/c7/ConvexFunction.svg', width=400)

Definition

A function is called convex if the following holds:

and concave when:

for all and for all .

We will give geometric intuition for this definition in the next section.

Geometric Intuition

show_sample_jensen_inequality(x=[2, 22])

An interactive visualization of the convex function: . We will use the same parabola during this post unless stated otherwise. You can use the slider to try different values of (, , where (from the definition) and .

We have a line segment that joins and . We can sample any point along the line segment with where and . For example:

  • When , we get the first point
  • When , we get the second point
  • And when , we get the middle point of the line segment
  • and so on… Try the slider above and notice how and are changing!

This point is visualized with a black point above. Let’s name it as A.

The light green point where the function graph intersects with the dotted line segment is represented by: . Let’s name it as B.

Then, the definition above is just asserting that and we also have . Note that we are only showing a single line segment, but this statement should be true for all similar line segments and for all .

Jensen’s Inequality

Jensen’s inequality is a generalization of the above convex function definition for more than 2 points.

Definition

Assume we have a convex function and in ’s domain, and also positive weights where . Then Jensen’s inequality can be stated as:

The equation is flipped for a concave function g:

Note that we arrive at the same definition for convex function when .

Geometric Intuition

A numerous proofs are already available by other posts. I encourage you to checkout the following resources:

Here I describe a geometric intuition, which resonates more with me.

Triangle

Let’s start with a triangle, i.e., :

show_sample_jensen_inequality(x=[2, 12, 27])

As before, you can use the slider to try different values of where .

We have a triangle that connects the points: .

In the case, we used and to sample a point along the line segment. In this case, it is similar, but we can sample any point inside or on the boundaries of the triangle with:

where . For example:

  • When where , we get the point
  • When , we get the center of mass of the triangle

The black point (named A) in the visualization represents this point.

Note that (, , ) describes the barycentric coordinate system. You don’t need to know it in this post, just sharing in case you’re already familiar with it.

The light green point where the parabola meets the dotted line segment is represented by:

If we name this point as B, then it is not difficult to see that Jensen’s inequality is the same as .

Four Points or More

It is easy to generalize for . I am adding it here for the sake of completeness:

show_sample_jensen_inequality(x=[2, 13, 22, 25])

In the general case, describes a point inside or on the boundary of the convex hull enclosing the points: . The convex hull is always above or on the convex function graph, which is why Jensen’s inequality holds true.

A few closing notes:

  • The convex hull may have any number of points, including n → ∞
  • We closely approximate the convex function in some interval with the convex hull as n approaches infinity
  • The convexity definitions for functions and polygons are the same once we have enough points, i.e., n → ∞
  • Jensen’s inequality is useful in a probability theory setting, since , including the continuous form with n → ∞.

Applications

The arithmetic mean-geometric mean inequality (AM-GM inequality) states that:

Let’s prove with Jensen’s inequality by rewriting the above with :

Since log is a concave and monotonic function, we can apply log to both sides.

The above equation is valid due to Jensen’s inequality. Note that the same proof works for the weighted version since the proof does not rely on the fact that for all .

The End

I hope you enjoyed this post. You can ask further questions on my telegram channel.

Subscribe to get a notification about future posts.

]]> convex function jensen https://maitbayev.github.io/posts/jensens-inequality/ Sat, 04 Jan 2025 00:00:00 GMT Why does L1 regularization encourage coefficients to shrink to zero? Madiyar Aitbayev https://maitbayev.github.io/posts/why-l1-loss-encourage-coefficients-to-shrink-to-zero/

Why does L1 regularization encourage coefficients to shrink to zero?

Introduction

Regularization is a common method for dealing with overfitting in Machine Learning (ML). The simplest and most widely used methods are L1 (Lasso) and L2 (Ridge). The L1 and L2 regularizations are well covered in numerous tutorials and books. However, I could not find any good geometric or intuitive explanation of why L1 encourages coefficients to shrink to zero. This post tries to address this.

The post contains collapsed code sections that are used to produce the visualizations. They’re optional, hence collapsed.

Feel free to leave feedback on my telegram channel.

Subscribe to get a notification about future posts.

Recap

The Lasso

The lasso regression is a linear regression model that shrinks the coefficients by imposing a constraint on their magnitude. Namely, it constrains the sum of absolute values of the coefficients:

The above equation is the same equation (3.51) from “The Elements of Statistical Learning” (ESL) book by Hastie, Tibshirani, and Friedman.

We can also write the lasso in the equivalent Lagrangian form (3.52), which penalizes the sum of the absolute values of the coefficients:

The equations (3.51) and (3.52) are equivalent under the correct and hyperparameters. The latter equation (3.52) is more preferred in ML.

Making sufficiently small will shrink some of the coefficients to be exactly zero, which we will give geometric intuition later. Thus, the lasso could be used for feature selection, i.e., for identifying less important features.

The Ridge

The ridge regression is similar to the lasso except it penalizes the sum-of-squares of the coefficients (3.41):

An equivalent way to write the ridge problem is (3.42):

The ridge regression will shrink the coefficient towards zero when is sufficiently small; however, the coefficients might not be exactly zero.

Ridge vs Lasso

Prior expalanations

The best post I found so far is https://explained.ai/regularization, which explains the difference between L1 and L2 empirically by simulating random loss functions. I really encourage you to check out the tutorial.

Another explanation is given by the amazing The Elements of Statistical Learning book:

Figure 3.11

The constraint region for L2 is the disk (right figure), while it is the diamond for L1 (left figure). Both methods find the first intersection point between the elliptical contours (loss) and the constrained region. The corners of the diamond have one parameter equal to zero. The diamond becomes a rhomboid in higher dimensional space, and has many corners; there are many more opportunities to intersect at the corners.

There are cool algebraic explanations as well, for example, given here https://www.reddit.com/r/MachineLearning/….

The explanations given by the book and other places are well-contained, but I could not fully grasp what’s so special about the corners w.r.t the other points on the edges. There are only 4 corners and unlimited points on the edges, so shouldn’t the probability to touch the other points be higher?

Geometric Explanation

Setup

We will use a simple loss function that illustrates circle contours instead of elliptical ones.:

Once we understand the intution for circles, it is easy to extend to other contours such as elliptical ones. We will use and most of the time.

code for loss setup and helper functions 
from enum import Enum

import mpl_toolkits.mplot3d.art3d as art3d
import numpy as np
from matplotlib import animation
from matplotlib import pyplot as plt
from matplotlib.patches import Circle, Polygon


def loss(b0, b1, cx, cy, scale=2.0, bias=100):
    return scale * (b0 - cx) ** 2 + scale * (b1 - cy) ** 2 + bias


class Reg(Enum):
    L1 = 1
    L2 = 2


def make_reg_shape(reg: Reg, t: float, color="k"):
    if reg == Reg.L1:
        return Polygon(xy=[(t, 0), (0, t), (-t, 0), (0, -t)], color=color, fill=False, linestyle='--')
    else:
        return Circle(xy=(0, 0), radius=t, color=color, fill=False, linestyle='--')


def argmin_within_constraint(reg: Reg, t: float):
    beta0 = np.linspace(*beta_range, 100)
    beta1 = np.linspace(*beta_range, 100)
    B0, B1 = np.meshgrid(beta0, beta1)
    Z = loss(B0, B1, cx=cx, cy=cy)
    if reg == Reg.L1:
        mask = np.abs(B0) + np.abs(B1) <= t
    else:
        mask = B0 * B0 + B1 * B1 <= t * t
    index = np.argmin(Z[mask])
    return B0[mask][index], B1[mask][index]


beta_range = -20, 20
cx, cy = 15, 5
vmax = 1000
code for plot3d 
def base_fig3():
    # Create a figure and a 3D Axes
    fig = plt.figure(figsize=(5, 5))
    ax = fig.add_subplot(111, projection='3d')
    ax.set_xlabel("$\\beta_1$", labelpad=0)
    ax.set_ylabel("$\\beta_2$", labelpad=0)
    ax.set_zlim(0, 500)
    ax.tick_params(axis='x', pad=0)
    ax.tick_params(axis='y', pad=0)
    ax.set_xlim(*beta_range)
    ax.set_ylim(*beta_range)
    # draw axes
    ax.plot(beta_range, [0, 0], color='k')
    ax.plot([0, 0], beta_range, color='k')
    return fig, ax


def plot3d(reg: Reg, t=3):
    fig, ax = base_fig3()

    # surface
    beta0 = np.linspace(*beta_range, 100)
    beta1 = np.linspace(*beta_range, 100)
    B0, B1 = np.meshgrid(beta0, beta1)
    Z = loss(B0, B1, cx=cx, cy=cy)
    ax.plot_surface(B0, B1, Z, alpha=0.7, cmap='coolwarm', vmax=vmax)

    # contours
    ax.plot([cx], [cy], marker='x', markersize=10, color='black')
    ax.contour(B0, B1, Z, levels=50, linewidths=.5, cmap='coolwarm', zdir='z', offset=0, vmax=vmax)
    
    # minima within regularization shape
    mx, my = argmin_within_constraint(reg, t)
    ax.plot([mx], [my], marker='.', markersize=10, color='r')

    # regularization contraints
    reg_shape = make_reg_shape(reg, t, color="black")
    ax.add_patch(reg_shape)
    art3d.pathpatch_2d_to_3d(reg_shape, z=0)

    ax.view_init(elev=39, azim=-106)
    plt.tight_layout()
    plt.show()

Let’s visualize our loss with the L1 constaint , i.e., in 3D:

plot3d(Reg.L1, t=5)

In Lasso Regression, we’re looking for and within the diamond that has the lowest loss, which is marked with the red point in the figure above. The global minima without any constraint is marked with “x”.

The same visualization but with the L2 constraint t=5, i.e., :

plot3d(Reg.L2, t=5)

The corresponding 2D visualizations that are similar to the ones given by the Elements of Statistical Learning book:

code for plot2d 
def loss_contour(ax):
    beta0 = np.linspace(*beta_range, 100)
    beta1 = np.linspace(*beta_range, 100)
    B0, B1 = np.meshgrid(beta0, beta1)
    Z = loss(B0, B1, cx=cx, cy=cy)

    ax.contour(B0, B1, Z, levels=50, linewidths=.5, cmap='coolwarm', vmax=vmax)
    # draw the global minima
    ax.plot([cx], [cy], marker='x', markersize=10, color='black')


def ax2d_init(ax):
    ax.set_aspect('equal')
    ax.set_xlabel("$\\beta_1$", labelpad=0)
    ax.set_ylabel("$\\beta_2$", labelpad=0)
    ax.tick_params(axis='x', pad=0)
    ax.tick_params(axis='y', pad=0)
    ax.set_xlim(*beta_range)
    ax.set_ylim(*beta_range)

    # draw axes
    ax.axhline(y=0, color='k')
    ax.axvline(x=0, color='k')


def plot2d(regs: [Reg], t: float):
    fig = plt.figure(figsize=(4 * len(regs), 4))
    axes = fig.subplots(1, len(regs))
    for ax in axes:
        ax2d_init(ax)
    for reg, ax in zip(regs, axes):
        # draw the regularization safe region
        ax.add_patch(make_reg_shape(reg=reg, t=t))
        loss_contour(ax)
        # draw minima within constraint
        mx, my = argmin_within_constraint(reg, t)
        ax.plot([mx], [my], marker='.', markersize=10, color='r')
    plt.tight_layout()
    plt.show()
plot2d([Reg.L1, Reg.L2], t=5)

Explanation

In both cases (L1, L2), we’re looking for a contour that just touches the constraint region. For example, when we have the following case:

code for plot_reg_and_circle 
def plot_reg_and_circle(reg: Reg, t: float, cx: float, cy: float, radius: float):
    fig = plt.figure(figsize=(3, 3))
    ax = fig.add_subplot()
    ax2d_init(ax)
    ax.add_patch(Circle(xy=(cx, cy), radius=radius, color='b', fill=False))
    ax.add_patch(make_reg_shape(reg, t=t))
    plt.tight_layout()
    plt.show()
plot_reg_and_circle(Reg.L1, t=7, cx=5, cy=5, radius=4)

Then it is more optimal to reduce the constraint region (i.e., the diamond) until they touch in a single point as shown below. Note that a contour has the same loss along its points.

plot_reg_and_circle(Reg.L1, t=(np.sqrt(50)-4) * np.sqrt(2), cx=5, cy=5, radius=4)

When does the Lasso pick the corner of the diamond over any other point on the edges? To simplify our problem, let’s fix a specific circle among the contours: a circle with a fixed radius . Now, we want to come up with a formula that gives the probability of a random tangent circle with radius touching our diamond at the corners versus at any other points.

The above problem becomes much simpler with a visualization:

code for html5_video_l1_tangent_circles 
def l1_tangent_circle_locations(t: float, radius: float) -> list[np.array]:
    def is_corner(v):
        return min(abs(v[0]), abs(v[1])) <= radius / np.sqrt(2) + 1e-3

    vertices = make_reg_shape(Reg.L1, t=t + np.sqrt(2) * radius).get_path().interpolated(50).vertices
    vertices = vertices.tolist()
    while is_corner(vertices[-1]):
        vertices.insert(0, vertices.pop())
    vertices.reverse()
    locations = []
    for i in range(8):
        should_be_corner = i % 2 == 0
        group = []
        while len(vertices) > 0 and is_corner(vertices[-1]) == should_be_corner:
            v = vertices.pop()
            if should_be_corner:
                if abs(v[0]) <= abs(v[1]):
                    corner = [0, np.sign(v[1]) * t]
                else:
                    corner = [np.sign(v[0]) * t, 0]

                vec = np.array(v) - np.array(corner)
                vec = (vec / np.linalg.norm(vec)) * radius
                group.append(corner + vec)
            else:
                group.append(v)
        locations.append(np.array(group))
    return locations


def l2_tangent_circle_locations(t: float, radius: float) -> list[np.array]:
    angles = np.linspace(0, 2 * np.pi, 200)
    vertices = np.column_stack([np.cos(angles), np.sin(angles)]) * (t + radius) 
    return [vertices]


def animate_tangent_circle_trajectories(reg: Reg, t: float, radius: float):
    fig = plt.figure(figsize=(4, 4))
    ax = fig.add_subplot()
    ax2d_init(ax)
    if reg == Reg.L1:
        diamond = make_reg_shape(Reg.L1, t=t)
        circle_locations = l1_tangent_circle_locations(t, radius)
        plots = []
        for i, locations in enumerate(circle_locations):
            color = 'g' if i % 2 == 0 else 'b'
            plots.append(ax.plot([], [], color=color, linewidth=2)[0])
    else:
        circle_locations = l2_tangent_circle_locations(t, radius)
        plots = [ax.plot([], [], color='b', linewidth=2)[0]]
    
    circle = Circle(xy=(t + radius, 0), radius=radius, color='g', fill=False)
    ax.add_patch(circle)
    ax.add_patch(make_reg_shape(reg, t=t))

    def update(frame):
        remaining = frame
        for plot, locations in zip(plots, circle_locations):
            need = min(remaining, len(locations))
            remaining -= need
            plot.set_xdata(locations[:need, 0])
            plot.set_ydata(locations[:need, 1])
            if remaining == 0 and need > 0:
                last = locations[need - 1]
                circle.set_center(last)
                circle.set_color(plot.get_color())

    num_frames = np.sum([len(l) for l in circle_locations])
    plt.tight_layout()
    ani = animation.FuncAnimation(fig, func=update, frames=range(0, num_frames, 2), interval=30)
    return ani


def html5_video_tangent_circle_trajectories(reg: Reg, t: float, radius: float):
    ani = animate_tangent_circle_trajectories(reg, t=t, radius=radius)
    html = ani.to_html5_video()
    plt.close()
    return html
from IPython.display import HTML
HTML(html5_video_tangent_circle_trajectories(Reg.L1, t=5, radius=6))

The animation above illustrates all tangent circle trajectories. The circle is green when it touches the diamond at the corners, and it is blue when it touches at other points. The total length of all blue (touching at other points) trajectories is given by , while the total length of all green (touching at the corners) trajectories is given by , where t is the constraint and r is the radius of the circle. This means that it is likely to touch at the corners when t is sufficiently small or r is sufficiently high. We illustrate one of the such cases below:

HTML(html5_video_tangent_circle_trajectories(Reg.L1, t=3, radius=10))

In Ridge (L2), we don’t have special points like corners, and all points along the disk have the same probability of touching a random tangent circle with radius :

HTML(html5_video_tangent_circle_trajectories(Reg.L2, t=5, radius=6))

End

I hope you enjoyed this post. You can ask further questions on my telegram channel.

Subscribe to get a notification about future posts.

]]> basics loss l1 l2 lasso ridge https://maitbayev.github.io/posts/why-l1-loss-encourage-coefficients-to-shrink-to-zero/ Fri, 27 Dec 2024 00:00:00 GMT WIP: High-Resolution Image Synthesis with Latent Diffusion Models Notes Madiyar Aitbayev https://maitbayev.github.io/posts/high-resolution-image-synthesis-with-latent-diffusion-models/

Notes for “High-Resolution Image Synthesis with Latent Diffusion Models”

My notes for the “High-Resolution Image Synthesis with Latent Diffusion Models” paper. Feel free to ask questions on my telegram channel

]]> stable diffusion paper https://maitbayev.github.io/posts/high-resolution-image-synthesis-with-latent-diffusion-models/ Sun, 21 Jul 2024 23:00:00 GMT Auto-Encoding Variational Bayes Notes Madiyar Aitbayev https://maitbayev.github.io/posts/auto-encoding-variational-bayes/

Notes for “Auto-Encoding Variational Bayes”

My notes for the “Auto-Encoding Variational Bayes” paper. Feel free to ask questions on my telegram channel

Method

Problem scenario

Let us consider some independently and identically distributed (i.i.d) dataset . We assume that the data involves an unobserved random variable . Then, the process consists of two steps:

  1. is generated from some prior distribution

  2. is generated from some conditional distribution

Unfortunately, the true parameters and the latent variables are unknown to us. Additionally, is intractable, so we approximate it with . The model is a probabilistic encoder model parameterized by . Similarly, is a probabilistic decoder model parameterized by .

The variational lower bound

Ideally, we would like to optimize the marginal likelihoods of the dataset X:

Where each term can be rewritten as:

We used the chain rule of probability:

The first term is the KL-divergence between the approximate and the true posterior. Since the KL-divergence is non-negative, the second term is called (variational) lower bound. Ideally, we would like to minimize the both terms. However, it is enough to optimize the lower bound w.r.t both parameters θand φ. Minimizing the lower bound will minimize the KL-divergence as well, since they sum up to a constant value.

The variational lower bound, also called as the evidence lower bound (ELBO) can be also rewritten as:

The reparameterization trick

The sampling from is a stochastic process which is not differentiable w.r.t. . We can use an alternative method for generating sample from , i.e., the reparameterization trick. We can often express the random variable z as a deterministic variable , where is an independent variable and is a function parameterized by .

The is commonly chosen to model a multivariate Gaussian with diagonal covariance, and the prior often is a standard Gaussian distribution:

where and we can choose

Therefore, by the reparameterization trick, sampling from an arbitrary Gaussian distribution can be performed by sampling from a standard Gaussian, scaling and shifting the result by the target mean and the deviation, which is deterministic and differentiable.

Variational Auto-Encoder

We use a neural network for the probabilistic encoder and where the parameters and are optimized jointly. We also assume that:

  • - the prior over the latent variables is a standard Gaussian

  • is a multivariate Gaussian (in case of real-valued data) or Bernoulli (in case of binary data)

  • is approximately Gaussian with an approximately diagonal covariance:

We use the reparameterization trick to sample from the posterior using . In this model both and are Gaussian; hence we compute the KL divergence in a closed form without estimation:

In the above equation , only the reconstruction error requires estimation by sampling, since the KL-divergence term is integrated analytically.

NoteProof
]]> diffusion paper autoencoder https://maitbayev.github.io/posts/auto-encoding-variational-bayes/ Sat, 20 Jul 2024 23:00:00 GMT Denoising Diffusion Probabilistic Models Madiyar Aitbayev https://maitbayev.github.io/posts/denoising-diffusion-probabilistic-models/

Notes for “Denoising Diffusion Probabilistic Models”

My notes for the “Denoising Diffusion Probabilistic Models” paper. Feel free to ask questions on my telegram channel

Background

Forward Process

We have latent variables of the same dimensionality as the image . The forward process or diffusion process is fixed to a Markov chain that gradually adds Gaussian noise to the image according to a variance schedule :

The above equation means that we start with our data , then we sample a noisy version of our image with the mean scaled by and the variance . Finally, we repeat this process times and arrive at pure standard Gaussian noise when small enough and large enough values are used. This process is fixed and not learned, unlike variational auto-encoder (VAE) models.

A nice benefit from the forward process is that we can sample at any timestep in closed form without simulating the forward process steps times. Let and :

Suppose we have access to random noise variables :


Here we used the fact that the sum of two independent Gaussian random variables remains a Gaussian with mean being sum of two means, and variance being the sum of two variances.

We can view the forward process as transitioning from our real-life complex distribution (image) to a simpler standard Gaussian distribution.

Reverse Process

The reverse process is a learned Gaussian transition starting at a pure standard Gaussian noise , then:

The above means that we start with a pure noise , then we gradually remove noises by sampling from learned means and variances parameterized by .

We can view the reverse process as transitioning from a simple standard Gaussian distribution to our real-life complex distribution inferred from the dataset. All possible images are reachable from the final standard Gaussian noise; however, fewer and fewer images will be reachable as we go in reverse. See the images below for reachable images from , , , ,

Loss

Training is performed by optimizing the variational lower bound (or the evidence lower bound) similar to the VAE:

Proof 1: Using Jensen’s Inequality

Proof 2: Using KL Divergence

The KL divergence is always non-negative, hence the lower bound approximation

The forward process variances are hyperparameters. The reverse process is Gaussian, when the forward process is Gaussian and are small.

Efficient training is possible by optimizing random terms of the loss, however it requires two random variables for each term, and , which introduces high variance Monte Carlo estimates. We can reduce the variance:

The equation above is tractable when conditioned on :

where:

and

All KL divergences in the loss are comparisons between Gaussians, which can be calculated in closed form instead of Monte Carlo estimates.

Diffusion Models and Denoising Autoencoders

Reverse Process and

The forward process variances are constants (hyperparameters), thus is a constant during training and can be ignored.

We now discuss . First, we set to untrained constant. Experimentally, and had similar results.

Second, the mean is parameterized by the following analysis of , where :

where C is a constant that does not depend on . The variance terms disappeared since they are the same between the forward and reverse processes. Hence, the most straightforward parameterization of is a model that predicts .

However, we can expand the equation above further by reparameterizing

The above equation reveals that must predict given . We may choose the following parameterization:

Where is a function approximator to predict from . We choose the above parameterization because it is in the same form as .

To sample is to compute , where . Furthermore, the loss simplifies to:

To summarize, we can train the reverse process mean function approximator to predict , or we can train to predict .

Simplified Training Objective

It is simpler to implement to train on the following variant of the lower variational bound, which is a weighted variational bound:

This also leads to better sample quality, since it down-weights loss terms corresponding to small t so that the network can focus on more difficult denoising tasks at larger t terms.

]]> diffusion paper https://maitbayev.github.io/posts/denoising-diffusion-probabilistic-models/ Sun, 14 Jul 2024 23:00:00 GMT