Skip to content
Plotwise

Level 03

Multi­variate.

Many variables at once. Humans cannot see beyond three dimensions, so we learn to flatten the space without losing what matters: rotate it, project it, and let points find their neighbours.

3.1

Multivariate analysis

Humans cannot interpret high-dimensional spaces.

  • Hard to visualize beyond 3 dimensions.
  • Difficult to understand relationships.

Dimensionality reduction enables 2D/3D visualization.

Principal Component Analysis (PCA)

  • Linear transformation
  • Maximizes global variance
  • Keeps overall data structure
  • Distances are meaningful at a global scale

t-distributed Stochastic Neighbor Embedding (t-SNE)

  • Nonlinear transformation
  • Preserves local neighborhoods
  • Points close in high dimensions stay close in low dimensions
  • Global distances are not reliable

3.2

Dimensionality reduction: beyond data understanding

As the number of variables grows, data becomes sparse and distances lose meaning: this is known as the curse of dimensionality.

  • Points tend to look equally distant from each other.
  • Neighborhood-based methods (e.g., clustering) degrade.
  • Models require exponentially more data.
Try it

Add dimensions, watch distances collapse

In real datasets, many variables are correlated or redundant. Techniques like PCA transform correlated variables into uncorrelated components, preserving most of the information.

  • Example: income, spending score and credit limit often overlap.
  • Keeping all variables adds noise without adding information.

High-dimensional data can cause:

  • Overfitting (the model learns noise).
  • Instability in parameter estimation.

Reducing dimensions restores structure and improves learning. These techniques come back again during the feature engineering step.

3.3

Principal Component Analysis

PCA is a linear dimensionality reduction technique grounded in linear algebra and statistics. Its goal is to transform a dataset into a new coordinate system where the axes (principal components) capture the maximum variance in decreasing order, while being orthogonal (uncorrelated).

PCA is based on the following assumptions:

  • Information–variance assumption: high-variance directions contain the most relevant information. Variance measures how much the data spreads along a direction.
    • High variance → data points are well spread → rich structure
    • Low variance → data points are compressed → little information
  • Linearity assumption: the relevant data structure can be captured through a global linear transformation and represented by a lower-dimensional linear subspace.

So when PCA maximizes variance, it is effectively finding directions that preserve as much information as possible.

Explore in 3D

Shape a cloud, find its principal components, flatten it

The projected 2D points y = Pᵀx

1D views original axes vs principal components

3.4

Linear projection

Let \bx be a 3D point:

\bx = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}

Given two linearly independent vectors (i.e., different directions) in 3D defining the projection plane:

\mathbf{p}_1 = (p_{1,1}, p_{1,2}, p_{1,3}) \qquad \mathbf{p}_2 = (p_{2,1}, p_{2,2}, p_{2,3})
The projected 2D point
\by = P^T\bx \quad \text{where} \quad P = \begin{bmatrix} p_{1,1} & p_{2,1} \\ p_{1,2} & p_{2,2} \\ p_{1,3} & p_{2,3} \end{bmatrix}
Carrying out the matrix multiplication
\by = \begin{bmatrix} y_1 \\ y_2 \end{bmatrix} = \begin{bmatrix} p_{1,1}x_1 + p_{1,2}x_2 + p_{1,3}x_3 \\ p_{2,1}x_1 + p_{2,2}x_2 + p_{2,3}x_3 \end{bmatrix}

Each 2D coordinate is a linear combination of the original 3D coordinates.

Challenge: beat PCA Scroll back to the 3D lab, switch the plane to Your plane and tilt it with the sliders. The live P matrix updates as you move. How close to the PCA variance can you get? (You can tie it. You can never beat it.)

3.5

Mathematical formulation for PCA

Which is the projection from the original dataset space \mathbb{R}^n to \mathbb{R}^k, with k < n, that summarizes the data at best? The data distribution is properly described by the covariance matrix in \mathbb{R}^n:

Covariance matrix
\Sigma = \frac{1}{n}\sum (\bx - \mu)(\bx - \mu)^T
Each entry is a covariance
Cov(\bx, \by) = \frac{1}{n}\sum_{k=1}^{n}(x_k - \xb)(y_k - \yb)

We need to find the \mathbb{R}^k space where the projected variance (i.e., the data distribution) is maximized. This can be solved as a constrained optimization problem that maximizes the variance of the projected data, called maximum variance projection or Rayleigh quotient maximization:

\max_{\mathbf{v}} \; \frac{\mathbf{v}^T\Sigma\,\mathbf{v}}{\mathbf{v}^T\mathbf{v}} \quad\Longrightarrow\quad \Sigma\,V_i = \lambda_i V_i

The solution is given by the transformation V determined by the eigenvectors of the covariance matrix V = [V_1, \dots, V_n].

  • Eigenvectors V are directions that remain invariant (up to sign) under a linear transformation.
  • In PCA, eigenvectors of the covariance matrix define a new orthogonal coordinate system.
  • The factor by which they are scaled (due to the data distribution) is called the eigenvalue \boldsymbol\lambda = [\lambda_1, \lambda_2, \dots, \lambda_n].

Each eigenvalue \lambda_i tells you how much variance lies along its eigenvector, i.e., how stretched the cloud is along each axis: large eigenvalue → important direction; small eigenvalue → less important.

Try it

The matrix that stays a direction

The data distribution is exactly preserved under the eigenvector transformation (just expressed in a new coordinate system). The new coordinate system does not apply any dimensionality reduction: data are still in \mathbb{R}^n but the axes changed. We carried out a change of basis in \mathbb{R}^n.

To carry out dimensionality reduction we:

  1. Order the eigenvectors V_i in descending order of the corresponding eigenvalues \boldsymbol\lambda = (\lambda_1, \lambda_2, \dots, \lambda_n) with \lambda_1 > \lambda_2 > \dots > \lambda_n.
  2. Pick the first k eigenvectors that explain enough variance.
  3. Given the percentage t of the informative content to be preserved (e.g., t = 95\%), select the minimum number of eigenvectors k such that the sum of the largest eigenvalues, with respect to the sum of all the eigenvalues, is higher than or equal to t:
k = \operatorname{argmin}_z \left( \frac{\sum_{i=1}^{z}\lambda_i}{\sum_{i=1}^{n}\lambda_i} \geq t \right)
Try it

How many components for t% of the variance?

Scree plot λᵢ and cumulative share

Covariance matrix Σ (correlations if standardized)

3.6

Projecting and retro-projecting the data

Projecting on the first k eigenvectors
\bz_k = V_k^T \bx
  • Data are represented in the \mathbb{R}^k data space with k < n.
  • Only the main components of the dataset are preserved.
Retro-projecting to the original space
\tilde{\bx} = V_k \bz_k
  • Data are represented in the original \mathbb{R}^n data space.
  • The original and the retro-projected points are different (\tilde{\bx} \neq \bx) since “details” are lost in projection.
Two distinct points \bx, \by \in \mathbb{R}^n that have the same first k components will be projected to the same point \bz \in \mathbb{R}^k and retro-projected to the same point \tilde{\bx} = \tilde{\by} \in \mathbb{R}^n. In the 3D lab, drag the projection slider to 100% with drop lines on: every line ending at the same spot on the plane is such a pair.
Try it

Rebuild the Iris flowers from k components

3.7

Reading the components: weights and biplot

Weights: each principal component is a linear combination of the original dimensions:

z_j = V_j^T\bx = v_{1j}x_1 + v_{2j}x_2 + \dots + v_{nj}x_n
  • v_{ij}, the i-th element of the eigenvector V_j, is the weight of the original dimension x_i in the component z_j.
  • Sign: + → x_i and z_j grow together; − → they move in opposite directions.
  • Magnitude: since \|V_j\| = 1, then \sum_i v_{ij}^2 = 1, so v_{ij}^2 is the share of z_j due to x_i.
  • Loading: v_{ij}\cdot\sqrt{\lambda_j} is the correlation between x_i and z_j (standardized data).
Standardize the data first Otherwise the weights reflect the units of measure and the dimension with the largest variance dominates the first component. Iris dataset, 4 standardized dimensions: PC1 ≈ flower size (petal length, petal width and sepal length have similar positive weights); PC2 ≈ sepal width, the only dimension with a large weight (0.92).
Try it

Iris: weights of the first two components

The biplot

A scatter plot of the reduced space \mathbb{R}^k that shows together:

  • the points, i.e., the projected data \bz = V_k^T\bx;
  • the original dimensions, one arrow for each x_i.

Where do the arrows come from? The unit vector \mathbf{e}_i of the original axis x_i is projected as V_k^T\mathbf{e}_i = (v_{i1}, \dots, v_{ik}): the i-th row of V_k, i.e., the weights of x_i in the k components (usually drawn as loadings, i.e., correlations).

How to read it

  • Direction: where x_i grows; points along the arrow have high values of x_i, points on the opposite side low values.
  • Length: how well x_i is represented in \mathbb{R}^k (length² = share of its variance explained); a short arrow → x_i lies mostly in the discarded components.
  • Angle between two arrows: its cosine ≈ correlation (0° positive, 90° uncorrelated, 180° negative).
  • Iris: petal length and width overlap (corr. 0.96) and point to virginica; sepal width forms an angle of ≈116° with them: cos = −0.43 = their correlation.
Try it

Iris biplot: tap two arrows to measure their angle

3.8

PCA at work, and when it fails

  1. Set 4 centroids in a 5-dimension dataspace.
  2. Generate a 400-point dataset.
  3. Apply PCA to 2D.
Centroid Note
[0, 0, 0, 0, 0] blob 1
[1, 1, 1, 1, 1] blob 2 (close to blob 1)
[10, 10, 10, −10, −10] blob 3 (far away)
[−10, −10, −10, 10, 10] blob 4 (far away)

When does PCA fail? PCA approximates the data through a linear subspace identified from the covariance structure. It is particularly effective when the data form an approximately elongated, ellipsoidal cloud around that subspace. In many cases this is not true:

  • Principal components summarize variance and linear dependence, but may fail to preserve nonlinear or local characteristics of the data distribution.
  • Banana-, circular- and spiral-shaped datasets.
  • Adding 2 high-variance random features to the original dataset: PCA spends its components on the noise.

In all these cases PCA loses relevant information and may yield misleading results. A nonlinear transformation is necessary.

Try it

Four blobs in 5D, seen in 2D

Try it

Banana, circles, spiral: one dimension to tell them apart

Original 2D data

Reduced to 1D PCA vs t-SNE

3.9

t-Distributed Stochastic Neighbor Embedding

t-SNE is a nonlinear dimensionality reduction method that preserves local neighborhood structure.

  • Mainly used to visualize high-dimensional data in 2D, 3D.
  • Reveals clusters and local neighborhoods: local structure is preserved, not global geometry.
  • Handles non-linear structure.
  • t-SNE is not linear and does not preserve global geometry because it does not apply a single global transformation (like a matrix multiplication); instead, it iteratively rearranges points to match pairwise similarities using a nonlinear objective.

It defines probability distributions over pairs of points in both the high- and the low-dimensional space, and minimizes their mismatch by letting the system learn Q from P.

How does t-SNE work?

1 · Convert high-dimensional distances into probabilities P

For each of the N points \bx_i, define a probability distribution over all other points using a Gaussian kernel:

p_{j|i} = \frac{\exp\left(-\|\bx_i - \bx_j\|^2 / 2\sigma_i^2\right)}{\sum_{k\neq i}\exp\left(-\|\bx_i - \bx_k\|^2 / 2\sigma_i^2\right)}

Then symmetrize P. Nearby points get high probability; distant points get near-zero.

p_{ij} = \frac{p_{j|i} + p_{i|j}}{2N}
2 · Define a similar distribution Q in low dimensions

Using a t-Student kernel:

q_{ij} = \frac{\left(1 + \|\by_i - \by_j\|^2\right)^{-1}}{\sum_{k\neq l}\left(1 + \|\by_k - \by_l\|^2\right)^{-1}}
3 · Learn Q by minimizing the Kullback–Leibler divergence
KL(P\,\|\,Q) = \sum_{i\neq j} p_{ij}\log\frac{p_{ij}}{q_{ij}}

KL is minimized when p_{ij} \approx q_{ij}.

Perplexity

\sigma_i^2 varies for each point \bx_i depending on the perplexity parameter, which smoothly defines the “effective number of neighbors” each point considers.

  • Low perplexity (e.g., 5–10): emphasizes very local structure and can produce fragmented clusters.
  • Medium perplexity (e.g., 20–30): balanced view of local clusters.
  • High perplexity (e.g., 40–100): captures broader/global structure but can merge small clusters.

Kernel

A kernel is a rule that turns the distance between two points into a similarity value: close points get high similarity, distant points get low similarity.

Live

Watch Q learn P: t-SNE vs PCA on the 5D blobs

PCA

t-SNE

When does t-SNE fail?

  • When you need a reusable mapping (out-of-sample data): t-SNE does not learn a function f(x). You can't take a new point and map it into the existing embedding; you'd rerun t-SNE on the whole dataset.
  • When global structure matters: t-SNE preserves local neighborhoods, not global geometry. Distances between clusters are often meaningless; cluster sizes may be distorted.
  • When you want quantitative analysis (not just visualization): the axes have no meaning; distances are not reliable metrics.
  • When results must be stable and reproducible: t-SNE depends on random initialization and is sensitive to hyperparameters (especially perplexity). Change the seed in the lab above and see.

3.10

Eating in the UK

What if our data have way more than 3 dimensions? Like, 17 dimensions?! The table holds the average consumption of 17 types of food in grams per person per week for every country in the UK.

Try it

Four countries, 17 foods, 2 components

PCA biplot

PC1 / PC2 loadings

The Northern Irish eat way more grams of fresh potatoes and way fewer of fresh fruits, cheese, fish and alcoholic drinks. That single fact is what separates Northern Ireland from the other three along PC1.

3.11 · Boss level

The political compass and the spatial modeling of politics

The Political Compass is a model used to represent political ideology in a two-dimensional space, rather than the traditional single left–right line. It is based on 62 propositions across economic and social axes:

  • Economic axis (left–right)
  • Social axis (authoritarian–libertarian)

The Political Compass goes beyond a dimensionality reduction problem, since it forces political questions into the two axes. Let's instead create a spatial model of political positions:

  1. Define the questionnaire (10 dimensions). These questions represent the 10-dimensional feature space. Questions define the dataspace: if you change the questions everything will change. “To what extent do you agree with the following statements?”, e.g.:
    • “Lower taxes are more important than expanding social welfare programs.”
    • “Immigration should be more strictly regulated and limited.”
    • “Environmental protection should be prioritized even if it slows economic growth.”
    • “The state should play a larger role in regulating the economy than the free market.”
  2. Collect answers from citizens. Each individual is a data point in the 10-dimensional space.
  3. Define political party profiles (Progressive, Conservative, Liberal, Populist). This encodes the ideological “position” of each party in the same feature space as individuals. Parties are “human feature engineering”, not data!
  4. Apply dimensionality reduction. Visualize global structure; identify general directions of variation in the data; analyze how individuals relate to political parties in the reduced space. Latent axes must be evaluated: they do not have a predefined meaning (e.g., left vs right).
Your turn

Answer the 10 questions, find your place on the map