A data understanding game
Every dataset
keeps a secret.
Thousands of points, drifting without meaning. Before any model, any prediction, any AI, someone has to understand them.
Scroll to decode
Phase I · Noise
At first, data is just noise: values without context, numbers without names.
Phase II · Structure
Look closer. Each point is an object: a customer, a passenger, a flower. Each object is described by a set of attributes. Rows and columns: the table appears.
Phase III · Pattern
Place the objects in the space of their attributes and they start to cluster. 150 iris flowers, three species, measured in centimetres. Drag to look around.
Phase IV · Understanding
One variable, then two, then many. That is the path you are about to walk. The type of data determines the methods, but data quality determines performance.
Begin the journey ↓0.1 · Prologue
Business & data understanding
Business understanding
Gain an in-depth knowledge about the domain and the business problem or opportunity.
- Involves interviews with the domain experts.
- Requires the ability to quickly adapt to the context and the domain-specific vocabulary.
- The most effective approach to carry out interviews is to verify the claims made by domain experts against the data.
Data understanding
Gain an in-depth knowledge about the data:
- Assessing the available data.
- Assessing data quality.
- Carrying out a manual analysis to get initial answers or to form hypotheses about the problem.
- Verifying the claims made by the domain experts and developing further questions to ask them.
The type of data determines the methods, but data quality determines performance.
0.2
Types of data
Different types of data imply different expressiveness and different techniques to be adopted to analyze them. Hover or tap a type.
0.3
Structured data
Structured data are composed of collections of objects described by a set of attributes.
- Synonyms for object: record, point, case, example, entity, instance, element.
- An attribute is a property or characteristic of an object. Synonyms: variable, field, characteristic.
Objects are rows, attributes are columns
0.4
Attribute types
An employee is described by an ID and an age, but it makes no sense to calculate the average employee ID!
In order to perform meaningful analysis the characteristics of the attributes must be known. The attribute type tells us what properties of the attribute are reflected in the value we use as a measure. A simple way to characterize them is based on the type of operator it makes sense to apply to the values it takes:
- Diversity =,\ \neq
- Ordering <,\ \leq,\ >,\ \geq
- Additivity +,\ -
- Multiplicativeness *,\ /
This determines 4 types of attributes: nominal, ordinal, interval and ratio. Each level inherits the operators of the previous one.
Pick a type: which operators unlock?
| Type | Description | Example | Statistical operator | |
|---|---|---|---|---|
| Categorical (qualitative) |
Nominal | Different names of values. We can only distinguish one from the others. | Gender, eye color, zip code, ID | Mode, correlation |
| Ordinal | Values enable us to sort objects based on the value of the attribute, but the distance between consecutive values is not measurable or necessarily equal. | Grade, hardness of a mineral | Median, percentile | |
| Numerical (quantitative) |
Interval | The difference between the values has a meaning, i.e., there is a unit of measurement (and equal numerical intervals). | Dates, temperature in Celsius and Fahrenheit | Mean, variance |
| Ratio | The ratio of values has meaning. | Age, mass, length, amount of money, temperature expressed in Kelvin | Geometric mean, harmonic mean |
Interval or ratio? Where is zero
Temperature in Celsius (−10 °C → 0 °C → 10 °C → 20 °C → 30 °C): the difference makes sense (30 °C − 10 °C = 20 °C), but the ratio doesn't: 20 °C is not “the double” of 10 °C, because 0 °C doesn't mean “absence of heat”. How would you describe −10 °C with respect to 20 °C in terms of ratio?
Weight in kg (0 kg → 10 kg → 20 kg → 30 kg): the difference makes sense (30 kg − 10 kg = 20 kg) and the ratio makes sense too: 20 kg is twice 10 kg, since 0 kg means no weight.
Is 20 °C twice as hot as 10 °C?
0.5 · Mini game
Sort the attributes
An attribute appears: decide its type. Three lives, one point per right answer. The explanation shows up after each guess.
0.6
Alternative classifications
Binary, discrete and continuous
- A discrete attribute has a finite number or an infinite countable set of values, normally represented by integers or labels.
- A continuous attribute takes real values.
- Nominal and ordinal attributes are typically discrete or binary, while interval and ratio attributes are continuous.
Countably infinite
An infinite set is said to be countably infinite if its elements can be enumerated by the natural numbers; equivalently, there exists a bijective mapping between the set and \mathbb{N}. Examples:
- Positive integers, integers
- Even / odd numbers
- The Cartesian product of two countably infinite sets
- Rational numbers
- All finite strings over a finite alphabet
Enumerating the integers with ℕ
0.7
Asymmetric attributes
For asymmetric attributes only the presence of a value (non-zero) is relevant; sharing a 0 (a common absence) does not make two objects similar.
- Study plan: a binary vector with one attribute for each exam offered by the University (1 = chosen, 0 = not chosen).
- The University offers m = 1{,}000 exams while each student chooses about 20 of them: more than 98% of the values are 0. Compared on all attribute values, every student would look very similar.
- The similarity between study plans must be based on the common exams only. f_{ab} = number of exams with value a in the first plan and b in the second one.
Counts both 1–1 and 0–0 matches.
Ignores 0–0 matches.
Anna and Marco: almost identical, or very different?
0.8
Beyond the plain table
Documents & texts
Documents are the objects of the analysis; they are described by a vector of terms. Each term is an attribute of the document, and its value is the number of times the term appears in the document.
Transactions
A record where:
- each record (transaction) involves multiple items;
- items come from a finite set;
- for example, in a supermarket, the set of products purchased by a customer during a visit constitutes a transaction, while the set of items is the set of all the products sold by the supermarket;
- the number of items may vary from transaction to transaction.
Pick an item: which baskets contain it?
Ordered data and time series
Both involve an order, but there's a fundamental difference related to the meaning of the order and time.
Ordered data
Data organized according to a specific order, but the order does not necessarily represent time. You can compare positions or rank data, but there is no implied time interval.
- Student rankings by score (1st, 2nd, 3rd)
- Clothing sizes (S, M, L, XL)
- Satisfaction levels (1–5 stars)
Time series data
Data collected in temporal sequence, where each value has a precise timestamp, often with regular or variable frequency. Time is intrinsic; patterns, trends and correlations depend on the temporal sequence.
- Daily temperature
- Stock prices minute by minute
- Hourly energy consumption
- Market research and user experience. In surveys like “rank your preferences”, users are asked to order products, services or features from most to least important.
- Psychology and cognitive sciences. Preference tests or cognitive games in which participants must rank stimuli (images, words, scenarios) according to liking, emotional response or perceived difficulty. The ranking becomes research data to understand perceptions and mental patterns.
- Medicine and health. In some clinical studies, patients rank symptoms or side effects by severity or impact on quality of life. The ranking helps researchers assess subjective priorities without requiring precise numerical scales.
- Recommendation systems. In Netflix, Spotify or Amazon, users can rank or prioritize movies, songs or products they prefer. The algorithm uses this information to personalize suggestions, even without explicit numerical ratings.
Ordering becomes an indirect method of data collection, because it allows subjective perceptions, preferences or relative priorities to be transformed into structured information without requiring absolute numbers.
Same values, two kinds of order
Spatio-temporal data
In spatio-temporal data, values are related to the “neighbors” data both in space and in time: a cell resembles the cells around it, and itself one month earlier.
Surface temperature through the year (synthetic)
0.9 · Your inventory
The datasets you will explore
Every lab in the next chapters runs on one of these. The real datasets ship with this site; the synthetic ones are generated in your browser with a fixed seed.
Bank data
bank-data.csv: 600 instances, no missing data. A
Personal Equity Plan (PEP) was a tax-advantaged investment account
promoted by the United Kingdom Government to (1) encourage household saving, (2)
promote long-term investment, (3) support capital markets.
The first rows of bank-data.csv
The Titanic dataset
The sinking of the Titanic is one of the most infamous shipwrecks in history. On April 15, 1912, during her maiden voyage, the widely considered “unsinkable” RMS Titanic sank after colliding with an iceberg. Unfortunately, there weren't enough lifeboats for everyone on board, resulting in the death of 1502 out of 2224 passengers and crew. While there was some element of luck involved in surviving, it seems some groups of people were more likely to survive than others. The Kaggle training set has 891 passengers.
Choose a level
Three levels, one dataset at a time
- 01 Univariate One variable at a time: center, spread, percentiles, skewness, kurtosis. Enter level →
- 02 Bivariate Two variables: Pearson, Spearman, spurious correlation, mutual information. Enter level →
- 03 Multivariate Many variables: curse of dimensionality, PCA in 3D, t-SNE, the UK diet, politics. Enter level →