Contents Dimensionality reduction Clustering Dimensionality reduction What is Dimensionality Reduction? Dimensionality reduction is the process of reducing the number of features or variables (also called dimensions) in a dataset, while retaining as much important information as possible. In simple terms, it means taking a dataset with many columns (features) and transforming it into one with fewer columns, without losing the core patterns or structure of the data. Why is Dimensionality Reduction Done? 1. Simplify the Model — faster to train and easier to interpret. 2. Reduce Overfitting — Too many features can cause models to memorize the training data 3. Improve Performance — Reduces computational cost: less memory, faster training and prediction. 4. Visualization — You can’t visualize data with 100 dimensions. Principal component Analysis Principal Component Analysis (PCA) is a technique used in dimensionality reduction. It helps to simplify complex datasets by reducing the number of variables (dimensions) while retaining the most important information. PCA is especially useful when there are strong correlations among variables — in such cases, reducing dimensions helps eliminate redundancy. Goal : to reduce the dimensions and identify patterns in data Core Idea: PCA works by finding the directions (principal components) in which the data varies the most. It then projects the original data onto a new set of axes formed by these components. These principal components are uncorrelated, and each one captures as much variance as possible: PC1 captures the most variance. PC2 captures the next most (and is orthogonal to PC1), and so on. If your data is 10-dimensional, PCA will give you 10 components. But often, just the first 2–3 components explain most of the variance. Steps of PCA Standardize the data Scale each feature so that it has a mean of 0 and a standard deviation of Compute the Covariance Matrix This matrix shows how features vary with each other — i.e., how strongly they are correlated. Find Eigenvalues and Eigenvectors Eigenvectors give the directions of the new feature space (principal components). Eigenvalues indicate how much variance each component captures. Create the Feature Vector Select the top k eigenvectors corresponding to the k largest eigenvalues. Recast the Data Multiply the original data by the feature vector to project it onto the new lower-dimensional space. Principal Components These components are linear combinations of the original features They are ordered by how much variance (information) they capture. The first component captures the most variance, the second captures the next most, and so on. A scree plot helps visualize the eigenvalues of each principal component. It shows how much variance each component captures. Clustering Clustering is an unsupervised learning technique used to group similar data points together based on their characteristics — without using predefined labels. The goal is to discover the natural structure or patterns in data. Each group formed is called a cluster, and data points in the same cluster are more similar to each other than to those in other clusters. Why Use Clustering? To find hidden patterns or groupings in data. To segment customers in marketing (e.g., by behavior, demographics). To detect anomalies (outliers often don’t belong to any cluster). To compress data by summarizing with cluster centers. To preprocess data for other tasks (like classification). K-Means Clustering K-Means is a partitional clustering algorithm that: Divides the dataset into k distinct clusters and Each cluster is represented by a centroid (center point) How the K-Means Algorithm Works Choose the number of clusters (k): The value of k is set by the user. Initialize centroids: Randomly select k points to serve as initial centroids (they don’t have to be data points). Assign data points to the nearest centroid: Each point is assigned to the cluster whose centroid is closest (based on distance, usually Euclidean). Compute new centroids: For each cluster, compute the mean of all the data points in that cluster to get the new centroid. Reassign points: Reassign each data point to the nearest new centroid. If any reassignment occurs, go back to step 4. If no changes happen, the algorithm converges , and clustering is complete. Choosing the Right Number of Clusters Choosing the optimal k is crucial. One popular method is using WCSS (Within-Cluster Sum of Squares). Elbow Method (Elbow Plateau Diagram) To find the optimal k : Run K-Means for a range of k values (e.g., 1 to 10) Calculate WCSS for each k Plot k vs. WCSS Look for the “elbow point” — where the rate of decrease sharply slows down This elbow point represents a balance between: Too few clusters (high WCSS, underfitting) Too many clusters (low WCSS, but possibly overfitting or unnecessary complexity) Spectral Clustering Traditional clustering algorithms like K-Means work well for spherical or convex clusters, but fail with non-convex, complex-shaped, or intertwined data (e.g., spirals). Spectral Clustering is designed to handle such cases by using graph theory and the spectrum (eigenvalues) of a matrix derived from the data. Key Advantages Makes no assumptions about the shape of the clusters. Works well for non-linearly separable data . Can cluster intertwined or overlapping structures better than K-Means. Spectral Clustering Process Step 1: Construct a Similarity Graph Represent data as a graph where: Each data point is a node . An edge connects each point to its k-nearest neighbors (K-NN) . The similarity (or weight of the edge) could be binary (connected or not) or based on distance (e.g., using a Gaussian kernel). Example: If you have 6 data points and set k = 2, the graph connects each point to its 2 closest points. Step 2: Spectral Embedding via Laplacian Matrix Construct the adjacency matrix (A) for the graph. Compute the degree matrix (D) — a diagonal matrix where each entry is the number of connections for that node. Compute the Laplacian matrix (L) : L=D−A Perform eigendecomposition on L . Use the eigenvector corresponding to the 2nd smallest eigenvalue (called the Fiedler vector ) to bi-partition the graph. For more than 2 clusters, use multiple eigenvectors (e.g., top k eigenvectors with smallest non-zero eigenvalues). Step 3: Apply Classical Clustering (e.g., K-Means) The selected eigenvectors transform the original data into a lower-dimensional space (spectral embedding). Run K-Means on this transformed data to produce the final clusters. Why Use the 2nd Smallest Eigenvalue? The smallest eigenvalue of a graph Laplacian is always zero (for a connected graph). The second smallest eigenvalue reflects how well the graph can be partitioned into two groups . Its corresponding eigenvector (Fiedler vector) provides a natural separation of data. Hierarchial Clustering Hierarchical clustering is an unsupervised machine learning algorithm used to group unlabeled data into clusters. Unlike K-Means clustering, it builds a tree-like structure of clusters using dendrograms to represent the hierarchy among data points. Dendrograms : A dendrogram is a tree-like diagram that records the merging process of clusters. X-axis : data points or features Y-axis : Euclidean distance between clusters You can cut the dendrogram at a chosen height to determine the number of clusters. Types of Hierarchical Clustering: Agglomerative Clustering (Bottom-up) Starts with each data point as its own cluster (singleton cluster). Compute a distance matrix based on all pairs of points Iteratively merges the two closest clusters based on minimum distance. Update distance matrix after each merge. Ends when all data points are merged into a single cluster. Divisive Clustering (Top-down) Starts with one large cluster. Recursively splits it into smaller clusters. Continues until each data point is its own cluster or a stopping criterion is met. Less commonly used due to its computational complexity Silhouette Score — Evaluating Clusters The silhouette score measures how well each data point fits within its assigned cluster: Where: a(i): average distance between point i and all other points in the same cluster. b(i): minimum average distance from point i to all points in other clusters. Interpretation of Silhouette Score: +1: Well-matched to its own cluster 0: On the boundary between two clusters –1: Poorly clustered Unsupervised Learning was originally published in GoPenAI on Medium, where people are continuing the conversation by highlighting and responding to this story.