SVD Working

Now that you know what SVD produces, let's walk through how it's actually computed and applied for dimensionality reduction.

Step-by-Step Working

1. Start with the original matrix A (m x n)
2. Compute A.A^T to find U (its eigenvectors)
3. Compute A^T.A to find V (its eigenvectors)
4. Take the square roots of the eigenvalues of A^T.A (or A.A^T) to get the singular values, forming Sigma
5. Arrange U, Sigma and V^T so that A = U . Sigma . V^T

Truncated SVD for Dimensionality Reduction

Just like PCA, SVD can be used to reduce dimensions by keeping only the top k singular values (and their corresponding vectors) instead of all of them, giving the best possible low-rank approximation of the original matrix.

from sklearn.decomposition import TruncatedSVD

svd = TruncatedSVD(n_components=2)
X_reduced = svd.fit_transform(X)

print("Explained Variance Ratio:", svd.explained_variance_ratio_)

SVD vs PCA

PCA first centers the data and works on the covariance matrix, while SVD works directly on the data matrix itself and doesn't require centering - which is why TruncatedSVD is often preferred over PCA for sparse data, like text data represented as a TF-IDF matrix.

Coming Up Next

With dimensionality reduction covered, the course now moves into a different area of unsupervised learning used heavily in retail - Market Basket Analysis.