14: Dimensionality Reduction (PCA)

Previous Next Index

Motivation 1: Data compression

Motivation 2: Visualization

Principle Component Analysis (PCA): Problem Formulation
PCA Algorithm
Algorithm description
Reconstruction from Compressed Representation
  • Earlier spoke about PCA as a compression algorithm
    • If this is the case, is there a way to decompress the data from low dimensionality back to a higher dimensionality format?
  • Reconstruction
    • Say we have an example as follows
    • We have our examples (x1x2 etc.)
    • Project onto z-surface
    • Given a point z1, how can we go back to the 2D space?
  • Considering 
    • z (vector) = (Ureduce)T * x
  • To go in the opposite direction we must do
    • xapprox = Ureduce * z
      • To consider dimensions (and prove this really works)
        • Ureduce = [n x k]
        • z [k * 1]
      • So
        • xapprox = [n x 1]
  • So this creates the following representation
  • We lose some of the information (i.e. everything is now perfectly on that line) but it is now projected into 2D space

Choosing the number of Principle Components
Advice for Applying PCA
Speeding up supervised learning algorithms

Applications of PCA