How I Chose Between Factor Analysis and Principal Component Analysis

7 min readSep 26, 2023

Key Takeaways

Factor analysis and principal component analysis are two techniques for dimensionality reduction that try to find a smaller set of variables that can explain the variation and correlation among large variables.
Factor analysis is based on a causal model that assumes that there are latent factors that influence the observed variables, while PCA is based on a mathematical transformation that does not assume any causal model or latent variables.
Factor analysis requires more assumptions and decisions than PCA, such as choosing between EFA and CFA, deciding how many factors to extract, choosing a method for estimating the factor loadings, and choosing a method for rotating the factors.
PCA is more straightforward and objective than factor analysis, as it only requires deciding how many components to retain and interpreting the meaning of the components based on their loadings and correlations with the original variables
Factor analysis can be useful for identifying the latent constructs or dimensions that underlie a set of variables, while PCA can be useful for finding the best way to represent the data in a lower-dimensional space.
The choice between factor analysis and PCA depends on the problem objective and the nature of the data. You should use factor analysis to find the latent factors or constructs that underlie your data and if you have prior knowledge or hypotheses about them. You should use PCA if you are interested in finding the best way to represent your data in a lower-dimensional space and do not have any prior knowledge or hypothesis about the latent factors or constructs.

While working on a data analysis project, I faced a common challenge: reducing the dimensionality of my data set without losing too much information. I had many variables correlated with each other, and I wanted to find a smaller set of variables that could capture the essence of my data. I knew there were two popular techniques for dimensionality reduction: factor analysis and principal component analysis (PCA).

But I was not sure which one to use and what were the differences between them. So, I decided to research and learn more about these two methods.

In this article, I will share what I learned and how I chose the best technique for my project.

What is Factor Analysis?

Factor analysis is a statistical method that explains the variation and correlation among a large set of observed variables in terms of a smaller number of unobserved latent variables called factors. The factors are assumed to be the underlying causes that influence the observed variables; read more about Factor Analysis.

Factor analysis can be divided into exploratory factor analysis (EFA) and confirmatory factor analysis (CFA). EFA is used when we do not have any prior knowledge or hypothesis about the factors and their relationships with the observed variables.

Exploratory factor analysis

EFA tries to discover the factors and their loadings (the coefficients that indicate how much each factor contributes to each observed variable) from the data.

Confirmatory factor analysis

CFA is used when we have some prior knowledge or hypothesis about the factors and their relationships with the observed variables. CFA tests whether the data fits the hypothesized model of factors and loadings.

Factor analysis can be useful.

Identifying the latent constructs or dimensions that underlie a set of variables
Reducing the number of variables for further analysis
Testing hypotheses about the structure and meaning of the data
Developing scales or instruments for measuring latent variables

What is Principal Component Analysis (PCA)?

Principal component analysis (PCA) is a technique for transforming a large set of correlated variables into a smaller set of uncorrelated variables called principal components. The principal components are linear combinations of the original variables that capture the maximum variation in the data.

PCA can be seen as a particular case of factor analysis, where the factors are orthogonal (uncorrelated) and account for all the variation in the data. Unlike factor analysis, PCA does not assume any underlying causal model or latent variables. PCA tries to find the best way to represent the data in a lower-dimensional space.

PCA can be useful

Reducing the dimensionality of the data
Simplifying the data for visualization or interpretation
Enhancing the signal-to-noise ratio in the data
Performing feature extraction or selection for machine learning algorithms

Difference Between Factor Analysis and Principal Component Analysis

The main difference between factor analysis and principal component analysis is that factor analysis is based on a causal model that assumes that there are latent factors that influence the observed variables, while PCA is based on a mathematical transformation that does not assume any causal model or latent variables.

As you can see, PCA tries to find orthogonal (perpendicular) components and explain as much variation as possible in the data. Factor analysis tries to find correlated factors and explain as much common variation as possible in the data while allowing for some unique variation and error terms.

Assumptions and Decisions

Another difference between factor analysis and principal component analysis is that factor analysis requires more assumptions and decisions than PCA. For example, in factor analysis, we need to:

Choose between EFA and CFA depending on our research question and prior knowledge
Decide how many factors to extract based on various criteria such as eigenvalues, scree plot, or parallel analysis
Choose a method for estimating the factor loadings, such as maximum likelihood, principal axis factoring, or generalized least squares
Choose a method for rotating the factors, such as varimax, quartimax, or oblimin
Interpret the meaning of the factors based on their loadings and theoretical relevance

In contrast, PCA is more straightforward and objective, as it only requires us to:

Decide how many components to retain based on the proportion of variance explained or other criteria
Interpret the meaning of the components based on their loadings and correlations with the original variables

Use Cases and Applications of Factor Analysis

It can be applied in various fields and domains where we want to understand our data’s underlying structure and meaning. Some examples are:

Psychology

It can be used to develop and validate psychological tests and scales that measure latent traits such as personality, intelligence, or attitudes. For example, the Big Five personality test is based on a factor analysis of various personality traits.

Marketing

It can be used to identify the key factors that influence consumer behaviour and preferences. For example, an analysis of customer satisfaction surveys can reveal the main dimensions of customer satisfaction and loyalty.

Education

It can evaluate the quality and validity of educational tests and assessments. For example, a student’s test scores can show the extent to which the test measures different skills and abilities.

Sociology

It can be used to explore the social and cultural factors that affect human behaviour and society. For example, demographic data can reveal a population’s major social groups and trends.

Use Cases and Applications of Principal Component Analysis

Principal component analysis can be applied in various fields and domains where we want to reduce the complexity and dimensionality of our data. Some examples are:

Image Processing

PCA can compress and enhance images by reducing the number of pixels or colours without losing much information. For example, PCA can be used to perform face recognition by extracting the main features of a face image.

Data Mining

PCA can be used to preprocess and transform data for machine learning algorithms by reducing the number of features or variables without losing much information. For example, PCA can detect anomalies by finding outliers in a data set.

Bioinformatics

PCA can analyze and visualize high-dimensional biological data such as gene expression or protein structure by reducing the number of dimensions without losing much information. For example, PCA can perform cluster analysis by finding groups of similar genes or proteins.

Finance

PCA can be used to analyze and model financial data such as stock prices or exchange rates by reducing the number of variables without losing much information. For example, PCA can optimise a portfolio by finding the optimal combination of assets that minimize risk and maximize return.

Which Method to Choose Based on the Problem Objective?

The choice between factor analysis and principal component analysis depends on the problem objective and the nature of the data. Here are some general guidelines that can help you decide which method to use:

Use factor analysis if you are interested in finding the latent factors or constructs that underlie your data and you have some prior knowledge or hypothesis about them. It is more suitable for exploratory or confirmatory research questions that involve causal inference or theory testing.
Use principal component analysis if you are interested in finding the best way to represent your data in a lower-dimensional space and have no prior knowledge or hypothesis about the latent factors or constructs. PCA is more suitable for descriptive or predictive research questions that involve data transformation or dimensionality reduction.

Of course, these are not strict rules, and there may be situations where both methods can be applied or compared. For example, you may use both factor analysis and principal component analysis to see how consistent they are in finding the underlying structure of your data, or you may want to use factor analysis as a preprocessing step before applying principal component analysis.

Conclusion

In this article, I have explained what factor and principal component analysis are, how they differ, and their use cases and applications. I have also shared with you how I chose these two methods for my data analysis project. I hope you have learned something new from this article and found it helpful for your data analysis projects.

Contact us at info@data03.online or visit our website at [ data03.online] if you need any help with your data analysis projects. We offer professional and affordable data analysis services using RStudio. Whether you need data cleaning, visualization, modelling, or reporting, we can help you achieve your goals.

We hope to hear from you soon. Thank you for choosing us as your data analysis partner. Happy data analyzing!

Thank you for reading this article. If you liked it, please share it with your friends and colleagues. And if you want to learn more about data analysis with RStudio, please subscribe to our newsletter or follow us on social media.

Originally published at https://www.data03.online.