On This Page:ToggleGoalsTechniquesMatching Techniques to Data and ObjectivesDescriptive vs. Explorative Analysis
On This Page:Toggle
On This Page:
Exploratory Data Analysis (EDA) is an approach to analyzing data that emphasizes exploring datasets for patterns and insights without any predetermined hypotheses.
The goal is to let the data “speak for themselves” and guide analysis, rather than imposing rigid structures or theories.
Goals
Exploratory data analysis (EDA) mainly analyzes and investigates datasets to discover patterns, spot anomalies, test hypotheses, and check assumptions.
EDA has several key goals:
In essence, EDA entails active investigation of what our data contains even before formal modeling to guide choices, reveal issues needing resolution, and ensure we squeeze all potential value from our data resources.
The flexibility and lack of stringent assumptions make EDA invaluable for open-ended understanding.
Techniques
Exploratory Data Analysis (EDA) emphasizes flexibility and exploring different approaches to let key aspects of datasets emerge, rather than rigidly testing hypotheses from the start.
It is an iterative cycle where we analyze, visualize, and transform data to extract meaning.
EDA principles underlie “data science” and complement traditional statistical inference. Smart use of EDA provides a rich understanding of phenomena that can guide the construction ofcausal theoriesand models.
Trying various statistical and machine learning techniques to understand different facets of datasets.Rather than sticking to predetermined analysis plans, the focus is using diverse tools suited for particular datasets.Techniques could include clustering algorithms, decision trees, linear regression,ANOVA, etc., based on the data characteristics and research goals.
Graphical techniques
EDA emphasizes using graphical techniques to reveal patterns, relationships, and anomalies within data.
Graphical techniques in EDA are not merely supplemental tools but are crucial for gaining a deeper understanding of the data, which quantitative summaries alone cannot achieve.
Graphical methods provide unparalleled power to explore data due to their ability to engage the analyst’s natural pattern recognition abilitiesPictures allow our powerful visual perception to notice things numerical summaries may miss.
Creating graphs, charts, and plots to visually inspect data distributions, relationships between variables, outliers, etc.
Summarizing
Describing key statistics of datasets to understand central tendency (mean, median), spread (variance, percentiles), shape (skewness), outliers, and so on.
These numerical summaries complement visual inspection.
Central Tendency: This refers to the “typical” or “middle” value of a dataset. Commonly used measures of central tendency include:
Spread: Also known as variability or dispersion, it describes how spread out the data values are. Key measures of spread include:
Shape: Describes the symmetry and peakedness of a distribution. Two important measures of shape are:
Tabulation helps to simplify complex datasets and identify dominant patterns. The insights gained from tabulation can then inform the selection of appropriate statistical methods for further analysis
Here’s how tabulation summarizes data:
Data transformations
When data deviates from anormal distribution, many statistical techniques, which are grounded in the assumption of normality, may yield misleading or incorrect results.
Transformations provide a means to reshape the data, bringing it closer to a normal distribution and thus enhancing the applicability of these statistical techniques.
Data transformations involve using mathematical functions to modify the structure of datasets to uncover patterns more easily.
For instance, if a variable like income is highly skewed, applying a logarithm transformation can normalize its distribution, addressing potential issues with specific statistical tests.
Beyond improving normality, transformations in EDA can also enhance the clarity of data patterns, linearize trends, and stabilize variance.
They play a vital role in ensuring that the chosen statistical analysis aligns with the characteristics of the data, ultimately leading to more accurate and reliable conclusions.
Lgarithms are recommended as a starting point for amounts, like times and rates, which are nonnegative real values. For counts, which are nonnegative integers, square roots and logarithms are good initial transformations.
When analyzing data with skewness, researchers often aim to normalize it. However, it’s essential to recognize that skewness can sometimes reflect a genuine nonlinear relationship between variables, not just a statistical anomaly.
For example, a quadratic relationship might exist between drug dosage and the number of symptoms, where a higher dosage initially reduces symptoms, but extremely high doses lead to their resurgence.
Outlier detection
Identifying anomalies that distort overall patterns, and either correcting erroneous values or analyzing outliers specifically since they often reveal useful insights about the phenomenon under study.
Outlier identification is not about simply discarding inconvenient data points. It’s about critically examining the data, understanding the stories behind extreme values, and making informed decisions about how to handle them to ensure accurate and meaningful conclusions.
Choosing the Right Tool: Matching Techniques to Data and Objectives
Histograms are a powerful tool for understanding the distribution of a variable. They provide insights into central tendency, spread, modality, and outliers.
By displaying the frequency or proportion of cases within specified ranges (bins), histograms offer a visual representation of the data distribution.
The shape of the histogram can reveal whether the data is symmetrical, skewed, unimodal, bimodal, or has other distinct patterns.
Trying different bin sizes can be helpful to reveal more or less detail in the shape of the distribution.
Descriptive vs. Explorative Analysis
Descriptive analysis focuses on summarizing what the data shows on the surface. Exploratory analysis digs deeper to uncover subtle patterns and non-obvious trends in the data.
Descriptive analysis might tell a dataset’s average, median, and standard deviation. Exploratory analysis would use visualizations, transformations, and interrogating the data with different techniques to model the relationships between variables beyond just summary statistics.
So descriptive analysis describes what the data shows, while exploratory analysis explores nuances in the data to extract deeper meaning.
But good data analysis uses both techniques – summary statistics to complement the graphs and visuals revealing relationships.
Descriptive Analysis
Exploratory Analysis
References
Behrens, J. T. (1997). Principles and procedures of exploratory data analysis.Psychological Methods, 2(2), 131–160.
Emerson, J. D., & Stoto, M. A. (1983). Transforming data. In D. C. Hoaglin, F. Mosteller, & J. W. Tukey(Eds.),Understanding robust and exploratory data analysis(pp. 97–128). Wiley.
Hoaglin, D. C., Mosteller, F., & Tukey, J. W. (Eds.). (1991).Fundamentals of exploratory analysis of variance. Wiley.
Tukey, J. W. (1977). Exploratory data analysis. Reading, MA: Addison-Wesley.
Velleman, P. F. (2008). Truth, damn truth, and statistics.Journal of Statistics Education: An International Journal on the Teaching and Learning of Statistics,16(2).
Olivia Guy-Evans, MSc
BSc (Hons) Psychology, MSc Psychology of Education
Olivia Guy-Evans is a writer and associate editor for Simply Psychology. She has previously worked in healthcare and educational sectors.
Saul McLeod, PhD
BSc (Hons) Psychology, MRes, PhD, University of Manchester
Saul McLeod, PhD., is a qualified psychology teacher with over 18 years of experience in further and higher education. He has been published in peer-reviewed journals, including the Journal of Clinical Psychology.
