Welcome

Contact us on LIVE CHAT for the completely plagiarism free work, in the best possible price, done by experienced Experts!

Introduction to Data Science

Introduction to Data Science

STAT 167 Introduction to Data Science
Exploratory data analysis
Wenxiu Ma
[email protected]
4/28/2022
© W Ma 2022 1/ 51
Exploratory data analysis
Wickham and Grolemund. “R for Data Science: Import, Tidy,
Transform, Visualize, and Model Data.
” 1st Edition (2017)
I Chapter 5 “Exploratory Data Analysis” in print.
I Chapter 7 “Exploratory Data Analysis” @
http://r4ds.had.co.nz/exploratory-data-analysis.html
© W Ma 2022 2/ 51
Recap – complete graphing template in ggplot2
ggplot(data = <DATA>) +
<GEOM_FUNCTION>(
mapping = aes(<MAPPINGS>),

stat = <STAT>, # optional
position = <POSITION> # optional
) +
<COORDINATE_FUNCTION> + # optional
<FACET_FUNCTION> +
<SCALE_FUNCTION> +
<THEME_FUNCTION>
# optional
# optional
# optional

© W Ma 2022 3/ 51
Recap – dplyr basic functions
I Pick observations by their values – filter()
I Reorder the rows – arrange()
I Pick variables by their names – select()
I Create new variables with functions of existing variables –
mutate()
I Collapse many values down to a single summary –
summarise()
I The above five can be used in conjunction with group_by()
which changes the scope of each function from operating on
the entire dataset to operating on it group-by-group.
These six functions provide the verbs for a language of
data
manipulation/wrangling
.
© W Ma 2022 4/ 51
Recap – All dplyr functions share similar grammar
I The first argument is a data frame.
I The subsequent arguments describe what to do with the data
frame, using the variable names (without quotes).
I The result is a new data frame.
Together these properties make it easy to chain together multiple
simple steps to achieve a complex result –
piping (%>%)!
© W Ma 2022 5/ 51
Combining the power of ggplot2 & dplyr
A typical data science project workflow
I Data exploration is the art of looking at your data, rapidly
generating hypotheses, quickly testing them, then repeating
again and again and again.
© W Ma 2022 6/ 51
Data set: nycflights13::flights
This data frame contains all 336,776 flights that departed from New York City in
2013. The data comes from the US Bureau of Transportation Statistics.
library(nycflights13)
flights
## # A tibble: 336,776 x 19
## year month day dep_time sched_dep_time dep_delay arr_time
## <int> <int> <int> <int> <int> <dbl> <int>
## 1 2013 1 1 517 515 2 830
## 2 2013 1 1 533 529 4 850
## 3 2013 1 1 542 540 2 923
## 4 2013 1 1 544 545 -1 1004
## 5 2013 1 1 554 600 -6 812
## 6 2013 1 1 554 558 -4 740
## 7 2013 1 1 555 600 -5 913
## 8 2013 1 1 557 600 -3 709
## 9 2013 1 1 557 600 -3 838
## 10 2013 1 1 558 600 -2 753
## # … with 336,766 more rows, and 12 more variables:
## # sched_arr_time <int>, arr_delay <dbl>, carrier <chr>,
## # flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
## # air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>,
## # time
© W Ma 2022 7/ 51
Exercise #1 – Study average delay per destination
Imagine that we want to explore the relationship between the
distance and average delay for each destination.
© W Ma 2022 8/ 51
Exercise #1 – Study average delay per destination
Imagine that we want to explore the relationship between the
distance and average delay for each destination.
-20
0
20
40
0 1000 2000 3000 4000 5000
dist
delay
count
4000
8000
12000
16000
© W Ma 2022 9/ 51
Calculate average delay per destination
by_dest <- group_by(flights, dest)
delays_by_dest
<- summarise(by_dest, count = n(),
dist = mean(distance),
delay = mean(arr_delay))
delays_by_dest
## # A tibble: 105 x 4

##
##
dest count dist delay
<chr> <int> <dbl> <dbl>
## 1 ABQ
## 2 ACK
## 3 ALB
## 4 ANC
## 5 ATL
## 6 AUS
## 7 AVL
## 8 BDL
## 9 BGR
## 10 BHM
254 1826 4.38
265 199 NA
439 143 NA
8 3370 -2.5
17215 757. NA
2439 1514. NA
275 584. NA
443 116 NA
375 378 NA
297 866. NA

## # … with 95 more rows
© W Ma 2022 10/ 51
How many missing values?
filter(delays_by_dest, is.na(delay))
## # A tibble: 100 x 4

##
##
dest count dist delay
<chr> <int> <dbl> <dbl>
## 1 ACK
## 2 ALB
## 3 ATL
## 4 AUS
## 5 AVL
## 6 BDL
## 7 BGR
## 8 BHM
## 9 BNA
## 10 BOS
265 199
439 143
17215 757.
2439 1514.
275 584.
443 116
375 378
297 866.
6333 758.
15508 191.
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA
## # … with 90 more rows
colSums(is.na(delays_by_dest)) # count # of NAs per column
## dest count dist delay
## 0 0 0 100

© W Ma 2022 11/ 51
Deal with missing values
1. use na.rm = TRUE in the summary function
delays_by_dest1 <- summarise(by_dest, count = n(),
dist = mean(distance),
delay = mean(arr_delay, na.rm = T))
delays_by_dest1
## # A tibble: 105 x 4

##
##
dest count dist delay
<chr> <int> <dbl> <dbl>
## 1 ABQ
## 2 ACK
## 3 ALB
## 4 ANC
## 5 ATL
## 6 AUS
## 7 AVL
## 8 BDL
## 9 BGR
## 10 BHM
254 1826
265 199
4.38
4.85
439 143 14.4
8 3370 -2.5
17215 757. 11.3
2439 1514. 6.02
275 584. 8.00
443 116
375 378
7.05
8.03
297 866. 16.9

## # … with 95 more rows
© W Ma 2022 12/ 51
Deal with missing values
2. filter out missing values from flights
# count # of NAs per column
colSums(is.na(flights))

##
##
year
0
month
0
dep_delay
8255
carrier
0
dest
0
minute
0
day
0
dep_time
8255
## sched_dep_time arr_time sched_arr_time
##
##
##
##
##
##
##
0
arr_delay
9430
origin
0
hour
0
8713
flight
0
air_time
9430
time_hour
0
0
tailnum
2512
distance
0

© W Ma 2022 13/ 51
Deal with missing values
2. filter out missing values from flights
filter(flights, is.na(arr_delay))
## # A tibble: 9,430 x 19

##
##
year month day dep_time sched_dep_time dep_delay arr_time
<int> <int> <int> <int>
1525
1528
1740
1807
1939
1952
2016
NA
NA
NA
<int>
1530
1459
1745
1738
1840
1930
1930
1630
1935
1500
<dbl>
-5
29
-5
29
59
22
46
NA
NA
NA
<int>
1934
2002
2158
2251
29
2358
NA
NA
NA
NA
## 1 2013
## 2 2013
## 3 2013
## 4 2013
## 5 2013
## 6 2013
## 7 2013
## 8 2013
## 9 2013
## 10 2013
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
## # … with 9,420 more rows, and 12 more variables:
## #
## #
## #
## #
sched_arr_time <int>, arr_delay <dbl>, carrier <chr>,
flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>,
time_hour <dttm>

© W Ma 2022 14/ 51
Deal with missing values
2. filter out missing values from flights
filter(flights, is.na(arr_delay)) %>% select(dep_time:arr_delay)
## # A tibble: 9,430 x 6

##
##
## 1
## 2
## 3
## 4
## 5
## 6
## 7
## 8
## 9
## 10
dep_time sched_dep_time dep_delay arr_time sched_arr_time arr_delay
<int>
1525
1528
1740
1807
1939
1952
2016
NA
NA
NA
<int>
1530
1459
1745
1738
1840
1930
1930
1630
1935
1500
<dbl>
-5
29
-5
29
59
22
46
NA
NA
NA
<int>
1934
2002
2158
2251
29
2358
NA
NA
NA
NA
<int>
1805
1647
2020
2103
2151
2207
2220
1815
2240
1825
<dbl>
NA
NA
NA
NA
NA
NA
NA
NA
NA
NA

## # … with 9,420 more rows
I NAs in dep_delay or arr_delay represent canceled flights
© W Ma 2022 15/ 51
Data cleaning – missing values
2. filter out missing values from flights and use non-canceled
flights instead
# NAs in dep_delay and arr_delay represent canceled flights
(not_canceled <- filter(flights, !is.na(dep_delay), !is.na(arr_delay)))
## # A tibble: 327,346 x 19

##
##
year month day dep_time sched_dep_time dep_delay arr_time
<int> <int> <int> <int>
517
533
542
544
554
554
555
557
557
558
<int>
515
529
540
545
600
558
600
600
600
600
<dbl>
2
4
2
-1
-6
-4
-5
-3
-3
-2
<int>
830
850
923
1004
812
740
913
709
838
753
## 1 2013
## 2 2013
## 3 2013
## 4 2013
## 5 2013
## 6 2013
## 7 2013
## 8 2013
## 9 2013
## 10 2013
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
## # … with 327,336 more rows, and 12 more variables:
## #
## #
sched_arr_time <int>, arr_delay <dbl>, carrier <chr>,
flight <int>, tailnum <chr>, origin <chr>, dest <chr>,
## # air_time <dbl>, distance <dbl>, hour <dbl>, minute <dbl>,
© W Ma 2022 16/ 51
## # ti

Data cleaning – missing values
2. filter out missing values from flights and use non-canceled
flights instead
not_canceled %>% group_by(dest) %>%
summarise(
count = n(), dist = mean(distance),
delay = mean(arr_delay))
## # A tibble: 104 x 4

##
##
dest count dist delay
<chr> <int> <dbl> <dbl>
## 1 ABQ
## 2 ACK
## 3 ALB
## 4 ANC
## 5 ATL
## 6 AUS
## 7 AVL
## 8 BDL
## 9 BGR
## 10 BHM
254 1826
264 199
4.38
4.85
418 143 14.4
8 3370 -2.5
16837 757. 11.3
2411 1514. 6.02
261 584. 8.00
412 116
358 378
7.05
8.03
269 866. 16.9

## # … with 94 more rows
© W Ma 2022 17/ 51
Summary – dealing with missing values
1. use na.rm = TRUE in the summary function
(delays_by_dest1 <- flights %>% group_by(dest) %>%
summarise(
count = n(), dist = mean(distance),
delay = mean(arr_delay, na.rm = T)))
## # A tibble: 105 x 4

##
##
dest count dist delay
<chr> <int> <dbl> <dbl>
## 1 ABQ
## 2 ACK
## 3 ALB
## 4 ANC
## 5 ATL
## 6 AUS
## 7 AVL
## 8 BDL
## 9 BGR
## 10 BHM
254 1826
265 199
4.38
4.85
439 143 14.4
8 3370 -2.5
17215 757. 11.3
2439 1514. 6.02
275 584. 8.00
443 116
375 378
7.05
8.03
297 866. 16.9

## # … with 95 more rows
© W Ma 2022 18/ 51
Summary – dealing with missing values
2. filter out missing values from flights and use non-canceled flights
instead
not_canceled <- filter(flights, !is.na(dep_delay), !is.na(arr_delay))
(delays_by_dest2
<- not_canceled %>% group_by(dest) %>%
summarise(
count = n(), dist = mean(distance),
delay = mean(arr_delay)))
## # A tibble: 104 x 4

##
##
dest count dist delay
<chr> <int> <dbl> <dbl>
## 1 ABQ
## 2 ACK
## 3 ALB
## 4 ANC
## 5 ATL
## 6 AUS
## 7 AVL
## 8 BDL
## 9 BGR
## 10 BHM
254 1826
264 199
4.38
4.85
418 143 14.4
8 3370 -2.5
16837 757. 11.3
2411 1514. 6.02
261 584. 8.00
412 116
358 378
7.05
8.03
269 866. 16.9

## # … with 94 more rows
© W Ma 2022 19/ 51
Are the results different?
I An inner join matches pairs of observations whenever their
keys are equal.
inner_join(delays_by_dest1, delays_by_dest2, by = “dest”)
## # A tibble: 104 x 7

##
##
dest count.x dist.x delay.x count.y dist.y delay.y
<chr> <int> <dbl>
254 1826
<dbl>
4.38
4.85
14.4
-2.5
11.3
6.02
8.00
7.05
8.03
16.9
<int> <dbl>
254 1826
<dbl>
4.38
4.85
14.4
-2.5
11.3
6.02
8.00
7.05
8.03
16.9
## 1 ABQ
## 2 ACK
## 3 ALB
## 4 ANC
## 5 ATL
## 6 AUS
## 7 AVL
## 8 BDL
## 9 BGR
## 10 BHM
265
439
199
143
264
418
199
143
8 3370 8 3370
17215 757. 16837 757.
2439 1514. 2411 1514.
275
443
375
297
584.
116
378
866.
261
412
358
269
584.
116
378
866.

## # … with 94 more rows
© W Ma 2022 20/ 51
Are the results different?
I anti_join(x, y) drops all observations in x that have a
match in
y.
anti_join(delays_by_dest1, delays_by_dest2, by = “dest”)
## # A tibble: 1 x 4

##
##
dest count dist delay
<chr> <int> <dbl> <dbl>
## 1 LGA 1 17 NaN

© W Ma 2022 21/ 51
Why is there a NaN?
filter(flights, dest == “LGA”)
## # A tibble: 1 x 19

##
##
year month day dep_time sched_dep_time dep_delay arr_time
<int> <int> <int> <int>
NA
<int>
106
<dbl>
NA
<int>
NA
## 1 2013 7 27
## # … with 12 more variables: sched_arr_time <int>, arr_delay <dbl>,
## #
## #
## #
carrier <chr>, flight <int>, tailnum <chr>, origin <chr>,
dest <chr>, air_time <dbl>, distance <dbl>, hour <dbl>,
minute <dbl>, time_hour <dttm>

© W Ma 2022 22/ 51
Why is there a NaN?
I na.rm = TRUE will produce NaN when all values are missing
# filter(flights, dest == “LGA”) %>% str()
filter(flights, dest == “LGA”) %>% glimpse()
## Rows: 1
## Columns: 19
## $ year <int> 2013
## $ month <int> 7
## $ day <int> 27
## $ dep_time <int> NA
## $ sched_dep_time <int> 106
## $ dep_delay <dbl> NA
## $ arr_time <int> NA
## $ sched_arr_time <int> 245
## $ arr_delay <dbl> NA
## $ carrier <chr> “US”
## $ flight <int> 1632
## $ tailnum <chr> NA
## $ origin <chr> “EWR”
## $ dest <chr> “LGA”
## $ air_time <dbl> NA
## $ distance <dbl> 17
## $ hour <dbl> 1
## $ minute <dbl> 6
## $ time_hour <dttm> 2013-07-27 01:00:00
© W Ma 2022 23/ 51
Exercise #1 – Study average delay per destination
Imagine that we want to explore the relationship between the
distance and average delay for each destination.
-20
0
20
40
0 1000 2000 3000 4000 5000
dist
delay
count
4000
8000
12000
16000
© W Ma 2022 24/ 51
not_canceled <- filter(flights, !is.na(dep_delay), !is.na(arr_delay))
(delays_by_dest2
<- not_canceled %>% group_by(dest) %>%
summarise(
count = n(), dist = mean(distance),
delay = mean(arr_delay)))
## # A tibble: 104 x 4

##
##
dest count dist delay
<chr> <int> <dbl> <dbl>
## 1 ABQ
## 2 ACK
## 3 ALB
## 4 ANC
## 5 ATL
## 6 AUS
## 7 AVL
## 8 BDL
## 9 BGR
## 10 BHM
254 1826
264 199
4.38
4.85
418 143 14.4
8 3370 -2.5
16837 757. 11.3
2411 1514. 6.02
261 584. 8.00
412 116
358 378
7.05
8.03
269 866. 16.9

## # … with 94 more rows
© W Ma 2022 25/ 51
ggplot(data = delays_by_dest2, mapping = aes(x = dist, y = delay)) +
geom_point(aes(
size = count), alpha = 1/3) +
geom_smooth(
se = FALSE)
## `geom_smooth()` using method = ‘loess’ and formula ‘y ~ x’
-20
0
20
40
0 1000 2000 3000 4000 5000
dist
delay
count
4000
8000
12000
16000
© W Ma 2022 26/ 51
What are the two points with largest distances?
arrange(delays_by_dest2, desc(dist))
## # A tibble: 104 x 4

##
##
dest count dist
<chr> <int> <dbl>
delay
<dbl>
## 1 HNL
## 2 ANC
## 3 SFO
## 4 OAK
## 5 SJC
## 6 SMF
## 7 LAX
## 8 BUR
## 9 LGB
## 10 PDX
701 4973. -1.37
8 3370 -2.5
13173 2578. 2.67
309 2576
328 2569
3.08
3.45
282 2521 12.1
16026 2469. 0.547
370 2465 8.18
661 2465 -0.0620
1342 2446. 5.14

## # … with 94 more rows
© W Ma 2022 27/ 51
What are the two points with largest distances?
top_n(delays_by_dest2, 2, dist)
## # A tibble: 2 x 4

##
##
dest count dist delay
<chr> <int> <dbl> <dbl>
## 1 ANC
## 2 HNL
8 3370 -2.5
701 4973. -1.37

© W Ma 2022 28/ 51
Detect outliers in delays_by_dest2$dist
ggplot(data = delays_by_dest2) +
geom_histogram(
mapping = aes(x = dist))
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
5 0
10
0 1000 2000 3000 4000 5000
dist
count
ggplot(data = delays_by_dest2) +
geom_boxplot(
mapping = aes( x = “”, y = dist)) +
coord_flip()
0 1000 2000 3000 4000 5000
dist
© x
W Ma 2022 29/ 51
not_canceled <- filter(flights, !is.na(dep_delay), !is.na(arr_delay))
delays_by_dest_filtered
<- not_canceled %>% group_by(dest) %>%
summarise(
count = n(), dist = mean(distance),
delay = mean(arr_delay)) %>%
filter(dest !=
“HNL”, dest != “ANC”)
ggplot(
data = delays_by_dest_filtered,
mapping = aes(x = dist, y = delay)) +
geom_point(aes(
size = count), alpha = 1/3) + geom_smooth(se = F)
## `geom_smooth()` using method = ‘loess’ and formula ‘y ~ x’
-20
0
20
40
0 1000 2000
dist
delay
count
4000
8000
12000
16000
© W Ma 2022 30/ 51
Problem with small n
I Whenever you do any aggregation, it’s always a good idea to
include either a count
n(), or a count of non-missing values
sum(!is.na(x)).
I That way you can check that you’re not drawing conclusions
based on very small amounts of data.
© W Ma 2022 31/ 51
Distribution of delays_by_dest_filtered$count
ggplot(delays_by_dest_filtered) +
geom_histogram(
mapping=aes(x=count))
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
5 0
10
15
20
0 5000 10000 15000
count
count
ggplot(data = delays_by_dest_filtered) +
geom_boxplot(
mapping = aes(x = “”, y = count)) + coord_flip()
0 5000 10000 15000
count
© x
W Ma 2022 32/ 51
Distribution of delays_by_dest_filtered$count
ggplot(delays_by_dest_filtered) +
geom_histogram(
mapping = aes(x = log10(count)))
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
0.0
2.5
5.0
7.5
0 1 2 3 4
log10(count)
count
ggplot(data = delays_by_dest_filtered) +
geom_boxplot(
mapping = aes(x = “”, y = log10(count))) + coord_flip()
0 1 2 3 4
log10(count)
© x
W Ma 2022 33/ 51
Detect small n in delays_by_dest_filtered$count
arrange(delays_by_dest_filtered, count)
## # A tibble: 102 x 4

##
##
dest count dist delay
<chr> <int> <dbl> <dbl>
## 1 LEX
## 2 SBN
## 3 HDN
## 4 MTJ
## 5 EYW
## 6 PSP
## 7 JAC
## 8 BZN
## 9 CHO
## 10 MYR
1 604 -22
10 645.
14 1728
14 1795
17 1207
6.5
2.14
1.79
6.35
18 2378 -12.7
21 1876. 28.1
35 1882
46 305
58 551.
7.6
9.5
4.60

## # … with 92 more rows
© W Ma 2022 34/ 51
Data transformation & cleaning – step by step
1. Filter out canceled flights;
2. Group non-canceled flights by destination;
3. Summarise grouped data to compute the number of flights,
average distance, and average arrival delay per destination;
4. Filter out outliers (longest distances – HNL and ANC);
5. Filter out noisy points (small flight counts – rare destinations).
© W Ma 2022 35/ 51
Data transformation & cleaning – step by step
1. Filter out canceled flights;
2. Group non-canceled flights by destination;
3. Summarise grouped data to compute the number of flights,
average distance, and average arrival delay per destination;
4. Filter out outliers (longest distances – HNL and ANC);
5. Filter out noisy points (small flight counts – rare destinations).
not_canceled <- filter(flights, !is.na(dep_delay), !is.na(arr_delay))
not_canceled_by_dest
<- group_by(not_canceled, dest)
delays_by_dest
<- summarise(not_canceled_by_dest, count = n(),
dist = mean(distance),
delay = mean(arr_delay))
delays_by_dest_filtered
<- filter(delays_by_dest,
dest !=
“HNL”, dest != “ANC”)
delays_by_dest_cleaned
<- filter(delays_by_dest_filtered, count >= 10)
© W Ma 2022 36/ 51
PIPING %>%
# original code – step by step
not_canceled <- filter(flights, !is.na(dep_delay), !is.na(arr_delay))
not_canceled_by_dest
<- group_by(not_canceled, dest)
delays_by_dest
<- summarise(not_canceled_by_dest, count = n(),
dist = mean(distance), delay = mean(arr_delay))
delays_by_dest_filtered
<- filter(delays_by_dest,
dest !=
“HNL”, dest != “ANC”)
delays_by_dest_cleaned
<- filter(delays_by_dest_filtered, count >= 10)
# alternative code using piping %>%
not_canceled <- filter(flights, !is.na(dep_delay), !is.na(arr_delay))
delays_by_dest_cleaned
<- not_canceled %>%
group_by(dest) %>%
summarise(
count = n(), dist = mean(distance), delay = mean(arr_delay)) %>%
filter(dest !=
“HNL”, dest != “ANC”, count >= 10)
I You can even pipe your dplyr output into ggplot2!
© W Ma 2022 37/ 51
Finally, visualization with ggplot2
ggplot(data = delays_by_dest_cleaned,
mapping = aes(x = dist, y = delay)) +
geom_point(
mapping = aes(size = count), alpha = 1/3) +
geom_smooth(
se = FALSE)
## `geom_smooth()` using method = ‘loess’ and formula ‘y ~ x’
0
20
40
0 1000 2000
dist
delay
count
4000
8000
12000
16000
© W Ma 2022 38/ 51
Comparison – filtering out outliers and noisy points
-20
0
20
40
0 1000 2000 3000 4000 5000
dist
delay
0
20
40
0 1000 2000
dist
delay
© W Ma 2022 39/ 51
Exercise #2 – Study the average delay per plane
Imagine that we want to investigate the distribution of average
delays among different planes.
© W Ma 2022 40/ 51
Exercise #2 – Study the average delay per plane
flights %>% glimpse()
## Rows: 336,776
## Columns: 19

## $ year
## $ month
## $ day
## $ dep_time
<int> 2013, 2013, 2013, 2013, 2013, 2013, 2013, 201~
<int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ~
<int> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ~
<int> 517, 533, 542, 544, 554, 554, 555, 557, 557, ~
## $ sched_dep_time <int> 515, 529, 540, 545, 600, 558, 600, 600, 600, ~
## $ dep_delay
## $ arr_time
<dbl> 2, 4, 2, -1, -6, -4, -5, -3, -3, -2, -2, -2, ~
<int> 830, 850, 923, 1004, 812, 740, 913, 709, 838,~
## $ sched_arr_time <int> 819, 830, 850, 1022, 837, 728, 854, 723, 846,~
## $ arr_delay
## $ carrier
## $ flight
## $ tailnum
## $ origin
## $ dest
## $ air_time
## $ distance
## $ hour
## $ minute
## $ time_hour
<dbl> 11, 20, 33, -18, -25, 12, 19, -14, -8, 8, -2,~
<chr> “UA”, “UA”, “AA”, “B6”, “DL”, “UA”, “B6”, “EV~
<int> 1545, 1714, 1141, 725, 461, 1696, 507, 5708, ~
<chr> “N14228”, “N24211”, “N619AA”, “N804JB”, “N668~
<chr> “EWR”, “LGA”, “JFK”, “JFK”, “LGA”, “EWR”, “EW~
<chr> “IAH”, “IAH”, “MIA”, “BQN”, “ATL”, “ORD”, “FL~
<dbl> 227, 227, 160, 183, 116, 150, 158, 53, 140, 1~
<dbl> 1400, 1416, 1089, 1576, 762, 719, 1065, 229, ~
<dbl> 5, 5, 5, 5, 6, 5, 6, 6, 6, 6, 6, 6, 6, 6, 6, ~
<dbl> 15, 29, 40, 45, 0, 58, 0, 0, 0, 0, 0, 0, 0, 0~
<dttm> 2013-01-01 05:00:00, 2013-01-01 05:00:00, 20~

© W Ma 2022 41/ 51
Exercise #2 – Study the average delay per plane
Data transformation – step by step
1. Filter out canceled flights;
2. Group non-canceled flights by tailnum;
3. Summarise grouped data to compute the number of flights and
average arrival delay per tailnum.
© W Ma 2022 42/ 51
Data transformation – step by step
1. Filter out canceled flights;
2. Group non-canceled flights by tailnum;
3. Summarise grouped data to compute the number of flights and average
arrival delay per tailnum.
(delays_by_tailnum <- not_canceled %>%
group_by(tailnum) %>%
summarise(
count = n(), delay = mean(arr_delay)))
## # A tibble: 4,037 x 3
## tailnum count delay
## <chr> <int> <dbl>
## 1 D942DN 4 31.5
## 2 N0EGMQ 352 9.98
## 3 N10156 145 12.7
## 4 N102UW 48 2.94
## 5 N103US 46 -6.93
## 6 N104UW 46 1.80
## 7 N10575 269 20.7
## 8 N105UW 45 -0.267
## 9 N107US 41 -5.73
## 10 N108UW 60 -1.25
## # … with 4,027 more rows
© W Ma 2022 43/ 51
Distribution of delays_by_tailnum$delay
ggplot(data = delays_by_tailnum, mapping = aes(x = delay)) +
geom_histogram(
mapping = aes(y=..density..), binwidth = 5,
colour = “black”, fill = “white”) +
geom_density(
alpha = .2, fill = “#FF6666”)
0.00
0.01
0.02
0.03
0.04
0 100 200 300
delay
© density
W Ma 2022 44/ 51
Problem with small n
ggplot(data = delays_by_tailnum,
mapping = aes(x = count, y = delay)) +
geom_point(
alpha = .1)
0
100
200
300
0 200 400
count
© delay
W Ma 2022 45/ 51
Detect small n in delays_by_tailnum$count
ggplot(delays_by_tailnum) +
geom_histogram(
mapping = aes(x = log10(count)))
## `stat_bin()` using `bins = 30`. Pick better value with `binwidth`.
0
100
200
300
0 1 2
log10(count)
count
ggplot(data = delays_by_tailnum) +
geom_boxplot(
mapping = aes(x = “”, y = log10(count))) + coord_flip()
0 1 2
log10(count)
© x
W Ma 2022 46/ 51
Detect small n in delays_by_tailnum$count
delays_by_tailnum %>% count(count)
## # A tibble: 356 x 2
## count n
## <int> <int>
## 1 1 168
## 2 2 99
## 3 3 67
## 4 4 50
## 5 5 78
## 6 6 46
## 7 7 49
## 8 8 32
## 9 9 27
## 10 10 24
## # … with 346 more rows
I It was not a good idea to name a variable as an existing function name!!
© W Ma 2022 47/ 51
Remove noisy points
delays_by_tailnum_cleaned <- filter(delays_by_tailnum, count >=3)
ggplot(
data = delays_by_tailnum_cleaned,
mapping = aes(x = count, y = delay)) +
geom_point(
alpha = .1)
0
50
100
150
0 200 400
count
© delay
W Ma 2022 48/ 51
Remove noisy points
ggplot(data = delays_by_tailnum_cleaned, mapping = aes(x = delay)) +
geom_histogram(
mapping = aes(y=..density..), binwidth = 5,
colour = “black”, fill = “white”) +
geom_density(
alpha = .2, fill = “#FF6666”)
0.00
0.01
0.02
0.03
0.04
0.05
0 50 100 150
delay
© density
W Ma 2022 49/ 51
Comparison – filtering out noisy points
0.00
0.01
0.02
0.03
0.04
0 100 200 300
delay
density
0.00
0.01
0.02
0.03
0.04
0.05
0 50 100 150
delay
density
© W Ma 2022 50/ 51
Exploratory Data Analysis (EDA)
EDA is an iterative cycle.
You:
I Generate questions about your data.
I Search for answers by visualizing, transforming, and modeling
your data.
I Use what you learn to refine your questions and/or generate
new questions.
© W Ma 2022 51/ 51

postadmin

Comments are closed.