LIS4317 - Visual Analytics - Module 1 Post

After checking out the internet for the most interesting data visualizations, I have decided to focus on The 50 Most Visited Websites in the World by Dorothy Neufeld and Joyce Ma of Visual Capitalist.

Here is the link to the visualization and their findings:

https://www.visualcapitalist.com/the-50-most-visited-websites-in-the-world/

To answer the question on why I chose this particular visualization, I chose it because it was easy to make sense of their findings and the idea of bubbles representing the number of monthly website visits made it very simple to interpret.

Relating the Keim et al. (2008) definition of visual analytics, it is said that data at first has no direct value until an individual makes a point to get information from it (p. 154). Therefore, the data in the visualization is monthly website visits and applying Keim et al.’s definition, one is aware the monthly website visits do not mean much unless there is some meaning giving to it. In this case, the greater the number of website visits, the more popular the website is. We can see this idea represented by the size of the bubbles like how the YouTube bubble is smaller than the Google bubble.

Moving on to the definition of visualization, Keim et al. (2008) suggests that the act of visualization takes both data and information and their associated processes and makes it communicable so that individuals can obtain a greater understanding and clarity about a particular dataset (pp.155-157). When an individual quickly glances at the visualization, they can immediately determine that Google is the website with the greatest number of monthly visits just by looking at the largest bubble. By using bubbles, the visualization captures various websites and how they are related by company and makes the data more expressive as a result.

As for knowledge, Keim et al. (2008) continuously stresses the importance of transforming “data into reliable and provable knowledge” through the use of methods and models (p. 155). Knowledge can be quite difficult to express and without careful consideration of the design of the visualization, the intended audience of the visualization will end up confused and no extraction of knowledge will occur. Thus, one can infer that visualization is very important for allowing individuals to make sense and derive information and knowledge from the data. So, visualization such as the choice of how one will represent their data goes hand in hand with knowledge extraction. That is, how will certain design choices impact the readability and/or clarity of the data presented? Just looking at the visualization, the creators clearly prioritized the design of it to make it as easy to read as possible.

According to Keim et al. (2008), models play a significant role in data analysis (p. 161). From preparing a training dataset to teach models to locate data samples based on classification and/or prediction to using association rules to identify the co-occurrence of data items, there is a variety of different models out there performing many things. Sadly, not much is known regarding the techniques used to gather data for this particular visualization. While the data was retrieved by SimilarWeb, they do not document how they went about getting and recording the data. Nor if any models were applied.

Link to SimilarWeb: https://www.similarweb.com/top-websites/

~ Katie

LIS4273 - UFO Sightings Final Project

Important Background Information:

For this project, I decided to analyze a dataset containing UFO sightings in the United States, Mexico, Canada, and certain places in Europe. However, to keep it simple I focused solely on the data records from the United States.

Here is the link to the original dataset: https://www.kaggle.com/datasets/NUFORC/ufo-sightings/data

Note: when you go on to download the dataset, you will actually download two datasets: scrubbed.csv and complete.csv

For this analysis, I worked from scrubbed.csv which had less entries. After some data munging, I was able to get it down to approximately 63,561 rows and 11 variables which will work as the base of my analysis.

Establishing the Hypothesis (Problem Description):

From this dataset, I intended to determine the following:

Is there any difference in the duration (seconds) of the sightings between the north and south regions of the United States?

From the flowchart provided in the project instructions, I will be working with two samples (North and South) and developed the null and alternative hypotheses.

Null Hypothesis (H0): The average duration of UFO sightings is the same in the North and South regions, and there is no significant difference in the variability of durations between the two regions.

Alternative Hypothesis (H1): The average duration of UFO sightings differs between the North and South regions, and there is a significant difference in the variability of durations between the two regions.

Through conducting the Levene Test, I was able to determine that there are equal variances, and a Two Sample t-Test would be suitable for this analysis.

Also, important to note, to determine what states could be classified as “North” or “South”, I created a new variable known as region which through an if-else statement classifies a state as “North” or “South” by checking if it is above or below my selected latitude bench line. Given that I am from Maryland, this latitude value is 39.718457 and is based off the Mason-Dixon Line which I have crossed on multiple occasions. Fun times!

Related Work:

As for how my problem and chosen method for analysis relates to what was learned during the semester, it can be traced back to Module #6 Random Variable(s) and Probability Distribution(s) and One-Sample and Two-Sample Tests. Through the provided table in the Module #6 One-sample and Two-sample tests page, I used it as a guide when it came to forming my hypotheses.

Solution:

To solve the problem, I used various R functions to conduct my analysis. To determine if there are equal variances, I made use of the Levene test function, and my primary methodology was using a two sample t-test.

Why a two sample t-test, you might ask? Well, this test was determined to be the best choice by following the guidelines in the Final Project flowchart.

Given that I had two samples (North and South), the first question I had to ask myself was whether the data was in pairs or not. In this case, it was not as for North there was only 28,423 records and for South there was 35,020 records. For data to be paired, the observations must be of equal length.

Therefore, I followed the “No” arrow and proceeded to the “Equal Variances?” prompt. This part was a little tricky as the p-value was 0.05573 as determined by the Levene test. It was just barely above the 0.05 significance level so it was natural to check “Assume Equal Variances”, but it could also be considered as no equal variances as well. I was concerned about a potential violation so I performed two t-tests with var.equal set to TRUE or FALSE to see if there was any difference. From the results, I was able to confirm that there was no significant difference. Thankfully, whether I selected “Yes” or “No” to the Equal Variances prompt, it led to the same Two-Sample t-test.

To understand why a two sample t-test was preferred over other analysis methods, I realized that I needed a test that could handle two samples so naturally, I crossed out the idea of using a one-sample t-test as I had two samples (North and South). Furthermore, I did not consider the one-way ANOVA test as you need to have a minimum of three independent samples to make it work. Also, because my data was not paired nor related to each other, I did not execute a paired t-test. Through these insights, I was able to conclude that the Two-Sample t-test was the best fit for the data given the choices in the flowchart. 

Findings and Conclusions:

Upon executing my first t-test will var.equal set to TRUE, I soon became aware that I was in marginal violation of the equal variance assumption as the p-value (0.05573) from the Levene test was only slightly above the significance level of 0.05. With this minor issue in mind, I wanted to see via two t-tests with var.equal set to TRUE or FALSE respectively if there was any major difference in output. Quickly, I saw that their differences were minor and essentially alluded to the same findings.

Thus, we can infer the following.

Seeing that the p-value is just above the significance level of 0.05, we can determine that there is not enough evidence to reject the null hypothesis. In other words, there is no significant difference in the average duration of UFO sightings in the north and south regions of the United States. While the means for duration do appear to differ between North and South, one must understand that there truly is no significant difference in the length of the sightings.

~ Katie

LIS4273 - Module 12 Assignment

 For this assignment, I will be answering the following questions:

The table below represents charges for a student credit card.

a. Construct a time series plot using R

b. Employ Exponential Smoothing Model as outlined in Avril Voghlan's notes and report the statistical outcome.

For these two parts, please see the following R code and its associated output:

c. Provide a discussion on time series and Exponential Smoothing Model result you led to.

Through the output, we can first see with the Time Series plot, it clearly shows the fluctuations in charges throughout the months and years. Furthermore, it does seem to peak in charges around mid-year, which may refer to high spending during the winter holiday months.

When we review the Exponential Smoothing model and its impact on the dataset, the first thing to take away is both parameters (beta and gamma) are set to FALSE which means that without trend and seasonality, it will emphasize the short-term variations of the data.

Looking at the high alpha value (0.8232442), this indicates that the forecasts are primarily influenced by recent observations. As for the coefficient ‘a’ value (62.44453), we can see that the model expects future values to revolve around this particular value. Now, when this new model is plotted, it more closely aligns with one another as opposed to the first plot which did not appear to be as closely aligned. Moving on the calculation of the sum-of-squared-errors (SSE), the value outputted was 835.38 which may infer that a model containing both trend and seasonality may better capture the underlying patterns in the data. Thus, it is key that you a strike a good balance between model refinement and goals of the analysis to increase model accuracy and reliability.

~ Katie

LIS4273 - Module 11 Assignment

 For this assignment, I will be answering the following questions:

10.1

From out textbook: pp. 188 Question 10.1

Set up an additive model for the ashina data, as part of the ISwR package

This data contains additive effects on subjects, period, and treatment. Compare the results with those obtained from t tests.

R code:

Result Interpretation:

Analyzing the results of the additive model, we first see that with the variable treat, its coefficient is -42.87 suggesting that the treatment group experienced a 42.87 unit decrease in vas as compared to the control group. As for period, its coefficient value is 80.50 which means that period 2 experienced an 80.50 unit increase in vas compared to period 1. Moving on to subjects, the first thing that sticks out are the significance levels and it is learned that for the intercept, treat, subject3, subject5, subject7, subject8, and subject10 are all significant (**) at 0.1 significance level where we can infer that their effects are not likely due to random chance. As for residuals, the range appears to go from -48.94 to positive 48.94. Ideally, these values should be randomly distributed around zero but a pattern such as this indicates the model is not capturing some aspect of data. As for the r-squared value, it is 0.7566 or 75.66% and while a good fit is a value close to 1, it means the 75.66% of the variability in vas scores is accounted for by the model. Lastly, with the F-statistic 2.914 and its associated p-value 0.02229, we can see that the model is indeed statistically significant.

Moving on to t test for treatment, we can see that there is a significant difference between the treatment and the control group via the p-value (0.02099). As for the t test for period, it is only marginally significant with the p-value (0.0672).

10.3 Consider the following definitions

Note:

The rnorm() is a built-in R function that generates a vector of normally distributed random numbers. The rnorm() method takes a sample size as input and generates that many numbers.

Your assignment:

Generate the model matrices for models z ~ a*b, z ~ a:b, etc. In your blog posting discuss the implications. Carry out the model fits and notice which models contain singularities.

Hint

We are looking for...

model.matrix(~ a:b); lm(z ~a:b)

R code:

Result Interpretation:

Through executing the code, the only model that contained singularities is model2 which held the expression a:b. The rest of the models yielded false when asked if they held singularities. What are the implications of this? Well, first off, it means the model2 has perfect collinearity. In other words, one predictor can be exactly predicted from the other which may lead to numerical instability in estimating the coefficients. Additionally, singularity means that there is an infinite number of solutions to model and R cannot uniquely determine the coefficients of a and b. Further, because a and b cannot be separated due to collinearity, the coefficient estimates will be unreliable.

As for the other models which yielded false when prompted for collinearity, we know that the coefficient estimates are reliable. Seeing that only one model had perfect collinearity and the rest did not, it might be best to focus on the non-collinear models for future analysis.

~ Katie

LIS4273 - Module 10 Assignment

 For this assignment, I will be answering the following questions:

Question 1:

From our textbook, Introductory Statistics with R pp. 159 Exercises, 9.1 and 9.2

9.1 I revised this question , so please follow my description only. Conduct ANOVA (analysis of variance) and Regression coefficients to the data from cystfibr (> data("cystfibr")) database. You can choose any variable you like. in your report, you need to state the result of Coefficients (intercept) to any variables you like both under ANOVA and multivariate analysis. I am specifically looking at your interpretation of R results.

Extra clue:

The model code:

In R:

Interpretation of the results:

Reviewing the output of the model, we can see that in terms of significance, the intercept is highly significant while the other variables: weight, bmp, and fev1 are all significant. Reading further into the results, we can take away the following ideas:

With age, we can see the pemax (maximum expiratory pressure) will decrease by approximately -3.4181 units for every one unit increase in age. However, the R results do not deem this variable as significant so we must move on.

As for weight, pemax (maximum expiratory pressure) will increase by 2.6882 units for every one unit increase in weight. R views this variable as statistically significant so this variable may be quite helpful in future models.

For bmp, which is body mass, the pemax variable will decrease by -2.0657 units for every one unit increase in bmp. This variable is also statistically significant and will definitely be helpful in future analyses.

Lastly, with fev1, which refers to forced expiratory volume, pemax will increase by 1.0882 units for every one unit increase in fev1.

Taking into account the p-values for each of the variables it seems that only fev1 gets rather close to 0.05 significance level at around 0.04695. The rest of the variables and their corresponding p-values are lower than fev1 with the exception of age, but that value is not at all significant to the analysis.

Moving on to the ANOVA table and its output, there are some surprising results. Looking first to the sum of squares which refers to concept of that a higher sum of squares equates to greater variability in the data. The variable age proves to have the highest value at 10098.5. The rest of the variables do not have as high of a sum of squares value. As we review each of the p-values and their corresponding significance levels we immediately see that age is highly significant at the three-star (***) range. Additionally, bmp and fev1 are considered as only marginally significant while there are no stars at all given to weight. Hence, age is a highly significant predictor and not far behind it are bmp and fev1 which alludes to the idea that they contribute to the variability in pemax variable.

Question 2:

9.2 The secher data (> data("secher")) are best analyzed after log-transforming birth weight as well as the abdominal and biparietal diameters. Fit a prediction weight as well as abdominal and biparietal diameters. For a prediction equation for birth weight.

How much is gained by using both diameters in a prediction equation?

The sum of the two regression coefficients is almost identical and equal to 3.

Can this be given a nice interpretation to our analysis?

Please provide a step by step on your analysis and code you use to find out the result.

Extra clue:

In R:

Interpretation of the results:

Analyzing the output of this model, we can see that when using both parameters bwt (birth weight) and ad (abdominal) in the prediction equation, it can be said that the log of ad indeed contributes to the prediction of log(bwt). As we look at the output, most notably the coefficients section, for each one unit increase of log(ad), the estimated change in log(bwt) is 2.2365. Now, given that the sum of the two regression coefficients is almost identical and equal to 3, this suggests that the model is inferring that the total effect on log(bwt) is additive when log(ad) as well as the intercept are considered. Just for reference, additive simply means that the effect of one predictor variable on the response variable is independent of the values of the other predictor variables. Moving on to the R-squared value, it is 0.7959 which means that 79.59% of the variability in log(bwt) can be explained by the model. As for the F-statistic, seeing its high value and the low p-value, one can infer that the model is indeed statistically significant.

~ Katie

LIS 4273 - Module 9 Assignment

For this assignment, I will be answering the following questions:

Question 1:

Your data frame is

 

Generate a simple table in R that consists of four rows: Country, age, salary, Purchased

Question 2

Generate a contingency table also known as rx C table using the mtcars dataset

Note: the following sections of this question can be found within the gist:

~ Katie

LIS4273 - Module 8 Assignment

 For this post, I will answering the following questions.

Question 1:

A researcher is interested in the effects of drug against stress reaction. She gives a reaction time test to three different groups of subjects: one group that is under a great deal of stress, one group under a moderate amount of stress, and a third group that is under almost no stress. The subjects of the study were instructed to take the drug test during their next stress episode and to report their stress on a scale of 1 to 10 (10 being most pain).

Report on the drug and stress level by using R. Provide a full summary report on the result of ANOVA testing and what does it mean? More specifically, report using the following R functions:

After running a summary report after using ANOVA, we first see that the degrees of freedom results in 2 meaning that there are two degrees of freedom associated with the variability in stress levels because of the stress group. As for Sum Sq or Sum of Squares, the value 82.11 represents the sum of squares associated with the stressGroup variable. Given the larger value, it can be interpreted that there is a significant amount of variability in stressLevel that is being explained by the stressGroup. Moving on to Mean Sq, 41.06 is the value associated with the stressGroup variable. Seeing how the value is larger than the residuals value, it can be taken away that the stress group indeed explains more variability in stressLevel than expected by random chance. For the F value, the summary produced the value of 21.36 which seems to indicate that it is a large value coupled with a rather small p-value (4.08e-05). Thus, the stress group variable has a statistically significant effect on stress levels.

Question 2:

From our Textbook: Introductory Statistics with R, Chapter 6, Exercises 6.1, pp. 127

The zelazo data (taken from textbook's R package IWsR) are in the form of a list of vectors, one for each of the four groups. Convert the data to a form suitable for the user of lm and calculate the relevant test. Consider t tests comparing selected subgroups or obtained by combining groups.

2.1 Consider ANOVA test (one-way or two-way) to this dataset (zelazo)

Recommendations

After reflecting upon the zelazo dataset, it can be seen that the one-way ANOVA test is more appropriate given that data set contains only one independent variable (type of training) with four levels like “active”, “passive”, “none”, and “ctr.w8” which means control. Seeing that one-way ANOVA tests work best when given a categorical independent variable (factor) with more than two levels (groups), it is clear that this test is more fitting to the problem than two-way ANOVA. We have four groups and one factor.

~ Katie