Data Analytics Workouts

More as an experiment, rather an attempt at sharing code and ideas, I created a Pinterest board devoted to my personal data analytics work, done with Python, R, or F#, as well as reviews of books, and was quite surprised with the result. The graphics could do with optimization, but otherwise...

As part of reading Fundamentals of Deep Learning: Designing Next-Generation Machine Intelligence Algorithms by Nikhil Buduma , I was expecting to work through some of the code examples with my own data, and for the examples, it recommended TensorFlow , which brings up competing alternatives, a primary one being Microsoft Cognitive Toolkit . Over the next few weeks, I will start exploring both in Python, as well as publishing some of the related work. A minor note, the darksigma/Fundamentals-of-Deep-Learning-Book: Code companion to the O'Reilly "Fundamentals of Deep Learning" book is available on GitHub.

This is a repost of a Goodreads' review I made in 2013, for a book I read in 2005, which seems relevant now, as the industry is adding a data-driven focus. Plus, the world is now being transformed by advances in artificial intelligence and machine learning (AI/ML), particularly deep learning, and the large data sets and complexity of donor actions should greatly benefit from analysis. Note, the tax changes for 2018 and beyond will increase the importance of major donors, attenuating the benefits of AI/ML, as data for high-net-worth individuals is sparse. Data Mining for Fund Raisers: How to Use Simple Statistics to Find the Gold in Your Donor Database Even If You Hate Statistics: A Starter Guide by Peter B. Wylie My rating: 4 of 5 stars My spouse, at times a development researcher of high-net worth individuals, was given this book because she was the 'numbers' person in the office. Since my undergraduate was focused on lab-design, including analysis of results using ...

ARIMA is an acronym for Autoregressive Integrated Moving Average, and one explanation describes ARIMA models as... ...another approach to time series forecasting. Exponential smoothing and ARIMA models are the two most widely-used approaches to time series forecasting, and provide complementary approaches to the problem. While exponential smoothing models were based on a description of trend and seasonality in the data, ARIMA models aim to describe the autocorrelations in the data. A detailed technical discussion can be found on Wikipedia. For the first exploration I developed several ARIMA models using financial data, varying the parameters, noted as P, D and Q. A general overview of the parameters is from Wikipedia: p is the order (number of time lags) of the autoregressive model d is the degree of differencing (the number of times the data have had past values subtracted) q is the order of the moving-average model # filtered to start on a date Portfolio.filtere...

While developing these demonstrations in logistic regression and neural networks, I used and discovered some interesting methods and techniques: Better Methods A few useful commands and packages...: update.packages() for updating installed packages in one easy action as.formula() for creating a formula that I can reuse and update in one action across all my code sections View() for looking at data frames fourfoldplot() for plotting confusion matrices neuralnet for developing neural networks caret , used with nnet , to create predictive model plotnet() in NeuralNetTools, for creating attractive neural network models Resources that I used or that I would like to explore... MS Azure Notebooks , for working online with Python, R, and F#, all part of MS's data workflows Efficient R Programming , that seems to have many good tips on working with R Data Mining Algorithms in SSAS, Excel, and R , showing various algorithms in each technology R Documentation , a ...

This is an example of neural networks using the neuralnet package on one of my sample data sets, walking through a Pluralsight training series, Data Mining Algorithms in SSAS, Excel, and R . In terms of results, the regression primarily predicts voter leanings based on five (5) traits, openness, conscientiousness, extraversion, agreeableness, and neuroticism, although only the first two (2) traits have a significant impact. Logistic regression is about 85% predictive, and slightly better at predicting Red states over its ability in predicting Blue states. [1] "Logistic Regression (All) - Correct (%) = 0.854166666666667" [1] "Logistic Regression (Red) - Correct (%) = 0.892857142857143" [1] "Logistic Regression (Blue) - Correct (%) = 0.8" For the neuralnet package, I created a loop to vary the hidden layers and the number of repetitions, since this is such a small number of records. This is obviously much less predictive than logistic regressi...

This is an example of Support Vector Machines, using one of my usual data sets, as part of a Pluralsight training presentation, Data Mining Algorithms in SSAS, Excel, and R . In terms of results, the prediction primarily predicts voter leanings based on two (2) traits, openness and conscientiousness, and although using all five (5) factors improved the prediction quality, plotting that is problematic. For this, the model is 96% predictive of Republican outcomes, but only 66% accurate in predicting Democratic leaning. Politics.prediction Blue Red Blue 15 1 Red 5 27 The code is below, as are some related graphs. Source data is here . # Clear memory rm(list = ls()) # set Working directory getwd() setwd('../Data') # load data Politics.df <- read.csv("BigFiveScoresByState.csv", na.strings = c("", "NA")) # clean data - remove NULLs Politics.df <- na.omit(Politic...

I noticed this very simple, very powerful article by James Marquez, Seven Easy Graphs to Visualize Correlation Matrices in R , in the Google+ community, R Programming for Data Analysis , so thought to give it a try, since I started some of my current analyses a decade ago by generating correlation matrices in Excel, which I've sometimes redone and improved in R. Some of these packages are only designed for display, or as extensions to ggplot2: corrplot: Visualization of a Correlation Matrix GGally: Extension to 'ggplot2' ggcorrplot: Visualization of a Correlation Matrix using 'ggplot2' These two are focused on more complex analysis: PerformanceAnalytics: Econometric tools for performance and risk analysis psych: Procedures for Psychological, Psychometric, and Personality Research As for data, I used Hofstede's culture dimensions, limited to developed countries. Using a broader and larger set of of countries would significantly reduce the correlation...

This decision tree demo is similar to a prior one I've done, but in this case, it uses the party package, that produces much higher-quality graphics than rpart, at least when used with plot. Although the idea for this analysis is mine, this was done as part of work for a Pluralsight training presentation, Data Mining Algorithms in SSAS, Excel, and R . Source data is here . # # Load Data # # Set working directory setwd("../Data") getwd() # read from data frame BigFivByState.df <- read.table("BigFiveScoresByState.csv", header = TRUE, sep = ",") # # Run Analysis # # Load package, install.packages('party', dependencies = TRUE) library(party) # train the model BigFivByState.dt <- ctree(data = BigFivByState.df, Liberal ~ Openness + Conscientiousness + Extraversion + Neuroticism + Agreeableness) # plot result plot(BigFivByState.dt, uniform = TRUE, main = ...

As part of another Pluralsight training presentation, Data Mining Algorithms in SSAS, Excel, and R , I worked through various exercises, and from that I've adapated Naive Basyes to one of my existing data sets. The code is below, as are some related graphs. Overall, the percent correct predicted based on Big Five personality traits using the Naive Bayes calculation is 66%. Source data is here . # # Load & Explore Data # # read from data frame BigFivByState.df <- read.table("BigFiveScoresByState.csv", header = TRUE, sep = ",") # review data head(BigFivByState.df) nrow(BigFivByState.df) summary(BigFivByState.df) names(BigFivByState.df) # various aggregations # as "count this value" ~ grouped by this + this Liberal.dist <- aggregate(State ~ Liberal, data = BigFivByState.df, FUN = length) head(Liberal.dist) RedBlue.dist <- aggregate(State ~ Politics, data = BigFivByState.df, FU...

As part of a Pluralsight training presentation, Understanding and Applying Logistic Regression , students worked through various exercises, one of which was predicting stock price changes, up or down, on Google, using Google and Spyder closing prices. As an ordered list of actions: Load data - Yahoo financials for each day for 5 years, taking only date and closing price for this analysis Transform sources: Merge sources, change column headings, cast the Date column as DATE type, sort descending Perform logistic regression Create a frame of actual versus predicted changes, and add a column for the correct/incorrect prediction result Find percent correct, on whether the price moved correctly up or down As a result, the lagged Google and SPY prices accurately predict next day prices about 63% of the time. Source data is here . # Clear memory rm(list = ls()) # Set working directory setwd("../Data") getwd() # load data # Data is Yahoo finan...

Thinking about social dimensions and innovation, it occurred to me that there might be a relationship with masculinity, but then quickly dismissed it, considering it much more likely to be predicated on science/math education. Even then, other cultural elements might be more likely correlated. What follows is an exploration of various correlations with patents per capita. Although Hofstede's Cultural Dimensions did have a significant correlation with patents per capita, somewhat surprisingly, PISA scores by country, education, nor average IQ, had a strong relationship with patent production, although if Asia was included, statistically it would. Notes: I often exclude Asia from analyses, as the initial driver of this work was looking at cultures that are similar, to tease out social effects. That is also why I ignore looking at all countries, as some relationships across the entire world disappear when limited to just developed economies. As an example, the value of work and...

Responses to a post on the correlation between country-average IQ and responding yes to a question on if religion matters are inversely correlated, but not strongly so, prompted me to dig up a more significant issue, the relationship between religiosity and inequality, as measured by the Gini coefficient. The correlation is quite high, at about .7, although this really says nothing about the cause, if religious countries tend toward inequality because of general tendencies, or if inequality drives people to religion, as a salve against suffering. In truth, they could both be reflective of some other aspect of a country, and not in any way causative. Example Code # Correlations on ReligionMatters and Gini Coeficients oecdData <- read.table("OECD - Quality of Life.csv", header = TRUE, sep = ",") #names(oecdData) religionMattersVector <- oecdData$ReligionMatters giniVector <- oecdData$Gini cor.test(giniVector, religionMattersVect...

I was reminded this morning by an article, New Study: Religious People Are Less Smart but Atheists Are Psychopaths . but realized that there are many studies showing the religious to be less intelligent, but rarely is this ever published. It would 'bite' most of the population, liberal and conservative alike. Note : I plan on redoing this with more solid numbers, and possibly different measures of religiosity. Example Code # Correlations on ReligionMatters and Average IQ oecdData <- read.table("OECD - Quality of Life.csv", header = TRUE, sep = ",") iqVector <- oecdData$IQ religionMattersVector <- oecdData$ReligionMatters cor.test(iqVector, religionMattersVector) lm1 <- lm(iqVector ~ religionMattersVector) summary(lm1) # Plot the chart. plot(iqVector, religionMattersVector, col = "blue", main = "IQ ~ Religion Matters Vector" , abline(lm(religionMattersVector ~ iqVector),...

This was a continuation of prior explorations, logistic regression predicting Red/Blue state dichotomy by income or by personality. This uses the same five personality dimensions, but instead builds a decision tree. Of the Big Five traits, only two were found to useful in the decision tree, conscientiousness and openness. Links to sample data, as well as to source references, are at the end of this entry. Example Code # Decision Tree - Big Five and Politics library("rpart") # grow tree input.dat <- read.table("BigFiveScoresByState.csv", header = TRUE, sep = ",") fit <- rpart(Liberal ~ Openness + Conscientiousness + Neuroticism + Extraversion + Agreeableness, data = input.dat, method="poisson") # display the results printcp(fit) # visualize cross-validation results plotcp(fit) # detailed summary of splits summary(fit) # plot tree plot(fit, uniform = TRUE, main = "Class...

This was a little bit of fun, after reading a few more chapters of Text Analysis with R for Students of Literature . Spinoza is a current interest, as I am also reading Radical Enlightenment: Philosophy and the Making of Modernity 1650-1750 . Example Code (Common to Subsections) # Text for this can be acquired as below Project Gutenberg, as below # http://www.gutenberg.org/cache/epub/989/pg989.txt, # or via a Sample Data at the end of this post: textToRead = 'pg989.txt' # Text for this can be acquired via Matthew Jockers site, as below, # http://www.matthewjockers.net/macroanalysisbook/expanded-stopwords-list/, # or via a Sample Data at the end of this post: exclusionFile = 'StopList_Extended.csv' # Read Text text.scan <- scan(file = textToRead, what = 'char') text.scan <- tolower(text.scan) # Create list text.list <- strsplit(text.scan, '\\W+', perl = TRUE) text.vector <- unlist(text.list) # Create ...

As part of the Data science is OSEMN module for Obtaining Data I worked through the exercises. Example Code # Exercises """http://people.duke.edu/~ccc14/sta-663/DataProcessingSolutions.html#exercises""" """1. Write the following sentences to a file “hello.txt” using open and write. There should be 3 lines in the resulting file. Hello, world. Goodbye, cruel world. The world is your oyster.""" str = 'Data\Test.txt' f = open(str, 'w') f.write('Hello, world.\r') f.write('Goodbye, cruel world.\r') f.write('The world is your oyster.') # Writes same thing, only in one statement f.write('\rHello, world.\rGoodbye, cruel world.\rThe world is your oyster.') f.close() with open(str, 'r') as f: content = f.read() print(content) """2. Using a for loop and open, print o...

I recently finished a tutorial I was working through in R - I have certainly not exhausted exploring R, just taking time to let it settle in - and am thinking of working with Python for a while. That would include using  Python Tools for Visual Studio :

This is a significant result, but instead of a logistic regression looking at the income average per state and the likelihood of being a Democratic state, it uses Chi-Square. Interpreting this is pretty straightforward, in that liberal states typically have cities and people that earn more money. When using adjusted incomes, by cost of living, this difference disappears. Example Code # R - Chi Square rm(list = ls()) stateData <- read.table("CostByStateAndSalary.csv", header = TRUE, sep = ",") # Create vectors affluence.median <- median(stateData$Y2014, na.rm = TRUE) affluence.v <- ifelse(stateData$Y2014 > affluence.median, 1, 0) liberal.v <- stateData$Liberal # Solve pol.Data = table(liberal.v, affluence.v) result <- chisq.test(pol.Data) print(result) print(pol.Data) Example Results Pearson's Chi-squared test with Yates' continuity correction data: pol.Data X-squared = 12.672, ...