Data Analytics Workouts

I've been interested in Hofstede's work since B-school, back in the early second millennia, and at one time considered publishing using his country characteristics as predictors for economic and social welfare outcomes. Nowadays, I use the results of his analyses frequently in small R programming demonstrations. He's an interesting researcher, who's done important work, as The Economist article describes : The man who put corporate culture on the map—almost literally—Geert Hofstede (born 1928) defined culture along five different dimensions. Each of these he measured for a large number of countries, and then made cross-country comparisons. In the age of globalisation, these have been used extensively by managers trying to understand the differences between workforces in different environments. The Economist article give a fuller picture of Geert Hofstede , and anyone interested in reading one of his works might enjoy Cultures and Organizations: Software of the Mi...

I was exploring Jupyter notebooks , that combines live code, markdown and data, through Microsoft's implementation, known as MS Azure Notebooks , putting together a small library of R and F# notebooks . As Microsoft's FAQ for the service describes it as : ...a multi-lingual REPL on steroids. This is a free service that provides Jupyter notebooks along with supporting packages for R, Python and F# as a service. This means you can just login and get going since no installation/setup is necessary. Typical usage includes schools/instruction, giving webinars, learning languages, sharing ideas, etc. Feel free to clone and comment... In R Azure Workbook for R - Memoisation and Vectorization Charting Correlation Matrices in R In F# Charnownes Constant in FSharp.ipynb Project Euler - Problems 18 and 67 - FSharp using Dynamic Programming

Efficient R Programming: A Practical Guide to Smarter Programming by Colin Gillespie My rating: 5 of 5 stars Simply a great book, chock full of tips and techniques for improving one's work with R. View all my reviews

Full of potential coding improvements, Efficient R Programming: A Practical Guide to Smarter Programming , the book makes two suggestions that are notable. Vectorization, explained here and here , and memoisation , caching prior results albeit with additional memory use, were relevant and significant. What follows is a demonstration of the speed improvements that might be achieved using these concepts. ################################ # performance # vectorization and memoization ################################ # clear memory between changes rm(list = ls()) #load memoise #install.packages('memoise') library(memoise) # create test function monte_carlo = function(N) { hits = 0 for (i in seq_len(N)) { u1 = runif(1) u2 = runif(1) if (u1 ^ 2 > u2) hits = hits + 1 } return(hits / N) } # memoise test function monte_carlo_memo <- memoise(monte_carlo) # vectorize function monte_carlo...

While developing these demonstrations in logistic regression and neural networks, I used and discovered some interesting methods and techniques: Better Methods A few useful commands and packages...: update.packages() for updating installed packages in one easy action as.formula() for creating a formula that I can reuse and update in one action across all my code sections View() for looking at data frames fourfoldplot() for plotting confusion matrices neuralnet for developing neural networks caret , used with nnet , to create predictive model plotnet() in NeuralNetTools, for creating attractive neural network models Resources that I used or that I would like to explore... MS Azure Notebooks , for working online with Python, R, and F#, all part of MS's data workflows Efficient R Programming , that seems to have many good tips on working with R Data Mining Algorithms in SSAS, Excel, and R , showing various algorithms in each technology R Documentation , a ...

As part of writing analyses using neural networks I thought of displaying a confusion matrix, but went looking for something more. What I found was either ugly and simple, or attractive but complicated. The following code demonstrates using fourfoldplot, with original insight gained from fourfoldplot: A prettier confusion matrix in base R , and a great documentation page, R Documentation's page for fourfoldplot . The code is below, as is the related graph. Source data can be found here . ############################################### # Load and clean data ################################################ Politics.df <- read.csv("BigFiveScoresByState.csv", na.strings = c("", "NA")) Politics.df <- na.omit(Politics.df) ################################################ # Neural Net with nnet and caret ################################################ # load packages library(nnet) library(caret) # set ...

This post is a demonstration using the caret and nnet packages on aggregate Big Five traits per state and political leanings, and a continuation of a series: Neural Networks (Part 1 of 4) - Logistic Regression and neuralnet on State 'Personality' and Political Outcomes Neural Networks (Part 2 of 4) - caret and nnet on State 'Personality' and Political Outcomes This process using a neural net process to train then test on data. The neural net itself would be envisioned as follows: Initially, the results seemed promising, with a prediction accuracy of ~.85, further analysis revealed that the bulk of accurate predictions were those that had no price change, and that price change classes were often inaccurate. I excluded security types without price change, and the average dropped even further. I then created a loop that calculated test and training data by security type, and those result indicated that the neural net very right and very wrong, for price change s...