Data Analytics Workouts

Although I've made many posts on my Data Analytics Workouts site, some generated more interest than others - nothing here was virally popular - and below are a handful of the most popular, sorted in descending order of views: Charting Correlation Matrices in R (877) Neural Networks (Part 4 of 4) - R Packages and Resources (507) Performance Improvements in R: Vectorization & Memoisation (322) Naive Bayes on Political Outcome Based on State-level Big Five Assessment (263) Decision Trees on Political Outcome Based on State-level Big Five Assessment (262) F# is Part of Microsoft's Data Science Workloads (223) Logistic Regression on Stock Data using Google and SPY (SPDR S&P 500) (196) Using Visual Studio Team Services for Personal Development (194) Microsoft Azure Notebooks - Live code - F#, R, and Python (186) Efficient R Programming - A Quick Review (178) Patents Per Capita and Hofstede's Cultural Dimensions (167) Comparing Performance in R Usin...

More as an experiment, rather an attempt at sharing code and ideas, I created a Pinterest board devoted to my personal data analytics work, done with Python, R, or F#, as well as reviews of books, and was quite surprised with the result. The graphics could do with optimization, but otherwise...

As part of reading Fundamentals of Deep Learning: Designing Next-Generation Machine Intelligence Algorithms by Nikhil Buduma , I was expecting to work through some of the code examples with my own data, and for the examples, it recommended TensorFlow , which brings up competing alternatives, a primary one being Microsoft Cognitive Toolkit . Over the next few weeks, I will start exploring both in Python, as well as publishing some of the related work. A minor note, the darksigma/Fundamentals-of-Deep-Learning-Book: Code companion to the O'Reilly "Fundamentals of Deep Learning" book is available on GitHub.

As part of my self-development, I wanted to rework a script, which are typically one-offs, and turn it into a reusable component, although there are existing packages for VaR. As such, this is currently a work in progress. This code is a Python-based class for VaR calculations , and for those unfamiliar with VaR, it is an acronym for value at risk, the worst case loss in a period for a particular probability. It is a reworking of prior work with scripted VaR calculations , implementing various high-level good practices, e.g., hiding/encapsulation, do-not-repeat-yourself (DRY), dependency injection, etc. Features: Requires data frame of stock returns, factor returns, and stock weights Expose a method to calculate and return a single VaR number for different variance types Expose a method to calculate and return an array of VaR values by confidence level Expose a method to calculate and plot an array of VaR values by confidence level Still to do: Dynamic factor usage Note...

The following modules linked below are based on a Pluralsight course, Understanding and Applying Financial Risk Modeling Techniques , and while the code itself is nearly verbatim, this is mostly for my own development, working through the peculiarities of Value at Risk (VaR) in both R and Python, and adding commentary as needed. The general outline of this process is as follows: Load and clean Data Calculate returns Calculate historical variance Calculate systemic, idiosyncratic, and total variance Develop a range of stress variants, e.g. scenario-based possibilities Calculate VaR as the worst case loss in a period for a particular probability The modules: In R: Financial Risk - Calculating Value At Risk (VaR) with R In Python: Financial Risk - Calculating Value At Risk (VaR) with Python

As part of understanding neural networks I was reading Make Your Own Neural Network by Tariq Rashid. A review is below: Make Your Own Neural Network by Tariq Rashid My rating: 4 of 5 stars The book itself can be painful to work through, as it is written for a novice, not just in algorithms and data analysis, but also in programming. For the neural network aspect, it jumped between overly simplistic and complicated, while providing neither in enough detail. That said, by the end I found it a worthwhile dive into neural networks, since once it got to the programming structure, it all made sense, but only because I stuck with it. View all my reviews

Introduction As part of understanding neural networks I was reading Make Your Own Neural Network by Tariq Rashid. The book itself can be painful to work through, as it is written for a novice, not just in algorithms and data analysis, but also in programming. Although the code is a verbatim transcription from the text (see Source section), I published it to better understand how neural networks are designed, made easy by the use of a Jupyter Notebook, not to present this as my own work, although I do hope that this helps others develop their talents with data analytics.  Github Source:  AzureNotebooks/Basic Three Layer Neural Network in Python.ipynb at master · JamesIgoe/AzureNotebooks (github.com) Overview The code itself develops as follows: Constructor set number of nodes in each input, hidden, output layer link weight matrices, wih and who weights inside the arrays are w_i_j, where link is from node i to node j in the next layer set learning rate activat...

My list is kind of paltry, but I’ve sat through others or started many but decided against finishing. The best courses I’ve finished have been along the lines of project management: Project Management for Software Engineers Project 2013 Fundamentals for Business Professionals I’ve also sat through this, and useful, although very rudimentary: Creating and Leading Effective Teams for Managers I do my own reading for data science, and have my own side projects, but I’ve also taken some data science courses via Pluralsight. The beginner demos are done well, although less informative than the intermediate ones, which are ultimately more useful. For the latter, I typically do simultaneous coding on my own data sets, which helps learn the material. Beginner Understanding Machine Learning Understanding Machine Learning with R Intermediate Understanding and Applying Logistic Regression (using Excel, Python, or R) Data Mining Algorithms in SSAS, Excel, and R ...

I was exploring Jupyter notebooks , that combines live code, markdown and data, through Microsoft's implementation, known as MS Azure Notebooks , putting together a small library of R and F# notebooks . As Microsoft's FAQ for the service describes it as : ...a multi-lingual REPL on steroids. This is a free service that provides Jupyter notebooks along with supporting packages for R, Python and F# as a service. This means you can just login and get going since no installation/setup is necessary. Typical usage includes schools/instruction, giving webinars, learning languages, sharing ideas, etc. Feel free to clone and comment... In R Azure Workbook for R - Memoisation and Vectorization Charting Correlation Matrices in R In F# Charnownes Constant in FSharp.ipynb Project Euler - Problems 18 and 67 - FSharp using Dynamic Programming

As part of the Data science is OSEMN module for Obtaining Data I worked through the exercises. Example Code # Exercises """http://people.duke.edu/~ccc14/sta-663/DataProcessingSolutions.html#exercises""" """1. Write the following sentences to a file “hello.txt” using open and write. There should be 3 lines in the resulting file. Hello, world. Goodbye, cruel world. The world is your oyster.""" str = 'Data\Test.txt' f = open(str, 'w') f.write('Hello, world.\r') f.write('Goodbye, cruel world.\r') f.write('The world is your oyster.') # Writes same thing, only in one statement f.write('\rHello, world.\rGoodbye, cruel world.\rThe world is your oyster.') f.close() with open(str, 'r') as f: content = f.read() print(content) """2. Using a for loop and open, print o...

OESMN is an acronym, for the elements of data science: Obtaining data Scrubbing data Exploring data Modeling data iNterpreting data As part of the Data science is OSEMN module  for Obtaining Data I performed the examples myself, as well as with variants. Example Code # Data science is OSEMN """http://people.duke.edu/~ccc14/sta-663/DataProcessingSolutions.html#data-science-is-osemn""" # Acquiring data """Plain text files We can open plain text files with the open function. This is a common and very flexible format, but because no structure is involved, custom processing methods to extract the information needed may be necessary. Example 1: Suppose we want to find out how often the words alice and drink occur in the same sentence in Alice in Wonderland. This example uses Gutenberg: http://www.gutenberg.org/wiki/Main_Page, a good source for files""" # the data file is linked below...

I worked through Exercises section of the Computational Statistics in Python tutorial . Below are the results, with some variations i generated to get a better understanding of the solutions: # Exercises """http://people.duke.edu/~ccc14/sta-663/FunctionsSolutions.html#exercises""" """1. Rewrite the following nested loop as a list comprehension ans = [] for i in range(3): for j in range(4): ans.append((i, j)) print ans""" arr = [(i,j) for i in range(3) for j in range(4)] arr """2. Rewrite the following as a list comprehension ans = map(lambda x: x*x, filter(lambda x: x%2 == 0, range(5))) print ans""" arr = [x**2 for x in range(5) if x%2 == 0] """3. Convert the function below into a pure function with no global variables or side effects x = 5 def f(alist): for i in r...

I worked through functools and itertools sections of the Computational Statistics in Python tutorial , and I found these promisingly powerfuul for data modeling and functional programming: # The functools module """The most useful function in the functools module is partial, which allows you to create a new function from an old one with some arguments “filled-in”.""" from functools import partial def power_function(power, num): """power of num.""" return num**power square = partial(power_function, 2) cube = partial(power_function, 3) quad = partial(power_function, 4) # The itertools module """This provides many essential functions for working with iterators. The permuations and combinations generators may be particularly useful for simulations, and the groupby gnerator is useful for data analyiss.""" from itertools import cyc...

I worked through Iterators, Generators and Decorators sections of the Computational Statistics in Python tutorial , and although this code is similar to the tutorial, it is modified in ways to make it retainable to myself: # Iterators can be created from sequences with the built-in function iter() """http://people.duke.edu/~ccc14/sta-663/FunctionsSolutions.html#iterators""" xs = [n for n in range(10)] x_iter = iter(xs) [x for x in x_iter] for x in x_iter: print(x) # Generators """http://people.duke.edu/~ccc14/sta-663/FunctionsSolutions.html#generators""" # Functions containing the 'yield' keyword return iterators # After yielding, the function retains its previous state def limit_seats(n): """Limits number and counts down""" for i in range(n, 0, -1): yield i counter = limit_seats(10) for count in ...

I worked through recursion in the Computational Statistics in Python tutorial , getting up to speed on Python, and although this code is similar to the tutorial, I did try to 'make it mine': # Recursion """http://people.duke.edu/~ccc14/sta-663/FunctionsSolutions.html#recursion""" # Correct Example def fibonacci(n): if n==0 or n==1: return 1 else: return fibonacci(n-1) + fibonacci (n-2) [fibonacci(n) for n in range(10)] def fibonacciWithCache(n, cache={0:1, 1:1}): try: return cache[n] except: cache[n] = fibonacci(n-1) + fibonacci (n-2) return cache[n] [fibonacciWithCache(n) for n in range(10)] # Correct Example def factorial(n): if n==0: return 1 else: return n * factorial(n-1) # Correct Example, with caching def factorialWithCache(n, cache={0:1}): try: return cache[n] except KeyError: cache[...

I am currently working through Computational Statistics in Python 's module Functions are first class objects , and for this - I have worked with delegates and higher order functions in other languages - I put together this example to better grasp higher order functions in Python: def power_function(num, power): """power of num.""" return num**power def add_function(num, more): """sum of num.""" return num+more def runFunction(fx, x, y): return fx(x,y) num = 2 powers = range(0,64) arrPower = list(map(lambda x: runFunction(power_function, num, x), powers)) arrAdd = list(map(lambda x: runFunction(add_function, num, x), powers)) arrPower arrAdd

To prepare for further exploration in data analytics, and give myself some time to absorb R, I decided to work in Python for a while. For this, I worked through Computational Statistics in Python module Introduction to Python . These are some of the code produced to solve the exercises: # Tutorial Answers """http://people.duke.edu/~ccc14/sta-663/IntroductionToPythonSolutions.html""" """1. Solve the FizzBuzz problem “Write a program that prints the numbers from 1 to 100. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multiples of both three and five print “FizzBuzz”.""" def fizz_buzz(begin, end): """Write a program that prints the numbers from 1 to 100. But for multiples of three print “Fizz” instead of the number and for the multiples of five print “Buzz”. For numbers which are multi...

I was working through a tutorial in Python, but found this one, Computational Statistics in Python from Duke , much better. It has exercises, which increase retention, and the material is well-presented.

I recently finished a tutorial I was working through in R - I have certainly not exhausted exploring R, just taking time to let it settle in - and am thinking of working with Python for a while. That would include using  Python Tools for Visual Studio :