Developers in New York City by Zip Code

Photo by Lukas Kloeppel from Pexels
In 2016, after reading a Dice Insight article, I downloaded data that had technology professional numbers by zip code, along with density. A recent NY Times article How Big Tech Is Turning New York Into a Silicon Valley Rival prompted me to resurrect the data, decorate it a bit with neighborhood names, and then import it into Google Maps, which was surprisingly easy.

The original data is available in Excel.

Cultural Dimensions and Coffee Consumption

Photo by Viktoria Alipatova from Pexels
Responding to a Treehugger article, Why Americans will never love tea as much as coffee, I initially wrote my personal preferences for tea and coffee, ending with, BTW, this has just given me an idea for comparing Hofstede's cultural dimensions and coffee and tea consumption. Afterward, I did some analysis in Excel, then ran the same processes in R using Visual Studio, then converted that to a Jupyter Notebook on Microsoft's Azure Notebooks.

Although this analysis is limited to 45 countries that have Hofstede's Cultural Dimensions, as well as per capita consumption for both coffee and tea, it would seem that coffee consumption correlates with power distance, individuality, and masculinity. Tea had small correlations with the dimensions and sometimes in the same direction as coffee. A fuller analysis is available on Microsoft's Azure Notebook, but some quick findings:
Higher power distance, lower coffee consumption: -.63 Higher…

VBA versus .NET

I was recently messaged by someone on LinkedIn, and since my response seemed full enough, I thought I'd share.

Question I see that you also program in VBA but you have made the jump to .NET. Unfortunately, I have found C#/Excel coding to be quite slow and just wanted to hear about your experiences. Responses Slow? It depends on what you mean. Honest, I have had to make the pitch when building apps that it should be in .NET rather than VBA for speed. One particular app had a form that needed to fill about 20 dropdowns on load, so using async operations were essential. That same app, while executing one SQL statement in the foreground, also executed 2 background statements that filled panels. It wouldn't have performed well if done in VBA.

If you mean that it takes longer, then yes, but that is a necessity for good code anyway. If you only need a local operation, non-threaded, that doesn't need to be used across the enterprise, VBA can make sense, but with .NET comes…

A Journey — if You Dare — Into the Minds of Silicon Valley Programmers

My responses in a NY Times comment section for the book, Coders: The Making of a New Tribe and the Remaking of the World by Clive Thompson:

#1 - Link

Although I've been a software developer for 15 years, and for longer alternating between a project manager, team lead, or analyst, mostly in finance, and now with a cancer center, I found it funny that you blame the people doing the coding for not seeing the harm it could cause. First, most scientific advancement has dark elements, and it is usually not the science but how it is used and sold by business people that is the problem. This leads to the second problem, in that it is not coding that is in itself problematic, but specifically how technology is harnessed to sell. It is normal and desirable to track users, to log actions, to collect telemetry, so as to monitor systems, respond to errors, and to develop new features, but that normal engineering practice has been used to surveil users for the purpose of selling. Blaming coders…

AI in Software Development

Even before AI, I would have thought that work being done now would be automated, and of course, AI will replace some work - since developers automate tasks themselves, using rules, patterns, and processes - but the idea is always to stay ahead of the 'crushing wave' of new tech, often automating oneself out fo a job, thereby keeping your job...

BTW, this mentions interesting tools leveraging AI to help coders, rather than simply replacing them, since the latter is not currently a realistic scenario.

Source:Will A.I. Take Over Your Programming Job?

Complexity: A Guided Tour

Complexity: A Guided Tour by Melanie Mitchell
My rating: 4 of 5 stars

I enjoy reading in systems and complexity, and this was a nice addition to my shelf, with a slightly different take than other books. I found a few areas in the first half a bit tedious, overly long, repetitive, and not illuminating, but generally, it's a great overview of seminal work and very thought-provoking. The first half overlaps but nicely differs from other books I've read, covering things like chaos and information processing, and the latter half of the book I found more engaging, focused on models, computation, network science, and scaling. As mentioned, although I found the first half a bit of a slog at times, the second half was very engaging.

View all my reviews

Self Review for 2017

My corporate annual review period recently passed, and I was reminded of all the skills developed and completed tasks over the past year, both in and out of work. Sincerely, remembering what I've done over the past year makes me feel good, and really reminds me of how much I enjoy learning.

Video Courses Although largely focused on reading to learn, I do partake of various streaming video resources via Pluralsight. The courses I've completed this past year:

Multiple courses on management and leadershipDeveloper to ManagerTechnical Writing: Documentation on Software Projects Quantitative & AI-related courses, accompanied by work in R, Python, or VBAUnderstanding Machine LearningUnderstanding Machine Learning with RData Mining Algorithms in SSAS, Excel, and RUnderstanding and Applying Logistic RegressionUnderstanding and Applying Factor Analysis and PCAUnderstanding and Applying Financial Risk Modeling Techniques
Software DevelopmentEfficient R Programming: A Practical…

Review: Complex Adaptive Systems: An Introduction to Computational Models of Social Life

Complex Adaptive Systems: An Introduction to Computational Models of Social Life by John H. Miller
My rating: 4 of 5 stars

A thought-provoking introductory exploration to modeling social systems, covering ideas for rule-based agents within a variety of rule-based systems, moving onto evolutionary-like automata and organization of agents to solve problems. Underlying some of the ideas, one could see references to deeper concepts, e.g., nonlinearity, attractors, emergence, and complexity, none of which was explained explicitly. At times, I did find the writing tedious, as some ideas were too obvious to spend time detailing, but overall, a well-written easy to digest text.

View all my reviews

My Most Popular Posts of 2017


Software engineers will be obsolete by 2060

In response to an article on Medium, Software engineers will be obsolete by 2060, I responded with the following:

Interesting article on The Economist titled Automation on Automation Angst,, that looks at several publications that look at the historical effects of automation, and although there is always a fear of being replaced, ultimately more jobs are created than destroyed. Software engineers disappear? So what! There will be other jobs, with different titles, and in the interim, the more people use tech, the more there will be a need for software engineers.

Because of this, a person asked for my opinion on maintaining their career as a .NET developer, to which I responded:

Although I am a .NET developer as well, I focus on expanding my project management and leadership skills, as well as developing skills in AI/ML. Rather than bore with all the details of my background, here is what I think:
You should develop your skills in AI/ML, if only by f…

Using Visual Studio Team Services for Personal Development


Microsoft provides free access to its online Visual Studio Team Services (VSTS), and for some time I've been using the service, I've wanted to restructure my code hierarchy, and recent changes in my work environment, automated build and deployment using Octopus, nudged me to finally take the task on, so in the past few weeks I've:
Restructured my Code library into one big project with sub-projects for Development, Websites, and WorkDeveloped my Work hierarchy of Epics, Features, Stories and Tasks, along with queries and sprint boardsAutomated all of my builds via check-in, adding extensions to evaluate code and build qualityDeveloped a dashboard to oversee the status of work
Code Library

I was frustrated with the limitations of working with my code library, and after reading opinions on best practices, I settled on one big project for my code, which I assumed would make it easier to manage my time and energy, and give me a global view of my individual projects.


Code: Pinterest as a Publication Channel for Data Analytics

More as an experiment, rather an attempt at sharing code and ideas, I created a Pinterest board devoted to my personal data analytics work, done with Python, R, or F#, as well as reviews of books, and was quite surprised with the result.

The graphics could do with optimization, but otherwise...

Deep Learning and Toolkits

As part of reading Fundamentals of Deep Learning: Designing Next-Generation Machine Intelligence Algorithms by Nikhil Buduma, I was expecting to work through some of the code examples with my own data, and for the examples, it recommended TensorFlow, which brings up competing alternatives, a primary one being Microsoft Cognitive Toolkit.

Over the next few weeks, I will start exploring both in Python, as well as publishing some of the related work.

A minor note, the darksigma/Fundamentals-of-Deep-Learning-Book: Code companion to the O'Reilly "Fundamentals of Deep Learning" book is available on GitHub.

Principal Component Analysis (PCA) on Stock Returns in R

Principal Component Analysis Principal Component Analysis is a statistical process that distills measurement variation into vectors with greater ability to predict outcomes utilizing a process of scaling, covariance, and eigendecomposition.
MS Azure Notebook The work for this is done in the following notebook, Principal Component Analysis (PCA) on Stock Returns in R, with detailed code, output, and charts. An outline of the notebook contents are below.
Overview of DemonstrationSupporting MaterialPluralsightExplained VisuallyWikipediaLoad Data: Format Data & SortPrep Data: Create ReturnsGenerate Principal ComponentsEigen Decomposition and Scree PlotCreate Principal ComponentsAnalysisFVX using PCA versus Logistic RegressionAlternative Libraries: Psych for the Social Sciences

Exercises in Programming Style by Cristina Videira Lopes

Exercises in Programming Style by Cristina Videira Lopes
My rating: 5 of 5 stars

An easily consumed, enjoyable read, and excellent review of the history of programming style, from older days of constrained memory and monolithic styles, through pipelining and object-oriented variants, to more recent patterns like model-view-controller (MVC), mapreduce, and representational state transfer (ReST). Along the way, each variant is described, along with its constraints, its history, and its context in systems design.

View all my reviews

Data Mining for Fund Raisers: How to Use Simple Statistics to Find the Gold in Your Donor Database Even If You Hate Statistics: A Starter Guide

This is a repost of a Goodreads' review I made in 2013, for a book I read in 2005, which seems relevant now, as the industry is adding a data-driven focus. Plus, the world is now being transformed by advances in artificial intelligence and machine learning (AI/ML), particularly deep learning, and the large data sets and complexity of donor actions should greatly benefit from analysis. Note, the tax changes for 2018 and beyond will increase the importance of major donors, attenuating the benefits of AI/ML, as data for high-net-worth individuals is sparse.

Data Mining for Fund Raisers: How to Use Simple Statistics to Find the Gold in Your Donor Database Even If You Hate Statistics: A Starter Guide by Peter B. Wylie

My rating: 4 of 5 stars

My spouse, at times a development researcher of high-net worth individuals, was given this book because she was the 'numbers' person in the office. Since my undergraduate was focused on lab-design, including analysis of results using statis…

Value-at-Risk (VaR) Calculator Class in Python

As part of my self-development, I wanted to rework a script, which are typically one-offs, and turn it into a reusable component, although there are existing packages for VaR. As such, this is currently a work in progress. This code is a Python-based class for VaR calculations, and for those unfamiliar with VaR, it is an acronym for value at risk, the worst case loss in a period for a particular probability. It is a reworking of prior work with scripted VaR calculations, implementing various high-level good practices, e.g., hiding/encapsulation, do-not-repeat-yourself (DRY), dependency injection, etc.

Requires data frame of stock returns, factor returns, and stock weightsExpose a method to calculate and return a single VaR number for different variance typesExpose a method to calculate and return an array of VaR values by confidence levelExpose a method to calculate and plot an array of VaR values by confidence level Still to do:
Dynamic factor usage Note: Data to valida…

Calculating Value at Risk (VaR) with Python or R

The following modules linked below are based on a Pluralsight course, Understanding and Applying Financial Risk Modeling Techniques, and while the code itself is nearly verbatim, this is mostly for my own development, working through the peculiarities of Value at Risk (VaR) in both R and Python, and adding commentary as needed.

The general outline of this process is as follows:
Load and clean Data Calculate returns Calculate historical variance Calculate systemic, idiosyncratic, and total variance Develop a range of stress variants, e.g. scenario-based possibilities Calculate VaR as the worst case loss in a period for a particular probability The modules:
In R: Financial Risk - Calculating Value At Risk (VaR) with R In Python: Financial Risk - Calculating Value At Risk (VaR) with Python

Review: Make Your Own Neural Network

As part of understanding neural networks I was reading Make Your Own Neural Network by Tariq Rashid. A review is below:

Make Your Own Neural Network by Tariq Rashid
My rating: 4 of 5 stars

The book itself can be painful to work through, as it is written for a novice, not just in algorithms and data analysis, but also in programming. For the neural network aspect, it jumped between overly simplistic and complicated, while providing neither in enough detail. That said, by the end I found it a worthwhile dive into neural networks, since once it got to the programming structure, it all made sense, but only because I stuck with it.

View all my reviews

Basic Three Layer Neural Network in Python

Introduction As part of understanding neural networks I was reading Make Your Own Neural Network by Tariq Rashid. The book itself can be painful to work through, as it is written for a novice, not just in algorithms and data analysis, but also in programming. Although the code is a verbatim transcription from the text (see Source section), I published it to better understand how neural networks are designed, made easy by the use of a Jupyter Notebook, not to present this as my own work, although I do hope that this helps others develop their talents with data analytics.

Overview The code itself develops as follows:
Constructor set number of nodes in each input, hidden, output layer link weight matrices, wih and who weights inside the arrays are w_i_j, where link is from node i to node j in the next layer set learning rate activation function is the sigmoid function Define the Training Function convert inputs list to 2d array calculate signals into hidden layer calculate…