Skip to main content

Plotting Text Frequency and Distribution using R for Spinoza's A Theological-Political Treatise [Part I]

This was a little bit of fun, after reading a few more chapters of Text Analysis with R for Students of Literature. Spinoza is a current interest, as I am also reading Radical Enlightenment: Philosophy and the Making of Modernity 1650-1750.

Example Code (Common to Subsections)

 # Text for this can be acquired as below Project Gutenberg, as below
 # or via a Sample Data at the end of this post:
 textToRead = 'pg989.txt'
 # Text for this can be acquired via Matthew Jockers site, as below,
 # or via a Sample Data at the end of this post:
 exclusionFile = 'StopList_Extended.csv'
 # Read Text  
 text.scan <- scan(file = textToRead, what = 'char')  
 text.scan <- tolower(text.scan)  
 # Create list  
 text.list <- strsplit(text.scan, '\\W+', perl = TRUE)  
 text.vector <- unlist(text.list)  
 # Create exclusion list  
 exclusion.file <- scan(exclusionFile, what='char')  
 exclusion.v <- strsplit(exclusion.file, ',', perl = TRUE)  
 exclusion.v <- tolower(exclusion.v)  
 # Remove exclusions
 not.exclusions.v <- which(!text.vector %in% exclusion.v)  
 text.vector <- text.vector[not.exclusions.v] 

 # Create sorted list  
 text.frequency = table(text.vector)  
 text.frequency.sorted = sort(text.frequency, decreasing = TRUE)  

Example Code for Frequency Plot

 # Plots  
 # Types: p = point, l = line, b = both, h = hist, s = stairs  
   , type="s"   
   , main = "Word Frequency"  
   , xlab = "Word Rank (Decreasing)"  
   , ylab = "Frequency")

Example Code for Dispersion Plots

 n.time.v = seq(1:length(text.vector))  
 textToPlot = 'Spinoza\'s A Theological-Political Treatise [Part I]'  
 wordToPlot = 'god'  
 wordToPlot = 'law'  
 wordToPlot = 'prophets'  
 wordToPlot = 'lord'  
 word.v <- which(text.vector == wordToPlot)  
 w.count.v <- rep(NA, length(n.time.v))  
 w.count.v[word.v] <- 1  
   , main = paste('Dispersion Plot of', wordToPlot, 'in', textToPlot, sep = " ")  
   , xlab = "Document Time"  
   , ylab = wordToPlot  
   , type = "h"  
   , ylim = c(0, 1), yaxt = 'n')  
Sample Data

Popular posts from this blog

Decision Tree in R, with Graphs: Predicting State Politics from Big Five Traits

This was a continuation of prior explorations, logistic regression predicting Red/Blue state dichotomy by income or by personality. This uses the same five personality dimensions, but instead builds a decision tree. Of the Big Five traits, only two were found to useful in the decision tree, conscientiousness and openness.

Links to sample data, as well as to source references, are at the end of this entry.

Example Code

# Decision Tree - Big Five and Politics library("rpart") # grow tree input.dat <- read.table("BigFiveScoresByState.csv", header = TRUE, sep = ",") fit <- rpart(Liberal ~ Openness + Conscientiousness + Neuroticism + Extraversion + Agreeableness, data = input.dat, method="poisson") # display the results printcp(fit) # visualize cross-validation results plotcp(fit) # detailed summary of splits summary(fit) # plot tree plot(fit, uniform = TRUE, main = "Classific…

Mean Median, and Mode with R, using Country-level IQ Estimates

Reusing the code posted for Correlations within with Hofstede's Cultural Values, Diversity, GINI, and IQ, the same data can be used for mean, median, and mode. Additionally, the summary function will return values in addition to mean and median, Min, Max, and quartile values:

Example Code
oecdData <- read.table("OECD - Quality of Life.csv", header = TRUE, sep = ",") v1 <- oecdData$IQ # Mean with na.rm = TRUE removed NULL avalues mean(v1, na.rm = TRUE) # Median with na.rm = TRUE removed NULL values median(v1, na.rm = TRUE) # Returns the same data as mean and median, but also includes distribution values: # Min, Quartiles, and Max summary(v1) # Mode does not exist in R, so we need to create a function getmode <- function(v) { uniqv <- unique(v) uniqv[which.max(tabulate(match(v, uniqv)))] } #returns the mode getmode(v1)
Example Results
> oecdData <- read.table("OECD - Quality of L…

Chi-Square in R on by State Politics (Red/Blue) and Income (Higher/Lower)

This is a significant result, but instead of a logistic regression looking at the income average per state and the likelihood of being a Democratic state, it uses Chi-Square. Interpreting this is pretty straightforward, in that liberal states typically have cities and people that earn more money. When using adjusted incomes, by cost of living, this difference disappears.

Example Code
# R - Chi Square rm(list = ls()) stateData <- read.table("CostByStateAndSalary.csv", header = TRUE, sep = ",") # Create vectors affluence.median <- median(stateData$Y2014, na.rm = TRUE) affluence.v <- ifelse(stateData$Y2014 > affluence.median, 1, 0) liberal.v <- stateData$Liberal # Solve pol.Data = table(liberal.v, affluence.v) result <- chisq.test(pol.Data) print(result) print(pol.Data)
Example Results
Pearson's Chi-squared test with Yates' continuity correction data: pol.Data X-squared = 12.672, df …