browserify with p5.serialcontrol

i’m working on the p5.serialcontrol app. one thing i’d like to fix is the way we use the p5 libraries.

currently, the app is packaged with static versions of p5, p5.dom, and p5.serial in the ‘libs’ folder.

this is not ideal. for one, we’re not pulling the most up-to-date versions of those libraries. i also got confused and was updating the p5.serialport library that comes packaged with p5.serialcontrol when i should have been working from the main p5.serialport repo.

it would be great if we could link to the live repositories in package.json. i spent some time researching this today and it seems like the way to link in the package.json, then use those modules on the client side, would require browserify or webpack. this would let us use the node require() syntax outside the node (server-side) context. i think the steps are:

  • update the package.json
  • remove the scripts from index.html

  • rewrite the existing sketch.js file with var p5 = require(‘p5’) at the top, like this
  • install and configure browserify for the project, something like this
  • compile the new electron app

this would solve several problems:

  • get rid of the entire ‘libs’ folder, which would be less confusing for development
  • make sure we always have the most up-to-date versions of those libraries

p5 serial

there are three parts of p5 serial:

  • p5.serialport library
  • p5 serial server
  • GUI (electron)

i’ll be thinking about to improve these tools. here are things i need to look into to get started:

initial needs:

  • serial server to list IP address and port so users can connect to it from their p5 sketches
  • compile latest p5.serialcontrol and p5.serialport into new release
  • figure out to demonstrate this model within the interface:

autocomplete next steps

surya spent several hours with me helping me strategize about next steps for autocomplete. i’m quickly putting some notes here so i remember:

  1. write the tern definition file for *one* piece of the p5 source code *by hand*. this will give me the ability to see how autocomplete works when it does what i want to. he suggested p5.image because it has the most logical schema, i.e. a prototype with private methods. much of the rest of p5 is add-ons to one global p5 object.
  2. specifically, parse every line that starts p5.image.prototype as well as the @method for that object
  3. once that’s done, test the def file in the tern.js demo. like the one here but forked and cloned with the new p5 def file added.
  4. once that’s working, create a python script to read into every p5 source file and repeat the process i went through with p5.image.

Google Summer of Code Final Submission

My original proposal was to implement autocomplete in the p5.js web editor. Here’s an excerpt:

 

The p5 web editor is currently in development, with plans to launch in time for fall semester classes. Currently, there’s no autocomplete code functionality and it’s not possible to access documentation from within the editor. We could use tern.js to solve both of these problems. Tern.js is a code-analysis engine for Javascript that works with CodeMirror, the code editor behind the p5 web editor. In addition to autocomplete, tern.js supports shortcuts to documentation and the ability to easily rename all occurrences of a variable. 

 

The p5.js web editor uses React for the front-end and is built on top of CodeMirror, a code editor for the browser.

I’ve been trying to hook up Tern.js—”a standalone Javascript analysis engine”—to implement autocomplete.

Tern relies on a JSON file called a Tern definition file in order to “specify the types of things” you code and trigger different autocomplete functions accordingly. You can read more about that file here.

Part of getting Tern working with p5.js is creating a Tern definition file for the p5.js library so that, when you’re writing in the editor, Tern can analyze what you’re writing. Tern has a utility to do this, but it didn’t work with the p5.js library.

As a hacky workaround, I adapted some scripts that convert YUIDoc documentation into Tern definition files. Repos are linked below. Because p5.js documentation is generated with YUI/YUIDoc, this sort of worked: you can trigger docs on p5 functions and autocomplete p5 function names.

But there’s a scoping problem where methods and variables that should not be available to certain functions still appear as autocomplete options.

As of August 2017, I’m trying a new approach to generating the Tern definition file.

Repositories

  1. My branch of the p5 web editor connected to Tern: https://github.com/kaganjd/p5.js-web-editor/tree/autocomplete. To get it up and running, you’d follow steps in the README.
  2. Code to generate the Tern definition file for the p5.js library: https://github.com/processing/p5-tern.yuidoc. This is set up to be a module dependency of the main p5.js repository and executed with a Grunt task from there.
  3. Documentation re: errors when running the Tern definition generator code: https://gist.github.com/kaganjd/32c87c49fda5f61b78aa2b88993daca4
  4. Next steps: https://gist.github.com/kaganjd/1e033541fa00af3e760c0acddf6fc197

The workflow for this whole thing in the wild will be:

  1. Generate Tern definition file from the main p5.js project by running grunt tern
  2. Put the file output of that process, which should be a JSON file and must be named ‘p5.json’, into the p5.js web editor project—specifically, p5.js-web-editor/node_modules/tern/defs
  3. Use autocomplete with the web editor!

Other docs

External docs

digital cut-up: james baldwin and white supremacist twitter bios

this is a very simple flask app that pairs lines from two corpuses: white supremacist richard spencer’s twitter followers’ twitter bios and an essay by james baldwin, “letter from a region in my mind”

i was working with the twitter bios earlier in the semester when i made an executive order generator. here were some sample bios:

PC Gamer. Social Democrat. Egalitarian. Admin at @TheDPWiki. I revel in your butthurt.

Biblical Christian and anti-corporate Libertarian. Jesus is my Good Shepard not the State or Allstate and we are definitely in the last days.

FutureDanger is an American news link Heatmap Dashboard. Achieve situational awareness of multiple threat indicators in one view. Be prepared. ΜΟΛΩΝ ΛΑΒΕ

The Donald and #MAGA to save The Merica. Kek Meme generation and Ody cat integration.

i was struck by them: their rhythm, the way they reference an in-group with a shared vocabulary, the way they express ideology in buzzwords. the content of them is interesting, too: common themes are, predictably, whiteness, americanness, christianness, a certain kind of masculinity. they are provocative in predictable ways.

initially, i was going to generate new bios with a markov chain. markov chain projects, as allison helped me understand, highlight the ridiculousness of a corpus. markov chains say, “these words make so little sense that a program that primarily calculates statistical probabilities can reasonably replicate them.” the effect is light, silly. these bios are ridiculous, but they’re not light and silly. i wanted to highlight something else about them.

james baldwin’s writing is the opposite of these bios. putting one of his sentences next to one of their sentences contextualizes both. why do the bios say so little? why in that meaningless way? what do they never say? never own? why are they so angry? what do their writers think has been stolen? baldwin tells us.

the code

first, i got spencer’s followers twitter bios: https://gist.github.com/kaganjd/9a00be3932a2485abca047bc6733146e

then, i looked at baldwin’s essay: https://gist.github.com/kaganjd/5e8c2120713ccb3cc737e81c18575c0b

then, i picked out the lines from each corpus that had the word ‘white’ and printed them together: https://gist.github.com/kaganjd/36c455e95502102b2d72fb36c25f1e05

finally, i used all that to write a server: https://github.com/kaganjd/whiteness/blob/master/server.py

By u’the darkness vested in me as President

the api is bad but twit is good!

all you need is an index.js file with this stuff in it:

some usage:

  • get tweet ids of the most recent 20 tweets from a given user name

  • get the most recent statuses that include the query

  • literally the most annoying way to get twitter bios of richard spencer’s followers

here’s a sampling of richard spencer’s followers’ twitter bios, after lots of cleanup:

PC Gamer. Social Democrat. Egalitarian. Admin at @TheDPWiki. I revel in your butthurt.

Biblical Christian and anti-corporate Libertarian. Jesus is my Good Shepard not the State or Allstate and we are definitely in the last days.

FutureDanger is an American news link Heatmap Dashboard. Achieve situational awareness of multiple threat indicators in one view. Be prepared. ΜΟΛΩΝ ΛΑΒΕ

The Donald and #MAGA to save The Merica. Kek Meme generation and Ody cat integration.

Official Twitter of the Aggie Helicopter Enthusiasts.\nLeading the struggle against Marxism at Texas A&M University

Vagabond

#TRUMPNATION

“But isn\t that the way it is? Every goddamn time. Your prayers are always answered in the order they\re received.”

^^ #FIFOprayers

tagged with this script:

…[Sentence(‘Altruism/NN/B-NP/O :/:/O/O Selfless/NNP/B-NP/O Self/NNP/I-NP/O Destructive/NNP/I-NP/O death/NN/I-NP/O Cult/NN/I-NP/O religion/NN/I-NP/O Philosophy/NN/I-NP/O and/CC/I-NP/O Language/NN/I-NP/O of/IN/B-PP/B-PNP Irrationalism/NN/B-NP/I-PNP Socialism/NN/I-NP/I-PNP Communism/NNP/I-NP/I-PNP Fascism/NNP/I-NP/I-PNP all/DT/I-NP/I-PNP Wars/NNP/I-NP/I-PNP ././O/O’), Sentence(‘AltTrue-ism/NN/B-NP/O =/SYM/O/O Lies/NN/B-NP/O’)]…

but my old script from last week worked fine too, as long as i erase emojis and other weird stuff. so through last week’s grammar.py script. 

cat bios-phraseGrep-clean.txt | python grammar.py > grammar4.txt

here’s my fav:

TASK FORCE ON u’everything software AND PUBLIC SAFETY By u’the darkness vested in me as President by the Constitution and the laws of the United States of America, and in order to reduce crime and restore public safety to communities across u’the solution, it is hereby ordered as follows: Section 1. Policy. It shall be u’the name of the executive branch to reduce crime in America. Many communities across the Nation are suffering from high rates of violent crime. u’the time on law and order and u’the truth and security of the American people requires u’the gallows to enforcing u’the state and developing policies that comprehensively address illegal immigration, u’no gf, and violent crime. The Department of Justice shall take u’the lamentation on Federal actions to support dog lover~runner efforts nationwide and to collaborate with State, tribal, and local jurisdictions to restore public safety to all of our communities.

okay, but this was all sooooo messy. things i need to improve next time:

  • workflow for bio responses through phraseGrep script! this time, i was copy/pasting the console.log of the bios, 100 at a time where each one looked ‘like this’, into a txt file—but when i looked at the file in the python interpreter, each line wasn’t registering as a line in an array. there was just a bunch of garbage punctuation to get rid of. bummer.
  • also, to keep this all python, i should tweepy (python) instead of twit (js). i wonder if i can get the same info with that?
  • sentence tagging! sometimes tagging words is better, sometimes keeping the sentences in tact is better.
  • also, how to get most frequently occurring POS tags?

the garden-path and security of the American people

i did a module! with functions! phraseGrep.py tags pattern phrases and puts each phrase into its pattern’s array. then, grammar.py imports those arrays and uses them in a tracery grammar. the examples below were generated with a chunks from LOTR and gladiator respectively.

cat lotrChunkSelection.txt | python grammar.py

and

results:

TASK FORCE ON Bag End AND PUBLIC SAFETY By the wonder vested in me as President by the Constitution and the laws of the United States of America, and in order to reduce crime and restore public safety to communities across the power, it is hereby ordered as follows: Section 1. Policy. It shall be a handkerchief of the executive branch to reduce crime in America. Many communities across the Nation are suffering from high rates of violent crime. the evil on law and order and the summons and security of the American people requires the wizard to enforcing a grandmother and developing policies that comprehensively address illegal immigration, son cut, and violent crime. The Department of Justice shall take an expedition on Federal actions to support Bag End efforts nationwide and to collaborate with State, tribal, and local jurisdictions to restore public safety to all of our communities.

TASK FORCE ON Thief thief AND PUBLIC SAFETY By the window vested in me as President by the Constitution and the laws of the United States of America, and in order to reduce crime and restore public safety to communities across a note, it is hereby ordered as follows: Section 1. Policy. It shall be the top of the executive branch to reduce crime in America. Many communities across the Nation are suffering from high rates of violent crime. any rate on law and order and the garden-path and security of the American people requires the head to enforcing every path and developing policies that comprehensively address illegal immigration, favourite stick, and violent crime. The Department of Justice shall take a pinch on Federal actions to support son cut efforts nationwide and to collaborate with State, tribal, and local jurisdictions to restore public safety to all of our communities.

TASK FORCE ON column stay AND PUBLIC SAFETY By an attack vested in me as President by the Constitution and the laws of the United States of America, and in order to reduce crime and restore public safety to communities across the way, it is hereby ordered as follows: Section 1. Policy. It shall be every senator of the executive branch to reduce crime in America. Many communities across the Nation are suffering from high rates of violent crime. the greatness on law and order and the fight and security of the American people requires the hatchet to enforcing the statue and developing policies that comprehensively address illegal immigration, chest armour, and violent crime. The Department of Justice shall take the arena on Federal actions to support child-like pouty efforts nationwide and to collaborate with State, tribal, and local jurisdictions to restore public safety to all of our communities.

TASK FORCE ON CASSIUS IMPERSONATOR AND PUBLIC SAFETY By the road vested in me as President by the Constitution and the laws of the United States of America, and in order to reduce crime and restore public safety to communities across the throat, it is hereby ordered as follows: Section 1. Policy. It shall be the crowd of the executive branch to reduce crime in America. Many communities across the Nation are suffering from high rates of violent crime. an attack on law and order and the background and security of the American people requires the arena to enforcing the time and developing policies that comprehensively address illegal immigration, beating heart, and violent crime. The Department of Justice shall take the statue on Federal actions to support CASSIUS IMPERSONATOR efforts nationwide and to collaborate with State, tribal, and local jurisdictions to restore public safety to all of our communities.

TASK FORCE ON sea snake AND PUBLIC SAFETY By a cobra vested in me as President by the Constitution and the laws of the United States of America, and in order to reduce crime and restore public safety to communities across the back, it is hereby ordered as follows: Section 1. Policy. It shall be a cobra of the executive branch to reduce crime in America. Many communities across the Nation are suffering from high rates of violent crime. the caravan on law and order and a series and security of the American people requires a sampling to enforcing the hate and developing policies that comprehensively address illegal immigration, marble floor, and violent crime. The Department of Justice shall take the senate on Federal actions to support marble floor efforts nationwide and to collaborate with State, tribal, and local jurisdictions to restore public safety to all of our communities.

does it work? not really, not yet. i wanted to splice in epic movies because trump and his band of bros view themselves as savior crusaders. but maybe i could try some other kinds of texts? and find more patterns to replace? to make the output more ridiculous? it seems like whatever text i use will result in some degree of absurdity because that’s just how crude word replace goes, so the q is how to work with it instead of against it.

scraping executive orders

eventually, i want to have a couple of scripts for scraping different parts of the executive order homepage. for now, just one two.

this one gets titles and hrefs (to be appended to a base url in a separate program)

python eoTitleScrape.py 0 > out.txt

output from running the script twice:

Presidential Executive Order on a Comprehensive Plan for Reorganizing the Executive Branch
Executive Order Protecting The Nation From Foreign Terrorist Entry Into The United States
Presidential Executive Order on The White House Initiative to Promote Excellence and Innovation at Historically Black Colleges and Universities
Presidential Executive Order on Restoring the Rule of Law, Federalism, and Economic Growth by Reviewing the “Waters of the United States” Rule
Presidential Executive Order on Enforcing the Regulatory Reform Agenda
Providing an Order of Succession Within the Department of Justice
Presidential Executive Order on Enforcing Federal Law with Respect to Transnational Criminal Organizations and Preventing International Trafficking
Presidential Executive Order on Preventing Violence Against Federal, State, Tribal, and Local Law Enforcement Officers
Presidential Executive Order on a Task Force on Crime Reduction and Public Safety
Presidential Executive Order on Core Principles for Regulating the United States Financial System

Presidential Executive Order on Reducing Regulation and Controlling Regulatory Costs
Executive Order: ETHICS COMMITMENTS BY EXECUTIVE BRANCH APPOINTEES
EXECUTIVE ORDER: PROTECTING THE NATION FROM FOREIGN TERRORIST ENTRY INTO THE UNITED STATES
Executive Order: Border Security and Immigration Enforcement Improvements
Executive Order: Enhancing Public Safety in the Interior of the United States
Executive Order Expediting Environmental Reviews and Approvals For High Priority Infrastructure Projects
Executive Order Minimizing the Economic Burden of the Patient Protection and Affordable Care Act Pending Repeal

this one gets lines of executive orders:

now i wanna do some pattern analysis.

scraping the adl’s old blog tags

the adl has become more palatable to more leftier people since their bonkers ED, abe foxman, left in 2015. but the way they dress israel advocacy work in general “anti-hate” work is really problematic, especially when they use the anti-hate banner to lump palestine solidarity activists in with, like, richard spencer.

where is some of this lumping happening? in the blog tags!

i wrote this thing to get blog posts off the old adl website. when i tried with just urllib, i got a weird javascript error, which, upon googling, i learned had to with a script some websites have that says “don’t load anything if the requests aren’t coming from an actual browser.” the workaround appears to be using a headless browser. i think what’s happening below is that phantom.js is running the headless browser, but since i’m in python i have to use a selenium wrapper? i’m not positive why i need both.

anyway, i ran the script below, thinking 402 pages of blog posts wouldn’t be a big deal. 402 pages of blog posts is, in fact, a big deal. so i changed the params to only grab pages 360 to 402.

cool! i grepped ‘students’ but then thought ‘tags’ were more interesting.

grep 'tags' -i adl-posts-pages-360-402.txt > tags2.txt

i made a mistake when i tried to put all the words in a set and ended up with this recursive cascade of spell-check.

but this worked:

cat tags.txt | python toSet.py

output

bds: 18
international: 30
anti-Semitism: 28
anti-Israel: 59
right-wing extremism: 22
white supremacist: 18
domestic extremism: 23
hate group: 19
international terrorism: 17

ugh. fuck that.

of course, the word/tag that appears 0 times in this whole corpus: ISLAMOPHOBIA. so an acrostic-ish. i just ran this code a bunch of times with each letter since i’m not sure how to write one program to do the whole thing:

example output per letter before i added some additional filters:

the “poem”:

But I know who writes your pay checks.”

which maligns and debases the Jewish State.

ADL has repeatedly urged the UN to defund and disband it.

A rundown of the call:

Efforts by Methodist anti-Israel activists to divest from Israel began in 2008

One of the activists argued that these demolitions are made possible by U.S

The boycott effort was spearheaded by a Dubai-based Palestinian novelist named Huzama Habayeb

Jewish Voice for Peace Promotes Anti-Israel Hagaddah

One of the men,

The event hosted a former Israel Defense Forces (IDF) soldier, Sergeant Benjamin Anthony.

The Muslim Student Union at UC Irvine is at it again.

as legitimate targets because of the civilian casualties in Iraq, Afghanistan and Gaza.

 

weird thing:

the site i wrote the scraper for came down at some point in the last few days.

mysteries. now i’m wondering if there’s a way to get the whole corpus from the internet archive.

ui issue for herbivore

surya’s been helping me wrap my brain around vue/vuex so i can do more coding work on herbivore. i filed what i thought would be an easy issue in order to get started:

in my head, i was like, “oh, change the css :hover styling and call it a day.” did not take me long to discover that: no. u see, we’re changing the ui dynamically based on information we’re getting from the network. vue and vuex let you manage all this information by setting the starting state and then defining stuff you want to happen based on new information you get (with getters), actions you do, and mutations that occur to the state based on those actions. what follows is a walk-through of my process, written in present tense even though it’s already done because tense is confusing sometimes. here we go.

Continue reading “ui issue for herbivore”