Monthly Archives: May 2016

Phil 5.31.16

7:00 – 4:30 VTX

  • Writing. Working on describing how maintaining many codes in a network contains more (and more subtle) information than grouping similar codes.
  • Working on the UrlChecker
    • In the process, I discovered that the annotation.xml file is unique only for the account and not for the CSE. All CSEs for one account are contained in one annotation file
    • Created a new annotation called ALL_annotations.xml
    • fixed a few things in Andy’s file
    • Reading in everything. Now to produce the new sets of lists.
    • I think it’s just easier to delete all the lists and start over.
    • Done and verified. You run UrlChecker from the command line, with the input file being a list of domains (one per line) and the ALL_annotations.xml file.
  • Need to add a Delete or Hide button to reduce down a large corpus to a more effective size.
  • Added. Tomorrow I’ll wire up the deletion of a row or cilumn and the recreation of the initialMatrix

Phil 5.30.16

7:00 – 10:00 Thesis/VTX

  • Built a new matrix for the coded lit review. I had coded a couple of more papers
  • Working on copying over the read papers into a new folder that I can run text analytics over
  • After carefully reading through the doc manager list and copying over each paper, I just discovered I could have exported selected.
  • Ooops: Exception in thread “JavaFX Application Thread” java.lang.IllegalArgumentException: Invalid column index (16384).  Allowable column range for EXCEL2007 is (0..16383) or (‘A’..’XFD’)
    • Going to add a limit of

      columns for now. Clearly that can be cut down.

    • Figuring out where to cut the terms. I’m summing the columns of the LSI calculation, starting at the highest value and then dividing that by the sum of all values. The top 20% of rank weights gives 280 columns. Going to try that first
    • Success! Some initial thoughts
      • The coded version is much more ‘crisp’
      • There are interesting hints in the LSI version
      • Clicking on a term or paper to see the associated items is really nice.
      • I think that document subgroups might be good/better, and it might be possible to use the tool to help build those subgroups. This goes back to the ‘hiding’ concept. (hide item / hide item and associated)

Phil 5.27.16

7:00 – 2:00 VTX

  • Wound up writing the introduction and saving the old intro to a new document – Themesurfing
  • Renamed the current document
  • Got the parser working. Old artifact settings.
  • Added some tweaks to show progress better. I’m kinda stuck with the single thread in JavaFx having to execute before text can get shown.
  • Need an XML parser to find out what sites have already been added. Added an IntelliJ project to the GoogleCseConfigFiles SVN file. Should be able to finish it on Tuesday.

Phil 5.26.16

7:00 – 5:00 VTX

Phil 5.25.16

7:00 – 4:30 VTX

  • Took the weekend off for the ESCN. Bailed on Saturday because of rain, then dodged rain for two days, then got a nice ride in on Tuesday.
  • Chatting with Aaron last night, I discovered that the REST API won’t work for Demo. I’ll need to get a new SQL dump from Heath. No, actually it works just fine in that it is accessable, but anything other than empty sets is a timeout.
  • Need to try to build a new jar file for the CorpusManager so it can have its own executable. Put the Manifest in the CM directory? not sure how to do that.
  • Writing
  • Looks like my old laptop finally bit the dust. Chromebook time?
  • Working on the Corpus manager to pull in links in the config file. Done!
  • So I’m having all kinds of problems getting the flag info from Jeremy’s rest service. I did realize that I can use the dashboard though and harvest the urls by following the links and build my list that way. Except that the flags are crap. Back to moby dick for the moment.
  • Actually, those are pretty bad too. Margarita put three urls up on confluence.
  • Got url scanning done through config file.
  • Ingested the first four chapters of Moby Dick. Pretty interesting. Ill try those three files tomorrow and we’ll see what we’ve got, at least for a sense of .gov sites…

Phil 5.20.16

7:00 – 3:30

  • Writing
  • Going to try LSI. I think the term clustering is simply the sum if the TF-IDF across docs by term. That should give a topic list. Then use that for centrality calculations? Take the top n words?
    • Actually, then the user could group words into concepts and that could make a smaller matrix where the concept count is the union of the counts of its component terms.
  • Have a LSI-lite version going that sums the TF-IDF scores and then sorts based on the sum of all scores * (number of docs with score / number of docs). Then sort and take the top n terms.
  • Need to multiply the matrix by something so that the count gets populated with something reasonable. Maybe 100? Tried that – it looks good.
  • Got the PDF parsing working. Need to get it to work with webpages next and try it on Moby Dick. Then output from the flag data
  • Need to make sure that I use the above pointing at the demo system. From Andy’s email:

    Yes …looks you are looking at dev….in Confluence, search on environment details…that Will give you the urls for the dashboards on dev, ci and demo…we are working on demo now.

Phil 5.19.16

7:00 – 5:00 VTX

  • Looks like I saved the wrong version of the code to dropbox, so I can’t update the app image.
  • More writing
  • System and Social trust, revisited: Algorithms, clickworkers, and the befuddled fury around Facebook Trends
  • GDELT uses some of the world’s most sophisticated natural language and data mining algorithms to extract more than 300 categories of “events” and the networks of people, organizations, locations, themes, and emotions that tie them together.
  • Working on the Corpus Processing Tool
  • Need to break apart calculateAndSave
  • Need to build matrix
  • Need to save spreadsheets
  • Need name to save too.
  • Start and stopwords
  • Add Latent Semantic Indexing? I have most of the pieces.

Phil 5.18.16

7:00 – 4:30 VTX

  • Writing
    • Wanted to show how a network could be used for intercoder agreement so I had to refresh my understanding of Cohen’s kappa
    • It occurs to me that if one coder’s rank can be mapped to another coder’s rank we have a kind of information distance measure. Although the math to do that eludes me. Rank comparison could make a lot of sense to compare centrality. Another possibility is to compare the network measures?
  • Adding ‘Filter’ field and button to LMN
  • This appears to be how you do it.
  • And it worked like a charm 🙂
  • Worked through scoring math with Aaron

Phil 5.17.16

7:00 -7:00

  • Great discussion with Greg yesterday. Very encouraging.
  • Some thoughts that came up during Fahad’s (Successful!) defense
    • It should be possible to determine the ‘deletable’ codes at the bottom of the ranking by setting the allowable difference between the initial ranking and the trimmed rank.
    • The ‘filter’ box should also be set by clicking on one of the items in the list of associations for the selected items. This way, selection is a two-step process in this context.
    • Suggesting grouping of terms based on connectivity? Maybe second degree? Allows for domain independence?
    • Using a 3D display to show the shared second, third and nth degree as different layer
    • NLP tagged words for TF-IDF to produce a more characterized matrix?
    • 50 samples per iteration, 2,000 iterations? Check! And add info to spreadsheet! Done, and it’s 1,000 iterations
  • Writing
  • Parsing Jeremy’s JSON file
    • Moving the OptionalContent and JsonLoadable over to JavaJtils2
    • Adding javax.persistence-2.1.0
    • Adding json-simple-1.1.1
    • It worked, but it’s junk. It looks like these are un-curated pages
  • Long discussion with Aaron about calculating flag rollups.

Phil 5.16.16

7:00 – 4:00 VTX

  • Writing
  • Extracting PDFs? Works! Built NaiveParser and added to JavaUtils2. Updating SVN and then trying against a small corpus of PDFs.
    • Need to add a way of adding strings with wildcard characters for things to delete, grab content, etc.
  • Fahad’s defense