Monthly Archives: August 2017

Phil 8.31.17

7:00 – 9:00, 10:00-4:30 IRAD

  • Something for my Visible from Space collection?
  • Saw this in the letters from CACM this month:

    Causal Connections for Predictive AI

    As part of the “ACM Panels in Print” section “Artificial Intelligence” (Feb. 2017), panelist David Blei said he believes computer science needs to identify the causal connections between data components, concluding that artificial intelligence, with its predictive capabilities, will be enhanced through causal inference. For example, the first step toward using AI in database systems is to analyze and create a data map of the complexity of the causal interconnections between the data components in a problem space. A data map connects a data component to other data components through causal interrelationships. A data map can be created through qualitative analysis of data collected for a particular problem. The qualitative analysis could then take the form of “thematic analysis,” (Braun, V. and Clarke, V. 2006) using systems diagramming to gain greater insight into the data. At that point it might be advantageous to start applying AI directly to the data.

    Researching the complexity of database systems, I have thus created such a data map, which is now ready to move to the next stage of automation where predictive analytics can help improve management of database systems. Using an analogy of the CODEX, or Control of Data Expediently, my research into causal connections has identified a potential role for AI in automating continuously changing best practice, thus representing an agile approach to deciphering the complexity of interconnections and promising to help create an autonomous way to deliver best practice in database management.

    Victoria Holt, Bath, U.K.

  • I looked at hew blog, research page and twitter account. Can’t find anything about mapping. Even looking at the thematic analysis and citing articles, I can’t find much about maps or mapping. Odd
  • Getting back to drawing the database. But first, updating my IDE, it seems…
  • Broke out DimensionInfo to TensorDimensionInfo since I need the mappingLow and mappingHigh to draw the borders
  • Changed the canvas drawBG to use StorageAndRetreival, since I need to know the projection I’m drawing
  • Killed JavaFX for a while. Had to reboot
  • More RFI work
  • Ok, got the background scaled correctly and drawing the border. Had to deal with more hard-coded values. SAD! Anyway, here’s all the parts working: stageWithBorder
Advertisements

Phil 8.30.17

6:30 – 6:00 ASRC Research, proposals and job stuff

  • Redid my resume this morning. The word doc went fine, but when I went to generate the html versions, I had a momentary lapse of good sense and had Word output the pages.  Horrible mistake. I had to go and use the very fine HTML Cleaner website and put everything back together again. So 15 minutes of adding a paragraph and a picture turned into over an hour of work. Sigh.
  • If I have time, the next step today is to get the StorageAndRetrieval code drawing on the canvas. Started setting this up. The actual cell coloring could be done with another evaluateTensor method? Maybe later. Get the basics working first.
  • In a significant miracle, my PhD work is now my job, at least until the end of the year. Sprinkled with some proposal writing.
  • Spent most of the day working on the NNSA RFI

Phil 8.29.17

7:00 – 6:30 ASRC Research

  • Probably not going to get the StorageAndRetreival class done and integrated by Sept 1, so I’ll need to work out the paper based on what I have. But let’s see how far we get…
  • I think I’m going to start simply by placing the agent type in the IR cell and color accordingly. There will be three states, NULL (white) , EXPLORE (red) and EXPLOIT (green)
  • Fixing some things along the way. Stage dimension initial size was hard coded. Oops.
  • Adding a clear() to LabeledTensor and StorageAndRetreival
  • Working out how to recurse though a set of arrays. Putting here for later reference:
    class Recurse{
        int linecount = 0;
    
        boolean fillMat(int[] curIndices, int[] maxIndices, int curIndex){
            if(curIndex > curIndices.length-1){
                //System.out.printf("curIndex(%d) > %d\n", curIndex, curIndices.length-1);
                return true;
            }
            for(int i = 0; i < maxIndices[curIndex]; ++i) {
                curIndices[curIndex] = i;
                boolean done = fillMat(curIndices, maxIndices, curIndex+1);
                if(done) {
                    linecount++;
                    System.out.println(linecount+" [" + curIndex + "]: " + Arrays.toString(curIndices) + " = " + done);
                }
            }
            return false;
        }
    
        public static void main(String[] args){
            int[] initial = {0, 0, 0};
            int[] max = {3, 3, 3};
            System.out.println("Recurse!");
            Recurse r = new Recurse();
            r.fillMat(initial, max, 0);
        }
    }
  • Here’s the final version, with an Interface callback
    public boolean fillTensor(int[] curIndices, int[] maxIndices, int curIndex, TensorCellEvaluator tce){
        if(curIndex > curIndices.length-1){
            // if we get here, we're done
            return true;
        }
        for(int i = 0; i < maxIndices[curIndex]; ++i) {
            curIndices[curIndex] = i;
            boolean done = fillTensor(curIndices, maxIndices, curIndex+1, tce);
            if(done) {
                tce.evaluate(curIndices, lt);
            }
        }
        // if we get here, we're not none, keep going
        return false;
    }
  • Here’s the call with the implemented interface. It doesn’t do anything yet, but the pieces all work
    stoRet.fillTensor(indicies, maxIndicies, 0, new TensorCellEvaluator() {
        @Override
        public String evaluate(int[] indicies, LabeledTensor lt) {
            System.out.println(Arrays.toString(indicies));
            return Arrays.toString(indicies);
        }
    });
  • Interview with USDA team. Not a fit at all

Phil 8.28.17

Another weekend of scrambling to get everything done in time. Plus a good bike ride 🙂

7:00 – 5:00 Research (ASRC IRAD)

  • Started up the computer this morning and it freaked out. After a restart, it rebuilt all the program indices and many of my configs were gone. I’m guessing that it was a bad disk sector. Scary, since this is a pretty new box
  • Adding in the Information Retrieval part that will mediate interactions between agents
  • And now the changes to JavaUtils2 aren’t being recognized. Deleted reference and reimported. And not the project is rebuilding super slow????? This is just not my morning.
  • And now IntelliJ is frozen. Restarted. The error is still there, but it compiles no problem. Had to invalidate caches.
  • Started StorageAndRetreival class and have a rough framing running
  • Long discussion with the interns about using mapping for the HR space
  • Wrote up some more paragraphs on normative mapping for paper justification
  • Fuzzy thought for today. The map is actually a set of pointers to the data used to produce it. For example, a discussion that produces normative poles has a history of collapse that the normative poles could reference back to.

Phil 8.25.17

7:00 – 5:00 ASRC IRAD

  • Got Understanding Ignorance, which has a section on Ignorance as a Place. More on that later
  • Since I’m in the middle of LabledTensor, I’m going to add the ability to string together multiple ArrayLists to accommodate Long indexing
    • Created class LongArrayList, based on part of List. Can’t implement the interface, since the parameters are int, rather than long. Done! Also fixed a few formatting errors in the toString().

Phil 8.24.17

6:45 – 3:00 ASRC IRAD

Phil 8.23.17

Research

  • Started the ball rolling on 899 approval and getting together waith Wayne for a chat
  • 899 is set up, Wayne is going to ECSCW, so sometime after that

8:30 – 5:30 ASRC

  • BRI suspended payment on the contract, so much churn. Lots of discussions with many people. Looks like an interview on Friday?
  • Had to reinstall Office. Getting coffee…
  • Connnected to my SVN and got the LabeledTensor code to work on. Need to add the support for labels and more than 32k entries.

Phil 8.18.17

7:00 – 8:00 Research

  • Got indexFromLocation() working. It took some fooling around with Excel. Here’s the method:
    public int[] indexFromLocation(double[] loc){
        int[] index = new int[loc.length];
        for(int i = 0; i < loc.length; ++i){
            double findex = loc[i]/mappingStep;
            double roundDown = Math.floor(findex);
            double roundUp = Math.ceil(findex);
            double lowdiff = findex - roundDown;
            double highdiff = roundUp - findex;
            if(lowdiff < highdiff){
                index[i] = (int)roundDown;
            }else{
                index[i] = (int)roundUp;
            }
        }
        return index;
    }
  • And here are the much cleaner results:
    • [0.00, 0.00] = [0, 0]
      [0.00, 0.10] = [0, 0]
      [0.00, 0.20] = [0, 1]
      [0.00, 0.30] = [0, 1]
      [0.00, 0.40] = [0, 2]
      [0.00, 0.50] = [0, 2]
      [0.00, 0.60] = [0, 2]
      [0.00, 0.70] = [0, 3]
      [0.00, 0.80] = [0, 3]
      [0.00, 0.90] = [0, 4]
      [0.00, 1.00] = [0, 4]

      [1.00, 0.00] = [4, 0]
      [1.00, 0.10] = [4, 0]
      [1.00, 0.20] = [4, 1]
      [1.00, 0.30] = [4, 1]
      [1.00, 0.40] = [4, 2]
      [1.00, 0.50] = [4, 2]
      [1.00, 0.60] = [4, 2]
      [1.00, 0.70] = [4, 3]
      [1.00, 0.80] = [4, 3]
      [1.00, 0.90] = [4, 4]
      [1.00, 1.00] = [4, 4]
  • Another thought that struck me as far as the (int) constraint is that I can have a number of ArrayLists that are embedded in a an object that has the first and last index in it. These would be linked together to provide unconstrained (MAX_VALUE or 2,147,483,647 lists) storage

8:30 – 4:30 BRI

  • I realized yesterday that the Ingest and Query microservices need to access the same GeoMesa Spring service. That keeps all the general store/query GeoMesa access code in one place, simplifies testing and allows for DI to provide the correct (hbase, accumulo, etc) implementation through a facade interface.
  • Got tangled up with getting classpaths right and importing the proper libraries
  • Got the maven files behaving, or at least not complaining on mvn clean and mvn compile!
  • Well that’s a new error: Error: Could not create the Java Virtual Machine. I get that running the new installation with the geomesa-quickstart-hbase
    • Ah, that’s what will happen when you paste your command-line arguments into the VM arguments space just above where it should go…
    • Wednesday’s goal will to verify that HBaseQuickStart is running correctly in its new home and start to turn it into a service.

Phil 8.17.17

BRI – one hour chasing down research hours from Jan – May

7:00 – 6:00 Research

  • Found this on negative flocking influences: The rise of negative partisanship and the nationalization of US elections in the 21st century.  Paper saved to Lit Review
    • One of the most important developments affecting electoral competition in the United States has been the increasingly partisan behavior of the American electorate. Yet more voters than ever claim to be independents. We argue that the explanation for these seemingly contradictory trends is the rise of negative partisanship. Using data from the American National Election Studies, we show that as partisan identities have become more closely aligned with social, cultural and ideological divisions in American society, party supporters including leaning independents have developed increasingly negative feelings about the opposing party and its candidates. This has led to dramatic increases in party loyalty and straight-ticket voting, a steep decline in the advantage of incumbency and growing consistency between the results of presidential elections and the results of House, Senate and even state legislative elections. The rise of negative partisanship has had profound consequences for electoral competition, democratic representation and governance.
  • Working on putting together an indexable high-dimension matrix that can contain objects. Generally, I’d expect it to be doubles, but I can see Strings and Objects as well.
  • Starting off by seeing what’s in the newest Apache Commons Math (v 3.6.1)
  • Found SimpleTensor, which uses the Efficient Java Matrix Library (EJML) and creates a 3D block of rows, columns and slices. THought it was what I wanted, but nope
  • Looks like there isn’t a class that would do what I need to do, or that I can even modify. I’m thinking that the best option is to use org.apache.commons.math3.linear.AbstractRealMatrix as a template.
  • Nope, coudn’t figure out how to do things as nested lists. So I’m doing it C-Style, where you really only have one array that you index into. Here’s a 4x4x4x4 Tensor filled with zeroes:
    Total elements = 256
    0.0:[0, 0, 0, 0], 0.0:[1, 0, 0, 0], 0.0:[2, 0, 0, 0], 0.0:[3, 0, 0, 0],
    0.0:[0, 1, 0, 0], 0.0:[1, 1, 0, 0], 0.0:[2, 1, 0, 0], 0.0:[3, 1, 0, 0],
    0.0:[0, 2, 0, 0], 0.0:[1, 2, 0, 0], 0.0:[2, 2, 0, 0], 0.0:[3, 2, 0, 0],
    0.0:[0, 3, 0, 0], 0.0:[1, 3, 0, 0], 0.0:[2, 3, 0, 0], 0.0:[3, 3, 0, 0],
    0.0:[0, 0, 1, 0], 0.0:[1, 0, 1, 0], 0.0:[2, 0, 1, 0], 0.0:[3, 0, 1, 0],
    ….
    0.0:[0, 2, 3, 3], 0.0:[1, 2, 3, 3], 0.0:[2, 2, 3, 3], 0.0:[3, 2, 3, 3],
    0.0:[0, 3, 3, 3], 0.0:[1, 3, 3, 3], 0.0:[2, 3, 3, 3], 0.0:[3, 3, 3, 3]
  • The only issue that I currently have is that ArrayLists are indexed by int, so the total size is 32k elements. That should be good enough for now, but it will need to be fixed.
  • set() and get() work nicely:
    lt.set(new int[]{0, 1, 0, 0}, 9.9);
    lt.set(new int[]{3, 3, 3, 3}, 3.3);
    
    System.out.println("[0, 1, 0, 0] = " + lt.get(new int[]{0, 1, 0, 0}));
    System.out.println("[3, 3, 3, 3] = " + lt.get(new int[]{3, 3, 3, 3}));
    
    [0, 1, 0, 0] = 9.9
    [3, 3, 3, 3] = 3.3
  • Started the indexFromLocation method, but this is too sloppy:
    index[i] = (int)Math.floor(Math.round(loc[i]/mappingStep));