Category Archives: cool

Phil 12.12.17

7:00 – 3:30 ASRC MKT

  • Schedule physical
  • Call Lisa Cross
  • Need to make sure that an amplified agent also has amplified influence in calculating velocity – Fixed
  • Towards the end of this video is an interview with Ian Couzin talking about how mass communication is disrupting our ability to flock ‘correctly’ due to the decoupling of distance and information
  • Write up fire stampede. Backups everywhere, one hole, antennas burn so the AI keeps trust in A* but loses awareness as the antennas burn: “The Los Angeles Police Department asked drivers to avoid navigation apps, which are steering users onto more open routes — in this case, streets in the neighborhoods that are on fire.” [LA Times] Also this slow motion version of the same thing: For the Good of Society — and Traffic! — Delete Your Map App
  • First self-driving car ‘race’ ends in a crash at the Buenos Aires Formula E ePrix; two cars enter, one car survives
  • Taking a closer look at Oscillator Models and Collective Motion (178 Citations) and Consensus and Cooperation in Networked Multi-Agent Systems (6,291 Citations)
  • Consensus and Cooperation in Networked Multi-Agent Systems
    • Reza Olfati-SaberAlex Fax, and Richard M. Murray
    • We discuss the connections between consensus problems in networked dynamic systems and diverse applications including synchronization of coupled oscillators, flocking, formation control, fast consensus in small world networks, Markov processes and gossip-based algorithms, load balancing in networks, rendezvous in space, distributed sensor fusion in sensor networks, and belief propagation. We establish direct connections between spectral and structural properties of complex networks and the speed of information diffusion of consensus algorithms (Abstract)
    • In networks of agents (or dynamic systems), “consensus” means to reach an agreement regarding a certain quantity of interest that depends on the state of all agents. A “consensus algorithm” (or protocol) is an interaction rule that specifies the information exchange between an agent and all of its (nearest) neighbors on the network (pp 215)
      • In my work, this is agreement on heading and velocity
    • Graph Laplacians are an important point of focus of this paper. It is worth mentioning that the second smallest eigenvalue of graph Laplacians called algebraic connectivity quantifies the speed of convergence of consensus algorithms. (pp 216)
    • More recently, there has been a tremendous surge of interest among researchers from various disciplines of engineering and science in problems related to multi-agent networked systems with close ties to consensus problems. This includes subjects such as consensus [26]–[32], collective behavior of flocks and swarms [19], [33]–[37], sensor fusion [38]–[40], random networks [41], [42], synchronization of coupled oscillators [42]–[46], algebraic connectivity of complex networks [47]–[49], asynchronous distributed algorithms [30], [50], formation control for multi-robot systems [51]–[59], optimization-based cooperative control [60]–[63], dynamic graphs [64]–[67], complexity of coordinated tasks [68]–[71], and consensus-based belief propagation in Bayesian networks [72], [73]. (pp 216)
      • That is a dense lit review. How did they order it thematically?
    • A byproduct of this framework is to demonstrate that seemingly different consensus algorithms in the literature [10], [12]–[15] are closely related. (pp 216)
    • To understand the role of cooperation in performing coordinated tasks, we need to distinguish between unconstrained and constrained consensus problems. An unconstrained consensus problem is simply the alignment problem in which it suffices that the state of all agents asymptotically be the same. In contrast, in distributed computation of a function f(z), the state of all agents has to asymptotically become equal to f(z), meaning that the consensus problem is constrained. We refer to this constrained consensus problem as the f-consensus problem. (pp 217)
      • Normal exploring/flocking/stampeding is unconstrained. Herding adds constraint, though it’s dynamic. The variables that have to be manipulated in the case of constraint to result in the same amount of consensus are probably what’s interesting here. Examples could be how ‘loud’ does the herder have to be? Also, how ‘primed’ does the population have to be to accept herding?
    • …cooperation can be informally interpreted as “giving consent to providing one’s state and following a common protocol that serves the group objective.” (pp 217)
    • Formal analysis of the behavior of systems that involve more than one type of agent is more complicated, particularly, in presence of adversarial agents in noncooperative games [79], [80]. (pp 217)
  • Not sure about this one. It just may be another set of algorithms to do flocking. Maybe some network implications? Flocking for Multi-Agent Dynamic Systems: Algorithms and Theory. It is one of the papers that the Consensus and Cooperation paper above leans on heavily though…
  • The Emergence of Consensus: A Primer
    • The origin of population-scale coordination has puzzled philosophers and scientists for centuries. Recently, game theory, evolutionary approaches and complex systems science have provided quantitative insights on the mechanisms of social consensus. However, the literature is vast and scattered widely across fields, making it hard for the single researcher to navigate it. This short review aims to provide a compact overview of the main dimensions over which the debate has unfolded and to discuss some representative examples. It focuses on those situations in which consensus emerges ‘spontaneously’ in absence of centralised institutions and covers topic that include the macroscopic consequences of the different microscopic rules of behavioural contagion, the role of social networks, and the mechanisms that prevent the formation of a consensus or alter it after it has emerged. Special attention is devoted to the recent wave of experiments on the emergence of consensus in social systems.
  • Critical dynamics in population vaccinating behavior
    • Complex adaptive systems exhibit characteristic dynamics near tipping points such as critical slowing down (declining resilience to perturbations). We studied Twitter and Google search data about measles from California and the United States before and after the 2014–2015 Disneyland, California measles outbreak. We find critical slowing down starting a few years before the outbreak. However, population response to the outbreak causes resilience to increase afterward. A mathematical model of measles transmission and population vaccine sentiment predicts the same patterns. Crucially, critical slowing down begins long before a system actually reaches a tipping point. Thus, it may be possible to develop analytical tools to detect populations at heightened risk of a future episode of widespread vaccine refusal.
  • For Aaron’s Social Gradient Descent Agent research (lit review)
    • On distributed search in an uncertain environment (Something like Social Gradient Descent Agents)
      • The paper investigates the case where N agents solve a complex search problem by communicating to each other their relative successes in solving the task. The problem consists in identifying a set of unknown points distributed in an n–dimensional space. The interaction rule causes the agents to organize themselves so that, asymptotically, each agent converges to a different point. The emphasis of this paper is on analyzing the collective dynamics resulting from nonlinear interactions and, in particular, to prove convergence of the search process.
    • A New Clustering Algorithm Based Upon Flocking On Complex Network (Sizing and timing for flocking systems seems to be ok?)
      • We have proposed a model based upon flocking on a complex network, and then developed two clustering algorithms on the basis of it. In the algorithms, firstly a k-nearest neighbor (knn) graph as a weighted and directed graph is produced among all data points in a dataset each of which is regarded as an agent who can move in space, and then a time-varying complex network is created by adding long-range links for each data point. Furthermore, each data point is not only acted by its k nearest neighbors but also r long-range neighbors through fields established in space by them together, so it will take a step along the direction of the vector sum of all fields. It is more important that these long-range links provides some hidden information for each data point when it moves and at the same time accelerate its speed converging to a center. As they move in space according to the proposed model, data points that belong to the same class are located at a same position gradually, whereas those that belong to different classes are away from one another. Consequently, the experimental results have demonstrated that data points in datasets are clustered reasonably and efficiently, and the rates of convergence of clustering algorithms are fast enough. Moreover, the comparison with other algorithms also provides an indication of the effectiveness of the proposed approach.
  • Done with the first draft of the white paper! And added the RFP section to the LMN productization version
  • Amazon Sage​Maker: Amazon SageMaker is a fully managed machine learning service. With Amazon SageMaker, data scientists and developers can quickly and easily build and train machine learning models, and then directly deploy them into a production-ready hosted environment. It provides an integrated Jupyter authoring notebook instance for easy access to your data sources for exploration and analysis, so you don’t have to manage servers. It also provides common machine learning algorithms that are optimized to run efficiently against extremely large data in a distributed environment. With native support for bring-your-own-algorithms and frameworks, Amazon SageMaker offers flexible distributed training options that adjust to your specific workflows. Deploy a model into a secure and scalable environment by launching it with a single click from the Amazon SageMaker console. Training and hosting are billed by minutes of usage, with no minimum fees and no upfront commitments. (from the documentation)

4:00 – 5:00 Meeting with Aaron M. to discuss Academic RB wishlist.

 

Advertisements

Phil 9.14.17

7:00 – 4:00 ASRC MKT

  • Reducing Dimensionality from Dimensionality Reduction Techniques
    • In this post I will do my best to demystify three dimensionality reduction techniques; PCA, t-SNE and Auto Encoders. My main motivation for doing so is that mostly these methods are treated as black boxes and therefore sometime are misused. Understanding them will give the reader the tools to decide which one to use, when and how.
      I’ll do so by going over the internals of each methods and code from scratch each method (excluding t-SNE) using TensorFlow. Why TensorFlow? Because it’s mostly used for deep learning, lets give it some other challenges 🙂
      Code for this post can be found in this notebook.
    • This seems important to read in preparation for the Normative Mapping effort.
  • Stanford  deep learning tutorial. This is where I got the links to PCA and Auto Encoders, above.
  • Ok, back to writing:
    • The Exploration-Exploitation Dilemma: A Multidisciplinary Framework
    • Got hung up explaining the relationship of the social horizon radius, so I’m going to change it to the exploit radius. Also changed the agent flocks to red and green: GPM
    • There is a bug, too – when I upped the CellAccumulator hypercube size from 10-20. The max row is not getting set

Phil 6.2.16

7:00 – 5:00 VTX

  • Writing
  • Write up sprint story – done
    • Develop a ‘training’ corpus known bad actors (KBA) for each domain.

      • KBAs will be pulled from http://w3.nyhealth.gov/opmc/factions.nsf, which provides a large list.
      • List of KBAs will be added to the content rating DB for human curation
      • HTML and PDF data will be used to populate a list of documents that will then be scanned and analyzed to prepare TF-IDF and LSI term-document tables.
      • The resulting table will in turn be analyzed using term centrality, with the output being an ordered list of terms to be evaluated for each domain.

  • Building view to get person, rating and link from the db – done, or at least V1
    CREATE VIEW view_ratings AS
      select io.link, qo.search_type, po.first_name, po.last_name, po.pp_state, ro.person_characterization from item_object io
        INNER JOIN query_object qo ON io.query_id = qo.id
        INNER JOIN rating_object ro on io.id = ro.result_id
        INNER JOIN poi_object po on qo.provider_id = po.id;
  • Took results from w3.nyhealth.gov and ran them through the whole system. The full results are in the Corpus file under w3.nyhealth.gov-PDF-centrality_06_02_16-13_12_09.xlsx and w3.nyhealth.gov-WEB-centrality_06_02_16-13_12_09.xlsx. The results seem to make incredibly specific searches. Here are the two first examples. Note that there are very few .com sites.:

Phil 5.3.16

7:00 – 3:30 VTX

  • Out riding, I realized that I could have a column called ‘counts’ that would add up the total number of ‘terms per document’ and ‘documents per terms ‘. Unitizing the values would then show the number of unique terms per document. That’s useful, I think.
  • Helena pointed to an interesting CHI 2016 site. This is sort of the other side of extracting pertinence from relevant data. I wonder where they got their data from?
    • Found it!. It’s in a public set of Google docs, in XML and JSON formats. I found it by looking at the GitHub home page. In the example code  there was this structure:
      source: {
          gdocId: '0Ai6LdDWgaqgNdG1WX29BanYzRHU4VHpDUTNPX3JLaUE',
          tables: "Presidents"
        }

      That gave me a hint of what to look for in the document source of the demo, where I found this:

      var urlBase = 'https://ca480fa8cd553f048c65766cc0d0f07f93f6fe2f.googledrive.com/host/0By6LdDWgaqgNfmpDajZMdHMtU3FWTEkzZW9LTndWdFg0Qk9MNzd0ZW9mcjA4aUJlV0p1Zk0/CHI2016/';
      

      And that’s the link from above.

    • There appear to be other useful data sets as well. For example, there is an extensive CHI paper database sitting behind this demo.
    • So this makes generalizing the PageRank approach much more simple since it looks like I can pull the data down pretty simply. In my case I think the best thing would be to write small apps that pull down the data and build Excel spreadsheets that are read in by the tool for now.
  • Exporting a new data set from Atlas. Done and committed. I need to do runs before meeting with Wayne.
  • Added Counts in and refactored a bit.
  • I think I want a list of what a doc or term is directly linked to and the number of references. Addid the basics. Wiring up next. Done! But now I want to click on an item in the counts list and have it be selected? Or at least highlighted?
  • Stored the new version on dropbox: https://www.dropbox.com/s/92err4z2posuaa1/LMN.zip?dl=0
  • Meeting with Wayne
    • There’s some bug with counts. Add it to the WeightedItem.toString() and test.
    • Add a ‘move to top’ button near the weight slider that adds just enough weight to move the item to the top of the list. This could be iterative?
    • Add code that compares the population of ranks with the population of scaled ranks. Maybe bootstrapping? Apache Commons Math has KolmogorovSmirnovTest, which has public double kolmogorovSmirnovTest(double[] x, double[] y, boolean strict), which looks promising.
  • Added ability to log out of the rating app.

Phil 2.9.16

7:00 – 4:00 VTX

  • Finished Publius: A robust, tamper-evident, censorship-resistant web publishing system
  • Starting Anonymity Loves Company – Anonymous Web Transactions with Crowds by Mike Reiter and Aviel Ruben, who was one of the co-authors on the Publius paper.
    • Crowds could probably be built with PeerJS. The ISP would still know traffic, but that’s it.
  • Found this nice article in Communications of the ACM: Schema.org: Evolution of Structured Data on the Web. Nice overview. Very current.
  • The Big List of Naughty Strings
  • Time to combine everything
    • Optional generation of Providers and queries – default is to load them from the DB
    • Run queries from the DB
      • Show the number available and allow a request – done
      • Iterating over the queries and pages. Need to create, append and persist a rating Done
      • Named queries for
        • Queries that have the lowest number of results.ratings – done-ish. Currently it looks for -1 as a flag. Should also look for queries that have unrated results.
        • Queries associated with ‘bad’ providers
        • Queries associated with ‘good’ providers
      • Connect to DB remotely
    • Wrap the app (done, with Launch4j. Very nice!) and test it on the other laptop. Note, it doesn’t have enough disk to install java on. That will have to wait.
    • Packing up the laptop. Debating bringing multi monitor support. I’ll have the other laptop…
    • Gratuitous screenshot: SwingFlashback

Phil 11.16.15

7:00 – 6:00 Leave

  • Found a new programmer resource that looks good – I Programmer. They pointed me to an article about Babel, which compiles JavaScript to… other things. It might even be able to monkeypatch modern JS to run on old browsers. Need to test one of these years. It’s based on plugins which really means that it can map from one thing to anything else. My only issue is that it could break debugging unless there is a mapping file like typescript has.
  • Discovered another communication app – Telegram. ISIS used it to announce Paris?
  • Noon – Thad Starner in ITE 459. Very interesting. Met Aaron Massey, who might be good on the Committee.
  • I’ve been reading Tefko Saracevic‘s paper RELEVANCE: A Review of and a Framework for the thinking on the Notion in Information Science. It’s full of really nice stuff, from a time when you couldn’t just throw processing power at problems and brute force an answer. It’s clarified my thinking about the client word-based network:
    • Search engines are pragmatic relevance engines (i.e topic-relatedness, quality, novelty, importance, credibility, etc.). The networks that they produce try to correlate knowledge at the ‘source’ – basically ‘in the world’
    • We, as individuals are pertinence/situational relevance machines (Wilson’s concerns, preferences and stock of knowledge). Our internal knowledge graph represents our view of the world. We are the ‘destination’ for information.
      • “Situational Relevance is relevance to a particular individual’s situation – but to the situation as he sees it, not as others see it, nor as it really is.”
      • The ‘shape’ of our internal knowledge graph, the sources of information that we lean more heavily on, the weights that we give to certain words (or possibly concepts) may be able to determine whether we are dependably credible or dependably counter-credible.
    • By enabling client-side weighting, we let users adjust components of a relevant search so that it becomes pertinent to us.
    • The information that we produce in this process (dictionaries, weights, etc) can be stored so that a well-structured record of what is pertinent to individuals (and more importantly, groups of individuals) becomes part of the world knowledge. Correlations with respect to internal credibility may then in turn be able to infer the credibility (or lack of) of information in the world.
  • Getting back to dictionary integration.
    • Re-upped my IntelliJ subscription for another two years
    • Updated files and DB. All seems to work
    • DbDictionary.removeDictionary returns a fail JSON message. Fixing. Fixed!
    • Adding ability to update an entry – done.
    • Finishing CreateDictionary. Finished and tested
    • Adding DeleteDictionary. Finished and tested
    • Adding ModifyDictionary. Finished and tested
    • Adding term extraction. Started poking, but that’s it. More tomorrow.