Monthly Archives: September 2009

Phil 9.30.2009

7:30 – 3:30

  • Deployed the new version – there are a few bugs:
    • Although remote files validate, they don’t read in – looks like the problem was that the remote files don’t terminate in .xml
    • Blanks are counted in the UniqueCounts view – fixed
    • Need to clear out the “where clauses” if the last item is unchecked – fixed
    • Need to keep the where clauses in the unique count query construction – fixed
  • Had a good discussion with Mike C about data
  • Scheduled a meeting with Anne to go over the new features
  • Burned a new disc and am ready to deploy tomorrow.

Phil 9.29.2009

7:30 – 3:3o

  • Integrating Dong’s code in prep for release – done
  • Burned a disc

Dong Shin Visibility Status – Sep '09

Week of 09/28

  • IE chokes when URL (endpoint) is not set up with full path; ***.***.com or IP address
  • Fixed tables returning from server not role/user based
  • Added publish status, changed mode to Overwrite to IngestConfirm window
  • Changed date in the table names to return YYYY-MM-DD format
  • Bug found in AutoIngestor
    • found that files with no ext (.xml, .xls, etc.) fails to return columns
    • input validation added to RemoteFileWindow and RoleSelectinCanvas
  • Added dataTips to AutoIngest DataGrid
  • Rearranged New and Remote File buttons in AutoIngestConfig
  • Extending File class to HTTPFile to use remote files
  • Adding HTTPS and HTTP capabilities to other types of files
  • All changes uploaded and downloaded.

Week of 09/21

  • Administrators can now delete any tables from the IngestManager
    • isUserAdmin() methods added to both client and server
  • Changed displaying ID of the user to Names of the Users in IngestPanel and IngestedDataConfiguration
    • TablesInfoObject.java
    • Database Query to join TABLE_USERS and TABLE_DATA tables
    • StyleFunction for the AdvancedDataGrid
  • AutoIngest now checks for existing data (file name and table name) on the client
  • Remote File Ingest to handle http and https
    • Server – HttpUtillities.java and SSLUtilities.java under com.edgeti.utils
    • Flex – added Remote File button in IngestVBox, RemoteFileWindow.mxml to handle remote file ingest
    • remote file added to Files Available after validation
    • change handleFileSelected() in IngestVBox to handle http and https and related events/services
    • columns returned from http and https
    • removed old remote file routines from AutoIngestConfigVBox and put RemoteFileWindow to use http and https autoingest…..
    • changed server codes to use full filename for remote (http) and uploadDir for local files.
    • ingest works!!!! seems nothing is broken….

Week of 09/14

  • disabled Use Remote location for Auto Ingest for now….
  • fixed auto ingest to run specified time, hourly, daily, weekly.
  • creating RoleColumns/RoleTree component so it can be used by IngestVBox and AutoIngest.

Week of 09/07

  • Resolved most of the bugs/fixes/enhancements Phil asked.
  • Reworking Ingestor, AutoIngest Config, and Ingested Data Config to use Roles and Role Columns. This requires changes to the Servlet, too.
  • Ingestor problem resolved
    • moved connect/disconnect for each insert SQL to east ingest. It seems ingestion is a lot faster now. XLS ingestor should be changed
  • Continued troubleshooting ingestor problem
    • Ingestor running too fast?
    • file too large?
  • Remote File ingestion
    • running Tomcat under SYSTEM prevents the Ingestor to read remote files
    • running different user (me) works fine.

Week of 08/31

  • Test Deploy to EdgeDemo – successful
    • change endpoint in assets/setupModel.xml
    • MakeFakeData in ManagementUtilsTest.html fails. It seems there is a problem in populateFakeUsers in UsersAndRoles.java
  • Having problems with ingestor. MySQL reports socket not available error
  • Continue looking into remote file ingesttion

Phil 9.28.2009

7:30 – 3:30

  • Gotta install sp2…. Well that was 20 minutes well spent! Wheee.
  • Working on multiple select = done. Leaving the single select code in place for a while out of, well, superstition and fear, actually.
  • I’m thinking about adding another chart that has the unique counts for each column – that way it would be possible to easily find out how many people we in a query, even if they had multiple skills. – done

Mike 9.27.09

  • Search now automatically scrolls to the selected item in the datagrid if it finds one
  • Saving / loading now once again works with the AdvQueryWidget
  • DynamicPanel may now go directly from minimized to maximized and vice versa
  • Updated all Layout Managers to handle the new panel feature

Phil 9.25.2009

7:30 – 3:30

  • Another meeting with John, Anne, Brian, and Mike this time
  • Based on getting new data (roster2) John is much happier with the tool. Mind you, there have been no changes to the tool.
  • Working on getting an “insert column” capability, *then* we might get to multiple selects – done

Phil 9.24.2009

7:30 – 4:00

  • Long and painful meeting with John and Anne. My notes follow:
    • Data names are not clear from source
    • Should be able to check a box and get a list of names
    • Eliminate redundant columns
    • Fix column names
    • Either the basic tree or the question tree needs to have some checkboxes that represent “where clauses” that would be OR-ed together
    • Collapse redundant entries in columns on Flex side – i.e. how many unique names there are. This may be a new chart view
  • Working on Question interface – first pass is done. Rolling up to bring in.
  • Tomorrow, work on AdvancedQueryWidget.onCreateChartClicked() so that multiple bars can be selected.

Phil 9.23.2001

7:30 – 5:00

  • Ingested data at the fort. Talked to John, who tried to find info with Pat yesterday. I said that the display is only as good as the data, but I don’t think it stuck.
  • Working on “Questions” interface
    • Modify – done
    • Add – done
    • Add as Child – done
    • Delete – done – turned out to be tricky. You have to use the delete operator.
    • Have “Set Query Builder” populate the _questionQuery.text field
    • Add a setTableQuestions(String questions,  String tableName) method in DbRemoteObject that a) Creates a table_questions table if there is none, and b) loads the xml as a blob in the table row (note: only one set of questions per table) See the dashboard entry for how to handle xml – done
    • Add a getTableQuestions(String tableName) method in DbRemoteObject that a) Creates a table_questions table if there is none, and b) either returns the xml if it exists or returns and empty <questions/> xml string. – done

Phil 9.22.2009

7:30 – 3:30

  • Deployed the new version with Mike’s fixes. Everything seems to be working fine
  • Talked to Anne about building special-purpose tables from the master table to make it easier to explore the data
  • Floated the question idea
  • Realized that we could also have all the available strings in a column listed in the (a?) hierachy view, so some level of drilldown could be available…
  • Changed all query double quotes to single quotes
  • Phone call from Anne – Meeting with here and John on Thursday. Might have a version by then.
  • Started on the question tab, mostly because it’s an easier problem to solve that might give us what we want. Go the part where you can click on a tree component “question” that runs a query.
    • Need to be able to interactively build the xml
    • Need to save the xml in a table-specific entry (only one entry per table holding the xml)

Phil 9.21.2009

7:30 – 4:00

  • Hey! It’s Fall! No wonder it’s dark when I get up these days
  • So I come in this morning and find that Mike has fixed half the issues that Anne and John listed. It’s like having pixies do the dishes at night!
  • VIS2 Meeting at 10:30
  • Writing up support for S2Dash
  • Need to make the system more Pat-freindly. Not sure what that means, but I’m leaning towards “Questions” that point to queries