Flattened Data

[benloh]

Merge after #194!!!

Branch: dev-bl/flatten

The original Net.Create data format was based on the Google Fusion export format that encapsulated arbitrary fields in an attributes object. When reading and writing data, we converted back and forth between an internal representation which used a flat set of variables (e.g. node.type) to the encapsulated format (e.g. node.attributes.NodeType). This added unnecessary complexity to data handling, especially with regards to exporting, importing data and template streamlining.

With the implementation of exporting, importing, and template improvements, this seemed a natural time to also update the data format. It would simplify the implementation of those features.

Features / Changes

Mapping Fields

The old attributes fields have been mapped to root level objects.

NODES

• node.attributes.Node_Type => node.type
• node.attributes["Extra Info"] => node.info
• node.attributes.Notes => node.notes

EDGES

• edge.attributes.Relationship => edge.type
• edge.attributes.Info => edge.info
• edge.attributes.Citations => edge.citation
• edge.attributes.Notes => edge.notes
• edge.attributes.Category => edge.category

Auto Migration

When you open a old pre-version 1.3 dataset file, we will automatically try to migrate the data to the new format. (You can see the migration function in ns-logic.m_MigrateData:1031).

The migrated dataset is then saved on top of the old data.

NOTE that the original attribute data remains in the .loki database file for reference. When you re-open the dataset, NetCreate will ignore the original attributes and just load the new attributes.

Export Labels set via Template's exportLabel property

Export labels can now be set via Template. Change the template's exportLabel to map internal variables to different csv field headers. For example, for the notes field, you can change the exportLabel to Description.

Implications

• Datasets created in this version of the app will not be backward compatible! You have to use Version 1.4+ (the code hasn't been tagged 1.4 yet, but will be) to read these datasets.
• This gets us about halfway towards a system where you can define arbitrary fields in the Template and have NetCreate automatically use them. e.g. You can add a new Gender field in the template and it would be automatically made available in the Node Editor, Node Table, Exports, and Filters.

---

To Test

1. git pull && git checkout dev-bl/flatten
2. ./nc.js --dataset=<existingproject> -- Open an existing project so you can see test the conversion
3. The dataset should open normally
4. Select a node and check that the node data (all fields) are present
5. Select an edge and check that all edge data fields are present
6. Review the Node Table and make sure the fields are correct.
7. Review the Edge Table and make sure the fields are correct.
8. Export a Node file and review the headers and data.
9. Export a Edge file and review the headers and data.
10. Try node and edge filters and make sure they work

[benloh]

@kalanicraig @jdanish Please test this. This data change will require migrating your old datasets to a new format, so make sure you save a copy of your old datasets before trying to open them.

If it's important, we can rename the data file so there's less confusion (like we did with the template file).

But this conversion was the precursor to being to custom define the export labels from the template. e.g. now you can use the .exportLabel field to map id to ID or UID or whatever-the-hell-you-want.

This PR builds on (and includes) the toml changes so you can just work with this.

[netcreateorg]

Sorry if this was noted somewhere and I missed it, but converted files appear to be semi-transparent whereas new ones work aok. Is that a bug or did I mess something up? Thanks.

[netcreateorg]

Random minor thing - in the template interface we currently need an empty node type to set the "no type selected" color. We should either add inline explanation to the editor (less ideal, maybe easier?) or have that be a special entry so that it is obvious what it does? This has always worked this way, but with noobs editing the templates I bet we run into quirks as a result.

[jdanish]

Sorry, last two questions were from me - I switched logins so that it won't read as "net.create org".

[benloh]

Sorry if this was noted somewhere and I missed it, but converted files appear to be semi-transparent whereas new ones work aok. Is that a bug or did I mess something up? Thanks.

This is due to a change in how we handle the default transparency setting for Nodes. There's actually two transparency settings you want:

1. default transparency for nodes in a normal graph
2. default unhighlighted transparency for nodes in a highlight graph
   The old default transparency setting was setting the transparency for the filters (2). The new default transparency is the default graph view (1). This was to match how we were treating the edge transparency -- there was some inconsistency there.

I should probably add a second default transparency value for the highlights. Unless you think my default values of 0.2 for nodes and 0.05 for edges is good enough?

The other approach here would be during the conversion process to force node default transparency to 1.0. I'm not sure why anyone would want a value other than 1.0...

In any case, changing the defaultTransparency for nodes to 1.0 will fix the transparency issue.

Random minor thing - in the template interface we currently need an empty node type to set the "no type selected" color. We should either add inline explanation to the editor (less ideal, maybe easier?) or have that be a special entry so that it is obvious what it does? This has always worked this way, but with noobs editing the templates I bet we run into quirks as a result.

Yeah, this is where the json-editor experience is less than ideal. Right now I'm thinking about either creating our own UI for managing node/edge types. Trying to hack json-editor to support things like combining or splitting is a bit of a pain. This is what I'm working through today.

[jdanish]

OK, I'll try changing the transparency and make sure it makes sense to me and then reply.

Re: the json editor, yeah. It might be enough on our end if you pre-fill it with "Unselected Nodes" or some such and then look for that when reading the template?

[benloh]

Remaining fixes in #200. OK to merge.