Skip to content

WIP System Overview for Demo

Flynn Duniho edited this page May 29, 2024 · 2 revisions

The Developer Experience (DX)

We have strived to make NetCreate's development environment easy-to-deploy and use. This not only makes it nicer to use for Inquirium, but we also hope it encourages collaboration by making operational concepts straightforward as we can. We think this will help generalist/casual programmers focus more on making something happen than having to muddle-through a lot of arbitrary boilerplate code.

An important secondary goal for designing the development environment is to make it helpful for learning programming concepts and best practices by playing with the code through modification. The motivation here is not to teach programming to people (though maybe it could be useful in that context), but is to help us be more productive in implementing features and extending the system. The primary way we're doing this is by making sure that there is enough context within the code itself to infer what it's doing and where to find more information about it. We've tried to organize the internals of the project (e.g. directory structure, code style) in a way that promotes clarity at the expense of being a bit verbose. As a general approach, our code tries to provide meaningful names and structures that help the programmer visualize the model of operation and the elements that comprise it. This is augmented with system descriptions and other reference links in comments. While this takes more time to write, we think it results in improved programmer productivity and better code quality overall so it balances out.

NetCreate DX Features

NOTE: not writing this for a general audience or explaining terms because that is not strictly part of the delivery, just calling out the technical whys and hows. Can explain in person during our calls if there is interest.

Development Tools

  • Using Brunch.io as our build tool. It's modern and fast, replacing three or four separate tools and simplifying our modular javascript. Specifically it handles transpiling JSX, enabling ES7 features, concatenating source libraries into single files, CommonJS-style module loading in the browser, and live code refresh-and-run features.
  • Using NPM as our package manager for both server and client side javascript, replacing Bower for client-side management as per current practice.
  • NPM is also now our main script runner, following current practice. These scripts invoke other tools (like Brunch), but the interface is consistent.
  • EditorConfig is used to enforce text encoding, tabbing, and line-ending standards. Requires plugin installation for some text editors, but this keeps our Git repo from flip-flopping between different-but-meaningless whitespace changes.

Main Front-facing Libraries

  • Bootstrap 4 is the most recent version of the popular CSS framework. We are using ReactStrap, a set of React components that implement Bootstrap 4-style widgets, which mirrors Bootstrap 4 conventions the React way. This makes writing components much less complicated compared to straight HTML Bootstrap, though there are some conventions (e.g. event handling) that are not immediately clear from the ReactStrap documentations.
  • React is the popular framework providing a nice way to create a web app GUI. It is relatively simple conceptually, though it is often not concisely explained. While the basic operation of React is straightforward, making it work with other code requires quite a few mental gymnastics.
  • D3 is the visualization library we are using. It is a framework that is somewhat like React in that it creates widgets, but is designed for information graphics rather than user interface components. Because it shares that role with React, they "fight each other" by having different approaches to managing the display.

Making Applications inside NetCreate

The NetCreate webapp uses Bootstrap 4, React, and D3 for the visual aspects of the application. Under the hood, we have to get them all to behave with each other by writing an operating system layer. We also need to make it obvious what code runs when, and when certain resources are allowed to be updated. This is made more difficult because of the asynchronous nature of web apps in general; javascript is very much reactive and "event driven", so understanding the events and designing an event management system is another significant challenge.

  • NetCreate's underlying code manages system services, the application lifecycle, and communication between components anywhere on the network in a logical manner. This module is called UNISYS by Inquirium, and we're expanding it to help us prototype more quickly by handling the messy stuff.
  • NetCreate's file structure is divided into directories for system code, server code, reusable components, and web app "views". A view is any code that displays something in the browser; you can think of a view as being the user-visible part of a program that's been written for our development environment. There can be multiple views associated with a program, and the NetCreate system can support multiple programs defined in the same environment.
  • NetCreate appears to the user as a web application with a NavBar. Each navigation link points to a View, which renders the associated HTML below the NavBar and loads any associated code. Every View is associated with a directory that contains that associated code and possibly image assets.
  • To create a new webapp within the system, you can duplicate an existing View folder and add a link to it the main AppShell file. The system is designed to concentrate changes to the minimum number of source files, so it should just work.

There are two kinds of Views we support: the default UNISYS view and a special "stand-alone HTML" view. The stand-alone view allows us to support arbitrary javascript code and display it within NetCreate. For example, if someone found a cool D3 demo on the Internet, they could copy it to the special "htmlviews" folder into a subdirectory, add a route to it in the AppShell file, and look at it inside our environment. This is good for several reasons:

  • Because it can be any stand-alone HTML+Javascript combination, you don't need to rewrite it so it works inside another program environment. This is quite time consuming because most Javascript example code is written assuming it has complete control over the browser. This lowers friction for collaborators too.
  • It uses NetCreate itself as the collaboration tool in conjunction with GitHub and its pull request model for contributing code. This makes it easier to distribute and compare the same user experience.
  • Our module system has been tweaked to grant access to UNISYS and all other system services, so you can optionally enhance the demo (with some limits) by loading our modules and using them. The primary example on our mind is using a DATASTORAGE module to provide data to the standalone demo, which is tweaked until it works. Or, contributing programmers who want to visualizing our data can write any Javascript code they want and access our DATASTORAGE module without understanding the complete system.

Web App Developer Convenience Features

A BIT OF HISTORY

One of the challenges of making a modular system like this is making it possible to write simple, declarative code that runs when you expect it to, in response to a particular condition. Javascript itself is not a language that has a syntax that makes it easy to write this kind of reactive code (what we'll call "asynchronous code" from this point onward). However, the browser environment itself requires this kind of coding. The best practices for writing asynchronous Javascript code has changed many times and are always changing as the community tries to come up with more elegant solutions.

There are several major steps we're taking in NetCreate to make this easier to handle, which should result in a more agile prototyping process

  • Lifecycle Management
  • Unified Event Triggering System
  • Unified Data Storage
  • Unified Data Passing Conventions
  • Unified State Change Management
  • Unified Network Messaging

Each of these systems define a way to manage common operations in an application that we've found to be a giant pain in the ass in our past work. We first implemented Unified Network Messaging in the second iteration of STEP, and the other modules are opinionated codifications of best practices that we've evolved.

So what are the benefits we are seeking?

Improving Code Robustness In our experience with STEP, we noticed that we were writing a lot of asynchronous code in different ways to handle different situations, and it was difficult to change. Also, figuring out where to put the code meant a programmer had to have a very sophisticated knowledge not only of asynchronous event handling in the browser, but also of how the entire program changed its internal state under certain conditions. This lead to code that tested for conditions to determine when it should run other code, but the conditions were not always complete and were written in a fragile matter prone to side effects. Our Lifecycle specifies an order of operations for starting an application, such that the condition of the entire system is known before each operation runs. For example, during the INITIALIZE lifecycle event, you can be assured that the React-based user interface has completely rendered, so you may make changes to it. It is easy to attach code to any particular lifecycle event. We also have created a Data Storage/State system that can be used to more easily inspect from any module.

Easier to Remember the One True Way to Do Something There are a lot of ways to respond to an event in Javascript. We counted at least 14 different applications of this in our own codebase, all using a slightly different variation of the same function. Some of these variations are due to the hodge-podge evolution of best practices in Javascript, while others grew out of expedience. We now have defined one way to handle Event-Triggered code that can apply to all of them, using the current best practice of Javascript Promises. We also have one way to handle global variable state and being informed of these changes using the aforementioned Event-Triggered methodology. All these features are managed through UNISYS. While this doesn't eliminate all the weirdnesses in Javascript callback-based functions, it makes it easier to hide the details so we can write our application through the definition of our app events+handlers, event chaining, state, data storage, and networked message/data passing.

Less Magical (i.e. "confounding and stupid") REACT Code One of the challenges of using REACT is figuring out how to do things "the REACT way" to change the appearance of a widget or read the value of it. While the React component model is pretty elegant in how predictable its states and props model is, it also results in bloated parent-child parameter passing for any component with more than a couple of visual parameters. Some of the best practices described by tutorials suggest using React-Redux as a way to get around this, but this merely makes the process of changing/reading a value even more round-about. We've thrown out React-Redux and written a much simpler replacement to provide the bridge between React components and our other system components with our State Change Manager. While it use of this manager still requires forethought to divide the function of the application into sets of operations, you no longer have to come up with a new way to get the data from HERE to THERE because there is ONE PLACE to store that stuff, and it automatically can distribute changes in either direction to any interesting piece of code in React, D3, or our own modules. React-Redux can do this too, but it uses an unwieldy action-creator model that adds several layers of indirection to the process in many different files.

Next Steps

There's one remaining UNISYS module to port, which is Unified Network Messaging. This is the ability to send events/data from one webapp on a computer (or even multiple webapps running on a single computer) to another. We have a version of this in STEP code, but as we are not focusing on networking for the April Demo this will become available later.

We also have not yet written the network database support, but our placeholder unified data model will remain the interface for this in code. Our goal is to create network state that is shared by all webapps, using the underlying messaging system to synchronize all instances. This should make a lot of interesting things possible for applications beyond NetCreate's immediate uses, such as shared datasets that update in realtime across all browsers without having to write any network code OR additional event handlers. We are currently leaning toward using MongoDB, a NoSQL database, for storing data on the server.

While we have a lot of new unified support features, we are still figuring out the actual way to make the GUI work in a polished and well-mannered way. This is the goal of post-April development work, but our underlying system infrastructure is the foundation for managing the all-important data robustly and reliably.

We're also thinking of what kind of developer utilities to provide for Kalani and Joshua for managing the system in realtime. The unified system features described here make the state of apps highly inspectable from any computer on the network, so it should be simple to create a custom webapp that implements monitoring/admin tools that receive and fire specific events to control the entire system. For a good system, forethought is required in naming and designing collections of data. Our system also allows for QUICK AND DIRTY data access too for on-the-fly service creation.

Moving forward from here, there are quite a lot of user interface, workflow things to resolve that we haven't yet addressed, but we are in a good place to start addressing those issues.

Clone this wiki locally