Retrieve data from Olympe datacloud, transform it and display it in a custom Visual Component

Introduction

The Olympe platform natively integrates a datacloud (DC) from and to which data can be pull or pushed. The datacloud can be seen as a distributed memory space accessible from any Olympe Virtual machine part of a Olympe cluster. In this tutorial, we will see how to use the Olympe DC API to retrieve data and exploit it in a visual component

Prerequisites

Once again, this tutorial piggybacks on the previous tutorial, to avoid creating all the boilerplate.

So it is assumed that the following tutorials have been completed already:

  • Initialize the versioning system
  • Extend DRAW with custom visual bricks made with CODE

Overview

We will essentially go through the following steps. First, we will define a couple of data structures that will determine how data will be stored and organized within the DC. Second, we will add some data into the DC - data organized along the lines of the data structures. These two steps will be performed in DRAW. Then we will move into CODE and redact a statement permitting to read these objects.

Create basic data model and access it in both CODE and DRAW

Create a new DRAW project and "describe it" in CODE

Create a new subfolder in your modules folder, and name it datacloud. Create or copy a module.json descriptor in it. Also create a src directory, already containing a _ns.js file defining const datacloud = {}, as usual. Do not forget to register this file into the module.json.

Add similarly a datacloudinit section under the sources section, with the following content

  "datacloudinit": {
    "defaultDomain": "bricks.olympe.ch",
    "initScriptDirectories": [
      "data"
    ]
  }

Make sure this module is imported in the root target (of course, you have thought to rename this new module datacloud).

Finally, connect to your local DRAW instance (localhost:8888) and create a new project in the Default repository. Then enter into it, identify its tag in the url and use this tag as rootTag in a snapshot-config.json file.

You may want to test your configuration by running the snapshooter and by re-initializing you DC with gulp rst.

Create data structure

Create a new Data Model in your Datacloud project. Create a model, name it Participant, then drag String to the Properties zone to add a string properties to Participant, and name this property name. Repeat the operation for a number, the new property should be named age.

datamodel

Generate corresponding code to access this structure in CODE

Once you're done adding properties to your model, click on the menu button (three dots) and select "Generate code skeleton". This should trigger the download of a JavaScript file.

skeleton

Locate the file in your downloads folder and move it into your CODE project, in modules/datacloud/src. Also add it to the module.json. The generated file should look as the following:

/**
 * @extends {olympe.dc.Sync}
 */
home.default.<your_name>.newProject.newDataModel.Participant = class Participant extends olympe.dc.Sync {
    /**
     *
     * @param {olympe.dc.Manager} manager DataCloud Manager instance
     * @param {string} tag Instance tag
     */
    constructor(manager, tag) {
        super(manager, tag)
    }

    /**
     * @return {olympe.df.OString | olympe.df.Proxy<olympe.df.OString>}
     * @osignature {olympe.df.OString}
     */
    getName() {
        return this.getPropertyAsString(home.default.<your_name>.newProject.newDataModel.Participant.nameProp);
    }

    /**
     * @return {olympe.df.ONumber | olympe.df.Proxy<olympe.df.ONumber>}
     * @osignature {olympe.df.ONumber}
     */
    getAge() {
        return this.getPropertyAsNumber(home.default.<your_name>.newProject.newDataModel.Participant.ageProp);
    }

};

// Hardcoded tags
home.default.<your_name>.newProject.newDataModel.Participant.entry = olympe.dc.Registry.registerSync('0173481c907139228b9f', home.default.<your_name>.newProject.newDataModel.Participant);
home.default.<your_name>.newProject.newDataModel.Participant.nameProp = home.default.<your_name>.newProject.newDataModel.Participant.entry.addProperty('01739af9ece44fb8b5da');
home.default.<your_name>.newProject.newDataModel.Participant.ageProp = home.default.<your_name>.newProject.newDataModel.Participant.entry.addProperty('0173481ca4b866c193a2');

Notice the namespace home.default.<your_name>.newProject.newDataModel. that we deduced from the breadcrumb in DRAW. We will have to modify this as the namespace system of CODE is not aligned on the one of DRAW (yet). Hence, press CMD-R to display the "replace in code" feature of WebStorm and replace home.default.<your_name>.newProject.newDataModel. by datacloud everywhere.

Once this is done, we can remark that the lines registering the class and its properties against a tag have been automatically generated.

// Hardcoded tags
datacloud.Participant.entry = olympe.dc.Registry.registerSync('0173481c907139228b9f', datacloud.Participant);
datacloud.Participant.nameProp = datacloud.Participant.entry.addProperty('01739af9ece44fb8b5da');
datacloud.Participant.ageProp = datacloud.Participant.entry.addProperty('0173481ca4b866c193a2');

We will later exploit these lines to access Participant's data.

Populate our datacloud with participant's data

Exit the datamodel editor and create a Data Set. Create a few instances of Participants and populate them with values - make sur you give an a name and an age to your participants.

dataset

Access the data in a brick

Now we move into CODE, where we will retrieve the participant's data. As a first step, we will reuse the code already written in BarChartImpl.js and adapt it to display the participant ages.

Previously, we were getting the data from the following command:

// barchart/src/BarChartImpl.js
const csvValue = this.getProperty(barchart.BarChart.csvProp, context, olympe.df.OString);

which was providing use a dataflow of String, that we were subsequently placing into a olympe.df.processFlows() construct. Every time the string was changing, we would split it and recreate bars.

In the current case, the approach is different. We need first to obtain a collection of Participants, then we will retrieve the age (thus the bar height) over each participant separately. But, of course, we want the adaptive behavior of Olympe to work here as well. Namely, we want bars to change, not only if ages of participant changes, but if participants are added or removed.

Obtain all instances of a model

To obtain a list of Participants, we will use an object called a ListDef, which stands for "definition of a list". A ListDef is always defined by a root tag, i.e. a position in the Olympe database (which is also a graph) where to walk from. In our case, to retrieve participants, we will walk from the Participant model we have recently defined. To retrieve its tag, we use the datacloud.Participant.entry entry.

A listdef can also include one or more Transformer. A transformer describes an operation to perform along the "walk" done over the graph. One of the most simple and frequently used transformer is the so called Related transformer which prescribe to follow a given relation. In the present case, we will follow the instances relation which relation a model to its instance.

Putting all these explanations together, to retrieve our list of Participant, we use the following statement:

// barchart/src/BarChartImpl.js
const participantsList = new olympe.dc.ListDef(datacloud.Participant.entry, [olympe.dc.Sync.instancesRel]);

Note the following:

  • We always pass an array of tranformers when we define a ListDef
  • Any relation in the Olympe world should have its definition in CODE. As a convention, the definition is suffixed by "Rel" and available in the file corresponding to its origin model. In the present case, the origin model is olympe.dc.Sync, which is the root type (thus the most generic type) of object in CODE.

Further explanations It may be constructive to visualize the participants graph and understand what the above query is achieving.

If we open up a Neo4j session in the browser (http://localhost:7474/browser/) and type the appropriate query (that we show afterwards), we obtain the following graph:

participantsGraph

Here, the node in the center Participant is a model and represents its counterpart JavaScript Participant class. Its tag, unsurprisingly, is the same that is indicated in datacloud.Participant.entry = ... (in our case, 0173481c907139228b9f).

We also see height nodes attached to the Participant model by the "is a model of" relation[1], they correspond to the height participants we have entered (and the label shows their respective age).

The ListDef object we have created represents a graph query. The first parameter is the "entry point" of the query, that is, where we start our walk over the graph. The second parameter specifies the transformations we perform when walking over the graph. If we refer to the above graph, we are interested in retrieving all instances of the Participant model, by following the "is a model of" relation backwards. We achieve this by using the olympe.dc.Sync.instancesRel transformer.

To obtain the above graph, we can type the following query in a Neo4j browser session:

MATCH (participantInstance)-[rel:ff022000000000000000]->(participantModel{t:"0173481c907139228b9f"})
    return participantInstance, rel, participantModel

Here, we are interested in retrieving the node representing the Participant model (node participantModel with tag 0173481c907139228b9f; note that the tag you have is very likely different), and all nodes participantInstance that are connected to participantModel by the relation ff022000000000000000 (which is the tag of the "is model of" relation).

"Iterate" over the instances

Once we have this list, we can "connect" a bar to the age of each of its element by calling the method forEach on it, which takes a callback as argument. In this callback, we will describe what to do with our "participants".

// barchart/src/BarChartImpl.js
        participantsList.forEach( (item, key, list) => {
            const participant = /** @type {datacloud.Participant} */item;
            const age = participant.getAge();
            const rank = list.getRank(key);
            const rectangle = new olympe.ui.std.Rectangle(new olympe.df.Vector2(20, age.mul(10)));
            rectangle.setBackgroundColor(
                this.getProperty(barchart.BarChart.colorProp, context, olympe.df.Color, olympe.df.Color.blue())
            );
            layout.appendChild(rectangle, 'rect' + key,
                new olympe.df.Vector2(rank.mul(30), layout.getHeight().minus(age.mul(10)))
            );
        });

As we can see, the callback passed to forEach takes three arguments: the list item to process, its key, and the list itself. At first, we perform a cast onto item. In this way, most IDE's code completion engines will be able to propose the method getAge() (that has been been generated and is available in the class Participant).

Before proceeding to the bar generation itself, we have to retrieve the rank of the item, which, unfortunately, is not given to us as argument in the callback. This rank is important since it determines the (unique) position of item in the list. We will come back to this point later.

Once the participant, age and rank are retrieved, the construction of the rectangle is straightforward. Just note that we need to "append" each rectangle to the layout using a different key. In this case, it is natural to reuse key.

Test the component

Once the code of the bar chart is ready, test it. You should see the ages of the participants you've added. To further test, open DRAW in two different windows, one displaying the dataset editor, the other a screeen with the (modified) bar chart. You should see that eights of bars are automatically adapted. But furthermore, if a participant is removed or added, the chart reacts accordingly.

Summary until there

At this point, you know how to structure the datacloud to put business data in it, how to insert data there, and how to retrieve this data in CODE.

Advanded data retrieval and aysychronous concepts

To further introduce the Olympe DC API, as well as its underlying concepts, we will extend this example in a couple of different ways.

Sorting

We might want not only to retrieve participants, but to retrieve them sorted along seniority. To do this, we can "simply" add a transformer to the listdef.

// barchart/src/BarChartImpl.js
        const participantsList = new olympe.dc.ListDef(datacloud.Participant.entry, [
            olympe.dc.Sync.instancesRel,
            new olympe.dc.transformers.Sort(
                new olympe.dc.comparators.Number(
                    new olympe.dc.valuedefs.NumberProperty(
                        datacloud.Participant.ageProp
                    )
                )
            )
        ]);

Yes, the API is cumbersome (we are working on it), but fundamentally it is no black magic: we define a Sort transformer that will be based on a Number comparison, the value coming from a NumberProperty which is ageProp.

Paste the above code in your example and test it: now your bars should be increasing.

Sorting and ranking

Change the age of a participant such that is provokes a reshuffling of the bars. The bar chart should react to the change and sort automatically. However, the callback given as argument to forEach is not recalled after any reshuffling (you can verify this by adding a console.log in it).

This is part of Olympe's magic, flowing through the rank concept. In the above code, when we retrieved the rank of an object, we didn't otbain a number, but a dataflow. As the positioning of the item changes, the position of the rectangle changes as well but the "piping" keeps constants. There is one rectangle per participant and that's it. But the positioning of this rectangle directly depend on the participant age and on its rank.

Altogether, it is very important to realize that the forEach call is not a loop. In it, we define an operation that applies to any participant. As participants are added, rectangles are added as well, as participants are removed, retangles are removed. Of course, if n participants already exist at the time the chart is rendered for the first time, the callback will be called n times. But the callback is not repeated as participants rank change, contrarily to traditional rendering where any change often triggers a full re-rendering of the full scene.

  1. If we compare the data model to an object-oriented programming language like Java, the Participant model can be thought as the Participant class, the height nodes as instances of Participant and the relation "is a model of" as "is instance of". ↩

← Extend DRAW with custom visual bricks made with CODEModifying the data on the data cloud →