Skip to content

Composable Modules

Composable Modules are functional blocks that have multiple inputs and outputs. A Module is analogous to a function in any programming language. A Module takes in one or many inputs, and produces one or many outputs. These outputs can then be connected to any number of other Module inputs.

A Module is an atomic processing element with strongly typed inputs and outputs. All information required for a Module to execute is retrieved from its inputs through connections. Modules can be reused easily and interchanged with other Modules.

Each input and output on a Module has a name, description and a type. Inputs and outputs are strongly typed. Objects of the same type can be assigned, and objects having types that extend base types can be assigned to inputs of the base types. For example, outputs of types integer can be fed into a Module input of type object. Convertibility is also permitted. For example, if a DataFlow author connects an output of type string to an input of type integer, then the system tries to perform the conversion automatically. Note that this can result in exceptions during the running of the DataFlow application. Composable is currently using the built-in conversion framework in the type system, and new input and output types can register conversions by creating casting operator.

First-Class Modules

Composable includes hundreds of first-class Modules that are developed by the product team and shipped with the product. First-class Modules are found in the Module Library, and classified under a specific category. For example, Modules for Image Processing can be found under the Images category, and Modules for integrating with Git can be found in the Git category.

Application Reference Module

Custom Modules may be created by creating a new DataFlow with externalized inputs and outputs. For more information, please see DataFlow Reuse. These Application Reference Modules may then be found under the My DataFlows or Search All DataFlows categories in the Module Library.

Custom Module Extensions

Custom Modules may also be created in code and brought into the Composable platform as Custom Module Extensions. To do this, developers may install a local copy of Composable, and also an IDE such as Visual Studio. Once installed, a "Composable Analytics Plugin" template should be available in Visual Studio under C#. Developers can select this template, select a name and location, and press the OK button. The default plugin template comes with an example Module plugin that can be modified to create a custom Module with custom inputs and outputs.

After importing a Custom Module Extension into the Composable instance, these custom Modules appear as first-class Modules in the Module Library. The following Custom Modules Extensions page includes more detailed information on authoring your own Modules.

Module Features

Composable DataFlows - Module Structure

  • Name: The name of the Module. A Module can be renamed by the user by double-clicking on the name. It is often helpful to document a DataFlow by renaming a Module to indicate its function. You are able to hover over the Module name to see its original name and description.
  • Inputs: Inputs can come from an output of another Module, or be defined in the Module itself. Values from other Modules take priority over the value defined in the Module. Hovering over the green node for each input will show a description and datatype of the input. Right-clicking on the input node will bring up Modules whose output object type matches the input object type. Clicking one of these Module options will add it to your DataFlow, and add a connection to the Module.
  • Outputs: Outputs have similar hover and right-click features as inputs. After running a DataFlow or viewing a run, clicking on the output connection node will display the outputted results of the Module. These can also be accessed by right-clicking the circle and selecting "View Results".
  • Module Settings: Clicking the settings icon will bring up a modal with Error Handling Options

    • Retry Count: Number of times to retry Module execution if it errors
    • Retry Delay: Allows the user to set a delay between retry executions
    • Continue on Error: An error will not prevent the rest of the DataFlow from running.
    • Cache Run Results: Caches the output values of the Module with the inputs when executing. When the same inputs are used on a later run within the user-defined caching period, the cached value is used instead of executing again. For example, if you have a slow SQL query from which the returned data is not expected to change often, you could choose to cache the results of your query Module for 30 minutes in order to avoid running the query as frequently
  • Module Details: Clicking the question mark icon will bring up a modal with the description of the Module as well as descriptions and object types for the inputs and outputs.

Composable DataFlows - Module Details

  • Right Click: Right clicking a Module, or a selection of Modules, brings up a context menu with several options
    • Disable/Enable: Disables/enables execution of the Module. Only applies to current Designer session, disabled status is not saved. Keyboard shortcut Ctrl+E
    • Duplicate: Adds a copy of the selection to the DataFlow.
    • Pin Results: Pinned Results allow you to carry a Module's output values from one run to the next, allowing you to skip executing that Module and guaranteeing that its results will not change. Only applies to current Designer session, pinned results are not saved. Intended as a tool to help with developing, testing and debugging.
    • Unpin Downstream Modules: Unpin any Modules downstream of the current Module to ensure that it will use the inputs generated for the current
    • Enable Forced Execution: Tells the execution engine to always execute this Module, regardless of whether it has results from a previous run it can draw from. In a sense, Pin Results and Force Execution are opposites
    • Statistics: Shows timing and duration history of an executed Module.
    • Show Children (loop Modules only): Adds a red border to Modules downstream of a loop Module.
    • Iterations (looped Modules only): Similar to statistics, displays the timing, duration and status for each iteration of a executed Module in a loop.
    • Create DataFlow from Selection: This will consolidate all of the selected Modules into one "Nested" Module and save it as a separate DataFlow. Double-clicking this nested Module will navigate you to the nested DataFlow.

Composable DataFlows - Module Context Menu

Module Upgrade

Modules are version controlled, and over time may be updated (with new inputs, outputs, logic, etc.) when the Composable instance is updated. If a Module is modified, the existing DataFlow will continue to use the version of the Module that was originally used when the DataFlow was first authored, and a clickable gear icon will appear on the top of the updated Module. The DataFlow author has the choice to continue using the original Module version, or click on the gear icon to upgrade the Module.

Module upgrades are available by clicking on the gear icon

When clicking on the gear icon, you are presented with a modal informing you of the changes to the Module, and are given an option to upgrade the Module to the latest version. In doing so, the Module will then be updated (with new inputs, outputs, etc.). Once updated, the Module cannot be reverted back to a previous version.

Module upgrades are available by clicking on the gear icon

Hovering over the Module name will show the current version of the Module.

Module version can be seen when hovering over the Module name

A detailed explanation on how to update custom Modules is provided at Versioning Modules.