Optional Settings

Several additional options allow you to customize the syncing behavior.

Permissions

By default, permissions are not synced. Any resources that already exist on the target will preserve their permissions while resources that are created by the deployment will grant no access except to the owner (which will be the user account set in TargetConnectionSettings). This behavior can be changed by setting SyncSettings.SyncPermissions.

Sync Permissions

If the boolean option SyncSettings.SyncPermissions is true, then permissions for all DataFlows, QueryViews, and Folders will be copied from the source to the target. This includes the setting to inherit folder permissions, and for Folders includes both content permissions and folder permissions. Permissions granted to "Everyone" or "Anyone with an Account" are always preserved, but other permission mapping is done by name, either by the name of a group or by the username for individual user permissions. If a group or user does not exist on the target those permissions are lost and a warning is printed.

Required Group Names

To avoid losing desired permissions accidentally, the SyncSettings.RequiredGroupNames option allows you to specify a list of groups by name that are expected to exist on the target server, usually because they have permissions you would like to set during the sync. If any of the required groups are missing, FolderSyncer.Sync will fail before saving any resources to the target. The error message will contain the name of the first missing group.

If SyncSettings.RequiredGroupNames is empty or not set no check will be made.

Keys and Other Identifiers

Every QueryView references a key for its database connection, and DataFlows may reference keys through modules. DataFlows may also refer directly to DataPortals and alerts. During sync, these references are mapped so they refer to the appropriate resource on the target so long as that resource exists. Identification of the matching target resource is primarily done by name, though an option exists to slightly modify this behavior for keys, described below.

When a resource is missing, the behavior depends on the type of resource.

  • If a DataPortal cannot be found, a warning is printed and the reference is nulled out. The resulting target DataFlow will not work correctly until the DataPortal is created and the reference is updated (either manually or by running another sync).
  • If an alert cannot be found, it will be automatically created and associated with the appropriate target dataflow and module. All DataFlows depending on this alert should behave as expected
  • If a key cannot be found, by default a warning is printed and the reference is nulled out. The resulting target DataFlow or QueryView will not work correctly until the key is created and the reference is updated (either manually or by running another sync). This behavior can be modified by the SyncSettings.MissingKeyBehavior option.

Missing Key Behavior

Four options exist for SyncSettings.MissingKeyBehavior:

  • MissingKeyBehavior.Warn: Null out the reference to the missing key and print a warning. This is the default behavior.
  • MissingKeyBehavior.Throw: Throw an exception, halting execution when a key is missing. Because resources are saved sequentially, this may result in a partial deployment of resources, but no individual resource will be saved without all key references.
  • MissingKeyBehavior.Create: Create a new key on the target with the appropriate name and (if applicable) folder. Update this and any other references to the missing key (within synced resources) to the newly created key. This newly created key will have no value, which will cause resources referencing it to not work until the key is updated to have the proper value, but no updates to the DataFlows and QueryViews themselves is required. A warning is printed for every created key.
  • MissingKeyBehavior.Copy: Copy the source key to the target, preserving its value. This is the option most likely to guarantee synced resources can run immediately after a deployment, but may result in undesired behavior if key values should be different between environments (e.g. dev and prod database connections). A warning is printed for every copied key. It is highly recommended you review all copied values to ensure they are correct for the new environment.

Unused Missing Keys

By default, SyncSettings.MissingKeyBehavior applies only to keys that will actually be loaded by the resource that references them. In some DataFlows, it is possible to reference a key in such a way that it will not actually be loaded when running the DataFlow; the most common way to do this is to refer to a key by name on a module input while also providing a connection to that input that will override the literal value.

SyncSettings.HandleUnusedMissingKeys allows such "unused" keys to be treated the same as all other keys. If this setting is true, then all keys referenced by any resource will follow the behavior required by SyncSettings.MissingKeyBehavior. Otherwise "unused" keys will be ignored.

Require Foldered Keys

By default, keys are matched by name only. If multiple keys exist with the same name but one of them is in a folder corresponding to the folder of the source key that one will be chosen, but no other folder requirements are applied. If SyncSettings.RequireFolderedKeys is true, then only keys in the target folder(s) will be considered potential matches. This option is most useful when the synced environments are on the same Composable instance but should not share keys.

Source Issues

Sometimes issues with the source resources themselves will impact the behavior of a sync task.

Missing Source Resources

One common issue is a resource missing from the source folder or instance. This can happen because the resource has been deleted or simply moved out of the set of folders that are being synced. By default, any missing resource referenced by a synced resource will throw an error, as it usually represents a problem with the source. The exception to this is "unused" resources (see above), which will by default only produce a warning.

This behavior can be modified by SyncSettings.MissingSourceResourceBehavior and SyncSettings.AllowUsedMissingSourceReferences. There are three options for MissingSourceResourceBehavior:

  • MissingSourceResourceBehavior.Ignore: References to resources that do not exist on the source will be set to null on the target.
  • MissingSourceResourceBehavior.Warn: References to resources that do not exist on the source will be set to null on the target and a warning will print. This is the default behavior.
  • MissingSourceResourceBehavior.Throw: References to resources that do not exist on the source will throw an exception. Because resources are saved sequentially, this may result in a partial deployment of resources, but no individual resource will be saved without all key references.

By default MissingSourceResourceBehavior applies only to "unused" resources. To apply it to all references, set AllowUsedMissingSourceReferences to true.

Outdated DataFlow References

When the inputs and outputs on a nested DataFlow change, an option appears on all references to that DataFlow to upgrade the reference. Depending on the nature of the change, unupgraded references may or may not work correctly or run at all. DataFlow upgrades potentially impact two parts of the sync process: outdated source DataFlows and outdated unsynced target DataFlows.

Outdated Source DataFlows

If a DataFlow reference has not been upgraded on the source, this may cause issues while syncing or running the DataFlow. By default such references will print a warning. This behavior can be modified by SyncSettings.SyncedOutdatedDataFlowReferenceBehavior, which has three options:

Outdated Unsynced Target DataFlows

If the target instance has DataFlows that reference synced DataFlows but are not themselves synced, these references will always be out of date when the inputs and outputs on the source version of the referenced DataFlow are changed. Thus the default behavior during a sync is to automatically upgrade any such target DataFlows. This behavior can be modified by SyncSettings.UnsyncedOutdatedDataFlowReferenceBehavior, which has three options:

Unsynced Resources

By default, syncing will affect only the resources that exactly match those in the source folder, with a couple minor exceptions (see here). This means that additional DataFlows and QueryViews can exist in the target folders. In some cases this may be useful, if those resources are only useful on the target server. On the other hand, this creates a disconnect between the two servers that can be confusing and sometimes is just additional clutter. One case in which the latter is more likely is when resources that were originally synced become unneeded for the project and are moved or deleted from the source folder. In this case, with the default behavior of FolderSyncer.Sync the target versions of those resources will remain in place unless moved or deleted manually.

UnsyncedResourceBehavior

SyncSettings.UnsyncedResourceBehavior provides options for what to do with DataFlows and QueryViews within target folders that are not also present in the corresponding source folders.

Other

Sync Recent Modifications

By default, all resources in the synced folder are synced, even if they already exist in the target folder. This behavior can be changed by using SyncSettings.SyncChangesAfterDateTimeOffset. If a DateTimeOffset is provided for this field, only resources that have been updated on the source after the provided date and time will be synced. For common iterated code promotion scenarios, use a DateTimeOffset slightly before the last time code was synced to the target to capture all new changes from the source.

Note that some resources might be updated on the target even if they have not been updated on the source if they depend on other resources that were synced.

Resource Transformation

For transformations not captured by the default mapping behavior or other optional settings, use SyncSettings.ApplicationTransform and SyncSettings.QueryViewTransform. These options allow you to call a method on every DataFlow or QueryView before saving it, making any required changes. This allows you to make essentially any change you'd like to one or more DataFlows or QueryViews.

For example, if you wanted to disable all timer-activated DataFlows except one called "SQL DB Backup", you could write

syncSettings.ApplicationTransform = (app) =>
{
    if (!app.Value.Name.Equals("SQL DB Backup", StringComparison.OrdinalIgnoreCase))
    {
        foreach (Module module in app.Modules)
        {
            if (module.ModuleType.Name == "Timer")
            {
                foreach (ModuleInput input in module.ModuleInputs)
                {
                    if (input.Name == "Enabled")
                    {
                        input.ValueObj = false;
                    }
                }
            }
        }
    }
};

The input app is a Resource<Application> where Application is a contract for a DataFlow. Through that object we check the DataFlow name, then loop through modules looking for one with module type "Timer". Then we loop through the module inputs to find the "Enabled" input and set its value to false.

If ApplicationTransform (or QueryViewTransform) is set, it will be called on every synced DataFlow (or QueryView), so be sure to write your tranform in such a way that it will only change DataFlows (or QueryViews) you want to change.

Source Search Parameters

The search string SyncSettings.SourceSearchParameters can be specified to filter which DataFlows and QueryViews from the source folders are synced. If specified, the search condition is applied at every source folder when selecting DataFlows and QueryViews to sync. The sync behavior of other resources is not impacted.

With SyncSettings.SourceSearchParameters = "(name : "app1") OR (name : "app2")", a DataFlow or QueryView from one of the source folders will only be synced if it has the name "app1" or "app2".

With SyncSettings.SourceSearchParameters = "(Type : "DataFlow")", only DataFlows will be synced.

Sync Run Retention Settings

If the boolean option SyncSettings.SyncRetentionSettings is true, then run retention settings for all DataFlows will be copied from the source to the target. These settings are visible on the runs page for a DataFlow and include things like "Days to Keep Runs".

If this option is false or not set, DataFlows that already exist on the target will preserve their run retention settings while DataFlows that are created by the deployment will have default settings.