Environments, variables, and secrets

Xill4 provides a centralized system for managing configuration values through flow variables and project environments. These two systems have distinct use cases and capabilities, but can both be used within components in virtually the same way, making it a powerful and flexible system.

Project variables and secrets (environments)

Project variables and secrets provide a secure way to store and share sensitive and/or project-wide configuration values across all components within the flows of a project. They are defined and organized using project environments. Multiple environments can be created in a single project, each with their own set of variables and secrets. This allows for the creation of separate configurations for different project stages, such as development, testing, and production. Environments, as well as the variables and secrets stored within them, are encrypted using OpenSSL with the encryption key from the XILL4_ENVIRONMENT_SECRET system environment variable that is set during installation.

info

To learn how to use or switch between multiple project environments, check out Switching between environments.

Managing project variables and secrets/environments

Project variables and secrets, and the environments they are stored in, are managed using the Environment Editor. The editor is accessed from the project view by clicking the Environment Editor button , or by pressing E. Through this editor, new environments can be created, and existing ones updated, renamed, duplicated, or deleted. Within each environment, a set of secrets and variables can be defined.

Creating a new environment

1. Select the project you want to create the new environment for.
2. Click on the Environment Editor button in the toolbar, or use the shortkey E to open the environment editor.
3. Click to add a new environment.
4. Provide a unique name for your environment (e.g., "development", "testing", "production") and click apply
5. Click to apply your changes and close the Environment Editor.

Renaming, duplicating, or deleting an existing environment

1. Select the project you want to manage the environments for.
2. Click on the Environment Editor button in the toolbar, or use the shortkey E to open the environment editor.
3. Click the menu icon next to the environment you want rename, duplicate, or delete.
4. Select the appropriate action by selecting either , , or .
   1. When renaming an environment, provide a new unique name for your environment and click apply .
5. Click to apply your changes and close the Environment Editor.

warning

Deleting an environment will permanently remove all associated secrets and variables.

Adding/updating variables and secrets in an environment

After creating a new environment or selecting an existing one in the Environment Editor, variables and secrets can be added, updated, or deleted within the environment. Variable and secret names are case sensitive, cannot contain spaces or special characters, and must be unique within an environment.

Adding an environment variable or secret

1. After opening the Environment Editor, select the environment add the variable or secret to.

2. Click to add a new secret (it can be changed to a variable later, if needed).

3. Provide a unique key and a value and, click apply .

4. If needed, toggle between the secret and variable types by pressing

   secret
   or
   variable
   respectively.
   note

   A

   secret

   can only be changed into a

   variable

   when it is first created. A

   variable

   can be changed into a

   secret

   later still.

5. After adding or updating one or more variables or secrets, click to permanently apply your changes and close the Environment Editor.

note

Secrets are stored encrypted and cannot be viewed again once saved. If you need to change a

secret

, you must update it with the new value.

tip

You can add, edit, or delete variables and secrets across multiple (new) environments without saving in-between, as switching or managing environments does not discard any pending changes.

Scope

Environment variables are suitable for, among others:

• Secret and sensitive items, such as connection strings, tokens, passwords, and certificates
• Variables that are shared across multiple flows, such as the path to an (output) directory
• Variables that change depending on the project stage (e.g, testing vs. production)

For variables that are not sensitive and specific to a particular flow, the use of flow variables is recommended.

Flow variables

Flow variables are defined per flow and are accessible only to components within that flow. Unlike environment variables and secrets, flow variables are not encrypted and are stored directly in the flow. This means that flow variables are included during export as well, making them well-suited for flow-specific settings and example configurations.

Managing flow variables

Flow variables are managed within a flow using the Variables management window, accessible through the flow designer by clicking the Variables button . In this window, new variables can be added, and existing updated and removed. Variable names are case sensitive, cannot contain spaces or special characters, and must be unique within the flow.

Adding a flow variable

1. Open the flow you want to add a flow variable to.

2. Click on the Variables button in the toolbar to open the Variables window.

3. Click to add a new variable.

4. Provide a unique key and a value and, click apply .

   note

   A flow variable is always of type

   variable

   and cannot be changed into a

   secret

   due to its unencrypted nature.

5. Click to apply your changes and close the Variables window.

warning

Flow variables are not encrypted and are exported with a flow. Make sure not to store sensitive information in flow variables and to use only example values when exporting a flow.

Scope

Flow variables are suitable for, among others:

• Configurations that are specific to a single flow, such as the extraction origin (e.g., File System or SharePoint Online)
• Quickly enabling or disabling flow features, such as downloading binaries, retrying failed items, removing old data, etc.
• Example configuration, this can include items such as a Mongo connection string, but these values should either:
  • Point to an environment variable (see: Using variables within variables)
  • Be example values, such as mongodb://example:password@localhost:27017/example_db

Sensitive information and secrets should never be defined in either flow variables, or directly in component configurations. Both are stored as unencrypted plain text and are included in flow exports, meaning they are unprotected against unauthorized access and create a significant risk of being shared accidentally with unauthorized parties. Encrypted project environment secrets should be used for such values instead.

Using variables and secrets

Variables and secrets, defined through either project environments or flows, can be references in component configurations using the variable syntax: %variableName%. When a component is activated, this variable token will automatically be replaced with the value stored in the variable or secret. A variable can be combined with plain text, as well as with multiple other variables:

Multiple variable usage example

prefix-%variable1% more text %variable2%suffix

Using the values value1 and value2 for the variables variable1 and variable2 respectively, this will become:

Multiple variable usage example (result)

prefix-value1 more text value2suffix

Usage example: Connection string

You want to use the environment variable instead of a plain-text connection string in, for example, a Document Retrieve component configuration.

1. Open up the environment editor for the project that contains the flow, containing the document retrieve component.
2. Add or select an existing environment, and add a secret called mongo_connection_string.
3. Save the changes and open up the flow containing the document retrieve component.
4. Open up the configuration settings for the Document Retrieve component, and find the option called Connection String.
5. Set the value of the Connection String option to %mongo_connection_string% and save the configuration.

When turning on the flow using the environment containing your new secret, the Document Retrieve component will automatically resolve the secret to the value entered in the Environment Editor.

Advanced example: JSON

Multiple variables can be used in a single field, even with advanced markup, such as SQL or JSON:

Example

{
  "endpoint": "%serverUrl%", // Notice the quotes, as string values are not automatically quoted,
                             // but they must be in order to result valid JSON.
  "followRedirects": true,
  "retry": {
    "enabled": true,
    "maxRetries": %maxRetries% // Here there are no quotes as the variable contains a number.
                               // Note that this is currently invalid JSON, but once the
                               // variable is resolved it is correct again.
  }
}

This will become:

Example (result)

{
  "endpoint": "https://example.com/api/v1/example",
  "followRedirects": true,
  "retry": {
    "enabled": true,
    "maxRetries": 3
  }
}

warning

Variable and secret values are replaced as-is, and are thus not automatically escaped. This means that, for this example, if the value of serverUrl contains any unescaped quotation marks, the resulting JSON would be invalid.

Escaping variables

If a percentage sign is part of the value and not meant to reference a variable it can be escaped with a backslash. If a backslash followed by a percentage sign is needed (for example in a path that includes a variable), the backslash itself must be escaped by using another backslash.

Example	Interpreted value
something%Variable%Something	somethingValueSomething
something\%Variable\%Something	something%Variable%Something
something\\\%Variable\%Something	something\%Variable%Something
something\\%Variable%Something	something\ValueSomething

note

The backslash \\ and percentage sign \% are the only tokens that can be escaped. Other common escape characters, such as \n, \t, or \s, are ignored and kept as-is in the result, including the preceding backslash.

Switching between environments

While environments are defined at project level, the active environment is set separately for every flow when it is turned on. If needed, a flow can be turned on multiple times concurrently with different environments. After turning on a flow, the active environment can still be changed, but only when the flow is idle. When a flow is running, no changes to the active environment can be made.

note

When the active environment of a flow is modified, the environment must be reloaded from the settings menu in the flow designer before the changes are reflected in the flow.

Variable look-up order and missing variables

While variables and secrets within an environment or within a flow cannot have the same name, it can be that a variable or secret with same name occurs in both the active environment and the flow variables. In this case, the flow variables takes precedence, and the variable or secret from the environment with the same name is ignored.

On the other hand, if a component references a variable or secret that does exist in either the flow variables nor the active environment, the component will throw an error when it tries to access it.

Using variables within variables

Variables and secrets can reference other variables and secrets in their values. This can be useful for mapping a flow variable to an environment variable with a different name or naming convention, or to create a new variable that is compounded of multiple variables, for example. Variables that used in other variables have the same look-up order as variables used in component configurations, and similarly an error is thrown when a non-existing variable is referenced.

When one or more secrets are used in either a flow variable or another environment variable, the item in question will be treated as a secret as well, despite being tagged as a

variable

. This doesn't mean that the variable referencing the secret suddenly becomes encrypted or hidden, but it does have implication for the use of the variable, as it will now have the same limitations as a secret.

Secret limitations

For security reasons, not all component settings accept variables and/or secrets. Each setting displays a badge

V

or

S

where variables and secrets are supported, respectively. In general, configuration settings related to authentication, such as connection strings, usernames, passwords, or certificates, accept secrets. If a secret is used in a setting that only accepts variables, the component will throw an error the moment it receives a message, not when turning on the flow or saving the configuration.