SPO-Migration

This component allows you to migrate to Sharepoint Online (using migration packages) into existing sites. It uses the Sharepoint Migration API. It does this by communicating with Azure Storage and Sharepoint Online to perform the migration. Packages and binaries are uploaded to Azure storage and then the migration is triggered from Sharepoint Online.

In migration mode the component creates two containers (folders) in Azure Storage blob storage, these can be found under Data storage -> Containers, named in Azure Storage as manifest container and source container. When the containers are created the following is being processed in order:

• manifest container: the manifest files from the package will be uploaded to this container as blobs. Documentation on the manifest files can be found here

• source container: the files (that will be migrated to Sharepoint Online) on the local file system or MongoDB will be uploaded to this container as blobs.

  Above actions will be performed using the Azure Blob Storage.

• When all blobs are uploaded into the manifest and source container, the content will be imported to Sharepoint Online from Azure Storage Containers using the Sharepoint Migration API.

Detailed process description

Configuration mode set to Migrate

The SPO Migration component requires an Azure connection string and incoming message with key packagePath that contains a path referencing a migration package.

These migration packages can be stored on the local file system or a cloud storage. In order to access the local filesystem, the XILL4_WORKDIRS environment variable must be set to the path of the directory to be accessed.

The packagePath has this format:

• File system: C://<migration package directory>.
• Cloud storage: <cloud-storage>://<migration package directory>.

As explained above, from this folder, the manifest files will be uploaded as blobs to a newly created manifest-#####-#####-##### container, this container will also contain the logs which can be retrieved with configuration mode set to Retrieve logs.

After the package is uploaded, a new container will be created: source-#####-#####-#####, this container will contain the source files as blobs.

note

The hashtags represent a GUID.

The component will use manifest.xml from the package to locate the files on the file system, cloud storage or MongoDB and upload them to Azure blob storage, the ParentWebUrl from the first SPO Object will be used as target site for the requests to Sharepoint.

In the manifest file, the component will search for all File Elements and use the values from the following attributes:

• SetupPathUser reference to path on the local file system, MongoDB or cloud storage
• FileValue reference to use as FileId on Azure Storage

Example of a File Element with a reference to a binary in the local file system:

<SPObjects>
    <SPObject ObjectType="SPFile">
        <File SetupPathUser="C://Pictures/Grand canyon.jpg" FileValue="5c764bc96b5b7d3a520e9c0305dc7a71"> </File>
    </SPObject>
</SPObjects>

Example of a File Element with a reference to a binary in MongoDB (GridFS):

<SPObjects>
    <SPObject ObjectType="SPFile">
        <File SetupPathUser="gridfs#969507351353" FileValue="5c764bc96b5b7d3a520e9c0305dc7a71"> </File>
    </SPObject>
</SPObjects>

Example of a File Element with a reference to a binary with a reference to a binary in a cloud storage:

<SPObjects>
    <SPObject ObjectType="SPFile">
        <File SetupPathUser="<cloud-storage>://Pictures/Grand canyon.jpg" FileValue="5c764bc96b5b7d3a520e9c0305dc7a71"> </File>
    </SPObject>
</SPObjects>

The created containers including the blobs can be found in the Azure Portal at storage account -> Data storage -> Containers.

After finishing all uploads to Azure, the migration job to Sharepoint will be triggered. After finishing the migration, the component will send the migrationJobId, azureContainerManifestUri, azureContainerSourceUri, manifestContainerName, sourceContainerName, azureQueueReportUri and optionally an encryptionKey depending on the setting provisioned containers enabled. The log files of this import to Sharepoint can be found in the Azure Portal in the created manifest container as blob(*.err/*.wrn/*.log). Err and wrn logs are optional and will only be created when errors or warnings occurred in the process.

The logs can also be retrieved using the component. Change the configuration setting Mode from Migrate to Retrieve logs. For retrieving logs from a migration using provisioned containers, the following keys are required in the incoming message: azureContainerManifestUri, azureQueueReportUri, migrationJobId and encryptionKey. When using a storage account during migration, only azureContainerManifestUri is required.

When using Retrieve logs a path to store logs is required in the configuration. A folder on the local file system or cloud storage can be used here.

Configuration

General

Mode

Options: Migrate, Retrieve logs, Delete containers.
Use Migrate to migrate content to Sharepoint Online or Retrieve logs to retrieve the logs from a previous migration.
When a migration is finished and the logs are retrieved, the option is there to delete the created containers in Azure Storage.
Use Delete containers to delete any container(s) by container name(s).

Path to store logs

The location where the logs are stored on the local file system or cloud storage when retrieving logs.

Database settings

Connection string

A MongoDB connection string.

Example: mongodb://<username>:<password>@localhost:27017/<databaseName>

Here <databaseName> is the database to retrieve the binaries.

Use TLS

Whether or not to use TLS in case your MongoDB requires TLS.

Allow Invalid Certificates

Checking this will disable certificate validation. Warning: specifying this option in a production environment makes your application insecure and potentially vulnerable to expired certificates and to foreign processes posing as valid client instances.

Certificate Authority File

One or more certificate authorities to trust when making a TLS connection. In order to access the local filesystem, the XILL4_WORKDIRS environment variable must be set to the path of the directory to be accessed.

Example: .\ca.pem

Azure & Sharepoint settings

Provision storage queue and containers

Usage of provisioned containers and queue, if not enabled, additional storage account connection string is required in configuration.

Azure Storage connection string

Connection string from Azure, retrieve it from Azure portal → storage account → Access keys → key1.
(key2 can be used instead if configured in the 'shared access signature' tab)

Example: DefaultEndpointsProtocol=https;AccountName=myAccount;AccountKey=myKey;EndpointSuffix=core.windows.net

Queue name

Naming for the storage queue in Azure portal. (default: xill4-queue)

Encrypt storage account migration data

Enables encryption for all blobs that are uploaded to Azure. Migration to Sharepoint from azure will be encrypted. (Migrated content will be available in Sharepoint decrypted)

SharePoint username

SharePoint username to use for authentication.

SharePoint password

SharePoint password to use for authentication. Do not use the password of the user if MFA is enabled, but use an app password instead.

Error Handling

Ignore missing binaries

If a file is missing during migration, the whole migration job will not be canceled but continues till finished.

Keep Azure containers on failed migration

After a failed migration, the containers created in Azure will not be deleted.

Rate limit

Request limit

The max amount of requests during the interval.

Interval

The interval in milliseconds in which the requests happen. Should be a multiple of 250.

Max concurrent

The maximum concurrent executions per message. API components that process an incoming message may need multiple requests towards an external API that maintains a request limit. To be able to keep the rate limiter aligned with this request limit and configurable per request, the message that is being processed in the component will be divided into multiple scheduled tasks that will be executed in order. This setting executes a batch of these scheduled tasks within the maximum request limit/interval.

Max concurrent incoming messages

The maximum number of incoming messages that can be processed concurrently.

Inputs | Outputs

SPO-Migration

Input

Output

Error

00 0 (ilysju47)

Config mode set to Migrate

The component expects a message with the key packagePath when mode is set to Migrate. In this example, the manifest (package) files are located in C://Training.

Example:

{
  "packagePath": "C://Training"
}

Config mode set to Retrieve logs

The component expects a message with key manifestContainerName when mode is set to Retrieve logs. In this example, the logs from the migration above are retrieved. By using the azureContainerManifestUri, azureQueueReportUri and migrationJobId from the output of that exampleWhen using provisioned containers make sure to add the encryptionKey as 4th required key.

Example: (using storage account):

{
  "azureContainerManifestUri": "manifest-sd233-2flg3-sss21"
}

Example: (using storage account if migrated with setting Encrypt storage account migration data):

{
  "azureContainerManifestUri": "manifest-sd233-2flg3-sss21",
  "encryptionKey": "key"
}

Example: (using provisioned containers enabled):

{
  "azureContainerManifestUri": "manifest-sd233-2flg3-sss21",
  "migrationJobId": "22-dfs2fsd-wew34dfs-wds",
  "azureQueueReportUri": "https://spodb1am1m038p2.blob.core.windows.net/22-dfs2fsd-wew34dfs-wds",
  "encryptionKey": "key"
}

Config mode set to Delete containers

The component expects a message with key containerNames when mode is set to Delete containers. In this example, the manifest and source container created in Azure Storage with the migration from the example above will be deleted. By using the manifestContainerName and sourceContainerName from the output of that example

Provisioned containers are deleted automatically by Sharepoint.

Example:

{
  "containerNames": ["manifest-sd233-2flg3-sss21", "source-saf3d-dw3fg-sdf32"]
}