Skip to main content
Version: 4.47.0

SPO-Migration

This component allows you to migrate to Sharepoint Online (using migration packages) into existing sites. It uses the Sharepoint Migration API. It does this by communicating with Azure Storage and Sharepoint Online to perform the migration. Packages and binaries are uploaded to Azure storage and then the migration is triggered from Sharepoint Online.

In migration mode the component creates two folders in Azure Storage, named in Azure Storage as manifest container and source container. When the containers are created the following is being processed in order:

  • manifest container: the manifest files from the package will be uploaded to this container as blobs. Documentation on the manifest files can be found here

  • source container: the files (that will be migrated to Sharepoint Online) on the local file system will be uploaded to this container as blobs.

    Above actions will be performed using the Azure Storage REST API.

  • When all blobs are uploaded into the manifest and source container, the content will be imported to Sharepoint Online from Azure Storage using the Sharepoint Migration API. In order to access the local filesystem, the XILL4_WORKDIRS environment variable must be set to the path of the directory to be accessed.

Detailed process description​

Configuration mode set to Migrate​

The Sharepoint Online API component requires an Azure connection string and incoming message with key packagePath that contains a path referencing a migration package.

As explained above, from this folder, the manifest files will be uploaded as blobs to a newly created manifest-#####-#####-##### container, this container will also contain the logs which can be retrieved with configuration mode set to Retrieve logs.

After the package is uploaded, a new container will be created: source-#####-#####-#####, this container will contain the source files as blobs.

Note the hashtags represent a GUID.

The component will use manifest.xml from the package to locate the files on the file system and upload them to Azure storage, the ParentWebUrl from the first SPO Object will be used as target site for the requests to Sharepoint.

In the manifest file, the component will search for all File Elements and use the values from the following attributes:

  • SetupPathUser reference to path on the local file system
  • FileValue reference to use as FileId on Azure Storage

Example of an File Element:

<SPObjects>
    <SPObject ObjectType="SPFile">
        <File SetupPathUser="C://Pictures/Grand canyon.jpg" FileValue="5c764bc96b5b7d3a520e9c0305dc7a71"> </File>
    </SPObject>
</SPObjects>

Example of an File Element with a reference to a binary in GridFS:

<SPObjects>
    <SPObject ObjectType="SPFile">
        <File SetupPathUser="gridfs#969507351353" FileValue="5c764bc96b5b7d3a520e9c0305dc7a71"> </File>
    </SPObject>
</SPObjects>

The created containers including the blobs can be found in the Azure Portal at storage account -> Data Storage -> Containers

After finishing all uploads to Azure, the migration job to Sharepoint will be triggered. After finishing the migration, the component will send the migrationJobId, azureContainerManifestUri, azureContainerSourceUri, manifestContainerName, sourceContainerName, azureQueueReportUri and optionally an encryptionKey depending on the setting provisioned containers enabled. The log files of this import to Sharepoint can be found in the Azure Portal in the created manifest container as blob(.err/.wrn/*.log). Err and wrn logs are optional and will only be created when errors or warnings occurred in the process.

The logs can also be retrieved using the component. Change the configuration setting Mode from Migrate to Retrieve logs. For retrieving logs from a migration using provisioned containers, the following keys are required in the incoming message: azureContainerManifestUri, azureQueueReportUri, migrationJobId and encryptionKey. When using a storage account during migration, only azureContainerManifestUri is required.

Configuration​

General​

Mode

Options: Migrate, Retrieve logs, Delete containers. Use Migrate to migrate content to Sharepoint Online or Retrieve logs to retrieve the logs from a previous migration. When a migration is finished and the logs are retrieved, the option is there to delete the created containers in Azure Storage. Use Delete containers to delete any container(s) by container name(s).

Path to store logs

The location where the logs are stored on the local file system when retrieving logs.

Database settings​

Connection string

A MongoDB connection string.

Example: mongodb://<username>:<password>@localhost:27017/<databaseName>

Here <databaseName> is the database to retrieve the binaries.

Use TLS

Whether or not to use TLS in case your mongoDB requires TLS.

Allow Invalid Certificates

Checking this will disable certificate validation. Warning: specifying this option in a production environment makes your application insecure and potentially vulnerable to expired certificates and to foreign processes posing as valid client instances.

Certificate Authority File

One or more certificate authorities to trust when making a TLS connection. In order to access the local filesystem, the XILL4_WORKDIRS environment variable must be set to the path of the directory to be accessed.

Example: .\ca.pem

Azure & Sharepoint settings​

Provision storage queue and containers

Usage of provisioned containers and queue, if not enabled, additional storage account connection string is required in configuration.

Azure Storage connection string

Connection string from Azure, retrieve it from Azure portal -> storage account -> Access keys -> key1. (key2 can be used instead if configured in the 'shared access signature' tab)

Example: DefaultEndpointsProtocol=https;AccountName=myAccount;AccountKey=myKey;EndpointSuffix=core.windows.net

Queue name

Naming for the storage queue in Azure portal. (default: xill4-queue)

Encrypt storage account migration data

Enables encryption for all blobs that are uploaded to Azure. Migration to Sharepoint from azure will be encrypted. (Migrated content will be available in Sharepoint decrypted)

SharePoint username

SharePoint username to use for authentication.

SharePoint password

SharePoint password to use for authentication. Do not use the password of the user if MFA is enabled, but use an app password instead.

Rate limit​

Request limit

The max amount of requests during the interval.

Interval

The interval in milliseconds in which the requests happen. Should be a multiple of 250.

Max concurrent

The maximum concurrent executions per message. API components that process an incoming message may need multiple requests towards an external API that maintains a request limit. To be able to keep the rate limiter aligned with this request limit and configurable per request, the message that is being processed in the component will be divided into multiple scheduled tasks that will be executed in order. This setting executes a batch of these scheduled tasks within the maximum request limit/interval.

Inputs​

Config mode set to Migrate​

The component expects a message with the key packagePath when mode is set to Migrate. In this example, the manifest (package) files are located in C://Training.

Example:

{
    packagePath: "C://Training"
}

Config mode set to Retrieve logs​

The component expects a message with key manifestContainerName when mode is set to Retrieve logs. In this example, the logs from the migration above are retrieved. By using the azureContainerManifestUri, azureQueueReportUri and migrationJobId from the output of that example When using provisioned containers make sure to add the encryptionKey as 4th required key.

Example (using storage account):

{
    "azureContainerManifestUri": "manifest-sd233-2flg3-sss21",
}

Example (using storage account if migrated with setting Encrypt storage account migration data):

{
    "azureContainerManifestUri": "manifest-sd233-2flg3-sss21",
    "encryptionKey": "key"
}

Example (using provisioned containers enabled):

{
    "azureContainerManifestUri": "manifest-sd233-2flg3-sss21",
    "migrationJobId": "22-dfs2fsd-wew34dfs-wds",
    "azureQueueReportUri": "https://spodb1am1m038p2.blob.core.windows.net/22-dfs2fsd-wew34dfs-wds",
    "encryptionKey": "key"
}

Config mode set to Delete containers​

The component expects a message with key containerNames when mode is set to Delete containers. In this example, the manifest and source container created in Azure Storage with the migration from the example above will be deleted. By using the manifestContainerName and sourceContainerName from the output of that example Provisioned containers are deleted automatically by Sharepoint.

Example:


{
    containerNames: ["manifest-sd233-2flg3-sss21", "source-saf3d-dw3fg-sdf32"]
}

Outputs​

Example output with configuration mode set to Migrate​

Outputs the migrationJobId, azureContainerManifestUri, azureContainerSourceUri, azureQueueReportUri, manifestContainerName, sourceContainerName and optionally encryptionKey.

{
    value: {
        packagePath: "C://Training"
    },
    result: {
        azureContainerManifestUri: "https://test.blob.core.windows.net/0-23233-232232", 
        azureContainerSourceUri: "https://test.blob.core.windows.net/0-23233-132232"
        azureQueueReportUri: "https://spodb1am1m038p2.blob.core.windows.net/22-dfs2fsd-wew34dfs-wds",
        encryptionKey: "key",  // this key will only be returned when using provisioned containers enabled 
        migrationJobId: "8abf0a04-e243-4e53-9c53-1b9ca00c303d"
        manifestContainerName:"manifest-sd233-2flg3-sss21",
        sourceContainerName:"source-saf3d-dw3fg-sdf32"
    }
}

Example output with configuration mode set to Retrieve logs​

Outputs hasResult and logPath as a result. When retrieving logs, the keys errPath and wrnPath are optional. When warnings or errors occur during the import to Sharepoint Online, the error or warning logs will be included in the output. In this example an error has occurred.

{
    value: {
        azureContainerManifestUri: "manifest-sd233-2flg3-sss21"
    },
    result: {
        hasResult: true,
        logPath:"C://sharepoint-logs/Import-90d47d76-5db6-449f-afcb-cec80d2ae337-1.log",
        errPath:"C://sharepoint-logs/Import-90d47d76-5db6-449f-afcb-cec80d2ae337-1.err"
    }
}

Example output with configuration mode set to Delete containers​

Outputs deletedContainerNames as a result. An error will be thrown for each container that failed to delete and will be excluded from the result.

{
    value: {
        containerNames: ["manifest-sd233-2flg3-sss21","source-saf3d-dw3fg-sdf32"]
    },
    result: {
        deletedContainerNames: ["manifest-sd233-2flg3-sss21","source-saf3d-dw3fg-sdf32"]
    }
}