Skip to main content
Version: 4.54.0

SharePoint Online (OneDrive, Teams) target connector

The SharePoint Online target connector allows you to migrate to pre-configured sites. It consists of multiple flows, which are described below. To migrate content to SharePoint Online certain identifiers are required. The SharePoint Online source connector should be used to retrieve those identifiers.

note

Please note that it is crucial to use the same versions for both the SharePoint source and target connectors to maintain compatibility and avoid any potential issues during the migration process.

Features​

  • Creation of (custom) folders
  • Creation of (custom) documents with their versions
  • Creation of links (.url files for internal and external destinations)
  • Setting (custom) permissions on folders and documents
  • Migrating to OneDrive
  • Migrating to Teams*
note

Migration to pre-configured Teams channels, to the file section, is supported. Any other Teams functionality is not supported.

The following SharePoint field types are supported:

  • Text (single and multiple)
  • Choice (single and multiple)
  • Number
  • Currency
  • Date and Time
  • Lookup (single and multiple)
  • Checkbox
  • Managed metadata (single and multiple)
  • Person or Group (single and multiple)
note
  • The following document kinds are supported: CONTAINER, RECORD, BINARY and ACL.
  • The maximum supported file size by SharePoint Online is currently 250 GB.
  • Document sets are officially not supported by the Microsoft Migration API which this connector uses. An event handler that is triggered when creating a document set through the UI is not triggered by the Migration API.
  • By default, SharePoint tries to map metadata in the migrated document to columns in the library. For further information, please refer to the SPO system documentation

The following storages are supported for writing and reading migration packages & logs.

note

When using a cloud storage, corresponding environment secret(s) must be set in the project to be able to connect to the specific cloud storage. See the documentation about cloud storages here. For storing created migration packages, set flow variable pathOutput in the accelerator SPO (3. Generate packages) to: <cloud-storage>://<directory migration-packages>. For storing retrieved logs, set flow variable pathOutput in the accelerator SPO (5. Monitor) to: <cloud-storage>://<directory migration-packages>.

Model​

The SharePoint metadata is populated from the Content Store. Please carefully read this section, as it contains import information for the transformation of objects.

For the ROOT documents an additional property needs to be set:

{
  "migration" : {
    // REQUIRED | STRING | this identifier should reference the `source.id` of a SPO object. These are stored by the SharePoint Online source connector with one of these content types: `documentLibrary`, `business` and `Folder`
    "id" : "42ecf431-36e6-4fc2-8e34-df5680e01eb0", 
    // REQUIRED | BOOLEAN | only children of ROOT documents with this value set to 'true' will be created in SharePoint Online
    "migrate" : true 
  },
  // All the Content Store required fields + optional ones if needed. Fields will be mapped to their SharePoint Online counter part automatically.
  "target" : {
    // REQUIRED | STRING | this path should hold the value of source.hierarchies[0] from the SPO object that is referenced in `migration.id`
    "hierarchies" : [
      "/site/library[/folder]" 
    ]
  }
}

The schemas for CONTAINER and RECORD documents are described below with an explanation of how the fields are translated to the SharePoint Online fields.

{
  "migration" : {
    // REQUIRED | BOOLEAN | Only objects where this value is set to 'true' will be created in SharePoint Online.
    "migrate" : true 
  },
  "target" : {
    // All the Content Store required fields + optional ones if needed. Fields will be mapped to their SharePoint Online counter part automatically.
    "contentType" : {
      // REQUIRED | STRING | this identifier should reference the source.contentType.systemName of a SPO content type. These are stored by the SharePoint Online source connector with content type: `ContentType`. When the displayName is provided, the connector will try to find the content type using that provided value. A random placeholder value can then be used for this field.
      "systemName" : "0x0101009E09C19FBC18AC478E34E7DF96733DD9", 
      // OPTIONAL | STRING | this value should reference the source.contentType.displayName of a SPO content type. These are stored by the SharePoint Online source connector with content type: `ContentType`. When the displayName is provided it will be used instead of the systemName.
      "displayName" : "Some Document" 
    },
    "created" : {
      "principal" : {
        // OPTIONAL | STRING | this identifier should reference the `source.id` of a SPO user. These are stored by the SharePoint Online source connector with content type: `User`.
        "systemName" : "cd5610a2-5abc-4ceb-805b-06fd28c71982" 
      }
    },
    "lastModified" : {
      "principal" : {
        // OPTIONAL | STRING | this identifier should reference the `source.id` of a SPO user. These are stored by the SharePoint Online source connector with content type: `User`.
        "systemName" : "cd5610a2-5abc-4ceb-805b-06fd28c71982" 
      },
    },
    "properties" : {
      // Any additional properties, using their technical property names defined in SharePoint Online.
      // The technical name of a property can be found on the content type extracted by the SharePoint Online source connector in source.properties.columns[].name.
      // When setting managed metadata properties, the path of the used term should be set as the property value. It can be retrieved from the source.hierarchies.0 value of a SPO term. These are stored by the SharePoint Online source connector with content type: `Term`.
      // When setting lookup properties, the displayName of the used list item should be set as the property value.
      // When setting person or group properties, the source.id of the used person or group should be set as the property value. These are stored by the SharePoint Online source connector with content types: `User` and `Group`.
      // When multiple values have to be set an Array containing those values should be used as the value of the property.
      // A versions's property value can be emptied by providing `null` as the value.
      // Retention labels can be set by populating the display name of the label in a property named `_RetentionLabel_`.
    }
  }
}
note

SharePoint content types defined in a site and used in a library will have similar identifiers. The identifier of the content type in the library will start with the identifier of the content type in the site. For example, a site content type with identifier "abc" will have "abcdef" as an identifier in the library for that specific content type. The same is true for all content types that are derived from a content type. When you reference the content type identifier as known on the site, SharePoint will use any content type that starts with the same identifier in the library. For that reason, it is best practice to always reference library content type identifiers in your transformation.

Each record that represents an actual file should have exactly one binary document.

{
  "target" : {
    // All the Content Store required fields + optional ones if needed. Fields will be mapped to their SharePoint Online counter part automatically.
    "properties" : {
      // OPTIONAL | STRING | comments for the version
      "checkinComment" : "Some comment",
      // OPTIONAL | STRING | used when migrating the same file again with additional versions. It will overwrite the generated migration.id. See note below for an explanation on how to use this.
      "migrationId" : "dc98eb3a597336bad16d9f35eceff22f" 
    }
  }
}
note

SharePoint will throw an error when adding new versions to an already migrated document. In order to fix this, set the target.properties.migrationId of the (new) current version BINARY to the migration.id value of the previously migrated current version BINARY. Furthermore, set the target.properties.migrationId of the previously migrated current version BINARY to a new md5 hashed value.

When setting permissions on CONTAINER and RECORD documents, the ACL schema should be used:

{
  // All the Content Store required fields + optional ones if needed. Fields will be mapped to their SharePoint Online counter part automatically.
  "target" : {
    // REQUIRED | ARRAY | this value should reference the `source.name.systemName` of a SPO permission level. These are stored by the SharePoint Online source connector with content type: `permissionLevel`. Only the first permission in the array is used. For a description of the permissions please reference SharePoint Online: Site permissions -> Advanced permissions settings -> Permission Levels.
    "permissions" : [
      "Full Control" 
	],
    // REQUIRED | ARRAY | this identifier should reference the `source.id` of a SPO user or group. These are stored by the SharePoint Online source connector with content types: `User` and `Group`
    "principals" : [
      {
        "systemName" : "cd5610a2-5abc-4ceb-805b-06fd28c71982"
      } 
    ]
  }
}

Flows​

SPO (1. Prepare)​

The first flow takes all ROOT documents, and uses the migration.id and target.hierarchies values to search for the object in the exported SharePoint Online objects. Using the information stored in that object, it sets the correct hierarchy value for all underlying objects and also the following identifiers used by the connector.

  • target.properties.spo.siteId
  • target.properties.spo.webId
  • target.properties.spo.webUrl
  • target.properties.spo.listId
  • target.properties.spo.listUrl

The migration.id value is generated by the flow for each object based on the _id value, and the full SharePoint Online URL for each object is stored in target.properties.spo.fullUrl.

After generating the fullUrl and documentID, the flow processes the link objects and creates binaries for the migratable links in a format acceptable to SharePoint Online for .url files. In cases where the SharePoint Online Document ID service is enabled and the documentID is set for the documents, the flow uses the documentID as the reference for the internal in-scope destinations. If the documentID is not available, the fullUrl will be used as the reference instead.

Note: The target.link.destination of the link objects with the internal destination, should match the target.versionInfo.seriesId of the internal destination, and the url will redirect to the current version of the destination file. The target.name.systemName and target.hierarchies of the link objects, should have .url extension, and this must be set in the transformation step, before running the SPO (1. Prepare) accelerator.

Furthermore, content types, fields, managed metadata values and users/groups set on each object are validated.

Settings​

mongoConnection​

The Mongo connection string to connect to.

mongoConnectionExport​

The Mongo connection string to connect to containing the exported SharePoint Online objects.

enableDocumentID​

Enables setting the Document ID metadata field for each RECORD with a value based on the migration.id value of its BINARY. Requires the SharePoint Online Document ID service to be enabled. Can either be true or false.

enableRetry​

This mode can be used to retry migrating objects that failed to upload correctly to SharePoint. When enabled, only objects that have migration.flags.retry set to true will be processed by this flow. This migration flag will have to be set by the user on the desired objects, including their versions. Can either be true or false.

SPO (2. Import Document Sets)​

Document sets and their ancestors are created by this flow using the REST API.

note

Modifier and modification date for document sets cannot be set by the used API. For the ancestors these metadata fields will be correctly set.

Settings​

mongoConnection​

The Mongo connection string to connect to.

sharepointUsername​

The SharePoint Online account email address used to create the content. This account should have site administrator access.

sharepointPassword​

The password belongs to the used SharePoint Online account. Do not use the password of the user if MFA is enabled, but use an app password instead.

note

The SharePoint username and password are required when using client credentials for authentication.

enableModernAuthentication​

Enforces authentication using a certificate uploaded to Azure portal instead of traditional basic authentication (username/password). For more information on how to create a certificate, go to Creating and Exporting a Self-Signed Certificate.

clientID​

The unique identifier for the client application registered in Azure Active Directory.

tenantID​

The unique identifier of the Azure Active Directory tenant where the application is registered.

certificateThumbprint​

A thumbprint is a unique identifier for the certificate uploaded to Azure Portal. It ensures the application is using the correct certificate.

certificatePrivateKeyPath​

The file path to the private key associated with the certificate.

note

Client ID, Tenant ID, Thumbprint certificate and Certificate private key file path are required when using a client certificate for authentication.

SPO (3. Generate packages)​

This flow creates import packages containing XMLs in a specified directory. These packages are generated per SharePoint list. The flow consists of three sub-flows:

  • Flow that fetches all current version RECORD documents to be migrated and, based on the maxPackageSizeInMB parameter, groups those into packages. The parent CONTAINER documents get the same package number. This means that CONTAINER documents can have multiple package numbers when it has children in multiple packages.
  • Flow that fetches all remaining empty folders and assigns a package number. For each package a RECORD document is created of content type SPOPackage.
  • Flow that fetches all RECORD documents with content type SPOPackage. For each package all migratable documents are retrieved and XML packages are generated. When the XMLs for a package are successfully generated, the RECORD package object is updated with the location of the import package.
note

Running this flow will clear all previously created RECORD documents with content type SPOPackage. Sub-flow 3 can be run independently in case of errors by disabling sub-flows 1 and 2 after they have completed.

Settings​

mongoConnection​

The Mongo connection string including the database name to connect to.

pathOutput​

The directory where the packages are created. This can be a directory to the local file system or a supported cloud storage.

maxPackageSizeInMB​

The maximum size in megabytes of all files in a package. By default, 250 is set and recommended by Microsoft.

maxPackageItems​

The maximum number of files including versions in a package. By default, 250 is set and recommended by Microsoft.

enableReset​

Removes all previously generated import packages (RECORD documents with content type SPOPackage) from the Content Store when set to true. By default, the value is set to false.

enableRetry​

This mode can be used to retry migrating objects that failed to upload correctly to SharePoint. When enabled, only objects that have migration.flags.retry set to true will be processed by this flow. This migration flag will have to be set by the user on the desired objects, including their versions. When objects were transformed again, be sure to run the prepare flow first. Can either be true or false.

SPO (4. Upload)​

The previously created SPOPackage documents are retrieved to fetch the locations of the import packages. The XMLs and files referenced in the Manifest.xml file in fields SetupPathUser are uploaded to Azure containers and then migrated to SharePoint Online. The id of the migration job and the created Azure container names are stored in the SPOPackage document.

Settings​

mongoConnection​

The Mongo connection string including the database name to connect to.

azureConnectionString​

The Azure storage account name and key which is needed to create the Azure containers. The connection string can be retrieved from the Azure portal (Storage account -> Access keys -> key1). Syntax of the Azure connection string DefaultEndpointsProtocol=[http|https];AccountName=myAccountName;AccountKey=myAccountKey

(key2 can be used instead if configured in the 'shared access signature' tab)

An Azure storage account is only required if option azureProvisionedContainers is disabled.

azureProvisionedContainers​

When set to true Azure will provision the containers and an Azure storage account is not required. Can either be true or false.

An azureConnectionString is not required when set to true.

When using Azure provisioned containers all data is encrypted and therefore the performance is lower than when using an Azure storage account.

sharepointUsername​

The SharePoint Online account email address used to create the content. This account should have site administrator access.

sharepointPassword​

The password belongs to the used SharePoint Online account. Do not use the password of the user if MFA is enabled, but use an app password instead. Furthermore, the SharePoint target connector cannot deal with any other non-Microsoft authentication mechanism (e.g. Federated). If this is in place, a Microsoft SharePoint Online service account that bypasses it completely is required.

note

The SharePoint username and password are required when using client credentials for authentication.

enableModernAuthentication​

Enforces authentication using a certificate uploaded to Azure portal instead of traditional basic authentication (username/password).

clientID​

The unique identifier for the client application registered in Azure Active Directory.

tenantID​

The unique identifier of the Azure Active Directory tenant where the application is registered.

certificateThumbprint​

A thumbprint is a unique identifier for the certificate uploaded to Azure Portal. It ensures the application is using the correct certificate.

certificatePrivateKeyPath​

The file path to the private key associated with the certificate.

note

Client ID, Tenant ID, Thumbprint certificate and Certificate private key file path are required when using a client certificate for authentication.

Creating and Exporting a Self-Signed Certificate​

For detailed instructions on creating a self-signed public certificate and exporting it along with its private key, refer to the Microsoft documentation.

Once the public certificate with its private key is exported in a .pfx file, an additional step is required to extract the private key and certificate for use in your flow.

Extracting the Private Key in PowerShell​

  1. Navigate to the directory containing the exported .pfx file.
  2. Run the following commands, replacing {certificateName} with the name of your exported .pfx file:
## private key:
openssl pkcs12 -in {certificateName}.pfx -nocerts -out private-key.pem -nodes
## certificate:
openssl pkcs12 -in {certificateName}.pfx -clcerts -nokeys -out certificate.pem

  1. When prompted, enter the password you used during the .pfx file export process. This will generate a private-key.pem file containing the extracted private key. Ensure this file is stored securely.

  2. Upload the exported certificate.pem to Azure Portal in App registrations -> Select App -> Certificates & secrets.

  3. Use the exported path for private-key.pem as certificatePrivateKeyPath in the flow. This can be a directory to the local file system or a supported cloud storage.

SPO (5. Monitor)​

When SharePoint Online has processed an import package, log files (.log, .wrn and .err) containing the result of the migration are written in the Azure containers. This flow retrieves the logs and stores the locations of those log files in the SPOPackage document. Furthermore, if there are errors within the package, the SPOPackage document will have its migration.failed flag set to true.

Settings​

mongoConnection​

The Mongo connection string including the database name to connect to.

azureConnectionString​

The Azure storage account name and key which is needed to create the Azure containers. The connection string can be retrieved from the Azure portal (SETTINGS > Access keys). Syntax of the Azure connection string DefaultEndpointsProtocol=[http|https];AccountName=myAccountName;AccountKey=myAccountKey

pathToStoreLogs​

The directory where the packages are created. This can be a directory to the local file system or a supported cloud storage.

SPO (6. Parse logs)​

The error logs that have been retrieved by the monitor flow are read and parsed. There are two possible outcomes:

  • There was a fatal error
  • Individual folders and/or files were not created

In the first outcome, all the documents belonging to the package will have their migration.failed flag set to true. In the second outcome, the individual documents that failed have their migration.failed flag set to true.

In all cases the migration.failedMessage contains the error message.

When the option to parse all logs has been selected, all package logs are parsed and stored on the migrated object in target.properties.spo.logs. Furthermore, three flags are set: migration.flags.hasLogs, migration.flags.hasWarnings and migration.flags.hasErrors.

Settings​

mongoConnection​

The Mongo connection string including the database name to connect to.

enableParseAllLogs​

When set to true all package logs are parsed and stored on the migrated object. Can either be true or false.

Re-running packages​

A failed package can be retried by fixing the issues for the objects in the package and recreating the package. When making changes to the name or term value(s) of an object, flow SPO (1. Prepare) has to be run again. Recreating the XML files in the package can be done by running the retry trigger in the third sub-flow of flow SPO (3. Generate packages).

SPO (7. Delete containers)​

After the migration has concluded, the created Azure containers can be deleted. This flow retrieves all SPOPackage documents for which the logs were retrieved and deletes the Azure containers.

This step is only required when using an Azure storage account. When using Azure provisioned containers, Azure will take care of deleting the containers.

Settings​

mongoConnection​

The Mongo connection string including the database name to connect to.

azureConnectionString​

The Azure storage account name and key which is needed to delete the Azure containers. The connection string can be retrieved from the Azure portal (SETTINGS > Access keys). Syntax of the Azure connection string DefaultEndpointsProtocol=[http|https];AccountName=myAccountName;AccountKey=myAccountKey

Known errors, warnings and failed messages​

SPO (1. Prepare)​

Warning: No terms found for​

Warning: No content types found for​

This warning occurs when terms or content types can't be found in the Content Store containing the exported SharePoint Online objects. To retrieve the missing terms and content types, run the second flow of the SharePoint Online source connector (SharePoint Online (2. Content Types & Term Store)).

Warning: No permission levels found for​

This warning occurs when permission levels can't be found in the Content Store containing the exported SharePoint Online objects. To retrieve the missing permission levels, run the third flow of the SharePoint Online source connector (SharePoint Online (3. Permission Levels)).

Warning: No children found for root​

This warning occurs when a ROOT document has no children with migration.migrate set to true. This might mean that you have an unexpected outcome of your transformation. Please check your transformation.

Error: Unknown content type: content type​

Error: Unknown field: field for content type: content type​

Error: Unknown term: value in field: field for content type: content type​

Error: Unknown creator user: value​

Error: Unknown modifier user: value​

Error: Unknown ACL user or group: value​

Error: Unknown ACL permission: value​

This error occurs when an object has a content type, field, term, user, group or permission level specified that is not available for the site or library in the Content Store containing the exported SharePoint Online objects. Please check your transformation and the exported content.

Error: textField property not found for term field: field for content type: content type`. Please check the SPO export and if needed rerun the content type export​

This error occurs when the `sub-flow that uses the REST API to get the term fields has not run successfully.

Error: Lookup list id values not found for list id: value configured in field: field for content type: content type​

This error occurs when the list items associated with the lookup field can't be found in the Content Store containing the exported SharePoint Online objects. Please make sure you have exported the list items.

Error: Multiple terms found for single term field: field for content type: content type​

Error: Multiple choices found for single choice field: field for content type: content type​

Error: Multiple lookups found for single lookup field: field for content type: content type​

Error: Multiple users or groups found for single personAndGroup field: field for content type: content type​

This error occurs when you try to set multiple values on a single value field. Please check your transformation.

Error: Invalid value: value in boolean field: field for content type: content type​

Error: Unknown choice: value in field: field for content type: content type​

Error: Unknown lookup: value in field: field for content type: content type​

Error: Value is not of type date for date(time) field: field for content type: content type​

Error: Unknown user or group: value in field: field for content type: content type​

Error: Group: value not allowed in person only field: field for content type: content type​

This error occurs when the set value is not allowed for that field. Please check your transformation.

Error: Hierarchy does not match with used document library. Rerun hierarchy `sub-flow!​

Error: Object name does not match with hierarchy. Rerun hierarchy `sub-flow!​

This error occurs when the name of hierarchy of the object was changed after the prepare flow executed. Please run the set hierarchies `sub-flow.

Error: URL length exceeds 400 characters: value​

Error: Object name contains forbidden characters: value​

Error: Object name contains starting and/or trailing whitespace: value​

Error: Folder name contains a dot at the end: value​

Error: Binary is larger than 250 GB​

Error: Date value is not between 1/1/1900 and 12/31/8900 in field: field for content type: content type​

Error: Length of value: value exceeds the configured maxLength max length for field: field for content type: content type​

This error occurs when the data is not transformed according to the SharePoint Online requirements. Please check your transformation.

Error: No migratable binary found​

Error: Binary target missing​

Error: Binary has no localReference or externalReference set​

Error: Binary has an invalid externalReference set. Should be a full path!​

This error occurs when you have a migratable binary that has not been downloaded or is not transformed correctly. Please check your export and/or transformation.

Error: Unable to find docId or fullUrl for the destination​

Error: "Unable to find docId or fullUrl for the destination"​

This error occurs when a link destination is missing fullUrl (target.properties.spo.fullUrl) or docId (target.properties.spo.docid._dlc_DocIdUrl). Please check your transformation and rerun the SPO (1. Prepare) accelerator to assign docId or fullUrl to the link destination.

Error: "The internal destination is not in scope"​

This error occurs when a link destination is out of the migration scope, therefore the migration.migrate should be set to false for the link objects with this failed message.

This error occurs when a link destination is missing the extension. Please check your transformation and rerun the SPO (1. Prepare) accelerator to set the hierarchies, so the link can be processed.

SPO (3. Generate Packages)​

Error: siteId, webId and listId cannot be empty!​

This error occurs when you are trying to package an object that has not gone through the prepare flow. Please run the prepare flow.

Error: Multiple versions with the same version numbers found​

This error occurs when you have multiple versions with the same version label. Please check your transformation.

SPO (6. Parse logs)​

Error: The folder with id '5da2e373-c7db-4ee2-9d69-8c6f6d613d6d' is present in the database but not as part of targeted web '5d7d831f-eea8-4f63-b0ff-faffdc7cab51'​

This error occurs when a folder with the same ID is already migrated to a different site. This can happen when the same folder is migrated to multiple sites. A folder with the same ID can't be migrated to different sites.