SharePoint Online (OneDrive, Teams) target connector
The SharePoint Online target connector allows you to migrate to pre-configured sites. It consists of multiple flows, which are described below. To migrate content to SharePoint Online certain identifiers are required. The SharePoint Online source connector should be used to retrieve those identifiers.
Features
- Creation of (custom) folders
- Creation of (custom) documents with their versions
- Setting (custom) permissions on folders and documents
- Migrating to OneDrive
- Migrating to Teams*
Note migration to pre-configured Teams channels, to the file section, is supported. Any other Teams functionality is not supported.
The following SharePoint field types are supported:
- Text (single and multiple)
- Choice (single and multiple)
- Number
- Currency
- Date and Time
- Lookup (single and multiple)
- Checkbox
- Managed metadata (single and multiple)
- Person or Group (single and multiple)
Note
- The following document kinds are supported:
CONTAINER
,RECORD
,BINARY
andACL
. - The maximum supported file size by SharePoint Online is currently 250 GB.
- Document sets are officially not supported by the Microsoft Migration API which this connector uses. An event handler that is triggered when creating a document set through the UI is not triggered by the Migration API.
- By default, SharePoint tries to map metadata in the migrated document to columns in the library. For further information, please refer to the SPO system documentation
Model
The SharePoint metadata is populated from the Content Store. Please carefully read this section, as it contains import information for the transformation of objects.
For the ROOT
documents an additional property needs to be set:
{
"migration" : {
"id" : "42ecf431-36e6-4fc2-8e34-df5680e01eb0", // REQUIRED | STRING | this identifier should reference the `source.id` of a SPO object. These are stored by the SharePoint Online source connector with one of these content types: `documentLibrary`, `business` and `Folder`
"migrate" : true // REQUIRED | BOOLEAN | only children of ROOT documents with this value set to 'true' will be created in SharePoint Online
},
"target" : {
// All the Content Store required fields + optional ones if needed. Fields will be mapped to their SharePoint Online counter part automatically.
"hierarchies" : [
"/site/library[/folder]" // REQUIRED | STRING | this path should hold the value of source.hierarchies[0] from the SPO object that is referenced in `migration.id`
]
}
}
The schemas for CONTAINER
and RECORD
documents are described below with an explanation of how the fields are translated to the SharePoint Online fields.
{
"migration" : {
"migrate" : true // REQUIRED | BOOLEAN | Only objects where this value is set to 'true' will be created in SharePoint Online
},
"target" : {
// All the Content Store required fields + optional ones if needed. Fields will be mapped to their SharePoint Online counter part automatically.
"contentType" : {
"systemName" : "0x0101009E09C19FBC18AC478E34E7DF96733DD9", // REQUIRED | STRING | this identifier should reference the source.contentType.systemName of a SPO content type. These are stored by the SharePoint Online source connector with content type: `ContentType`. When the displayName is provided, the connector will try to find the content type using that provided value. A random placeholder value can then be used for this field
"displayName" : "Some Document" // OPTIONAL | STRING | this value should reference the source.contentType.displayName of a SPO content type. These are stored by the SharePoint Online source connector with content type: `ContentType`. When the displayName is provided it will be used instead of the systemName
},
"created" : {
"principal" : {
"systemName" : "cd5610a2-5abc-4ceb-805b-06fd28c71982" // OPTIONAL | STRING | this identifier should reference the `source.id` of a SPO user. These are stored by the SharePoint Online source connector with content type: `User`
}
},
"lastModified" : {
"principal" : {
"systemName" : "cd5610a2-5abc-4ceb-805b-06fd28c71982" // OPTIONAL | STRING | this identifier should reference the `source.id` of a SPO user. These are stored by the SharePoint Online source connector with content type: `User`
},
"properties" : {
// Any additional properties, using their technical property names defined in SharePoint Online
// The technical name should reference the source.name.systemName of a SPO Field. These are stored by the SharePoint Online source connector with content type: `Field`
// When setting managed metadata properties, the path of the used term should be set as the property value. It can be retrieved from the source.hierarchies.0 value of a SPO term. These are stored by the SharePoint Online source connector with content type: `Term`
// When setting lookup properties, the displayName of the used list item should be set as the property value
// When setting person or group properties, the source.id of the used person or group should be set as the property value. These are stored by the SharePoint Online source connector with content types: `User` and `Group`
// When multiple values have to be set an Array containing those values should be used as the value of the property
// A versions's property value can be emptied by providing `null` as the value
}
}
Note: SharePoint content types defined in a site and used in a library will have similar identifiers. The identifier of the content type in the library will start with the identifier of the content type in the site. For example, a site content type with identifier "abc" will have "abcdef" as an identifier in the library for that specific content type. The same is true for all content types that are derived from a content type. When you reference the content type identifier as known on the site, SharePoint will use any content type that starts with the same identifier in the library. For that reason, it is best practice to always reference library content type identifiers in your transformation.
Each record that represents an actual file should have exactly one binary document.
{
"target" : {
// All the Content Store required fields + optional ones if needed. Fields will be mapped to their SharePoint Online counter part automatically.
"properties" : {
"checkinComment" : "Some comment" // OPTIONAL | STRING | comments for the version, this is the only supported property on a binary
}
}
}
When setting permissions on CONTAINER
and RECORD
documents, the ACL
schema should be used:
{
"target" : {
// All the Content Store required fields + optional ones if needed. Fields will be mapped to their SharePoint Online counter part automatically.
"permissions" : [
"Full Control" // REQUIRED | ARRAY | this value should reference the `source.name.systemName` of a SPO permission level. These are stored by the SharePoint Online source connector with content type: `permissionLevel`. Only the first permission in the array is used. For a description of the permissions please reference SharePoint Online: Site permissions -> Advanced permissions settings -> Permission Levels
],
"principals" : [
{"systemName" : "cd5610a2-5abc-4ceb-805b-06fd28c71982"} // REQUIRED | ARRAY | this identifier should reference the `source.id` of a SPO user or group. These are stored by the SharePoint Online source connector with content types: `User` and `Group`
]
}
}
Flows
SPO (1. Prepare)
The first flow takes all ROOT
documents, and uses the migration.id
and target.hierarchies
values to search for the object in the exported SharePoint Online objects.
Using the information stored in that object, it sets the correct hierarchy value for all underlying objects and also the following identifiers used by the connector.
target.properties.spo.siteId
target.properties.spo.webId
target.properties.spo.webUrl
target.properties.spo.listId
target.properties.spo.listUrl
The migration.id
value is generated by the flow for each object based on the _id
value, and the full SharePoint Online URL for each object is stored in target.properties.spo.fullUrl
.
Furthermore, content types, fields, managed metadata values and users/groups set on each object are validated.
Settings
mongoConnection
The Mongo connection string to connect to.
mongoConnectionExport
The Mongo connection string to connect to containing the exported SharePoint Online objects.
enableDocumentID
Enables setting the Document ID metadata field for each RECORD
with a value based on the migration.id
value of its BINARY
. Requires the SharePoint Online Document ID service to be enabled. Can either be true
or false
.
SPO (2. Import Document Sets)
Document sets and their ancestors are created by this flow using the REST API.
Note: Modifier and modification date for document sets cannot be set by the used API. For the ancestors these metadata fields will be correctly set.
Settings
mongoConnection
The Mongo connection string to connect to.
sharepointUsername
The SharePoint Online account email address used to create the content. This account should have site administrator access.
sharepointPassword
The password belongs to the used SharePoint Online account. Do not use the password of the user if MFA is enabled, but use an app password instead.
SPO (3. Generate packages)
This flow creates import packages containing XMLs in a specified directory. These packages are generated per SharePoint list. The flow consists of three sub-flows:
- Flow that fetches all current version
RECORD
documents to be migrated and, based on themaxPackageSizeInMB
parameter, groups those into packages. The parentCONTAINER
documents get the same package number. This means thatCONTAINER
documents can have multiple package numbers when it has children in multiple packages. - Flow that fetches all remaining empty folders and assigns a package number. For each package a
RECORD
document is created of content typeSPOPackage
. - Flow that fetches all
RECORD
documents with content typeSPOPackage
. For each package all migratable documents are retrieved and XML packages are generated. When the XMLs for a package are successfully generated, theRECORD
package object is updated with the location of the import package.
Note:
Running this flow will clear all previously created RECORD
documents with content type SPOPackage
. Sub-flow 3 can be run independently in case of errors by disabling sub-flows 1 and 2 after they have completed.
Settings
mongoConnection
The Mongo connection string including the database name to connect to.
pathOutput
The directory where the packages are created.
maxPackageSizeInMB
The maximum size in megabytes of all files in a package. By default, 250 is set and recommended by Microsoft.
maxPackageItems
The maximum number of files including versions in a package. By default, 250 is set and recommended by Microsoft.
enableReset
Removes all previously generated import packages (RECORD
documents with content type SPOPackage
) from the Content Store when set to true
. By default, the value is set to false
.
SPO (4. Upload)
The previously created SPOPackage
documents are retrieved to fetch the locations of the import packages. The XMLs and files referenced in the Manifest.xml
file in fields SetupPathUser
are uploaded to Azure containers and then migrated to SharePoint Online. The id of the migration job and the created Azure container names are stored in the SPOPackage
document.
Settings
mongoConnection
The Mongo connection string including the database name to connect to.
azureConnectionString
The Azure storage account name and key which is needed to create the Azure containers. The connection string can be retrieved from the Azure portal (Storage account -> Access keys -> key1). Syntax of the Azure connection string DefaultEndpointsProtocol=[http|https];AccountName=myAccountName;AccountKey=myAccountKey
(key2 can be used instead if configured in the 'shared access signature' tab)
An Azure storage account is only required if option azureProvisionedContainers
is disabled.
azureProvisionedContainers
When set to true
Azure will provision the containers and an Azure storage account is not required. Can either be true
or false
.
An azureConnectionString
is not required when set to true
.
When using Azure provisioned containers all data is encrypted and therefore the performance is lower than when using an Azure storage account.
sharepointUsername
The SharePoint Online account email address used to create the content. This account should have site administrator access.
sharepointPassword
The password belongs to the used SharePoint Online account. Do not use the password of the user if MFA is enabled, but use an app password instead.
SPO (5. Monitor)
When SharePoint Online has processed an import package, log files (.log, .wrn and .err) containing the result of the migration are written in the Azure containers. This flow retrieves the logs and stores the locations of those log files in the SPOPackage
document. Furthermore, if there are errors within the package, the SPOPackage
document will have its migration.failed
flag set to true
.
Settings
mongoConnection
The Mongo connection string including the database name to connect to.
azureConnectionString
The Azure storage account name and key which is needed to create the Azure containers. The connection string can be retrieved from the Azure portal (SETTINGS > Access keys). Syntax of the Azure connection string DefaultEndpointsProtocol=[http|https];AccountName=myAccountName;AccountKey=myAccountKey
pathToStoreLogs
The directory where the retrieved package log files are stored.
SPO (6. Parse logs)
The error logs that have been retrieved by the monitor flow are read and parsed. There are two possible outcomes:
- There was a fatal error
- Individual folders and/or files were not created
In the first outcome, all the documents belonging to the package will have their migration.failed
flag set to true
.
In the second outcome, the individual documents that failed have their migration.failed
flag set to true
.
In all cases the migration.failedMessage
contains the error message.
Settings
mongoConnection
The Mongo connection string including the database name to connect to.
Re-running packages
A failed package can be retried by fixing the issues for the objects in the package and recreating the package. When making changes to the name or term value(s) of an object, flow SPO (1. Prepare)
has to be run again. Recreating the XML files in the package can be done by running the retry trigger in the third sub-flow of flow SPO (3. Generate packages)
.
SPO (7. Delete containers)
After the migration has concluded, the created Azure containers can be deleted. This flow retrieves all SPOPackage
documents for which the logs were retrieved and deletes the Azure containers.
This step is only required when using an Azure storage account. When using Azure provisioned containers, Azure will take care of deleting the containers.
Settings
mongoConnection
The Mongo connection string including the database name to connect to.
azureConnectionString
The Azure storage account name and key which is needed to delete the Azure containers. The connection string can be retrieved from the Azure portal (SETTINGS > Access keys). Syntax of the Azure connection string DefaultEndpointsProtocol=[http|https];AccountName=myAccountName;AccountKey=myAccountKey
Known errors and warnings
SPO (1. Prepare)
Warning: No terms found for
Warning: No content types found for
This warning occurs when terms or content types can't be found in the Content Store containing the exported SharePoint Online objects. To retrieve the missing terms and content types, run the second flow of the SharePoint Online source connector (SharePoint Online (2. Content Types & Term Store)).
Warning: No permission levels found for
This warning occurs when permission levels can't be found in the Content Store containing the exported SharePoint Online objects. To retrieve the missing permission levels, run the third flow of the SharePoint Online source connector (SharePoint Online (3. Permission Levels)).
Warning: No children found for root
This warning occurs when a ROOT
document has no children with migration.migrate
set to true
. This might mean that you have an unexpected outcome of your transformation. Please check your transformation.
Error: Unknown content type: content type
Error: Unknown field: field
for content type: content type
Error: Unknown term: value
in field: field
for content type: content type
Error: Unknown creator user: value
Error: Unknown modifier user: value
Error: Unknown ACL user or group: value
Error: Unknown ACL permission: value
This error occurs when an object has a content type, field, term, user, group or permission level specified that is not available for the site or library in the Content Store containing the exported SharePoint Online objects. Please check your transformation and the exported content.
Error: textField property not found for term field: field
for content type: content type`. Please check the SPO export and if needed rerun the content type export
This error occurs when the `sub-flow that uses the REST API to get the term fields has not run successfully.
Error: Lookup list id values not found for list id: value
configured in field: field
for content type: content type
This error occurs when the list items associated with the lookup field can't be found in the Content Store containing the exported SharePoint Online objects. Please make sure you have exported the list items.
Error: Multiple terms found for single term field: field
for content type: content type
Error: Multiple choices found for single choice field: field
for content type: content type
Error: Multiple lookups found for single lookup field: field
for content type: content type
Error: Multiple users or groups found for single personAndGroup field: field
for content type: content type
This error occurs when you try to set multiple values on a single value field. Please check your transformation.
Error: Invalid value: value
in boolean field: field
for content type: content type
Error: Unknown choice: value
in field: field
for content type: content type
Error: Unknown lookup: value
in field: field
for content type: content type
Error: Value is not of type date for date(time) field: field
for content type: content type
Error: Unknown user or group: value
in field: field
for content type: content type
Error: Group: value
not allowed in person only field: field
for content type: content type
This error occurs when the set value is not allowed for that field. Please check your transformation.
Error: Hierarchy does not match with used document library. Rerun hierarchy `sub-flow!
Error: Object name does not match with hierarchy. Rerun hierarchy `sub-flow!
This error occurs when the name of hierarchy of the object was changed after the prepare flow executed. Please run the set hierarchies
`sub-flow.
Error: URL length exceeds 400 characters: value
Error: Object name contains forbidden characters: value
Error: Object name contains starting and/or trailing whitespace: value
Error: Folder name contains a dot at the end: value
Error: Binary is larger than 250 GB
Error: Date value
is not between 1/1/1900 and 12/31/8900 in field: field
for content type: content type
This error occurs when the data is not transformed according to the SharePoint Online requirements. Please check your transformation.
Error: No migratable binary found
Error: Binary target missing
Error: Binary has no localReference or externalReference set
This error occurs when you have a migratable binary that is not transformed correctly. Please check your transformation.
SPO (3. Generate Packages)
Error: siteId, webId and listId cannot be empty!
This error occurs when you are trying to package an object that has not gone through the prepare flow. Please run the prepare flow.
Error: Multiple versions with the same version numbers found
This error occurs when you have multiple versions with the same version label. Please check your transformation.
SPO (6. Parse logs)
Error: The folder with id '5da2e373-c7db-4ee2-9d69-8c6f6d613d6d' is present in the database but not as part of targeted web '5d7d831f-eea8-4f63-b0ff-faffdc7cab51'
This error occurs when a folder with the same ID is already migrated to a different site. This can happen when the same folder is migrated to multiple sites. A folder with the same ID can't be migrated to different sites.