Skip to main content
Version: Latest (4.61.0)

SharePoint Online source connector

The SharePoint Online source connector consists of multiple flows, which are described below. These flows use the MS Graph API v1 and the SharePoint Online REST API.

Features​

  • Exporting content structure
  • Exporting users
  • Exporting groups
  • Exporting user OneDrives
  • Exporting content types
  • Exporting managed metadata

Requirements​

For the connector to work, an Azure application with certain permissions is required. An Azure administrator is needed to create this application and to grant permissions.

Graph API​

In order to connect to the Graph API an application needs to be registered using the Azure portal. For more information see MS Graph API v1 authentication.

The following Graph API permissions are required using the Application Permissions tab:

  • Sites.Selected for specific sites or Sites.Read.All for the entire tenant
  • User.Read.All
  • Group.Read.All (optional for retrieving groups)
  • TermStore.Read.All (optional for retrieving managed metadata)

When limited site access is given with the Sites.Selected permission, an administrator has to grant access to the sites. For example, this PnP PowerShell command can be used: Grant-PnPAzureADAppSitePermission. Example script using a certificate:

# Azure application with Graph API `Sites.FullControl.All` permission
$clientId = '5f5b92f0-2097-4a14-a152-3e8172951d67'
$certificatePassword = (ConvertTo-SecureString -AsPlainText 'certificate_password' -Force)

# connect to SPO using a certificate
Connect-PnPOnline -Url "https://contoso.sharepoint.com" -ClientId $clientId -CertificatePath 'C:/spo/certificate.pfx' -CertificatePassword $certificatePassword -Tenant 'contoso.onmicrosoft.com'

# grant permission to the site for the Azure application that the connector will use
$SiteURL = "https://contoso.sharepoint.com/sites/test"
Grant-PnPAzureADAppSitePermission -AppId '9114b753-f288-41be-b81c-16afaa7c79ae' -DisplayName 'Xill4' -Site $SiteURL -Permissions 'FullControl'

For more information see Updates on controlling app specific access on specific SharePoint sites (Sites.Selected)

Furthermore, a self-signed certificate will have to be generated, which is then registered in the Azure application.

Creating and Exporting a Self-Signed Certificate​

For detailed instructions on creating a self-signed public certificate and exporting it along with its private key, refer to the Microsoft documentation.

Once the public certificate with its private key is exported in a .pfx file, an additional step is required to extract the private key and certificate for use in your flow.

Extracting the Private Key in PowerShell​

  1. Navigate to the directory containing the exported .pfx file.
  2. Run the following commands, replacing {certificateName} with the name of your exported .pfx file:
## private key:
openssl pkcs12 -in {certificateName}.pfx -nocerts -out private-key.pem -nodes
## certificate:
openssl pkcs12 -in {certificateName}.pfx -clcerts -nokeys -out certificate.pem

  1. When prompted, enter the password you used during the .pfx file export process. This will generate a private-key.pem file containing the extracted private key. Ensure this file is stored securely.

  2. Upload the exported certificate.pem to the registered application in the Azure Portal: App registrations -> Select App -> Certificates & secrets. The Thumbprint value of the registered certificate is needed by the connector.

SharePoint API​

In order to connect to the SharePoint API, permissions need to be granted to the previously registered Azure application. The following SharePoint API permission is required using the Application Permissions tab:

  • Sites.Selected for specific sites or Sites.FullControl.All for the entire tenant

When limited site access is given with the Sites.Selected permission, an administrator has to grant FullControl access to the sites using previously mentioned PnP powershell command: Grant-PnPAzureADAppSitePermission.

Rate limit​

Every application has its own limits in a tenant, which are based on the number of licenses purchased per organization. In the HTTP request components, the limits are set to make sure throttling is avoided for the lowest amount of license count (0 - 1k), with a safe margin. More info can be found here.

Flows​

SharePoint Online (1. Content)​

The first flow exports the tenant content and users.

Settings​

mongoConnection​

The Mongo connection string including the database name to connect to.

tenantID​

The ID of the tenant to connect to.

tenantName​

The name of the tenant to connect to.

clientID​

The client ID of the application.

clientSecret​

The client secret of the application.

clientCertificateThumbprint​

A thumbprint is a unique identifier for the certificate uploaded to Azure Portal. It ensures the application is using the correct certificate.

clientCertificatePrivateKey​

The file contents of the private key associated with the certificate.

note

Authentication can be done using the clientSecret or clientCertificateThumbprint together with clientCertificatePrivateKey. For more information on how to create a certificate, go to Creating and Exporting a Self-Signed Certificate.

rootPaths​

The rootPaths setting determines where to start crawling within a SharePoint tenant. Multiple starting points can be given. rootPaths is expected to be an array and required to be filled when setting retrieveAllSites to false. Every entry in rootPaths represents a relative path to a site within the tenant.
The relative paths should use the following format:

  • site(s) created directly under the main tenant URL: ["/siteName1", "/siteName2"].
  • site(s) created under the /sites managed path: ["/sites/siteName"].
note

Any amount of relative paths can be given within the rootPaths array. The relative path should always point to a site.
When retrieveAllSites is set to true, rootPaths will be ignored, the requested rootPath(s) will already be retrieved by getting all sites including sub sites.

retrieveAllSites​

When set to true all sites and sub sites within the tenant will be retrieved.

note

Microsoft Teams are also retrieved when this setting is enabled.

retrieveUsers​

When set to true the users are retrieved.

retrieveOneDrives​

When set to true the OneDrives of the users are retrieved.

note

Setting retrieveUsers to true is a prerequisite for retrieving personal OneDrives.

retrieveGroups​

When set to true the Azure AD groups are retrieved. When variable tenantName has also been set and access has been granted on a tenant level, non-AD and domain groups will also be retrieved.

retrieveHiddenLists​

When set to true the hidden lists are retrieved.

retrieveListItems​

When set to false all site lists/drives are retrieved, but children from lists and drives are ignored.

extractionDepth​

Specifies the maximum depth of objects that will be retrieved from drives (document libraries). When set to -1, all objects of a drive will be retrieved and when set to 0 the drive will be not be crawled. A positive number indicates the object depth to be crawled. For example, when set to 1 only the drive its direct children are retrieved.

note

When retrieveListItems has been set to false, the extraction depth will not be applied.

discardMetadata​

Cleans the source.properties object for all stored objects by removing default extracted metadata and unused properties.

Only a minimal set of metadata fields required by schemas and processing logic is retained.

Retained metadata per object type
Containers​

The default properties from the source.properties is reduced to:

["sharepointIds", "webUrl"]
Records​

The default properties from the source.properties is reduced to:

["fields", "sharepointIds"]
Binaries​

The default properties from the source.properties is reduced to:

["sharepointIds"]
Principals​

For users the source.properties is reduced to:

["accountEnabled", "deletedDateTime"]

For Domain groups the source.properties is reduced to:

["deletedDateTime", "fields"]

All non-default (custom) metadata is retained.

origin​

Specifies the origin of the document in the Content Store.

SharePoint Online (2. Content Types & Term Store)​

This flow exports the content types for each stored site and document library in the Content Store. Furthermore, for each site it also exports the term store.

Settings​

mongoConnection​

The Mongo connection string including the database name to connect to.

tenantID​

The ID of the tenant to connect to.

tenantName​

The name of the tenant to connect to.

clientID​

The client ID of the application.

clientSecret​

The client secret of the application.

clientCertificateThumbprint​

A thumbprint is a unique identifier for the certificate uploaded to Azure Portal. It ensures the application is using the correct certificate.

clientCertificatePrivateKey​

The file contents of the private key associated with the certificate.

retrieveTerms​

Determines which term groups are retrieved from the term store. The term store has global term groups that are available in the entire tenant and site collection terms which are only available on the sites in that site collection. By default the value is set to all and both global and site collection term groups are retrieved. By setting this value to either global or siteCollection the retrieval is limited to either of those term groups. It is also possible to not retrieve the term groups by setting this value to none.

resetContentTypes​

Removes all previously crawled content types from the Content Store when set to true. By default, the value is set to false.

resetTerms​

Removes all previously crawled term groups, term sets and terms from the Content Store when set to true. By default, the value is set to false.

discardMetadata​

Cleans the source.properties object for all stored objects by removing default extracted metadata and unused properties.

Retained metadata per object type
Content types​

The default properties from the source.properties is reduced to:

["columns"]
TermGroups, TermSets and Terms​

No metadata properties are retained.

origin​

Specifies the origin of the document in the Content Store.

note

Authentication can be done using the clientSecret or clientCertificateThumbprint together with clientCertificatePrivateKey. For more information on how to create a certificate, go to Creating and Exporting a Self-Signed Certificate.

SharePoint Online (3. Permission Levels)​

This flow exports all permission levels of each site in the Content Store. It is only required to run this flow when (custom) permissions are set on objects that are migrated.

Settings​

mongoConnection​

The Mongo connection string including the database name to connect to.

tenantID​

The ID of the tenant to connect to.

tenantName​

The name of the tenant to connect to.

clientID​

The client ID of the application.

clientSecret​

The client secret of the application.

clientCertificateThumbprint​

A thumbprint is a unique identifier for the certificate uploaded to Azure Portal. It ensures the application is using the correct certificate.

clientCertificatePrivateKey​

The file contents of the private key associated with the certificate.

resetPermissions​

Removes all previously crawled permission levels from the Content Store when set to true. By default, the value is set to false.

discardMetadata​

Cleans the source.properties object for all stored objects by removing default extracted metadata and unused properties.

Retained metadata per object type
Permissions​

The default properties from the source.properties is reduced to:

["Description", "Hidden", "Id", "Name", "Order", "permMask", "RoleTypeKind"]
origin​

Specifies the origin of the document in the Content Store.

note

Authentication can be done using the clientSecret or clientCertificateThumbprint together with clientCertificatePrivateKey. For more information on how to create a certificate, go to Creating and Exporting a Self-Signed Certificate.