Horizontally scaling MongoDB

Doing large enterprise migrations is inherently a high-load operation. Each document typically undergoes between 200 and 300 operations during a migrations, and the Content Store can easily grow to millions of documents. To maximize the throughput of the Content Store, we advise to not store the binaries in GridFS, and do your migrations in batches. However, if you still need to scale the Content Store, but you can't batch or scale vertically, you can use MongoDB sharding to horizontally scale the Content Store.

Use this setup when:

• MongoDB already runs on high-specification hardware;
• Vertical scaling is no longer possible or desirable;
• Binaries are not stored in GridFS;

This guide shards the documents collection over two MongoDB servers. The same configuration can be used for more shards, but the recommended starting point is two. The documents collection is the only collection that requires sharding.

Configure sharding before the migration

Set up and validate the complete sharded cluster before migration starts. Changing a populated standalone database into a sharded deployment can be a hassle.

Recommended operating system

Use Linux for the MongoDB hosts. MongoDB runs on Windows, but Linux with XFS provides the most predictable storage behavior and simpler service, file-permission, and no scanning by Windows Defender. Use the Windows procedure only when Linux is not available in the local infrastructure.

Recommendation

Use this configuration:

Setting	Value
Shard key	{ _id }
Data shards	2 to start
Binaries	External File Share, not GridFS
Router	One local mongos on every Xill4 execution host
Xill4 MongoDB address	mongodb://<username>:<password>@localhost:27017/source

Hashed _id is present and immutable on every Content Store document. The accelerators use _id for most per-document inserts, stores, and updates, allowing those commands to target one shard. Broad hierarchy, flag, count, and aggregation operations contact both shards and execute their share of the work in parallel.

In a simulation with approximately 2 million documents in the Content Store, two hash buckets differed by fewer than 25 documents. Across eight buckets, the largest and smallest buckets differed by approximately 0.5%. Use hashed _id as the default without a separate source-specific key.

Indexes

The existing indexes require a small but mandatory adjustment. The complete index commands to be executed are found in Create the Content Store. This is just an overview of the changes.

Keep

Keep all normal Content Store indexes. The accelerator-created indexes are non-unique and valid on a hashed _id sharded collection.

Change

The standard migrationId, sourceId, and targetId indexes are unique in a normal deployment. MongoDB does not permit those unique secondary indexes when hashed _id is the shard key. Create them as non-unique sparse indexes instead.

This means MongoDB no longer enforces global uniqueness for those three fields. _id remains the globally distributed identity and shard key.

Add

Add these indexes:

• { _id: "hashed" } for sharding;
• { name: 1 } on mappings; and
• { worker: 1 } on logs.

The mappings and logs indexes must be created manually because this setup enables database.skipCreateIndexes.

Local topology

Host	Private address	Processes
Xill4	192.168.50.10	Xill4, local mongos, and the cfgReplSet member
Shard 1	192.168.50.21	shard01 member
Shard 2	192.168.50.22	shard02 member

MongoDB requires every shard and the config servers to use replica-set topology. This setup is focussed on performance, and not redundancy. Therefore, each data shard and the config-server replica set in this speed-focused setup has one member. The config server runs alongside Xill4 and mongos; documents are stored only on the two dedicated shard hosts.

Add these entries to /etc/hosts on Linux or C:\Windows\System32\drivers\etc\hosts on Windows:

192.168.50.10  xill4
192.168.50.10  cfg
192.168.50.21  shard1
192.168.50.22  shard2

Use host aliases in replica-set configuration because MongoDB requires hostnames. Bind MongoDB only to localhost and the appropriate private address. Allow ports 27018 and 27019 only inside the private migration network. The mongos router listens only on localhost port 27017 on the Xill4 host.

Linux setup (recommended)

These instructions assume MongoDB 8 is installed and its service account is mongodb.

Prepare every MongoDB host

Create one shared cluster key and copy the same file to every config, shard, and Xill4 host:

sudo openssl rand -base64 756 | sudo tee /etc/mongodb-keyfile > /dev/null
sudo chown mongodb:mongodb /etc/mongodb-keyfile
sudo chmod 400 /etc/mongodb-keyfile

Create data and log directories on the Xill4 host and both shard hosts:

sudo install -d -o mongodb -g mongodb /var/lib/mongodb /var/log/mongodb

Configure the config server on the Xill4 host

On the Xill4 host, write /etc/mongod.conf:

/etc/mongod.conf

storage:
  dbPath: /var/lib/mongodb
systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
net:
  port: 27019
  bindIp: 127.0.0.1,192.168.50.10
replication:
  replSetName: cfgReplSet
sharding:
  clusterRole: configsvr
security:
  keyFile: /etc/mongodb-keyfile

Start the config-server service on the Xill4 host:

sudo systemctl enable --now mongod

Configure the two shard hosts

On each shard host, write /etc/mongod.conf. The example is for shard1; use private address 192.168.50.22 and replica-set name shard02 on the second host.

/etc/mongod.conf

storage:
  dbPath: /var/lib/mongodb
systemLog:
  destination: file
  path: /var/log/mongodb/mongod.log
  logAppend: true
net:
  port: 27018
  bindIp: 127.0.0.1,192.168.50.21
replication:
  replSetName: shard01
sharding:
  clusterRole: shardsvr
security:
  keyFile: /etc/mongodb-keyfile

Start the service on both shard hosts:

sudo systemctl enable --now mongod

Configure the local router on the Xill4 host

Write /etc/mongos.conf:

/etc/mongos.conf

systemLog:
  destination: file
  path: /var/log/mongodb/mongos.log
  logAppend: true
net:
  port: 27017
  bindIp: 127.0.0.1
sharding:
  configDB: cfgReplSet/cfg:27019
security:
  keyFile: /etc/mongodb-keyfile

Create /etc/systemd/system/mongos.service:

/etc/systemd/system/mongos.service

[Unit]
Description=MongoDB Router
After=network-online.target

[Service]
User=mongodb
Group=mongodb
ExecStart=/usr/bin/mongos --config /etc/mongos.conf
Restart=on-failure

[Install]
WantedBy=multi-user.target

Register the router service, but do not start it yet:

sudo systemctl daemon-reload
sudo systemctl enable mongos

Continue with Initialize the cluster.

Windows setup

These instructions assume MongoDB 8 is installed under C:\Program Files\MongoDB\Server\8.0.

Prepare every MongoDB host

Create the directories required by the process running on that host:

New-Item -ItemType Directory -Force -Path @(
  "C:\MongoDB\config",
  "C:\MongoDB\security",
  "D:\MongoDB\data",
  "D:\MongoDB\log"
)

Generate the shared key once, copy it to every MongoDB and Xill4 host, and grant the MongoDB service identity read access:

$key = New-Object byte[] 756
[Security.Cryptography.RandomNumberGenerator]::Fill($key)
[Convert]::ToBase64String($key) |
  Set-Content -NoNewline "C:\MongoDB\security\cluster.key"

icacls "C:\MongoDB\security\cluster.key" /inheritance:r
icacls "C:\MongoDB\security\cluster.key" /grant "MongoService:R"
icacls "D:\MongoDB" /grant "MongoService:(OI)(CI)M"

Replace MongoService with the Windows account that runs the MongoDB services.

Configure config and shard hosts

Use the same YAML settings shown in the Linux section, but replace paths as follows:

storage:
  dbPath: D:\MongoDB\data
systemLog:
  destination: file
  path: D:\MongoDB\log\mongod.log
  logAppend: true
security:
  keyFile: C:\MongoDB\security\cluster.key

Keep the role-specific net, replication, and sharding settings from the Linux examples. Save the completed file as C:\MongoDB\config\mongod.yml and install it as a Windows service:

& "C:\Program Files\MongoDB\Server\8.0\bin\mongod.exe" `
  --config "C:\MongoDB\config\mongod.yml" `
  --install `
  --serviceName "MongoDB-Sharded"

Start-Service "MongoDB-Sharded"

Configure the local router on the Xill4 host

Save this as C:\MongoDB\config\mongos.yml:

mongos.yml

systemLog:
  destination: file
  path: D:\MongoDB\log\mongos.log
  logAppend: true
net:
  port: 27017
  bindIp: 127.0.0.1
sharding:
  configDB: cfgReplSet/cfg:27019
security:
  keyFile: C:\MongoDB\security\cluster.key

Install the router service, but do not start it yet:

& "C:\Program Files\MongoDB\Server\8.0\bin\mongos.exe" `
  --config "C:\MongoDB\config\mongos.yml" `
  --install `
  --serviceName "MongoDB-Router"

Initialize the cluster

Run these commands after completing either the Linux or Windows process setup.

Initialize the config server

Connect locally to port 27019 on the Xill4 host and run:

rs.initiate({
  _id: "cfgReplSet",
  configsvr: true,
  members: [{ _id: 0, host: "cfg:27019" }],
});

Wait until the config member becomes primary.

Initialize the shards

Connect locally to each shard on port 27018 and run the matching command:

shard1

rs.initiate({
  _id: "shard01",
  members: [{ _id: 0, host: "shard1:27018" }],
});

shard2

rs.initiate({
  _id: "shard02",
  members: [{ _id: 0, host: "shard2:27018" }],
});

Create users and add shards

Start the local router after the config and shard replica sets are initialized:

Linux

sudo systemctl start mongos

Windows

Start-Service "MongoDB-Router"

Connect to mongodb://localhost:27017/admin on the Xill4 host. Create the first administrator while the localhost exception is active:

use admin;

db.createUser({
  user: "cluster-admin",
  pwd: passwordPrompt(),
  roles: [{ role: "root", db: "admin" }],
});

Reconnect as cluster-admin, then add both shards:

sh.addShard("shard01/shard1:27018");
sh.addShard("shard02/shard2:27018");
sh.status();

Create the Xill4 user:

use admin;

db.createUser({
  user: "xill4-user",
  pwd: passwordPrompt(),
  roles: [
    { role: "readWrite", db: "source" },
    { role: "dbAdmin", db: "source" },
  ],
});

Create the Content Store

Connect through local mongos as cluster-admin and run:

use xill4;

db.createCollection("documents");
db.documents.createIndex({ _id: "hashed" }, { name: "documentIdHashed" });
sh.shardCollection("source.documents", { _id: "hashed" });

db.documents.createIndexes([
  { key: { kind: 1 }, name: "kind" },
  { key: { "migration.id": 1 }, name: "migrationId", sparse: true },
  { key: { "migration.migrate": 1 }, name: "migrationMigrate" },
  { key: { "migration.failed": 1 }, name: "migrationFailed" },
  { key: { "source.id": 1 }, name: "sourceId", sparse: true },
  { key: { "source.parentIds": 1 }, name: "sourceParentIds" },
  { key: { "source.hierarchies": 1 }, name: "sourceHierarchies" },
  {
    key: {
      "source.versionInfo.seriesId": 1,
      "source.versionInfo.major": -1,
      "source.versionInfo.minor": -1,
    },
    name: "sourceVersionInfo",
  },
  { key: { "source.versionInfo.isCurrent": 1 }, name: "sourceVersionInfoIsCurrent" },
  { key: { "source.contentType.systemName": 1 }, name: "sourceContentTypeSystemName" },
  { key: { "target.id": 1 }, name: "targetId", sparse: true },
  { key: { "target.parentIds": 1 }, name: "targetParentIds" },
  { key: { "target.hierarchies": 1 }, name: "targetHierarchies" },
  {
    key: {
      "target.versionInfo.seriesId": 1,
      "target.versionInfo.major": -1,
      "target.versionInfo.minor": -1,
    },
    name: "targetVersionInfo",
  },
  { key: { "target.versionInfo.isCurrent": 1 }, name: "targetVersionInfoIsCurrent" },
  { key: { "target.contentType.systemName": 1 }, name: "targetContentTypeSystemName" },
]);

db.mappings.createIndex({ name: 1 }, { name: "name" });
db.logs.createIndex({ worker: 1 }, { name: "worker" });

Accelerators create their additional non-unique indexes when they run.

Configure Xill4

Disable automatic default-index creation because its three unique indexes are incompatible with the shard key. Point Xill4 at the local router:

config.yml

database:
  connectionString: mongodb://xill4-user:password@localhost:27017/xill4?authSource=admin
  skipCreateIndexes: true

Use the same connection string for %mongo_connection% and other project MongoDB connections. Every Xill4 execution host should run its own local mongos on port 27017.

Before extraction, verify the cluster:

sh.status();
db.documents.getShardDistribution();
db.documents.getIndexes();

Do not start extraction until both shards are visible and the expected indexes exist.

Operating the cluster

• Run one flow at a time unless concurrent flows were tested and sized in advance.
• Keep the balancer enabled while initially loading the empty hashed collection so chunks remain distributed.
• Start the performance-critical transformation and SPO stages only after chunk movement has settled.
• Monitor document count, stored bytes, CPU, memory, and disk throughput per shard.
• Monitor local mongos CPU and network usage during aggregations and sorted queries.
• Add shards before existing shards run out of memory or disk; redistribution needs spare capacity.

Expected performance

Hashed _id distributes the canonical document IDs evenly, but speedup is not linear. Queries without _id contact both shards, and mongos must merge some results. Two shards primarily increase capacity and parallelize broad work; they do not halve every flow's runtime.

For MongoDB-specific details, see the official documentation on sharding, hashed sharding, and query routing.