Catalog File Shard Manifest¶
Enumerate Catalog files and save the list as sharded JSONL manifests for later processing.
Use it to divide a large file set among several tasks. The node creates file lists without parsing file contents.
Configure source and shards¶
Setting |
Description |
|---|---|
Data source |
Catalog source to enumerate, or a bound set of file references |
Destination |
Optional Catalog volume or directory for manifests |
Output file prefix |
Required and nonempty; distinguishes this manifest set |
Total shards |
Required positive integer determining the number of manifest shards |
Page size |
Entries per request when enumerating a volume in pages |
Choose one source set and a positive shard count.
Use the output¶
manifest_file_ids lists manifest IDs in shard_index order. manifest_files describes the shards. count is the total number of file records enumerated; shard_total is the number of shards.
Records contain file IDs, paths, and sequence information. Later nodes read these IDs and process the original files.
Send each generated shard to its corresponding processing flow.
Example: prepare email archive shards¶
Flow: Catalog File Shard Manifest → Sharded Email Archive Parsing.
Select the archive source, set a positive shard count and unique prefix, and pass each manifest to the matching email-processing shard.