Clean¶
Clean whitespace, tags, and selected noise from text, or clean a document array while retaining source metadata.
Use documents when parsed content will continue to chunking, extraction, or indexing. Use text for a standalone text value.
Choose the input¶
Supply nonempty text or documents, but not both.
The node normalizes whitespace, which can merge line breaks and repeated spaces. Check whether this is suitable for content that depends on indentation, line breaks, or table layout.
Configure cleaning rules¶
The following options are off by default.
Setting |
Effect |
|---|---|
Mask sensitive information |
Replace email and phone-number pattern matches with |
Normalize Unicode |
Convert text to Unicode NFC |
Traditional to Simplified Chinese |
Convert Traditional Chinese characters to Simplified Chinese |
Remove URLs |
Remove matching HTTP(S) and www links |
Remove invisible characters |
Remove control and Unicode formatting characters |
Remove HTML tags |
Strip HTML tags from text |
The special-character ratio and N-Gram repetition ratio thresholds accept values from 0 to 1. The default, 0, disables the check. Repetition is checked using 3-Grams.
Exceeding a threshold causes an error and stops processing. Empty content after cleaning also causes an error.
Use the output¶
Text input produces text; document input produces documents with source metadata retained. Continue passing the array rather than rebuilding records from plain text.
Example: clean webpage content¶
Flow: General Document Parsing → Clean → Segment.
Bind documents and enable HTML tag and URL removal. Check a small sample for missing content before enabling ratio thresholds.