Preprocessing Pipeline
Text normalisation, tokenisation, linguistic feature extraction, and stop word removal with negation preservation
Pipeline Steps
Applied in sequence to every post in all splits
Unicode Normalisation
ftfy + NFKCFix encoding artefacts and normalise to NFKC form. Handles mojibake, smart quotes, and non-standard whitespace.
HTML Stripping
bleachRemove all HTML tags while preserving the underlying text content. Reddit posts sometimes contain markdown-encoded HTML.
Noise Removal
regex + emojiRemove URLs (http/https), Reddit mentions (@user), subreddit references (r/sub), hashtags, and emoji characters.
Text Normalisation
regexLowercase all text and collapse runs of 3+ repeated characters to 2 (e.g. 'sooooo' → 'soo'). Normalise all whitespace.
Tokenisation & Tagging
spaCy en_core_web_smSentence segmentation, word tokenisation, part-of-speech tagging, and lemmatisation. NER and dependency parsing are disabled for speed.
Stopword Removal
spaCy + customRemove stopwords from the spaCy list. Negations (not, never, n't, etc.) are explicitly preserved as they carry semantic signal.
Token Filtering
customDiscard tokens that are non-alphabetic or fewer than 2 characters. Retain only meaningful lexical tokens.
Linguistic Feature Extraction
customCompute 13 surface and psycholinguistic features per post: word count, char count, sentence count, avg sentence length, type-token ratio, pronoun/negation/social/emotion/noun/verb/adj/adv ratios.
Extracted Linguistic Features
13 features computed per post after preprocessing
| Feature | Description | Psycholinguistic Relevance |
|---|---|---|
| word_count | Total token count (all alpha tokens ≥ 2 chars) | Post length proxy |
| char_count | Character count of cleaned text | Verbosity indicator |
| sentence_count | Sentence count (spaCy sentencizer) | Structural complexity |
| avg_sentence_length | word_count / sentence_count | Prose density |
| type_token_ratio | unique_tokens / total_tokens | Lexical diversity |
| pronoun_ratio | First-person pronouns / total (I, me, my, mine, myself) | Self-focus |
| negation_ratio | Negation words / total (not, never, n't, etc.) | Negative framing |
| social_word_ratio | Social vocabulary / total (friend, alone, isolated, etc.) | Relational content |
| emotion_word_ratio | Emotion vocabulary / total (sad, empty, hopeless, etc.) | Affective content |
| noun_ratio | NOUN tokens / total | Referential density |
| verb_ratio | VERB tokens / total | Action/state density |
| adj_ratio | ADJ tokens / total | Descriptive density |
| adv_ratio | ADV tokens / total | Modification density |
Sample Post Explorer
Browse preprocessed examples from the dataset. See the full preprocessing pipeline output for each post.
0 total · page 1/1
No data available
Run run_pipeline.py on Google Colab, then upload the results/ directory to HuggingFace Spaces.