TF-IDF (Term Frequency–Inverse Document Frequency) was the foundation of Stashmetrics’ content analysis pipeline, applied to three sources: user profile bios, post content, and the text of external links shared in posts.
Baseline and term extraction
Rather than computing IDF against the indexed corpus alone, we used a sample of the Twitter firehose as a baseline - scoring terms against what was common across the platform as a whole. This meant that terms salient to a specific niche community would score highly even if they appeared frequently within that community, as long as they were relatively uncommon in general discourse.
The top-N terms extracted per user or post formed a sparse term vector representing that profile or piece of content.
Community grouping and topic modelling
Community discovery was graph-based: follower relationships and shared list membership were traversed starting from a keyword-seeded set of profiles, expanding outward through the social graph to identify a connected community. Once a community was identified, TF-IDF term vectors for its members were aggregated and k-means clustering applied to identify the topics and themes most characteristic of that group. This allowed the platform to surface content that was popular specifically within a niche rather than relying on platform-wide popularity signals.
Building vectors
The extracted TF-IDF terms served double duty: they were both the basis for community clustering and the input to the vector similarity search pipeline. Each profile and content item was represented as a sparse set of its most salient terms, which could then be indexed and queried efficiently using a BK-tree structure.