hierarchical-taxonomy-clustering

1# Hierarchical Taxonomy Clustering
2 
3Create a unified multi-level taxonomy from hierarchical category paths by clustering similar paths and automatically generating meaningful category names.
4 
5## Problem
6 
7Given category paths from multiple sources (e.g., "electronics -> computers -> laptops"), create a unified taxonomy that groups similar paths across sources, generates meaningful category names, and produces a clean N-level hierarchy (typically 5 levels). The unified category taxonomy could be used to do analysis or metric tracking on products from different platform.
8 
9## Methodology
10 
111. **Hierarchical Weighting**: Convert paths to embeddings with exponentially decaying weights (Level i gets weight 0.6^(i-1)) to signify the importance of category granularity
122. **Recursive Clustering**: Hierarchically cluster at each level (10-20 clusters at L1, 3-20 at L2-L5) using cosine distance
133. **Intelligent Naming**: Generate category names via weighted word frequency + lemmatization + bundle word logic
144. **Quality Control**: Exclude all ancestor words (parent, grandparent, etc.), avoid ancestor path duplicates, clean special characters
15 
16## Output
17 
18DataFrame with added columns:
19- `unified_level_1`: Top-level category (e.g., "electronic | device")
20- `unified_level_2`: Second-level category (e.g., "computer | laptop")
21- `unified_level_3` through `unified_level_N`: Deeper levels
22 
23Category names use ` | ` separator, max 5 words, covering 70%+ of records in each cluster.
24 
25## Installation
26…