sn-da-large-file-analysis

1# Large Scale Excel Analysis Skill
2 
3## Mandatory Rules
4 
5> **When total rows >= 10,000, you MUST use the methods in this skill.**
6 
7| Data Scale | Read Strategy | Reason |
8|-----------|---------------|--------|
9| < 10k rows | `pd.read_excel()` directly | No memory pressure |
10| 10k–100k rows | `pd.read_excel()` → convert to Parquet → `pd.read_parquet()` for analysis | Avoid repeated slow reads |
11| 100k–1M rows | **openpyxl `read_only` + `iter_rows` streaming** → Parquet | `pd.read_excel()` will OOM or timeout |
12| > 1M rows | Streaming read + **multi-sheet split** (Excel max 1,048,576 rows per sheet) | Must chunk |
13 
14**Prohibited:**
15- Do NOT use `pd.read_excel()` to fully load 100k+ row files
16- Do NOT search for fonts with `fc-list`, `find ... fonts`, or install packages with `pip install`
17- Do NOT use `df.iterrows()` on large DataFrames (use `itertuples()` or vectorized ops)
18- Do NOT use `df.apply(lambda...)` for operations that can be vectorized
19 
20---
21 
22## Environment Setup
23 
24```python
25import pandas as pd
26import numpy as np
27import os
28import gc
29 
30pd.options.mode.copy_on_write = True
31 
32# CJK font setup (fixed paths — do NOT search for fonts)
33# ⚠️ Copy this block as-is. Do NOT use fc-list, find, subprocess, or glob to locate fonts.
34import matplotlib
35import matplotlib.pyplot as plt
36import matplotlib.font_manager as fm
37 
38_FONT_PATHS = [
39    '/mnt/afs_agents/SimHei.ttf',
40    '/mnt/afs_agents/mnt/data/SimHei.ttf',
41    os.path.expanduser('~/.fonts/SimHei.ttf'),
42…