tabullm.ClusterExplainer¶

class tabullm.ClusterExplainer(llm, text_transformer, observations, text_fields, llm_context_length=200000, llm_cost_per_1M_tokens=None, custom_prompts=None)[source]¶

Bases: object

Generate LLM-based explanations for text data clusters.

Provides automatic handling of large datasets through recursive summarization, cost transparency via token counting, and flexible prompt customization.

Parameters:

llm (BaseLanguageModel) – LangChain language model for generating explanations.
text_transformer (TextColumnTransformer) – Fitted or unfitted TextColumnTransformer. Uses prep_X() for consistent text formatting.
observations (str) – Description of what observations represent (e.g., “job postings”, “patient records”). Injected into prompts for LLM context.
text_fields (str) – Description of text content (e.g., “titles and descriptions”). Injected into prompts for LLM context.
llm_context_length (int, default=200_000) – LLM context window in tokens (Claude: 200K, GPT-4: 128K, Gemini 1.5: 1M). Enables automatic recursion for large datasets.
llm_cost_per_1M_tokens (float or None, default=None) – Cost in USD per 1M tokens (as of Feb 2026). If provided, cost estimates included in count_tokens_only output.
custom_prompts (dict or None, default=None) –
Override default prompts. Valid keys:
- 'label_direct': overrides prompt_label_direct (used in _explain_1stage)
- 'label_from_summaries': overrides prompt_label_from_summaries (used in _explain_2stage stage 2)
- 'summarize_observations': overrides prompt_summarize_observations (used in _recursive_summarize at depth 0)
- 'combine_summaries': overrides prompt_combine_summaries (used in _recursive_summarize at depth > 0)
- 'synthesize': overrides prompt_synthesize (used in _synthesize when synthesize=True)

Examples

>>> from tabullm import load_fraud, TextColumnTransformer, GMMFeatureExtractor, ClusterExplainer
>>> from langchain_huggingface import HuggingFaceEmbeddings
>>> from langchain_openai import ChatOpenAI
>>>
>>> X, y, metadata = load_fraud()
>>> text_cols = ['title', 'description']
>>>
>>> # Embed and cluster
>>> transformer = TextColumnTransformer(model=HuggingFaceEmbeddings(
...     model_name='sentence-transformers/all-MiniLM-L6-v2'
... ))
>>> X_embedded = transformer.fit_transform(X[text_cols])
>>> gmm = GMMFeatureExtractor(n_components=10, random_state=42).fit(X_embedded)
>>> cluster_labels = gmm.labels_
>>>
>>> # Initialize explainer (requires an OPENAI_API_KEY in the environment)
>>> explainer = ClusterExplainer(
...     llm=ChatOpenAI(model='gpt-4o-mini'),
...     text_transformer=transformer,
...     observations="job postings",
...     text_fields="titles and descriptions",
...     llm_cost_per_1M_tokens=0.80
... )
>>>
>>> # Estimate cost first
>>> info = explainer.explain(X[text_cols], cluster_labels, preview=True)
>>> print(f"Cost: ${info['estimated_cost']:.2f}")
>>>
>>> # Generate explanations
>>> explanations = explainer.explain(X[text_cols], cluster_labels, strategy='auto')

Methods

`__init__`	Initialize the ClusterExplainer.
`explain`	Generate cluster explanations using LLM with automatic context handling.
`preview_prompts`	Print and return all fully rendered prompt strings.

Details

preview_prompts()[source]¶

Print and return all fully rendered prompt strings.

Useful for inspecting what will be sent to the LLM before calling explain(). Injects the instance’s observations and text_fields; remaining variables are filled with descriptive placeholders.

Returns:: dict[str, str] – Rendered prompt strings keyed by prompt name: 'label_direct', 'summarize_observations', 'combine_summaries', 'label_from_summaries'.
Return type:: dict

explain(X, cluster_labels, y=None, y_label=None, strategy='auto', max_summary_words=200, max_label_words=50, max_members_per_cluster=None, random_state=None, observation_stats=None, stat_labels=None, correction=None, synthesize=False, preview=False, verbose=False)[source]¶

Generate cluster explanations using LLM with automatic context handling.

Parameters:

X (DataFrame) – Original data with text columns. Uses text_transformer.prep_X() for formatting.
cluster_labels (array-like) – Cluster assignments for each sample.
y (array-like or None, default=None) –
Optional outcome variable for statistical association tests. Accepted formats:
- Numeric with exactly 2 unique non-NaN values → binary (Fisher’s exact test per cluster, Chi-square globally)
- Boolean dtype → binary
- Numeric with more than 2 unique values → continuous (independent t-test per cluster, one-way ANOVA globally)
Non-numeric y (strings, categories) is not supported; encode as numeric before passing (e.g., 0/1 for binary labels). Validated at call time with a descriptive error for unsupported inputs.
y_label (str or None, default=None) – Natural-language description of the outcome variable (e.g., "fraudulent job posting (1=fraud, 0=legitimate)"). Used only in the synthesis step when synthesize=True. Deliberately withheld from all cluster-labeling prompts so that cluster descriptions reflect text content, not the outcome. When y is provided but y_label is omitted, synthesis uses the generic phrase “the outcome variable”.
strategy ({'auto', 'one-stage', 'two-stage'}, default='auto') – Explanation strategy: - ‘auto’: Intelligently selects based on data size - ‘one-stage’: Direct explanation of all clusters - ‘two-stage’: Hierarchical (cluster summaries → final labels)
max_summary_words (int, default=200) – Maximum words for cluster summaries (stage 1 in two-stage).
max_label_words (int, default=50) – Maximum words for final cluster labels.
max_members_per_cluster (int or None, default=None) – Maximum number of members to include per cluster. If a cluster has more members, a random subsample of this size is used. None uses all members.
random_state (int or None, default=None) – Seed for cluster subsampling reproducibility. Only relevant when max_members_per_cluster is set.
observation_stats (pd.DataFrame or None, default=None) –
Optional per-observation statistics, one row per sample in X. When provided:
- Cluster-level means are computed and appended as columns to the output table (always).
- Each column is tested for correlation with y when y is also provided: t-test for binary y, Pearson correlation for continuous y. Results are returned as a third element.
Intended use: pass GMMFeatureExtractor.assignment_confidence_stats(X) here to include GMM-derived diagnostics in the explanation output.
stat_labels (dict or None, default=None) –
Human-readable descriptions for observation_stats columns, used as inline annotations in the synthesis prompt so the LLM knows what each statistic measures. Keys are column names; values are short descriptions. Ignored when synthesize=False or observation_stats is None. Example:
```
stat_labels = {
    'max_posterior':    'avg. cluster assignment confidence',
    'entropy':          'avg. membership entropy across clusters',
    'log_joint_margin': 'avg. log-joint margin over nearest rival',
}
```
correction ({'bonferroni', 'holm', 'fdr_bh'} or None, default=None) – Multiple testing correction for the K per-cluster hypothesis tests. 'bonferroni' is the most conservative; 'holm' controls the family-wise error rate with greater power; 'fdr_bh' (Benjamini-Hochberg) controls the false discovery rate and is recommended for exploratory work. When provided, a P-value (adjusted) column is appended to result_df (and to stat_assoc_df when observation_stats is also supplied). The global association p-value is a single omnibus test and is not adjusted. Ignored when y=None.
synthesize (bool, default=False) – If True, performs a final LLM pass that interprets the full output (cluster labels, association statistics, observation_stats results) as a coherent narrative. The synthesis prompt uses y_label when y is also provided; without y it produces a descriptive landscape summary. The synthesis text is appended as the last element of the return tuple (see Returns below).
preview (bool, default=False) – If True, prints a cost/strategy summary and returns without calling the LLM. Useful for inspecting estimated tokens, cost, and auto-selected strategy before committing to a full run.
verbose (bool, default=False) – Print progress messages during processing.

Returns:

DataFrame or tuple or dict –

If preview=True:

dict with keys: total_tokens, estimated_cost, strategy

If preview=False and y=None and observation_stats=None:

synthesize=False: DataFrame with columns: Group Number, Short Description, Long Description
synthesize=True: (result_df, synthesis_text)

If preview=False and y=None and observation_stats provided:

synthesize=False: DataFrame with cluster-mean stat columns appended
synthesize=True: (result_df, synthesis_text)

If preview=False and y provided and observation_stats=None:

synthesize=False: (result_df, global_results)
synthesize=True: (result_df, global_results, synthesis_text)

If preview=False and both y and observation_stats provided:

synthesize=False: (result_df, global_results, stat_assoc_df)
synthesize=True: (result_df, global_results, stat_assoc_df, synthesis_text)

synthesis_text is a string containing the LLM-generated narrative. When synthesize=False (the default), return shapes are identical to pre-synthesis behaviour.

Return type:

DataFrame

Notes

Uses text_transformer.prep_X() for consistent text formatting. Automatically handles large datasets with recursive summarization. Strategy selection (when strategy=’auto’) is based on full unsampled data size. Token count shown in preview reflects subsampling if max_members_per_cluster is set.