tabullm.TextColumnTransformer¶

class tabullm.TextColumnTransformer(model, colnames=None, colsep=' || ', prefix='X_', normalize=False)[source]¶

Bases: BaseEstimator, TransformerMixin

A transformer for converting one or more text columns to a numeric matrix using Langchain embedding models.

Parameters:

model (Embeddings) – The Langchain text embedding model to use for transforming the text columns.
colnames (dict or None, default=None) – Optional display names for columns in embedding payload. Format: {actual_column_name: display_name} If None, uses actual column names. Example: {‘desc’: ‘Product Description’, ‘title’: ‘Title’}
colsep (str, default=' || ') – The column separator for concatenating multiple text columns, if applicable.
prefix (str, default='X_') – The prefix for the returned embedding columns.
normalize (bool, default=False) – If True, L2-normalize each embedding vector after generation. Projects all vectors onto the unit hypersphere, making cosine similarity equivalent to dot product. Useful when the downstream model (e.g. GMM) is sensitive to vector magnitude or when switching between embedding models that differ in their output norms.

Examples

>>> import pandas as pd
>>> from langchain_openai import OpenAIEmbeddings
>>>
>>> # Requires an OPENAI_API_KEY in the environment
>>> embedding_model = OpenAIEmbeddings()
>>> transformer = TextColumnTransformer(
...     model=embedding_model,
...     colsep=' || ',
...     prefix='emb_',
...     colnames={'col1': 'Column 1', 'col2': 'Column 2'}
... )
>>> df = pd.DataFrame({'col1': ['hello', 'world'], 'col2': ['foo', 'bar']})
>>> embeddings = transformer.fit_transform(df)

Methods

`__init__`
`estimate_tokens`	Estimate token count and cost for embedding operation without calling API.
`fit`	Fit the transformer on the data.
`fit_transform`	Fit to data, then transform it.
`get_feature_names_out`	Get output feature names for transformation.
`get_metadata_routing`	Get metadata routing of this object.
`get_params`	Get parameters for this estimator.
`prep_X`	Preprocess the input DataFrame X by concatenating all column values into a single string for each row.
`set_output`	Set output container.
`set_params`	Set the parameters of this estimator.
`transform`	Transform the data into embeddings.

Details

prep_X(X)[source]¶: Preprocess the input DataFrame X by concatenating all column values into a single string for each row.

estimate_tokens(X, chars_per_token=4, cost_per_1M_tokens=None)[source]¶

Estimate token count and cost for embedding operation without calling API.

Useful for cost estimation before expensive embedding operations.

Parameters:

X (pandas.DataFrame) – Data to estimate tokens for.
chars_per_token (float, default=4) – Character-to-token ratio heuristic. Typical values: - English text: ~4 characters per token - Code: ~3 characters per token Adjust based on your text type.
cost_per_1M_tokens (float or None, default=None) – Embedding model pricing in USD per 1 million tokens (as of Feb 2026). Examples: - AWS Titan Text V2: 0.02 - AWS Cohere Embed v4: 0.10 - OpenAI text-embedding-3-small: 0.02 - OpenAI text-embedding-3-large: 0.13 - HuggingFace (local): 0.0 (free) If None, cost is not estimated.

Returns:

dict – Dictionary with keys: - ‘n_samples’: Number of samples - ‘total_chars’: Total character count - ‘estimated_tokens’: Estimated token count (total_chars / chars_per_token) - ‘estimated_cost’: Estimated cost in USD (None if cost_per_1M_tokens not provided)

Notes

Uses character count heuristic (not exact tokenization). Accuracy typically 75-80% for English text. Sufficient for cost estimation purposes.

Examples

>>> import pandas as pd
>>> from tabullm import TextColumnTransformer
>>> from langchain_huggingface import HuggingFaceEmbeddings
>>>
>>> transformer = TextColumnTransformer(
...     model=HuggingFaceEmbeddings(model_name='sentence-transformers/all-MiniLM-L6-v2')
... )
>>> df = pd.DataFrame({'text': ['hello world', 'foo bar baz']})
>>>
>>> # Estimate for a free, local model
>>> info = transformer.estimate_tokens(df)
>>> print(f"Will embed {info['estimated_tokens']:,} tokens")
>>>
>>> # Estimate with cost (e.g. AWS Bedrock pricing)
>>> info = transformer.estimate_tokens(df, cost_per_1M_tokens=0.02)
>>> print(f"Estimated cost: ${info['estimated_cost']:.2f}")

fit(X, y=None)[source]¶

Fit the transformer on the data.

Parameters:

X (pandas.DataFrame) – The input data to fit.
y (None) – Ignored.

Returns:

self (object) – Returns the instance itself.

transform(X)[source]¶

Transform the data into embeddings.

Parameters:: X (pandas.DataFrame) – The input data to transform.
Returns:: embeddings (pandas.DataFrame) – The transformed embeddings.

get_feature_names_out(input_features=None)[source]¶

Get output feature names for transformation.

Parameters:: input_features (array-like of str, optional) – Input features (ignored).
Returns:: output_feature_names (array-like of str) – Output feature names.