Cross-validation

Blocked K-fold cross-validation for choosing the number of latent classes. Folds are drawn at the decision-maker level so that an individual's entire choice history sits in exactly one fold.

Experimental

The cross-validation utility is functional but still under active refinement. See the model-selection tutorial for a worked example. And please feel free to suggest improvements or extensions; I've never done any scholarship on hyperparameter tuning, so I'd be excited to hear about recent developments in this area!

lcl._cross_validation.cv_optimal_classes(data, alts_col=None, cases_col=None, panels_col=None, num_classes_list=None, formula=None, utility_formula=None, membership_formula=None, choice_col=None, case_varnames=None, dem_varnames=None, dems_data=None, numeraire=None, folds=5, seed=42, *, spec=None, fit_options=None, optimization_options=None, em_alg_config=None, mle_config=None)

Perform blocked K-Fold Cross Validation to determine the optimal number of latent classes.

Splits the data safely at the panel (decision-maker) level to ensure that the same decision-maker does not appear in both the training and test folds.

The recommended high-level call mirrors :func:lcl.fit: pass the same :class:~lcl.spec.LCLSpec you fit with, then sweep num_classes_list::

cv_optimal_classes(data, spec, num_classes_list=[2, 3, 4, 5])

The lower-level keyword form (alts_col=..., case_varnames=..., em_alg_config=...) remains supported for backward compatibility.

Parameters:

Name	Type	Description	Default
data	Any	The main dataset containing choice situations and alternatives.	required
alts_col	str | LCLSpec | None	Either the alternative-identifier column name or, in the high-level form, the :class:~lcl.spec.LCLSpec to reuse across the sweep. An LCLSpec passed here is equivalent to passing it through spec=.	None
cases_col	str | None	Name of the column grouping observations into distinct choice situations. Optional when an LCLSpec supplies the identifier columns.	None
panels_col	str | None	Name of the column mapping observations to specific decision-makers. Optional when an LCLSpec supplies the identifier columns.	None
num_classes_list	Sequence[int]	A sequence of integers specifying the numbers of latent classes to evaluate (e.g., [2, 3, 4, 5, 10, 15, 20]).	None
formula	str | None	Backward-compatible combined Formulaic string, for example "choice ~ price + C(brand) | income".	None
utility_formula	str | None	Formulaic string for the utility design, such as "choice ~ price + C(brand)".	None
membership_formula	str | None	Right-hand-side Formulaic string for class-membership demographics, such as "~ income + C(segment)".	None
choice_col	str | None	Name of the boolean/binary column indicating chosen alternatives.	None
case_varnames	Sequence[str] | None	List of alternative-specific variables.	None
dem_varnames	Sequence[str] | None	List of demographic variables.	None
dems_data	Any | None	A separate dataset containing panel-level demographics.	None
numeraire	str | None	The variable to be constrained as strictly negative (e.g., "price") via softplus.	None
folds	int	Number of cross-validation folds.	5
seed	int	Random seed for replicable panel splitting.	42
spec	:class:`~lcl.spec.LCLSpec` | None	Declarative specification supplying the identifier columns, utility and membership variables, formula, and numeraire constraint. Explicit keyword arguments override the corresponding spec fields.	None
fit_options	:class:`~lcl._struct.FitOptions` | None	High-level EM fit options applied to every fold. Translated internally to an :class:~lcl._struct.EMAlgConfig.	None
optimization_options	:class:`~lcl._struct.OptimizationOptions` | None	High-level M-step optimizer options applied to every fold.	None
em_alg_config	:class:`~lcl._struct.EMAlgConfig`	Lower-level configuration for the EM algorithm loop. Ignored when fit_options is supplied.	None
mle_config	:class:`~lcl._struct.MleConfig`	Lower-level configuration for the inner optimization routines. Ignored when optimization_options is supplied.	None

Returns:

Type	Description
DataFrame	A DataFrame containing the Average Out-of-Sample Log-Likelihood for each specified number of classes.

Source code in src/lcl/_cross_validation.py

def cv_optimal_classes(
    data: Any,
    alts_col: str | LCLSpec | None = None,
    cases_col: str | None = None,
    panels_col: str | None = None,
    num_classes_list: Sequence[int] | None = None,
    formula: Optional[str] = None,
    utility_formula: str | None = None,
    membership_formula: str | None = None,
    choice_col: Optional[str] = None,
    case_varnames: Optional[Sequence[str]] = None,
    dem_varnames: Optional[Sequence[str]] = None,
    dems_data: Optional[Any] = None,
    numeraire: Optional[str] = None,
    folds: int = 5,
    seed: int = 42,
    *,
    spec: LCLSpec | None = None,
    fit_options: FitOptions | None = None,
    optimization_options: OptimizationOptions | None = None,
    em_alg_config: EMAlgConfig | None = None,
    mle_config: MleConfig | None = None,
) -> pl.DataFrame:
    """Perform blocked K-Fold Cross Validation to determine the optimal number of latent classes.

    Splits the data safely at the panel (decision-maker) level to ensure that
    the same decision-maker does not appear in both the training and test folds.

    The recommended high-level call mirrors :func:`lcl.fit`: pass the same
    :class:`~lcl.spec.LCLSpec` you fit with, then sweep ``num_classes_list``::

        cv_optimal_classes(data, spec, num_classes_list=[2, 3, 4, 5])

    The lower-level keyword form (``alts_col=...``, ``case_varnames=...``,
    ``em_alg_config=...``) remains supported for backward compatibility.

    Parameters
    ----------
    data : Any
        The main dataset containing choice situations and alternatives.
    alts_col : str | LCLSpec | None
        Either the alternative-identifier column name or, in the high-level form,
        the :class:`~lcl.spec.LCLSpec` to reuse across the sweep.  An ``LCLSpec``
        passed here is equivalent to passing it through ``spec=``.
    cases_col : str | None
        Name of the column grouping observations into distinct choice situations.
        Optional when an ``LCLSpec`` supplies the identifier columns.
    panels_col : str | None
        Name of the column mapping observations to specific decision-makers.
        Optional when an ``LCLSpec`` supplies the identifier columns.
    num_classes_list : Sequence[int]
        A sequence of integers specifying the numbers of latent classes to evaluate
        (e.g., [2, 3, 4, 5, 10, 15, 20]).
    formula : str | None, optional
        Backward-compatible combined Formulaic string, for example
        ``"choice ~ price + C(brand) | income"``.
    utility_formula : str | None, optional
        Formulaic string for the utility design, such as
        ``"choice ~ price + C(brand)"``.
    membership_formula : str | None, optional
        Right-hand-side Formulaic string for class-membership demographics, such
        as ``"~ income + C(segment)"``.
    choice_col : str | None, optional
        Name of the boolean/binary column indicating chosen alternatives.
    case_varnames : Sequence[str] | None, optional
        List of alternative-specific variables.
    dem_varnames : Sequence[str] | None, optional
        List of demographic variables.
    dems_data : Any | None, optional
        A separate dataset containing panel-level demographics.
    numeraire : str | None, optional
        The variable to be constrained as strictly negative (e.g., "price")
        via softplus.
    folds : int, default=5
        Number of cross-validation folds.
    seed : int, default=42
        Random seed for replicable panel splitting.
    spec : :class:`~lcl.spec.LCLSpec` | None, optional
        Declarative specification supplying the identifier columns, utility and
        membership variables, formula, and numeraire constraint.  Explicit keyword
        arguments override the corresponding spec fields.
    fit_options : :class:`~lcl._struct.FitOptions` | None, optional
        High-level EM fit options applied to every fold.  Translated internally to
        an :class:`~lcl._struct.EMAlgConfig`.
    optimization_options : :class:`~lcl._struct.OptimizationOptions` | None, optional
        High-level M-step optimizer options applied to every fold.
    em_alg_config : :class:`~lcl._struct.EMAlgConfig`, optional
        Lower-level configuration for the EM algorithm loop.  Ignored when
        ``fit_options`` is supplied.
    mle_config : :class:`~lcl._struct.MleConfig`, optional
        Lower-level configuration for the inner optimization routines.  Ignored
        when ``optimization_options`` is supplied.

    Returns
    -------
    pl.DataFrame
        A DataFrame containing the Average Out-of-Sample Log-Likelihood for each
        specified number of classes.
    """
    if isinstance(alts_col, LCLSpec):
        if spec is not None:
            raise ValueError(
                "Pass an LCLSpec either positionally or via spec=, not both."
            )
        spec = alts_col
        alts_col = None

    if spec is not None:
        alts_col = alts_col or spec.ids.alt
        cases_col = cases_col or spec.ids.case
        panels_col = panels_col or spec.ids.panel
        choice_col = choice_col or spec.ids.choice
        formula = formula if formula is not None else spec.formula
        utility_formula = (
            utility_formula if utility_formula is not None else spec.utility_formula
        )
        membership_formula = (
            membership_formula
            if membership_formula is not None
            else spec.membership_formula
        )
        if formula is None and utility_formula is None:
            case_varnames = case_varnames if case_varnames is not None else spec.utility
        if formula is None and membership_formula is None:
            dem_varnames = dem_varnames if dem_varnames is not None else spec.membership
        numeraire = numeraire or spec.numeraire

    if num_classes_list is None:
        raise ValueError("num_classes_list is required.")
    if alts_col is None or cases_col is None or panels_col is None:
        raise ValueError(
            "alts_col, cases_col, and panels_col are required unless an "
            "LCLSpec supplies them."
        )

    if em_alg_config is None:
        em_alg_config = fit_options.to_em_config() if fit_options else EMAlgConfig()
    if mle_config is None:
        mle_config = optimization_options if optimization_options else MleConfig()
    # Unify input into Polars for easy fold splitting, avoiding pandas coercion bugs
    if isinstance(data, pl.DataFrame):
        df = data
    elif hasattr(data, "columns"):
        df = pl.from_pandas(data)
    else:
        df = pl.DataFrame(data)

    unique_panels = df[panels_col].unique().to_numpy()

    rng = onp.random.default_rng(seed)
    shuffled_panels = rng.permutation(unique_panels)
    fold_panel_lists = onp.array_split(shuffled_panels, folds)

    results = []

    for num_classes in num_classes_list:
        if num_classes < 2:
            logger.warning(
                "Skipping evaluation for %s classes. The model requires at least 2 latent classes.",
                num_classes,
            )
            continue

        logger.info("Evaluating model with %s latent classes.", num_classes)
        fold_lls = []

        for f in range(folds):
            test_panels = fold_panel_lists[f]

            # Split data at the dataframe level
            test_df = df.filter(pl.col(panels_col).is_in(test_panels))
            train_df = df.filter(~pl.col(panels_col).is_in(test_panels))

            # 1. Instantiate and Fit Model on Training Fold
            model = LatentClassConditionalLogit(
                num_classes=num_classes, numeraire=numeraire
            )

            try:
                res = model.fit(
                    data=train_df,
                    alts_col=alts_col,
                    cases_col=cases_col,
                    panels_col=panels_col,
                    formula=formula,
                    utility_formula=utility_formula,
                    membership_formula=membership_formula,
                    choice_col=choice_col,
                    case_varnames=case_varnames,
                    dem_varnames=dem_varnames,
                    dems_data=dems_data,
                    em_alg_config=em_alg_config,
                    mle_config=mle_config,
                )

                # 2. Package Test Data using the ingestion engine
                # This guarantees that the test fold gets properly ranked contiguous IDs
                parsed_test = model._ingest_data(
                    data=test_df,
                    alts_col=alts_col,
                    cases_col=cases_col,
                    panels_col=panels_col,
                    formula=formula,
                    utility_formula=utility_formula,
                    membership_formula=membership_formula,
                    choice_col=choice_col,
                    case_varnames=case_varnames,
                    dem_varnames=dem_varnames,
                    dems_data=dems_data,
                )

                test_data, *_ = model._setup_data(parsed_test)
                test_diff = _diff_unchosen_chosen(test_data)

                # 3. Evaluate Out-of-Sample Log Likelihood
                oos_ll = res._full_loglik_fn(res.flat_params, test_diff, test_data)
                fold_lls.append(float(oos_ll))

            except Exception as e:
                logger.warning(
                    "Model evaluation failed for %s classes, fold %s: %s",
                    num_classes,
                    f + 1,
                    e,
                )
                fold_lls.append(onp.nan)

        avg_oos_ll = onp.nanmean(fold_lls)
        results.append({"Num_Classes": num_classes, "Avg_OOS_LL": avg_oos_ll})

    return pl.DataFrame(results)