clinCompare: Dataset Comparison with CDISC Validation

Description

A comprehensive toolkit for comparing clinical trial datasets. Provides functions for dataset comparison including variable-level and observation-level differences, data type checking, and missing value analysis. Integrates CDISC validation for SDTM and ADaM datasets.

Main Functions

compare_datasets

High-level comparison of two datasets

compare_variables

Compare variable names and types

compare_observations

Row-wise value comparison

cdisc_compare

Compare datasets with CDISC validation

validate_cdisc

Validate a dataset against CDISC standards

detect_cdisc_domain

Auto-detect CDISC domain or ADaM dataset

CDISC Standards Supported

SDTM

DM, AE, LB, VS, EX, CM, MH, DS, SV, TA, TE domains

ADaM

ADSL, ADAE, ADLB, ADTTE, ADEFF datasets

Author(s)

Maintainer: Siddharth Lokineni sidhu871@gmail.com

See Also

Useful links:

• https://github.com/siddharthlokineni/clinCompare

• Report bugs at https://github.com/siddharthlokineni/clinCompare/issues

Package-Level Settings Environment

Description

Internal environment used to store package settings without modifying global options.

Usage

.clincompare_env

Format

An object of class environment of length 2.

Print Observation-Level Differences (Internal Helper)

Description

Shared helper used by both print.dataset_comparison and print.cdisc_comparison. Prints a summary line, a per-variable table, and up to n rows of the top variable's differing observations.

Usage

.print_observation_diffs(obs, n = 30, id_details = NULL, n_total_obs = NULL)

Arguments

Value

Called for side effects (prints to console). Returns NULL invisibly.

Build Metadata Comparison

Description

Internal function to compare metadata attributes (types, labels, lengths, formats, and column order) between two datasets.

Usage

build_metadata_comparison(df1, df2)

Arguments

Value

A list with:

Build Unified Comparison Table

Description

Internal function that merges attribute differences (type, label, length, format) and value differences into a single data frame, giving a consolidated per-variable view of all differences.

Usage

build_unified_comparison(meta, obs_comp, id_vars, df1, df2)

Arguments

Value

A data frame with columns: variable, diff_type, row_or_key, base_value, compare_value. The diff_type column indicates whether the row is a Type, Label, Length, Format, or Value difference.

Compare Two Datasets with CDISC Validation

Description

Flagship function that compares two datasets AND runs CDISC validation on both. Combines dataset comparison with CDISC conformance analysis to provide comprehensive insights into both differences and regulatory compliance.

Usage

cdisc_compare(
  df1,
  df2,
  domain = NULL,
  standard = NULL,
  id_vars = NULL,
  vars = NULL,
  ts_data = NULL,
  detect_outliers = FALSE,
  tolerance = 0,
  where = NULL
)

Arguments

Value

A list containing:

Examples

# Create sample SDTM DM domains
dm1 <- data.frame(
  STUDYID = "STUDY001",
  USUBJID = c("SUBJ001", "SUBJ002"),
  DMSEQ = c(1, 1),
  RACE = c("WHITE", "BLACK OR AFRICAN AMERICAN"),
  stringsAsFactors = FALSE
)

dm2 <- data.frame(
  STUDYID = "STUDY001",
  USUBJID = c("SUBJ001", "SUBJ003"),
  DMSEQ = c(1, 1),
  RACE = c("WHITE", "ASIAN"),
  ETHNIC = c("NOT HISPANIC", "NOT HISPANIC"),
  stringsAsFactors = FALSE
)

# Positional matching (default)
result <- cdisc_compare(dm1, dm2, domain = "DM", standard = "SDTM")

# Key-based matching by ID variables
result <- cdisc_compare(dm1, dm2, domain = "DM", id_vars = c("USUBJID"))
names(result)

Check Compatibility of Two Datasets for Comparison

Description

Checks if two datasets are compatible for comparison by verifying their dimensions, column names, and data types. Returns a list indicating whether the datasets are compatible and detailing any structural differences.

Usage

check_compatibility(df1, df2)

Arguments

Value

A list containing details about the compatibility of the datasets, including information on dimension equality and common columns.

Clean Dataset

Description

Removes duplicate rows, standardizes column names and text values to uppercase or lowercase, and performs basic data cleaning on a data frame.

Usage

clean_dataset(
  df,
  variables = NULL,
  remove_duplicates = TRUE,
  convert_to_case = NULL
)

Arguments

Value

A cleaned data frame.

Examples

  df <- data.frame(name = c("Alice", "Bob", "Alice"),
                   score = c(90, 85, 90),
                   stringsAsFactors = FALSE)
  clean_dataset(df, remove_duplicates = TRUE, convert_to_case = "upper")

Compare Two Datasets by Group

Description

Compares two datasets within subgroups defined by grouping variables. Performs separate comparisons for each group and returns results organized by group.

Usage

compare_by_group(df1, df2, group_vars)

Arguments

Value

A list of comparison results for each group.

Examples

  df1 <- data.frame(region = c("A", "A", "B"), value = c(10, 20, 30),
                    stringsAsFactors = FALSE)
  df2 <- data.frame(region = c("A", "A", "B"), value = c(10, 25, 30),
                    stringsAsFactors = FALSE)
  compare_by_group(df1, df2, group_vars = "region")

Compare Two Datasets

Description

Compares two datasets at three levels in a single call:

1. Dataset level – dimensions, column overlap, missing-value totals.

2. Variable level – column name discrepancies and data-type mismatches (delegates to compare_variables()).

3. Observation level – row-by-row value differences on common columns. Uses positional matching by default, or key-based matching when id_vars is provided.

The return value is a list with class "dataset_comparison", which has a tidy print() method. The same object is accepted by generate_summary_report(), generate_detailed_report(), and compare_by_group().

Usage

compare_datasets(df1, df2, tolerance = 0, vars = NULL, id_vars = NULL)

Arguments

Value

A dataset_comparison list containing:

Examples

# Positional matching (default)
df1 <- data.frame(id = 1:3, val = c(10, 20, 30))
df2 <- data.frame(id = 1:3, val = c(10, 25, 30))
result <- compare_datasets(df1, df2)
result

# Key-based matching (for different row counts or row orders)
df1 <- data.frame(id = c(1, 2, 3), val = c(10, 20, 30))
df2 <- data.frame(id = c(2, 3, 4), val = c(20, 35, 40))
result <- compare_datasets(df1, df2, id_vars = "id")
result
result$unmatched_rows

Compare Observations of Two Datasets

Description

Performs row-by-row comparison of two datasets on common columns, identifying specific value differences at the cell level. Returns discrepancy counts and details showing which rows differ and how their values diverge.

Usage

compare_observations(df1, df2, tolerance = 0)

Arguments

Value

A list containing discrepancy counts and details of row differences.

Examples

  df1 <- data.frame(id = 1:3, value = c(1.0, 2.0, 3.0))
  df2 <- data.frame(id = 1:3, value = c(1.0, 2.5, 3.0))
  compare_observations(df1, df2)
  compare_observations(df1, df2, tolerance = 0.00001)

Compare Observations by ID Variables

Description

Internal function to match rows between two datasets using specified key variables, then compare values on matched rows. Also identifies unmatched rows in either dataset.

Usage

compare_observations_by_id(df1, df2, id_vars, common_cols, tolerance = 0)

Arguments

Value

A list with:

Batch Compare CDISC Datasets Across Submission Directories

Description

Scans two directories for matching dataset files, runs cdisc_compare() on each pair, and optionally generates a consolidated Excel report.

Usage

compare_submission(
  base_dir,
  compare_dir,
  format = NULL,
  id_vars = NULL,
  tolerance = 0,
  output_file = NULL
)

Arguments

Value

Named list of cdisc_compare() results, one per matched domain.

Examples

## Not run: 
  # Auto-detects format from directory contents
  results <- compare_submission("v1/", "v2/",
                                 output_file = "submission_diff.xlsx")

  # Explicit format
  results <- compare_submission("v1/", "v2/", format = "csv")

## End(Not run)

Compare Variables of Two Datasets

Description

Compares the structural attributes of two datasets including column names, data types, and variable ordering. Identifies common columns and reports columns that exist in only one dataset.

Usage

compare_variables(df1, df2)

Arguments

Value

A list containing variable comparison details and discrepancy count.

Examples

  df1 <- data.frame(id = 1:3, name = c("A", "B", "C"))
  df2 <- data.frame(id = 1:3, name = c("A", "B", "C"), score = c(90, 80, 70))
  compare_variables(df1, df2)

Converts the data types of specified variables in a dataset.

Description

Converts columns in a data frame to specified types based on a named list mapping column names to target types. Supports conversion to numeric, character, factor, integer, logical, and other R data types.

Usage

convert_data_types(df, conversions)

Arguments

Value

A data frame with converted variable types.

Create CDISC Conformance Comparison

Description

Internal function to compare CDISC validation results from two datasets and identify which issues are unique to each or common to both.

Usage

create_conformance_comparison(val_df1, val_df2)

Arguments

Value

A data frame showing CDISC issue distribution across datasets, with columns:

Detect CDISC Domain Type

Description

Detects whether a data frame looks like an SDTM domain or ADaM dataset by comparing column names against known CDISC standards. Calculates a confidence score based on the percentage of expected variables present.

Auto-detection is a convenience for exploratory use. For anything important – validation reports, regulatory submissions, scripted pipelines – always pass domain and standard explicitly. Datasets with common columns (STUDYID, USUBJID, etc.) can match multiple domains, and a warning is issued when the top two candidates score within 10 percentage points of each other.

Usage

detect_cdisc_domain(df, name_hint = NULL)

Arguments

Value

A list containing:

Examples

# Create a sample SDTM DM domain
dm <- data.frame(
  STUDYID = "STUDY001",
  USUBJID = "SUBJ001",
  SUBJID = "001",
  DMSEQ = 1,
  RACE = "WHITE",
  ETHNIC = "NOT HISPANIC OR LATINO",
  ARMCD = "ARM01",
  ARM = "Treatment A",
  stringsAsFactors = FALSE
)

result <- detect_cdisc_domain(dm)
print(result)

Detect Outliers Using Z-Score Method

Description

Internal function to detect potential outliers in numeric columns of both datasets using the z-score method. Values with |z| > 3 are flagged. Results are returned as advisory notes for the user.

Usage

detect_outliers_zscore(df1, df2, threshold = 3)

Arguments

Value

A data frame with columns: dataset, variable, row, value, zscore. Empty data frame if no outliers found.

Export Comparison Report to File

Description

Exports a dataset or CDISC comparison result to a file in multiple formats. Automatically detects format from file extension (.html, .txt, .xlsx).

Usage

export_report(result, file, format = NULL)

Arguments

Details

Supported formats:

• HTML (.html): Self-contained HTML report with styling and interactive charts.

• Text (.txt): Plain text report suitable for console review.

• Excel (.xlsx): Multi-sheet workbook with tabbed data:

  • "Summary": Dataset dimensions, domain, standard, matching type, tolerance

  • "Variable Diffs": Metadata attribute differences

  • "Value Diffs": Unified diff data frame from get_all_differences()

  • "CDISC Validation": Combined validation results (for CDISC comparisons only)

The result object can be either a dataset_comparison (from compare_datasets()) or cdisc_comparison (from cdisc_compare()). All features are supported for both.

Value

Invisibly returns the input result (useful for piping).

Examples

# Create sample datasets
df1 <- data.frame(
  ID = c(1, 2, 3),
  NAME = c("Alice", "Bob", "Charlie"),
  AGE = c(25, 30, 35)
)

df2 <- data.frame(
  ID = c(1, 2, 3),
  NAME = c("Alice", "Bob", "Charles"),
  AGE = c(25, 30, 36)
)

# Compare datasets
result <- compare_datasets(df1, df2)

# Export to different formats (write to tempdir)
export_report(result, file.path(tempdir(), "report.html"))
export_report(result, file.path(tempdir(), "report.txt"))

# Explicit format specification
export_report(result, file.path(tempdir(), "report.xlsx"), format = "excel")

Extract CDISC Version from TS Domain

Description

Reads a Trial Summary (TS) dataset and extracts the CDISC standard version information. Looks for SDTM IG version (TSPARMCD = "SDTIGVER" or "CDISCVER") and ADaM IG version (TSPARMCD = "ADAMIGVR") parameters.

Usage

extract_cdisc_version(ts_data)

Arguments

Value

A list containing:

Format Validation Results as HTML

Description

Internal function to format validation results as an HTML table.

Usage

format_validation_html(validation_df)

Arguments

Value

Character vector of HTML lines.

Format Validation Summary

Description

Internal function to format validation results as text.

Usage

format_validation_summary(validation_df)

Arguments

Value

Character vector of formatted lines.

Generate CDISC Validation Report

Description

Generates a formatted report from the results of cdisc_compare(). Supports both text-based console output and HTML reports with professional styling and color-coding.

Usage

generate_cdisc_report(cdisc_results, output_format = "text", file_name = NULL)

Arguments

Details

The report includes:

• Dataset Comparison Summary

• CDISC Compliance for each dataset

• CDISC Conformance Comparison

For text output, formatting uses console-friendly layout. For HTML output, a self-contained report is generated with color-coded severity levels: red for ERROR, orange for WARNING, blue for INFO.

Value

Invisibly returns the input cdisc_results (useful for piping).

Examples

## Not run: 
# Create sample datasets
dm1 <- data.frame(
  STUDYID = "STUDY001",
  USUBJID = c("SUBJ001", "SUBJ002"),
  DMSEQ = c(1, 1),
  RACE = c("WHITE", "BLACK OR AFRICAN AMERICAN")
)

dm2 <- data.frame(
  STUDYID = "STUDY001",
  USUBJID = c("SUBJ001", "SUBJ003"),
  DMSEQ = c(1, 1),
  RACE = c("WHITE", "ASIAN")
)

result <- cdisc_compare(dm1, dm2, domain = "DM")

# Generate text report to console
generate_cdisc_report(result, output_format = "text")

# Generate HTML report to file
out <- file.path(tempdir(), "report.html")
generate_cdisc_report(result, output_format = "html", file_name = out)

## End(Not run)

Generate Visualization for Data Comparison

Description

Creates a ggplot2 bar chart visualization showing the number of discrepancies per variable from comparison results. Provides a clear visual summary of data differences across variables in the datasets being compared.

Usage

generate_comparison_visualization(comparison_results)

Arguments

Value

A plot object visualizing the comparison results.

Generate a Detailed Report of Dataset Comparison

Description

Creates a detailed report outlining all the differences found in the comparison, including variable differences, observation differences, and group-based discrepancies.

Usage

generate_detailed_report(
  comparison_results,
  output_format = "text",
  file_name = NULL
)

Arguments

Value

The detailed report. For 'text', prints to console. For 'html', writes to file.

Examples

## Not run: 
  generate_detailed_report(comparison_results, output_format = "text")

## End(Not run)

Generate HTML Report

Description

Internal function to generate a self-contained HTML report with styling.

Usage

generate_html_report(cdisc_results)

Arguments

Value

Character string containing the HTML report.

Generate a Summary Report of Dataset Comparison

Description

Provides a summary of the comparison results, highlighting key points such as the number of differing observations and variables.

Usage

generate_summary_report(
  comparison_results,
  detail_level = "high",
  output_format = "text",
  file_name = NULL
)

Arguments

Value

The summary report. For 'text', prints to console. For 'html', writes to file.

Examples

## Not run: 
  generate_summary_report(comparison_results, detail_level = "high", output_format = "text")

## End(Not run)

Generate Text Report

Description

Internal function to generate a formatted text report from CDISC comparison results.

Usage

generate_text_report(cdisc_results)

Arguments

Value

Character string containing the formatted text report.

ADaM Metadata

Description

Returns metadata for ADaM datasets following CDISC standards. Provides information about required, conditional, and other variables for each ADaM analysis dataset.

Usage

get_adam_metadata(version = "1.3")

Arguments

Details

Variable definitions are based on the published CDISC ADaM Implementation Guide. The canonical machine-readable source is the CDISC Library API (https://www.cdisc.org/cdisc-library), which requires CDISC membership. The metadata shipped with clinCompare is hand-curated from the published IG specifications.

Value

A named list where keys are ADaM dataset names and values are data.frames with columns:

variable

Variable name (character)

label

Variable label/description (character)

type

Data type: "Char" for character or "Num" for numeric

core

Importance level: "Req" (Required), "Cond" (Conditional)

Extract All Differences as a Unified Data Frame

Description

Converts per-variable observation differences into a single long-format data frame suitable for filtering with dplyr, writing to CSV, or programmatic analysis. This is the R equivalent of SAS PROC COMPARE's OUT= dataset with _TYPE_ and _DIF_ variables.

Accepts output from compare_datasets(), cdisc_compare(), or any list containing an observation_comparison element with the standard discrepancies / details / id_details structure.

Usage

get_all_differences(comparison_results)

Arguments

Value

A data frame with one row per differing cell. Columns:

Variable

Character: column name where the difference was found.

Row

Integer: row index in df1 (positional matching).

Base

The value in df1 (base dataset).

Compare

The value in df2 (compare dataset).

Diff

Numeric: Base - Compare (NA for character columns).

PctDiff

Numeric: absolute percentage difference relative to Base (NA when Base is 0 or column is character).

When key-based matching was used (id_vars), the ID columns are prepended to the left of the data frame.

Returns an empty data frame with the expected columns when no differences exist or observation comparison was skipped.

Examples

df1 <- data.frame(id = 1:3, value = c(10, 20, 30), name = c("A", "B", "C"))
df2 <- data.frame(id = 1:3, value = c(10, 25, 30), name = c("A", "B", "D"))
result <- compare_datasets(df1, df2)
diffs <- get_all_differences(result)
head(diffs)

SDTM Metadata

Description

Returns metadata for SDTM domains following CDISC standards. Provides information about required, expected, and permissible variables for each SDTM domain.

Usage

get_sdtm_metadata(version = "3.4")

Arguments

Details

Variable definitions are based on the published CDISC SDTM Implementation Guide. The canonical machine-readable source is the CDISC Library API (https://www.cdisc.org/cdisc-library), which requires CDISC membership. The metadata shipped with clinCompare is hand-curated from the published IG specifications and should be cross-referenced with the official CDISC Library for regulatory submissions.

Value

A named list where keys are SDTM domain codes and values are data.frames with columns:

variable

Variable name (character)

label

Variable label/description (character)

type

Data type: "Char" for character or "Num" for numeric

core

Importance level: "Req" (Required), "Exp" (Expected), or "Perm" (Permissible)

Get Tolerance Level for Comparisons

Description

Retrieves the currently set tolerance level for numeric comparisons.

Usage

get_tolerance()

Value

The current tolerance level as a numeric value.

Handle Missing Values in Dataset

Description

Handles missing values (NA) in a data frame using one of several strategies: exclude rows, replace with a value, fill with column mean, fill with column median, or flag with an indicator column.

Usage

handle_missing_values(df, method = "exclude", replace_with = NULL)

Arguments

Value

A data frame after handling missing values.

Initialize Settings for Data Comparison

Description

Initializes default settings for dataset comparison including tolerance and other parameters stored in a package environment.

Usage

initialize_comparison_settings(tolerance = 0, missing_value_method = "ignore")

Arguments

Value

Invisible NULL. Called for its side effect of updating package settings.

Prepare Datasets for Comparison

Description

Prepares two datasets for comparison by optionally sorting by specified columns and filtering rows based on a condition.

Usage

prepare_datasets(df1, df2, sort_columns = NULL, filter_criteria = NULL)

Arguments

Value

A list containing two prepared datasets.

Examples

  df1 <- data.frame(id = c(3, 1, 2), score = c(70, 90, 80))
  df2 <- data.frame(id = c(2, 3, 1), score = c(80, 75, 90))
  prepare_datasets(df1, df2, sort_columns = "id", filter_criteria = "score > 75")

Print CDISC Comparison Results

Description

Prints a concise summary of CDISC comparison results. Shows dataset dimensions, domain, number of differences, and a pass/fail verdict based on CDISC validation errors.

Usage

## S3 method for class 'cdisc_comparison'
print(x, ...)

Arguments

Value

Invisibly returns x.

Print Dataset Comparison Results

Description

Print Dataset Comparison Results

Usage

## S3 method for class 'dataset_comparison'
print(x, ...)

Arguments

Value

Invisibly returns x.

Print CDISC Validation Results

Description

Pretty-prints CDISC validation results to the console with a summary and grouped output by category. Displays counts of errors, warnings, and info messages.

Usage

print_cdisc_validation(validation_result)

Arguments

Details

Output includes:

• Summary counts of errors, warnings, and info messages

• Issues grouped by category

• Each issue displayed with its variable name and message

Value

Invisibly returns the input (useful for piping).

Examples

## Not run: 
# Validate a dataset
dm <- data.frame(
  STUDYID = "STUDY001",
  USUBJID = c("SUBJ001", "SUBJ002"),
  DMSEQ = c(1, 1),
  RACE = c("WHITE", "BLACK OR AFRICAN AMERICAN")
)

validation_result <- validate_cdisc(dm, domain = "DM", standard = "SDTM")
print_cdisc_validation(validation_result)

## End(Not run)

Generate a Report of Differences Found in Dataset Comparison

Description

Creates a formatted report summarizing all differences found between two data frames, including column-level and value-level differences.

Usage

report_differences(variable_diffs, observation_diffs)

Arguments

Value

A structured report of the differences, typically a list or a data frame.

Reset Comparison Settings to Defaults

Description

Resets all comparison settings back to their defaults, clearing any custom tolerance or other parameters.

Usage

reset_comparison_settings()

Value

Invisible NULL. Called for its side effect of resetting package settings.

Set Tolerance Level for Comparisons

Description

Sets the numeric tolerance for floating-point comparisons, allowing small differences within the tolerance to be treated as equal.

Usage

set_tolerance(tolerance = 0)

Arguments

Value

Invisible NULL. Called for its side effect of updating the tolerance setting.

Summarize CDISC Comparison Results

Description

Returns a concise one-row data frame summarizing the comparison: domain, standard, row/col counts, number of differences, and CDISC error/warning counts.

Usage

## S3 method for class 'cdisc_comparison'
summary(object, ...)

Arguments

Value

A one-row data frame with summary metrics.

Transform Variables in a Dataset

Description

Applies mathematical or logical transformations to specified columns in a data frame based on a named list of transformation functions.

Usage

transform_variables(df, transformations)

Arguments

Value

A data frame with transformed variables.

Validate ADaM Compliance

Description

Validates a data frame against a specific ADaM dataset specification. Similar to validate_sdtm() but uses ADaM metadata and treats Conditional variables differently.

Usage

validate_adam(df, domain)

Arguments

Details

Severity levels:

• ERROR: Required variable is missing

• WARNING: Data type mismatch detected

• INFO: Conditional variable missing, non-standard variable, or variable information

Value

A data frame with validation results containing columns:

Validate CDISC Compliance

Description

Main validation entry point that checks whether a data frame conforms to CDISC standards. If domain and standard are not provided, they are automatically detected via detect_cdisc_domain(). Dispatches to validate_sdtm() or validate_adam() as appropriate.

Usage

validate_cdisc(df, domain = NULL, standard = NULL)

Arguments

Value

A data frame with columns:

Examples

# Auto-detect domain
dm <- data.frame(
  STUDYID = "STUDY001",
  USUBJID = "SUBJ001",
  DMSEQ = 1,
  RACE = "WHITE",
  stringsAsFactors = FALSE
)
results <- validate_cdisc(dm)
print(results)

# Validate with explicit domain specification
results <- validate_cdisc(dm, domain = "DM", standard = "SDTM")

Validate SDTM Compliance

Description

Validates a data frame against a specific SDTM domain specification. Checks for missing required/expected variables, data type mismatches, and non-standard variables.

Usage

validate_sdtm(df, domain)

Arguments

Details

Severity levels:

• ERROR: Required variable is missing

• WARNING: Expected variable is missing or data type mismatch detected

• INFO: Non-standard variable present or variable information

Value

A data frame with validation results containing columns: