CDISC in the storage layer: enforcing structure where it actually matters

New to SDTM? There’s a quick reference at the end of this post covering the domains, datasets, and variables mentioned here.

The gap nobody talks about

Ask data teams whether their CDISC submissions are compliant and most of them will say yes. Files are named correctly, folders follow the right structure, a define.xml gets produced before submission, Pinnacle 21 runs clean, or close enough.

However, none of that tells you whether the data is actually structured correctly.

A file called ae.xpt sitting in a folder called SDTM is a file with a name. The storage layer holding it does not know what SDTM is. It does not know that AEBODSYS is required, that AESER can only take values from a controlled list, or that every record in SUPPAE needs a matching record in AE. It simply holds a file.

Structural problems accumulate unnoticed through the study. The first time anyone looks thoroughly at the data is during submission readiness review, six weeks before the FDA deadline. Some problems that are uncovered could be easy to fix, but others may require re-running analyses or going back to vendors for corrected data that should have originally been rejected two years earlier.

This is the gap between CDISC-named and CDISC-structured. The former is a folder convention. The latter is a property of the data, and it needs to be enforced from the beginning, when data enters your system, rather than when you are already trying to leave.

In one of the previous posts, we covered how to build a storage layer with hard boundaries between landing, validated, and analytical data. This article is about what happens at those boundaries: checking a file structurally before it reaches the validated layer, storing derivation context alongside ADaM datasets, and generating define.xml from storage metadata instead of writing it by hand.

What valid CDISC actually means

CDISC compliance isn’t a single check. It’s four distinct layers: required variables, controlled terminology, SUPPQUAL relationships, and cross-domain relationships. Most automated tools stop at the first. Here’s what each one catches, and where the blind spot is.

Required variables

Every SDTM domain has a defined set of variables that must be present. For AE, that includes STUDYID, USUBJID, AETERM, AEBODSYS, AESER, and others. Missing any of them is a hard failure, especially because it’s one of the easiest checks run by most of the teams: compare the columns in the file against the SDTM Implementation Guide for that domain.

Controlled terminology

Many variables can only take values from an official list maintained by the NCI Thesaurus. AESER accepts Y or N. SEX accepts M, F, U, or UNDIFFERENTIATED. A value of Male or Yes is a violation, even if the intent is obvious.

This check requires a current copy of the NCI Thesaurus codelist for each controlled variable, and it runs against every row. A domain with fifty thousand adverse event records needs fifty thousand checks on AESER alone. Teams that run this only at submission often find that a vendor has been delivering non-compliant values for eighteen months.

SUPPQUAL relationships

When a domain needs to carry a non-standard variable, it goes into a supplemental qualifier dataset. SUPPAE holds extra variables for adverse events, SUPPDM for demographics.

Every record in SUPPAE has USUBJID, IDVAR, and IDVARVAL fields pointing to a specific record in AE. If that parent record does not exist, the SUPPQUAL record is orphaned. Orphaned records are a validation failure, and they are very easy to create when the parent domain gets corrected after the SUPPQUAL was already built.

Cross-domain relationships

SDTM domains are not independent. Every subject in AE should exist in DM. Every record in SUPPAE should have a parent in AE. Exposure records in EX should trace back to subjects in DM.

These relationships are not enforced by the file format. XPT files do not have foreign keys. The checks have to be explicit, and they have to run across multiple files at the same time.

Submission validation tools check all four layers. The question is just when. Running them at submission readiness means finding problems at the worst possible time. Running them at ingestion, before data reaches the validated layer, means a structurally invalid file never enters the system.

Enforcing structure at ingestion

The previously mentioned post on the clinical lake architecture described a promotion gate between the landing zone and the validated layer. A data manager reviews the file, signs off, and a manifest records the decision. That gate is the right place to add structural validation. The file is in landing and has not been certified yet. Rejecting it here costs just an email to a vendor. Rejecting it after it has been in the validated layer for six months costs considerably more.

A schema registry for SDTM and ADaM

You need somewhere to register the expected schema for every SDTM domain and ADaM dataset you work with. AWS Glue Data Catalog is one option: it stores table definitions and can be queried by validation jobs running in the same AWS environment. Other teams also use a simpler approach: a version-controlled set of JSON or YAML schema files stored in the same repository as the validation code. The specific tool matters less than the principle: schemas live in one place, validation jobs read from that place, and changes to schemas go through the same review process as changes to code.

For each domain, you register a table definition: column names, data types, and which columns are required. For controlled terminology variables, you store the valid value set alongside the schema, either directly in the table properties or in a separate reference table the validation job reads at runtime.

SDTM files arrive as XPT. Most schema registries and validation frameworks do not natively parse XPT, so the pipeline needs a parsing step before validation runs. Libraries like haven in R or xport in Python handle this reliably.

A minimal schema entry for AE:

{
  "domain": "AE",
  "required_columns": [
    {"name": "STUDYID",  "type": "string"},
    {"name": "USUBJID",  "type": "string"},
    {"name": "AETERM",   "type": "string"},
    {"name": "AEBODSYS", "type": "string"},
    {"name": "AESER",    "type": "string", "controlled_term": "NY"}
  ]
}

NY refers to the NCI codelist for Yes/No responses. The validation job resolves that reference against the current NCI Thesaurus version at check time.

The validation pipeline

When a file lands in the landing zone, an ingestion pipeline runs before the data manager sees it. Four checks, in sequence.

1. The file is parsed. A corrupt XPT or encoding issue fails here before anything else runs.
2. Required columns are checked against the registered schema for that domain. Any missing column produces a structured error with the column name and the SDTM IG reference.
3. Controlled terminology is checked. For each column with a registered codelist, every row is scanned and violations are collected. The output is not a pass/fail. It is a count of violations per column with a sample of offending values, so the vendor knows what to fix.
4. SUPPQUAL relationships are checked if a supplemental dataset is present. Every record is matched against its parent. Orphaned records are reported with their IDVAR and IDVARVAL so they can be traced back to the source.

If any check fails, the file does not move forward. The pipeline writes a rejection report to the landing zone, sends a notification, and stops. The data manager sees the report before deciding whether to escalate to the vendor.

If all checks pass, the file is flagged as structurally valid and the data manager can proceed with content review and promotion.

What this does not replace

The pipeline catches structural and terminology violations. It does not catch a value that is technically valid but clinically wrong. AESER=N for an event that should be serious will pass every automated check. That judgment belongs solely to a medical reviewer.

Automated checks eliminate a class of problems that should never reach human review, however, they do not eliminate the human review itself.

ADaM lineage: storing the derivation, not just the result

An ADaM dataset is the end of a derivation chain. ADSL was built from DM, DS, EX, and probably several other SDTM domains. A specific version of each, using a specific script, with specific package versions installed at the time it ran.

Most teams store the dataset, but very few store everything else.

That matters when an FDA inspector asks you to reproduce an analysis. The answer is a deterministic rerun that produces the same output from documented inputs. If you do not know which version of DM fed into ADSL, or which version of the admiral package was installed when the script ran, you cannot give that answer.

What lineage metadata needs to capture

For each ADaM dataset, a derivation manifest is stored alongside it in the validated layer:

{
  "output_dataset": "adsl",
  "output_version": "2024-11-03T14:22:00Z",
  "sdtm_inputs": [
    {"domain": "DM", "validated_layer_path": "validated/study-123/sdtm/dm/2024-10-15/", "checksum": "a3f9..."},
    {"domain": "DS", "validated_layer_path": "validated/study-123/sdtm/ds/2024-10-15/", "checksum": "b72c..."},
    {"domain": "EX", "validated_layer_path": "validated/study-123/sdtm/ex/2024-10-15/", "checksum": "d41d..."}
  ],
  "derivation_code": {
    "repository": "github.com/org/study-123-adam",
    "commit": "f3a8c21",
    "script": "programs/adsl.R"
  },
  "package_versions": {
    "R": "4.3.1",
    "admiral": "0.12.3",
    "dplyr": "1.1.3",
    "haven": "2.5.3"
  },
  "run_environment": "validated-r-env:2024-10",
  "produced_by": "j.kowalski@org.com",
  "produced_at": "2024-11-03T14:22:00Z"
}

The SDTM input paths point to specific versions in the validated layer. The checksums confirm those files have not changed since the derivation ran. The commit hash pins the exact code. The package versions pin the exact computational environment.

Why package versions matter more than people think

Package behavior changes between versions. For example, admiral 0.11.x and admiral 0.12.x handle certain edge cases differently. If the manifest records only the script and not the package versions, you can rerun the code but cannot guarantee the same result.

Teams that upgrade packages mid-study without updating their lineage records sometimes find they cannot reproduce earlier datasets exactly. The derivation runs but produces slightly different output. Explaining that discrepancy to an inspector is not a position you want to put yourself in.

Where the manifest lives

The derivation manifest goes into the validated layer alongside the ADaM dataset, under the same Object Lock policy, not a log file in a database. A file, with a checksum, with a retention period, stored next to what it describes.

When a new version of an ADaM dataset is produced, a new manifest is written. Old manifests are not overwritten. The full derivation history is preserved, including runs superseded by later corrections.

This is where the architecture from the previous post pays off. SDTM inputs in the validated layer have checksums and Object Lock. When a manifest records a path and checksum there, that is a stable, verifiable reference. The input files will still be there in fifteen years, unchanged, ready to feed back into a rerun. Without immutable storage underneath, lineage metadata is a record of what you intended to use, not a guarantee of what you can retrieve.

define.xml as a byproduct, not a deliverable

define.xml is the document that tells FDA reviewers what is included in your submission. Every dataset, every variable, every controlled term, every relationship between domains. Reviewers use it to understand the data before they open a single file. Pinnacle 21 uses it to validate the submission package.

The standard way to produce it is manually, or semi-manually through a tool like Pinnacle 21 Enterprise or an internal SAS/R script that reads dataset metadata and writes XML. A programmer builds it towards the end of the project, checks it against the actual datasets, fixes discrepancies, and submits it.

The problem is timing and drift. define.xml gets built when datasets are mostly stable. Then a dataset changes. Then another. Each change creates a potential discrepancy between the document and the data it describes. Variable labels get updated in the dataset but not in define.xml. A new controlled term gets added. A SUPPQUAL variable gets renamed. By submission time, someone is manually reconciling two sources of truth that should never have been separate.

Generating from metadata

If the schema registry already holds the schema for every SDTM domain and ADaM dataset, define.xml is an export of that metadata.

The registry knows the column names, data types, labels, and controlled terminology references for every registered dataset. The derivation manifests from the previous section know the relationships between ADaM and SDTM inputs. The promotion records know when each dataset was validated and by whom.

A define.xml generator reads from those three sources and produces the document. It runs at any point, not just at submission time. Every time it runs, it reflects the current state of the validated layer.

The R ecosystem has tools like metacore and xportr that work with structured metadata to produce define.xml. The exact implementation depends on how your schema registry is structured, but the principle is the same: the generator reads machine-readable schema definitions rather than relying on a programmer to type variable labels by hand.

The output comes from the same metadata that drove ingestion validation in the previous section. If a variable passed schema enforcement when the file was promoted, it is in the registry. If it is in the registry, it is in define.xml. The document and the data cannot drift apart because they share the same source.

What you still need to check

Generating define.xml from metadata does not mean it is correct by definition. The metadata has to be right. If a variable was registered with the wrong label, define.xml will carry that wrong label.

The difference is where errors live. In a manual process, errors can exist in the document without existing in the data, and vice versa. In a metadata-driven process, an error in define.xml traces back to an error in the registry. You fix the registry entry, regenerate, and the document updates. One source, not two.

Where the machine stops and the programmer starts

Everything in this post describes what a machine can do: schema checks, controlled terminology validation, SUPPQUAL relationship enforcement, lineage capture, define.xml generation. All of it is automatable and all of it should be automated. None of it is enough on its own.

What automated checks catch

Structural violations: missing required variables, wrong data types, values outside controlled codelists, orphaned SUPPQUAL records, broken cross-domain references. These are binary. Either AESER contains a value from the NY codelist or it does not. Either every record in SUPPAE has a parent in AE or it does not. A machine checks these faster and more consistently than a human.

Automated checks are also good at catching drift. A vendor who delivered clean files for six months and then quietly changed an encoding convention will trip a controlled terminology check. Without enforcement at ingestion, that change might not surface until submission review.

What automated checks do not catch

AESER=N for an event a clinical reviewer would flag as serious. A derivation window in ADSL that is valid CDISC but uses the wrong reference date for the per-protocol population. An exposure record in EX where the dose units are correctly coded but the values look implausible given the drug and indication.

None of these fail a schema check. All of them require a statistical programmer or medical reviewer who knows the study, knows the protocol, and knows what the data should look like.

Automated checks enforce structure and controlled vocabulary. Human review enforces scientific and clinical correctness. The line between them is clear, and neither side substitutes for the other.

Where version control and package management come in

Reproducibility requires more than validated storage and lineage metadata. It requires that derivation code can be rerun in an environment identical to the one that produced the original output.

Derivation code lives in version control with the same discipline as the data. Commit messages reference the ADaM dataset and version being produced. Tags mark the exact commit used for each submission deliverable. A programmer looking at a derivation manifest from two years ago can check out that commit and know they have the right code.

The R or SAS environment is pinned and reproducible. For R, that means renv lockfiles committed alongside the code. renv.lock captures the exact version of every installed package and can restore that environment on any machine. SAS is harder: there is no equivalent of renv for SAS macros, so teams typically rely on controlled macro libraries with version numbers and documented SAS version requirements. It works, but it requires more manual discipline than the R approach. The package versions in the derivation manifest are a specification for the environment needed to rerun, not just informational metadata.

Teams that treat package management as an afterthought find the problem when they need to rerun a derivation for an FDA query and the package the code depends on has changed behavior in two years. The code runs. The output differs. That is a conversation worth avoiding.

Putting it together

The sections above each cover one layer of the system. Here is how they connect as a sequence, and where a human needs to make a call versus where the pipeline runs on its own.

Vendor delivers a file. The file lands in the landing zone. The system records a timestamp and computes a checksum. No human involvement at this stage.

Automated validation runs. The ingestion pipeline checks structure, controlled terminology, and SUPPQUAL relationships against the registered schema. It produces either a pass or a structured rejection report.

Human decision: reject or proceed. A data manager reads the report. If the file fails, they decide whether to escalate to the vendor or investigate. If it passes automated checks, they review the content, the things no schema validator can assess, and decide whether to promote it.

Promotion to the validated layer. The data manager signs off. A manifest is written. Object Lock is applied. The file is now immutable.

ADaM derivation. A statistical programmer runs derivation code against SDTM inputs from the validated layer. The derivation manifest records the exact inputs, code commit, and package versions used.

Human decision: review the derivation. The programmer, and usually a second reviewer, checks that the ADaM dataset is scientifically correct. Does the population flag logic match the protocol? Are the baseline windows right? Automated checks confirmed the structure. This step confirms the content.

define.xml generation. The generator reads from the schema registry and produces define.xml. This can run at any point. It is not a submission-time activity.

Submission readiness. Pinnacle 21 runs against the final package. Structural and terminology violations were caught at ingestion. Pinnacle 21 should be confirming compliance, not discovering it.

What the full picture looks like

The storage layer handles immutability and structural validity. Lineage manifests record what went into each derivation. Version control handles the code. Package lockfiles handle the environment. define.xml comes out of the metadata rather than being written by hand.

None of these layers work in isolation. A valid dataset in immutable storage is useless for reproducibility if the derivation code is not version controlled. Pinned package versions mean nothing if the SDTM inputs they ran against are not retained with checksums. A gap in any one of them surfaces at the worst possible time, which is usually six weeks before an FDA deadline.

The alternative is not painless upfront. Setting up schema validation at ingestion takes a few weeks at the start of a study. Building a derivation manifest template takes a sprint. Wiring define.xml generation to your schema registry is a one-time build. None of it is urgent when a study is just starting.

Done late, or not at all, the cost is different. Retroactive data migration with an unbroken audit trail. Re-running analyses against corrected inputs. Reconstructing derivation lineage for datasets produced eighteen months ago by a programmer who has since left. All of it under deadline pressure, all of it documentable to a regulator who will ask why it was not in place from the start. Clinical data infrastructure has a way of making you pay for shortcuts twice.

Quick reference

SDTM domains referenced in this post

‍

ADaM datasets referenced in this post

‍

Key variables