Vendor onboarding without the pain: standardising CRO and vendor handoff
Every automated pipeline is only as good as the conventions it depends on. If every new vendor is a custom integration project, the automation you built around data delivery will spend most of its time being patched instead of running.
In the previous post, we walked through how event-driven ingestion turns a file drop into an auditable, traceable pipeline. The Lambda picks up the file, extracts metadata from the S3 key path (vendor identity, study identifier, data type) and routes everything from there. Checksums, audit logs, promotion to the clinical data bucket, notifications, quarantine on failure. All automated, all structural.
But that entire pipeline rests on one assumption: that vendors upload files to consistent, predictable paths. The S3 key structure is the routing table. When it's clean, the automation runs itself. When it's not, someone is writing exception logic at 11 PM because a new CRO decided their folder layout should look different from everyone else's.
That's what this post is about. Not the pipeline itself, but the organizational pattern that keeps it working as new vendors, new studies, and new data types get added over time.
Every vendor is a snowflake (until you decide they're not)
Here's what vendor onboarding looks like at most organizations we've worked with. A new CRO wins a study contract. Someone on the data management team gets an email. A meeting gets scheduled. Over the next few weeks, there's a back-and-forth about folder structure, file naming, which CDISC domains they'll deliver, how they format dates, whether they'll include a manifest file, and what their upload schedule looks like.
The IT team creates credentials. Someone provides a directory. The path convention is whatever seemed reasonable at the time, or whatever the previous vendor used, or whatever the person setting it up found in the wiki that may or may not reflect current practice. There's no template. There's no checklist that lives in version control. The knowledge of how to onboard a vendor lives in the heads of two or three people, and the specifics vary by vendor because nobody enforced a standard when the first few came through the door.
Fast forward eighteen months. You have eight vendors, each with a slightly different folder layout. Some nest study data under a vendor prefix. Some nest it under the study ID directly. One uses underscores where everyone else uses hyphens. Two deliver blinded and unblinded data into the same directory. Your ingestion pipeline has grown a thicket of conditional logic to handle these variations, and every new vendor means another round of custom path-parsing rules.
The problem isn't technical. It's organizational. Nobody decided early on that vendor onboarding should be a repeatable pattern. Every handoff was treated as a one-off, and the automation inherited that debt.
What standardised onboarding actually looks like
The fix is boring in the best way: a single, codified onboarding template that every vendor goes through, no exceptions. The specifics will vary by organization, but the pattern we've landed on has three layers.
Path conventions come first. Every vendor gets a namespace in the landing bucket. Under that namespace, every study gets a prefix. Under every study, data types are separated into predictable subdirectories. Blinded and unblinded data live at different prefixes with different access policies. They never share a path, and they never share an access surface. The full key structure looks something like {vendor}/{study}/{blinded|unblinded}/{data-type}/, and it's non-negotiable. When a vendor says “we normally organize things differently,” the answer is: this is how our platform ingests data, and here's the spec. Most vendors have seen this before from other sponsors and adapt without friction.
This matters because the path structure is what the ingestion Lambda reads. Vendor identity, study routing, data classification, blinded/unblinded segregation: it all comes from parsing the S3 key. When the convention is consistent, the Lambda doesn't need per-vendor logic. When it's inconsistent, every vendor is a special case.
Access isolation comes second. Each vendor gets scoped IAM policies that restrict their SFTP session to their own namespace in the landing bucket. They can write to their prefixes and nothing else. They can't see other vendors' data. They can't list the bucket root. In AWS, S3 Access Points make this straightforward. Each Access Point has its own policy scoped to specific prefixes, giving you the same security boundary as a separate bucket without the operational overhead of managing dozens of buckets as your vendor count grows. Provisioning a new vendor's access is a Terraform PR: add the namespace, attach the policy, create the Access Point, done. Revoking access at the end of a study is the reverse. Another PR, another review, another auditable change.
Credentials and identity come third. With Transfer Family, vendor authentication ties into your identity provider. Each vendor identity maps to a specific IAM role with the scoped policies described above. SSH key management, rotation schedules, and access expiry are all managed in infrastructure-as-code. Onboarding a vendor's credentials is part of the same Terraform module that provisions their namespace and access. Offboarding removes the whole stack in one change. No stale credentials sitting in a shared directory. No SSH keys that nobody remembers who they belong to.
The whole point is that adding a new vendor looks exactly the same every time. A data management lead fills in the parameters (vendor name, study identifiers, data types, blinded/unblinded requirements) and the platform team runs the Terraform. What used to be a multi-week, multi-email, multi-meeting process becomes a standardised PR that can be reviewed, approved, and deployed in a day.
Why this matters for compliance
Vendor onboarding isn't just an operational convenience problem. It's a compliance surface.
When an auditor asks “who has access to study data, and since when?” you need a clear answer. In the bespoke onboarding model, that answer lives in a combination of email threads, ticket systems, shared spreadsheets, and whoever remembers what happened eighteen months ago. Producing it takes time, and the confidence level is never as high as anyone would like.
When onboarding is codified in Terraform and version-controlled in Git, the answer is in the commit history. Every vendor namespace, every access policy, every credential provisioning event is a reviewed, timestamped, attributable change. Revoking access produces the same kind of record. You don't reconstruct the access history. You read it. Git becomes the audit trail for who could access what, when, and who approved the change.
The blinded/unblinded separation enforced at the prefix level is another compliance property that's hard to bolt on later. When blinded and unblinded data share a path, access control depends on people being careful about which files they open. When they're at different prefixes with different IAM policies, a statistician who shouldn't see unblinded data literally can't reach it. The control isn't procedural. It's structural. That's a much easier story to tell during an inspection than “we trained everyone to only open the right folders.”
21 CFR Part 11 asks for access controls that limit system access to authorized individuals. Infrastructure-as-code onboarding gives you that: least-privilege policies scoped to specific vendors and studies, provisioned through a reviewable change process, revocable through the same process. The audit trail isn't a byproduct of running the pipeline. It's a byproduct of managing who gets to use it.
Where this takes you
Once onboarding is standardised, the downstream benefits compound. The ingestion pipeline from the previous post stops accumulating per-vendor exceptions. Schema validation rules can reference the same path conventions to match incoming data against expected CDISC structures per study. Monitoring and alerting can key off the vendor namespace to detect late deliveries or missing files. Data lineage traces back through a clean, consistent path hierarchy instead of a patchwork of vendor-specific conventions.
More practically, it changes the conversation with vendors. Instead of a weeks-long negotiation about how they'll deliver data, onboarding becomes a handshake: here's the spec, here are your credentials, here's where your data goes. Most CROs have delivered to enough sponsors to recognize this pattern. The ones who push back are usually the ones who would have caused the most integration pain anyway. Catching that early is a feature, not a bug.
The pattern also sets up a natural conversation about what happens when SFTP itself isn't the right tool. Some vendors deal with datasets too large for SFTP to handle efficiently. Others have APIs that could feed data directly into your pipeline. Some organizations are looking at managed file transfer platforms or direct S3 uploads for partners who can handle them. That's the next post in this series: when SFTP isn't enough, and what the alternatives actually look like in a regulated environment where the compliance requirements don't change just because the transfer protocol does.
