×

To enhance security, Multi-Factor Authentication (MFA) is mandatory for all HL7 users. More information

 
Return to Search

Testing as an Integrated Component of the FHIR Implementation Guide Authoring and Publishing Process (PSS-2770)

Return to Search

Basics

Testing as an Integrated Component of the FHIR Implementation Guide Authoring and Publishing Process (PSS-2770)

2007
Ballot - Informative

FHIR

Guidance (e.g. Companion Guide, Cookbook, etc)

Create New Standard

Informative
Details

h1. Executive Summary

The ability to exchange healthcare data reliably depends not only on well-specified Implementation Guides, but on demonstrable evidence that those specifications have been implemented correctly. Today, the FHIR community lacks a consistent, structured approach to demonstrating that. Testing is treated as an afterthought — assembled under pressure in advance of a Connectathon, scoped narrowly, and disconnected from the conformance requirements it is meant to verify.  Testing as an Integrated Component of the FHIR Implementation Guide Authoring and Publishing Process will seek to change that.  Sponsored by the HL7 FHIR Infrastructure (FHIR-I) Workgroup and co-sponsored by the HL7 Electronic Healthcare Record Workgroup, this project seeks to establish the frameworks, guidance, and reference examples that make comprehensive, traceable conformance testing a first-class discipline with every Implementation Guide that HL7 publishes — converting the theoretical promise of FHIR interoperability into a measurable, verifiable reality.

Why This Project Matters

  • Objective, Measurable Conformance Evidence — this effort will aid the community in shifting the focus away from self-attestation and informal Connectathon-driven coverage toward a traceable, Requirements Traceability Matrix (RTM)-driven chain of evidence connecting every IG conformance requirement to an executable test and a verifiable implementation result. For implementers, this means knowing — not guessing — that their system conforms. For regulators, payers, and healthcare organizations, it means the interoperability claims behind every standard are backed by proof.
  • Accelerated IG Adoption and Reduced Implementation Risk — Standardized Testing Guides give implementers clear, unambiguous validation paths from the moment an IG is published, eliminating the interpretive gaps that slow adoption and introduce costly defects into production systems. Evidence from mature FHIR programs and internal analysis suggests that access to comprehensive, calibrated test guidance can reduce implementation cycle times by 30% or more — compressing the path from IG publication to production deployment and returning value to stakeholders measurably faster.
  • Interoperability Assurance at Scale — Conformance to a profile is necessary for interoperability, but it is not sufficient. Through this effort, our goal is to advance testing beyond basic FHIR validation into comprehensive, scenario-based verification that systems can exchange data correctly across organizational boundaries. This level of assurance is foundational for the federated ecosystems driving modern healthcare — CMS-aligned networks, TEFCA, payer-to-payer and payer-to-provider exchange — where the consequences of a failed interaction are measured in patient outcomes, not debug logs.
  • FHIR Standards Maturity and HL7 Governance Credibility — Integrating testing artifacts into the IG ballot process improves the quality and depth of community review, accelerates IG progression toward Normative publication, and creates an auditable record of implementation evidence that supports regulatory compliance and certification programs. For HL7 as a standards development organization, this is the difference between publishing specifications and publishing verifiable, implementable standards — a distinction that carries increasing weight in an era of heightened regulatory scrutiny and global interoperability expectations.

The promise of FHIR — _implement once, exchange infinitely_ — will only be realized when the community can demonstrate, not merely assert, that implementations are correct. Through this effort, we will provide the structural foundation to make that demonstration routine across every IG in HL7's portfolio. By establishing standardized Testing Guides as a required companion to every Implementation Guide, and by building the framework that makes authoring them tractable, this project gives implementers the confidence to move fast, gives stakeholders the evidence to trust what they receive, and gives HL7 the credibility to lead healthcare interoperability standards globally — not just in specification, but in verified, trustworthy practice.


h1. Introduction

The Health Level Seven (HL7) International FHIR Infrastructure (FHIR-I) Workgroup is following on work previously undertaken as part of the HL7 FAST Accelerator (Testing and Versioning) and Health and Human Services (HHS) Office of the National Coordinator (ONC) {_}FAST{_}: Testing, Conformance and Certification Tiger Team, seeking to continue advancing these efforts under a new project entitled “Testing as an Integrated Component of the FHIR Implementation Guide (IG) Authoring and Publishing Processes”  Approved by the HL7 FHIR Infrastructure (FHIR-I) Workgroup, the project will continue the work completed to date and published proposal content under the [PSS-2717|https://jira.hl7.org/browse/PSS-2717] initiative, while working to align with the new HL7 FHIR-I Project Scope Statement, and HL7 FHIR Best-Practices approach to Testing.

The final ONC FAST Task Force: Testing, Conformance and Certification Tiger Team published Solution Document can be found {_}[here|https://confluence.hl7.org/display/FAST/Testing+and+Versioning?preview=/134941709/175606493/FAST%20Testing%20Certification%20Tiger%20Team%20v3%20Solution%20Final%20Document%20111320.docx]{_}[_+[1]+_|https://confluence.hl7.org/spaces/PSS/pages/453905966/Testing+as+an+Integrated+Component+of+the+FHIR+Implementation+Guide+Authoring+and+Publishing+Process#_ftn1]

The final HL7 FAST Accelerator: Testing and Versioning project published documents can be found [here|https://confluence.hl7.org/spaces/FAST/pages/134941709/Testing+and+Versioning] [[2]|https://confluence.hl7.org/spaces/PSS/pages/453905966/Testing+as+an+Integrated+Component+of+the+FHIR+Implementation+Guide+Authoring+and+Publishing+Process#_ftn2]

Originally conceived with a focus on Testing, Conformance, and Certification, the HL7 FAST Accelerator combined Conformance under the Testing initiative, while the Certification effort remained on the HL7 FAST Backlog in a “paused” state, pending identification of a sponsor to lead it in the future. The HL7 FAST Steering Committee, FAST PMO, and FAST Technical Director subsequently chose to close the FAST Testing and Versioning effort. This effort {-}{-}now continues the work under formal HL7 FHIR Infrastructure (FHIR-I) governance, carrying forward the “Change the Context of Testing FHIR” approach central to the prior FAST effort.

This project continues and formalizes that prior work. Now sponsored by the HL7 FHIR Infrastructure (FHIR-I) Workgroup and co-sponsored by the EHR Workgroup, the project establishes a structured framework for integrating testing as a first-class component of the FHIR IG authoring and publishing lifecycle. With FHIR R5 now providing the TestPlan, Requirements, and ActorDefinition resources needed to realize this vision, these resources are now positioned to deliver the community guidance, templates, and reference examples that IG authors need to develop comprehensive Testing Guides that can operate in parallel with their specifications.

Such an integrated ecosystem provides a safe testing environment that closely approximates what a production community of information exchange participants will look and act like in a real-world setting. Further technical details on this approach will be developed and published as part of the previous project deliverables.

h3. In Scope

Building on the prior effort (FAST) analysis and through discussions within the HL7 FHIR-I Workgroup and the broader HL7 community, this effort will initially focus on techniques and concepts to further Maturity of Requirements, the benefits of publishing a Minimum Level of Conformance, and Timing-related issues. Our goal is to address these by capturing a set of “Testing Requirements” and documenting “best practices” as guidance documents, culminating in a White Paper offered for a Comment-Only ballot.

h3. Out of Scope

Tooling (and Test Platforms) and Certification Governance are initially considered out of scope for this project.

 

[[1]|https://confluence.hl7.org/spaces/PSS/pages/453905966/Testing+as+an+Integrated+Component+of+the+FHIR+Implementation+Guide+Authoring+and+Publishing+Process#_ftnref1] [https://confluence.hl7.org/display/FAST/Testing+and+Versioning?preview=/134941709/175606493/FAST%20Testing%20Certification%20Tiger%20Team%20v3%20Solution%20Final%20Document%20111320.docx]

[[2]|https://confluence.hl7.org/spaces/PSS/pages/453905966/Testing+as+an+Integrated+Component+of+the+FHIR+Implementation+Guide+Authoring+and+Publishing+Process#_ftnref2] [https://confluence.hl7.org/spaces/FAST/pages/134941709/Testing+and+Versioning]



May 2027 - Informative


https://confluence.hl7.org/spaces/FHIR/pages/491914478/Testing+-+Introduction+to+TestScript+FHIR+Testing+methodologies+Track+Report


Current State of FHIR Testing and Related HL7 Activities FHIR R4 and Earlier: Base Testing Artifacts Through FHIR R4, the core specification includes a dedicated Testing FHIR page and defines three canonical base resources that support conformance testing: • CapabilityStatement (formerly the Conformance resource) declares the capabilities a server or client supports; • TestScript defines an executable set of tests against a FHIR implementation; • TestReport records the results of executing a TestScript. FHIR DSTU2 (v1.0.2, October 2015) first introduced these resources. Several organizations — including MITRE, InteropLab, and AEGIS — have built FHIR test tooling on them over the intervening decade, and TestScript has been exercised extensively at HL7 FHIR Connectathons. Future HL7 testing initiatives, including this project, should leverage these resources rather than introduce parallel constructs. [Verify current FMM rating for TestScript in R5 before citing a specific level.] FHIR R5: Requirements, Actors, Test Plans, and Obligations FHIR R5 extended this foundation in two ways that this project depends on directly. First, R5 introduced three base resources that connect specification content to testing: Requirements (formal statements of what an implementation must do), ActorDefinition (the roles that participate in an exchange), and TestPlan (the overall test methodology, including its constituent Test Cases). Together these provide the vocabulary for tracing a written conformance requirement forward to an executable test. Second, R5 introduced Obligations to clarify the conformance expectations behind MustSupport [1]. MustSupport has long been a source of implementation ambiguity; Obligations give IG authors a way to state precisely what an application must do with an element, and give testers something concrete to verify. Alignment with Related HL7 Testing Activities This project will proceed in parallel with, and build on, several ongoing efforts. FHIR Shorthand (FSH) and SUSHI. MITRE created and maintains FHIR Shorthand and the SUSHI compiler, which are now the dominant authoring path for HL7 IGs. MITRE and AEGIS are jointly extending FSH to support authoring TestScript resources so test artifacts can be written and versioned alongside profile content. Federal partner support. HHS/CMS has contracted Lantana (with AEGIS) to support a number of FHIR IGs across the Da Vinci, CARIN Blue Button, and FAST Accelerators. That support has included varying levels of testing for the IGs' profiles and operations, testing the reference implementations (RI), and represents existing in-kind investment this project can draw on. FMG IG Publisher tooling. The FHIR Management Group (FMG), a primary stakeholder in this project, is sponsoring two IG Publisher enhancements being developed by the FHIR Product Director with members of the AEGIS FHIR technical team [2]: 1. Decoupled artifact packages. The IG Publisher can now publish an IG's Examples and Testing artifacts (TestPlan, Test Cases, TestScripts) as separate packages, so that updating an example or a test no longer requires republishing the IG itself. 2. R5 backport without breaking changes. R5 resources and features (e.g., the Requirements resource) can be used within R4-based IGs, allowing current IG sponsors to adopt the R5-and-later IG architecture without a version migration. Work on integrating testing artifacts into the IG authoring and publishing process continues under AEGIS leadership with oversight from the FHIR Product Director [3]. Reference Model: HL7 FAST Scalable Consent Management Testing Guide The HL7 FAST Scalable Consent Management Testing Guide (v1.0.0-ballot, package hl7.fhir.us.consent-management-tg) is a working example of the practice this project proposes to make standard across HL7's IG portfolio. Developed by AEGIS under the FAST Accelerator and now in formal ballot, it is published as a companion package — structurally separate from, but aligned with, the parent Scalable Consent Management IG — demonstrating the decoupled publication model the FMG tooling enhancements are designed to support at scale. Its value as a reference model lies in the completeness and traceability of its artifacts: • Actors. The guide formally defines its participants (an Admin Server and a Client), anchoring every test scenario to roles established in the parent IG. • Requirements Traceability Matrix (RTM). Presented in three views — Requirements listing, Test Cases inventory, and Coverage analysis — the RTM shows where each conformance requirement is exercised and where validation gaps remain. This structure establishes the normative expectation for Testing Guides, replacing the informal, Connectathon-driven assessment that has historically served as the de facto measure of IG coverage. • Executable tests. Eighteen or more TestScript resources, organized into test groups covering capability verification, read and simple search, custom operations (File a Consent, Revoke a Consent, Record a Disclosure), and basic REST-hook Subscription workflows, supported by staged fixtures (Consent, Patient, Organization, DocumentReference) aligned to the IG's profiles. • TestPlan. Serves as the overarching test methodology and the anchor for traceability from written requirement to executable artifact. The guide's own Guidance page identifies an outstanding gap: the IG Publisher does not yet automatically produce a testing package as a distinct publication output. Closing that gap is a direct target of the IG Publisher work described above and one of the concrete tooling deliverables this project will address in later phases. The Testing Guide will serve as the reference case study as the broader framework is developed and refined: https://wildfhir4.wildfhir.org/ig/fhir-consent-management-tg/index.html

EHR, PHR Vendors,Health Care IT Vendors,Health Care IT,Payer/Third Party Administrator,Regulatory Agency,Vendor/Manufacturer



Testing as an Integrated Component of the FHIR Implementation Guide "Changing the Context of Testing" https://confluence.hl7.org/spaces/PSS/pages/453905966






Organization & Realm

Universal

People

Mario Hyland

Richard Ettema


FHIR Infrastructure Work Group

Electronic Health Records Work Group

Status & Dates

Active

Active Project (Resources assigned to pjt)

09/29/2026

05/01/2028

2028 May WGM/Ballot

05/01/2027

2027 May WGM/Ballot

Approvals



Not Needed


Not Needed


Pending


System Info

280879

PSS-2770

https://jira.hl7.org/browse/PSS-2770
View Ballot Items (with NIBs)

09/29/2026

Jira Project Service


Return to Search