Validated Assurance Reports, and What Happens when Validation Fails

By Jeremy Huval, Chief Compliance Officer

As discussed in HITRUST’s recent article, How Do I Know if an Assurance Report is Rely-Able, HITRUST communicated how accuracy, transparency, consistency, and integrity of our assurance reports, or what we call rely-ability, is paramount to their acceptance in the market. We recognize that HITRUST CSF reports have value only because of the quality and consistency of their underlying assessment and reporting processes. That article outlined the numerous mechanisms in place to help ensure the quality of our assurance reports, including the multiple reviews each HITRUST CSF Assessment must successfully undergo. This blog post explains what it means for a HITRUST CSF report to be “validated,” and also addresses the necessary, but unfortunate, fact that not all validated HITRUST CSF assessment efforts lead to the issuance of a CSF Validated report.

HITRUST currently issues three types of assurance reports:

HITRUST Readiness Reports (previously called “Self-assessment Reports”), which contain the unaudited description of an organization’s implementation of the HITRUST CSF. Readiness reports are not meant to provide reliable assurances to third parties but can be used to demonstrate the achievement of an important milestone in the path towards HITRUST CSF certification.
Validated HITRUST CSF Reports, which are issued by HITRUST when the control scores are below the level required for certification.
Validated HITRUST CSF Reports with Certification, which are issued by HITRUST when the control scores are at or above the level required for certification.

For a HITRUST-issued report to be deemed “validated”, the following conditions must be met:

The assessed entity’s HITRUST CSF compliance levels reflected in the report must be confirmed by an Authorized HITRUST External Assessor Organization (i.e., an “external assessor”).
The documentation prepared by both the assessed entity and its external assessor must pass HITRUST’s quality assurance (“QA”) review.

The concerted efforts of many parties are necessary for these two conditions to be met. First, management of the assessed entity must implement the HITRUST CSF, and they must also attest to their organization’s level of compliance with the HITRUST CSF. Then, the assessed entity must demonstrate to its external assessor that the HITRUST CSF has been implemented in the environment to the stated levels. The external assessor must perform assessment procedures to confirm the assessed entity’s HITRUST CSF implementation, and these assessment procedures must be executed in accordance with HITRUST’s assessment performance and documentation requirements. Last, the assessment must pass the HITRUST Assurance function’s QA review as well as a subsequent review by HITRUST’s Compliance department.

What does HITRUST’s QA review entail?

HITRUST’s QA review consists of the following:

Automated checks: The assessment’s scoring, commentary, and accompanying documentation is subjected to over four dozen automated quality checks designed to identify common assessment scoring and commentary errors and omissions.
Core QA: A sample of randomly selected HITRUST CSF requirement statements is reviewed to confirm the sufficiency of the external assessor’s basis for agreement with the assessed entity’s scoring.
Test of N/A’s: The documented rationale for deeming any HITRUST CSF requirement statements as “not applicable” is reviewed for reasonableness, consistency, and appropriateness.
Test of measured and managed: All HITRUST CSF requirement statements where the measured and managed PRISMA control maturity levels were scored are reviewed to confirm the sufficiency of the external assessor’s basis for agreement with the assessed entity’s scoring.

What does it mean for a HITRUST CSF Assessment to fail QA?

It is normal for HITRUST’s QA Analysts to raise questions about an assessment as they perform QA procedures. Normally, these questions are resolved through collaboration between the HITRUST QA Analyst, the external assessor’s engagement team, and the assessed entity. However, in some instances these questions cannot be satisfactorily resolved. In order to protect the quality and rely-ability of our assurance reports and CSF Assurance Program, HITRUST will not issue a Validated HITRUST CSF Report if there are unresolved concerns about the rigor of the assessed entity’s HITRUST CSF implementation and/or the adequacy of the external assessor’s procedures.

Note that an assessment failing QA is very different from an assessment result reflective of low control maturity. The concept of an audit / assessment / inspection failing a review typically means that a reviewer of the audit (e.g., a regulator, an internal reviewer, a peer reviewer) uncovered significant issues about the audit itself. These issues typically stem from either (a) the auditor’s failure to uncover issues in the audited environment, and/or (b) the auditor’s failure to adhere to applicable auditing standards in the performance or documentation of the audit. In the HITRUST context, a validated assessment failing HITRUST’s QA review means that HITRUST had significant enough concerns about the assessment itself to prevent issuance of a Validated HITRUST CSF Report (with or without certification).

What’s HITRUST’s process to confirm that an assessment has failed its QA?

Given how impactful this outcome is to all involved parties, multiple reviewers internal to HITRUST must concur with the “failed QA” outcome.

If the HITRUST QA Analyst’s procedures yield questions which cannot be resolved through the QA Analyst’s collaboration with the external assessor, the assessment is escalated to the HITRUST VP of Assurance Services. The VP of Assurance Services reviews the QA Analyst’s work and may independently review additional requirement statements not previously reviewed by the QA Analyst. If the VP of Assurance Services agrees that the assessment failed HITRUST’s QA review, the submission is escalated to the Compliance department. A Compliance team member reviews all QA work performed by the Assurance team, and often independently reviews even more requirement statements. If the Compliance team member agrees that the assessment failed HITRUST’s QA review, HITRUST’s unresolved questions and concerns are discussed over one or more meetings attended by the external assessor.

A “failed QA” outcome is only reached when (a) leadership of both the Assurance and Compliance departments conclude that this outcome is warranted, and (b) the external assessor is unable to resolve HITRUST’s questions and concerns. Because HITRUST’s QA focuses on a subset of the external assessor’s testing, unresolved concerns identified during QA are viewed as indicative of problems in the larger validated assessment.

What could cause a HITRUST CSF Assessment to fail HITRUST’s QA?

While nothing is typical about a “failed QA” outcome, the root causes leading to this potential outcome can be grouped as follows:

The assessed entity failed to implement the CSF to a degree warranting certification yet scored itself within HITRUST’s certification scoring threshold, and the external assessor failed to identify and/or push back on the assessed entity’s inaccurate control maturity scoring.
The assessed entity failed to effectively demonstrate its implementation of the CSF to the external assessor, and the external assessor failed to identify and/or push back on the assessed entity’s unsubstantiated control maturity scoring.
The assessed entity and the external assessor did not correctly leverage HITRUST’s Control Maturity Scoring Rubric when respectively determining and confirming the organization’s control maturity scoring.
The external assessor failed to conduct and/or document the validated assessment in accordance with HITRUST’s assessment requirements.
The external assessor failed to incorporate changes to the HITRUST CSF Assurance Program into its assessment methodology.

Examples of actual issues which have led to validated assessments failing HITRUST’s QA include:

The external assessor attempted to “test the spirit of the control” instead of testing the actual HITRUST CSF implementation requirements.
The assessed entity represented (and the external assessor agreed) that the policy and procedure PRISMA levels were “Fully Compliant,” yet the organization’s documented policies and procedures did not sufficiently address the HITRUST CSF implementation specifications.
The assessed entity represented (and the external assessor agreed) that the implemented PRISMA levels were “Fully Compliant,” yet only policy and procedure documents were collected to substantiate control operation and implementation.
Neither the assessed entity or the external assessor properly understood concepts related to measuring the ongoing operation of internal controls, leading to inflated scoring in both the measured and managed PRISMA control maturity levels.

What happens if an assessment fails HITRUST’s QA?

When a validated assessment submission fails HITRUST’s QA review, no validated report is issued. HITRUST will instead issue a letter to the assessed entity describing the unresolved concerns leading to the “failed QA” outcome. The assessed entity must undergo a completely new validated assessment if they choose to proceed towards a Validated HITRUST CSF Report. Dependent on the age of previously collected evidence (relative to the external assessor’s fieldwork dates of the repeated validated assessment), fresh copies of audit evidence likely must be collected. If a Validated HITRUST CSF Report with Certification is issued as a result of the re-performed validated assessment effort, it will be dated as of the end-date of that re-performed validated assessment effort (and not the original / failed assessment).

What should an assessed entity do to avoid a failed QA outcome?

The top two things an assessed entity should do to prevent a “failed QA” outcome are exercising meaningful due diligence when selecting an external assessor and building HITRUST literacy within the organization.

Due diligence in selecting an external assessor

Don’t assume that all Authorized HITRUST External Assessor Organizations are created equal. Some of these independent firms specialize in CSF readiness consulting, others focus on performing SOC 2 + HITRUST examinations, others focus only on customers within a specific industry, and others simply do not perform validated assessments. Although HITRUST vets all professional services firms applying to become Authorized HITRUST External Assessor Organizations, this vetting process focuses on evaluating their capability to perform validated assessments and does not guarantee successful performance.

Basic vendor selection due diligence can go a long way in selecting an external assessor that is a good fit for performing an organization’s validated HITRUST CSF assessment. First, ensure that all the potential professional services firms are Authorized HITRUST External Assessor Organizations. Only firms listed on HITRUST’s website are authorized to perform validated assessments. Also, contact one or more of the firm’s previous HITRUST customers to confirm their experience.

Additionally, organizations should ask purposeful questions to help them understand the firm’s experience and qualifications. Example questions to consider asking potential external assessors include:

How long has it been since your firm last performed a validated HITRUST assessment?
Has your firm performed any validated assessments as complex as mine?
Have any of your validated assessments ever failed HITRUST’s QA review?
How many Certified CSF Practitioners (CCSFPs) do you currently have on staff? (Note that all Authorized HITRUST External Assessor Organizations are required to have at least five CCSFPs.)
How many Certified HITRUST Quality Professionals (CHQPs) do you have on staff? (Note that all Authorized HITRUST External Assessor Organizations are required to have at least two CHQPs.)
Is your firm currently under any quality-related corrective action agreements with HITRUST?
Have any of your firm’s CCSFPs ever had their CCSFP designation revoked by HITRUST?
What kinds of other (non-HITRUST) IT assurance work does your firm perform? (Look for experience in a regulated audit context where an external reviewer performs an after-the-fact review of the audit.)

Build the organization’s internal HITRUST literacy

Designate one or more HITRUST champions within your organization, and see that all HITRUST champions are sufficiently educated on the HITRUST CSF and on the HITRUST CSF Assurance Program. While not required, many organizations choose to have their HITRUST champions become CCSFPs. If the organization’s internal HITRUST champion does nothing else, we highly recommend that they read and understand the following documents prior to embarking on any HITRUST validated assessment effort:

The more knowledgeable your organization is about the normal course of events along the path to HITRUST CSF Certification, the more equipped you are to identify and remedy when things aren’t going normally. For example, knowing about the potential for a validated assessment to have a “failed QA” outcome should cause you to recognize a “guaranteed HITRUST certification” claim from any professional services firm as a red flag. As another example, if an assessor calls several months after the validated assessment ends asking you to sign the HITRUST management representation letter, a knowledgeable internal HITRUST champion would recognize this to be a red flag as the management representation letter should have been signed no more than two weeks after the assessor’s fieldwork ends.

Closing thoughts

Unlike other information assurance mechanisms, every validated report issued by HITRUST is subjected to numerous quality checks designed to ensure reporting consistency, accuracy, and integrity. Some HITRUST CSF assessments fail these quality checks and, as a result, do not lead to a Validated HITRUST CSF Report. When HITRUST determines it is unable to issue a validated report, it does so for the benefit of those who rely on HITRUST reports as the gold-standard of rely-able assurance mechanisms. Assessed entities and their external assessors should exercise due professional care in the preparation for and performance of validated HITRUST assessments to help ensure that their validated assessments pass through HITRUST’s quality assurance checks without issue.