Can We Trust the Dashboard?
This page teaches teams how to evaluate EHR dashboards, certified quality tools, registries, and population health vendors when quality scores do not line up with the applicable program, measure version, data lineage, or submission format.
🧭 First rule: “Vendor” does not mean one thing
EHR / CEHRT Vendor
The certified health IT vendor responsible for certified functionality such as recording, exporting, importing, calculating, or reporting CQMs when those modules are certified.
Certified Quality Tool or Registry Vendor
A third-party tool may have certified modules even if it is not the EHR of record. Certification is module-specific.
Non-certified Pop Health / Analytics Vendor
A useful dashboard may still be an analytics tool using extracts, claims, interfaces, APIs, or proprietary mappings rather than certified reporting logic.
🚦 Dashboard trust levels
| Trust level | What it means | What to ask next |
|---|---|---|
| 🟢 High trust | Dashboard uses the applicable reporting-year logic, value sets, structured data, and certified reporting evidence. | Show program-specific calculation and reconciliation proof. |
| 🟡 Operational trust | Dashboard is directionally useful but uses separate extracts, timing, claims, or mappings. | Ask how gaps are reconciled to the certified module and required submission format. |
| 🔴 Low trust | Vendor cannot identify program, measure version, value set version, source field, or submission alignment. | Do not use as source of truth for official reporting without validation. |
🏥 Questions for the EHR / CEHRT vendor
Certification and reporting-year readiness
- Are the quality-reporting functions we use certified under ONC/ASTP certification criteria for the applicable reporting year?
- Which certification criteria apply to the functions we rely on: record/export, import/calculate, report, or other modules?
- Does the CMS EHR Certification ID match the applicable CMS reporting-year requirement?
- What date will production be updated for the current reporting year's eCQM specifications, VSAC value sets, QRDA implementation guide, Schematron files, and validation rules?
UI dashboard vs certified reporting output
- Is the UI score calculated from the same data model, value sets, logic, and measure version used for the applicable program output?
- If the UI and reporting output disagree, which certified module and artifact govern submission?
- Can you provide patient-level reconciliation showing denominator, numerator, exclusion, exception, and failure logic?
- Can you export a sample QRDA-I file and show where the supporting evidence appears?
Annual update governance
- Do you maintain prior-year logic separately so 2025 reports are not recalculated using 2026 logic?
- Do you provide release notes for measure logic changes, value set changes, retired codes, added codes, and deleted codes?
- Can we run old and new logic side by side before the change goes live?
📊 Questions for certified third-party quality or registry vendors
Certification
Are you ONC-certified for the specific quality reporting functions we are using, and where can we verify that certification?
Reporting year
Which reporting-year eCQM specifications, VSAC value sets, QRDA implementation guide, and validation package are loaded?
Alignment
If your output differs from the certified module's applicable report, which system is authoritative and how is the discrepancy reconciled?
Proof
Can you show the exact code, source field, date, value set OID, and measure logic step that caused a patient to count or fail?
🧪 Questions for non-certified pop health / analytics vendors
These vendors may still be valuable, but the team needs to know whether the dashboard is official reporting logic or an operational approximation.
| Question | Why it matters |
|---|---|
| Are you certified health IT for this measure calculation/reporting function, or are you an analytics/dashboard tool? | This defines how much regulatory certification confidence the organization can place in the output. |
| Are you using the same structured data elements the certified module uses, or a separate extract, claims feed, flat file, interface, API, or proprietary mapping layer? | Separate pipelines often create different answers. |
| Can your system reconcile against the applicable certified-module calculation and required submission format? | This is key when the population-health dashboard and official output disagree. |
| How often do you update VSAC value sets, measure logic, terminology versions, and mapping tables? | A stale value set can turn correct care into a reported failure. |
| Is your update cadence contractually guaranteed, or best effort? | If it is not in the contract, it may not happen on the organization's timeline. |
| Can you provide customer-facing release notes and before/after patient-level impact reports? | This shows whether annual updates are controlled and auditable. |
🧬 Data lineage: the magic phrase
Documentation
Where did the user enter it?
Structured field
Was it stored as data or only note text?
Code and value set
Which code and value set made it count?
Measure result
Why denominator, numerator, exclusion, exception, or failure?
✅ Printable call-prep checklist
Before a vendor call, select what you need to ask.
0 selected.
🎮 Vendor scenario quiz
1. The EHR dashboard says a patient passed, but the QRDA-I export does not contain the numerator evidence. What should you ask first?
2. A pop health vendor says, “Our dashboard is close enough.” What is the best response?
3. A vendor cannot tell you which VSAC release or measure version is loaded. What trust level is that?
🧾 Copy-paste vendor request
Suggested message:
Please provide documentation showing the program, reporting or performance year, eCQM measure version, VSAC release, applicable QRDA implementation guide, validation package, terminology versions, and production update date. Confirm whether the UI and certified reporting output use the same structured data and measure logic, identify the required submission format, and provide a reconciliation example from documentation to standard code to value set to final measure status.
📌 Source notes
Authoritative starting points: eCQI QRDA overview, ONC conformance test tools, and ONC CQM record and export. Check the applicable program and reporting-year requirements before operational use.