The National Evaluation of theCHIPRA Quality Demonstration Grant ProgramEvaluation Highlight No.10, August 2014The CHIPRA QualityDemonstration Grant ProgramIn February 2010, the Centers for Medicare &Medicaid Services (CMS) awarded 10 grants,funding 18 States, to improve the quality ofhealth care for children enrolled in Medicaidand the Children’s Health Insurance Program(CHIP). Funded by the Children’s HealthInsurance Program Reauthorization Act of2009 (CHIPRA), the Quality DemonstrationGrant Program aims to identify effective,replicable strategies for enhancing quality ofhealth care for children. With funding fromCMS, the Agency for Healthcare Researchand Quality (AHRQ) is leading the nationalevaluation of these demonstrations.The 18 demonstration States areimplementing 52 projects in five generalcategories: Using quality measures to improve childhealth care. Applying health information technology (IT)for quality improvement. Implementing provider-based deliverymodels. Investigating a model format for pediatricelectronic health records (EHRs). Assessing the utility of other innovativeapproaches to enhance quality.The demonstration began on February 22,2010, and will conclude on February 21,2015. The national evaluation of the grantprogram started on August 8, 2010, and willbe completed by September 8, 2015.How are CHIPRA quality demonstrationStates testing the Children’s ElectronicHealth Record Format?Author: Leslie FosterThis Evaluation Highlight is the 10th in a series that presents findings from theChildren’s Health Insurance Program Reauthorization Act of 2009 (CHIPRA)Quality Demonstration Grant Program. Two States—North Carolina andPennsylvania—are part of an effort to test the Children’s Electronic Health Record(EHR) Format (the Format), which was commissioned by the Centers for Medicare& Medicaid Services (CMS) and the Agency for Healthcare Research and Quality(AHRQ) and intended to improve the quality of health care for children enrolledin Medicaid and CHIP. The Highlight focuses on the States’ activities from 2012 toearly 2014.KEY MESSAGES State and provider stakeholders in North Carolina and Pennsylvania generallyagreed that the Format addresses many child-specific functions not addressed bycurrent EHRs. (The current version of the Format is available at Practices and health systems discovered that their EHRs did not meet manyFormat requirements, although some requirements were available through thepurchase of an EHR upgrade. When requirements were already present in EHRsbut could not be accessed readily, the Format drove discussions about the needsand expectations of EHR users. Incorporating the Format requirements into current EHRs was challenging.Pennsylvania health systems prioritized the changes they would try to make totheir EHRs, whereas EHR coaches in North Carolina chose to focus on trainingpractices to improve their use of EHRs. State, health system, and practice staff in North Carolina and Pennsylvania saidthat EHR vendors were reluctant to engage in their projects because of otherpriorities. An EHR certification module for child health, even if limited to a subset of highpriority Format requirements, could help spur desired change in child-specificEHR functionality.

How are CHIPRA quality demonstration States testing the Children’s Electronic Health Record Format?BackgroundFigure 1. Evolution of the Children’s EHR FormatHealth care providers, payers, andFederal and State policymakersincreasingly look to EHRs as a tool formeasuring and improving health carequality. However, existing EHRs do notfully support the provision of highquality care to children from prenataldevelopment through adolescence.1,2For example, weight-based medicationdosing, immunization tracking, andmonitoring of children’s growthagainst standardized charts are routineclinical practices that many EHRs donot fully support.In 2009, the reauthorization of CHIPspecifically required the developmentof a Children’s EHR Format and thusbecame an impetus for change. CMSfunded and collaborated with AHRQon a development process that drewon existing work and specificationsin children’s health informationtechnology (IT) and the contributionsof health IT and child healthinformatics experts.The resulting Format is a set ofrecommended requirements forEHR data elements, data standards,usability, functionality, andinteroperability. The Format’s currentset of 700 requirements is sortedinto 21 topic areas relevant to thecare of children in ambulatory eleased to NC and PAwith 568 requirements20132014Released to the publicwith 700 requirements20152016NC and PA CHIPRAprojects concludeNote: AHRQ and CMS are responsible for the development and refinement of the Format.Refinements identified during a contract ending in late 2015 may be implemented later.inpatient settings. Topic areas includeprenatal and newborn screening,immunizations, growth data, childrenwith special health care needs, wellchild/preventive care, patient portalavailability, medication management,and the reporting of child abuse. Theindividual requirements in each topicare prioritized by whether they shall,should, or may be present in EHRs.The information in this Highlightcomes from semi-structured interviewsconducted by the national evaluationteam in spring 2012 and spring 2014.The team interviewed each State’sCHIPRA quality demonstration staffand the staff of participating healthsystems and primary care practices.This Highlight focuses on the rolesof North Carolina and Pennsylvaniain the Format’s evolution (Figure 1).Both States are using CHIPRA qualitydemonstration grant funds to test (1)how well the Format’s requirementssupport the provision of primary careto children and (2) how readily therequirements can be incorporated intoexisting EHRs.The two States took differentapproaches to testing the usabilityand functionality of the Format.Pennsylvania tested it with five healthsystems that serve children: threechildren’s hospitals and affiliatedambulatory practice sites, one federallyqualified health center (FQHC), andone small hospital. North Carolinaused EHR “coaches” to reach out to 30individual practices about testing theFormat (Table 1).FindingsTable 1. Overview of Health Systems and Practices Testing the Format, as of Spring 2014StatePennsylvaniaProvider TypesPrior EHR ExperienceTotal Number of EHRsin Usea3 children’s hospitalsand affiliated ambulatorypractice sitesNot required4StatewideRural, urban, andsuburbanRequired6StatewideRural, urban, andsuburbanService Areas1 FQHC1 small hospitalNorth Carolina30 independent pediatricor family practicesFQHC federally qualified health centerThe EHRs are sold by vendors offering products that are certified under the CMS EHR Incentive Program.aThe National Evaluation of theCHIPRA Quality Demonstration Grant ProgramPage 2Evaluation Highlight No. 10, August 2014

How are CHIPRA quality demonstration States testing the Children’s Electronic Health Record Format?Pennsylvania’s health systemsworked independentlyPennsylvania gave the five healthsystems freedom to test the Formatas they saw fit and designated oneof the children’s hospitals to supportand loosely organize the work of theother four systems. Accordingly, eachsystem developed its own projectobjectives and implementation plansand reported to the State on theirprogress and setbacks. The State gavethe systems this much leeway becausethe extent of their experience withEHRs varied widely; some systems hada great deal of experience, while othershad very little.North Carolina used EHR coachesto recruit and guide practicesNorth Carolina hired, trained, andsupervised four EHR coaches whoseprofessional backgrounds rangedfrom nursing to practice managementto health IT. According to projectstakeholders, the coaches’ interpersonalskills and knowledge of health carehave been especially germane to thecoaching job.Each coach recruited practices inan assigned area of the State to testthe Format. Coaches oversaw thecompletion of a survey that askedpractices and EHR vendors to compareexisting EHRs to the Format. Thecoaches also have acted as a liaisonbetween practices and vendors inconsidering next steps, and they havebegun training practices to use EHRfunctionalities that already meetFormat requirements.Practices and health systems weremotivated by a desire for betterEHRsPractices in North Carolina and healthsystems in Pennsylvania joined the“For the practices, [the Format]is about having a better EHRsystem. The practices have alot of angst about the systemsthey are currently using. TheirEHR may be missing an asthmaaction plan and a growth chart.It’s missing key things theyneed.”— North Carolina DemonstrationStaff, May 2014CHIPRA quality demonstration eitherbecause they were dissatisfied withtheir EHRs’ capacity for supportinghigh-quality children’s health care,because they saw the CHIPRA qualitydemonstration as an opportunityto improve their EHRs, or both.Practices and health systems viewedthe Format as a tool for learning moreabout their EHRs, and they used theCHIPRA quality demonstration as astructure through which they couldcommunicate their unmet needs tovendors.In addition, North Carolina practiceswere drawn to the project by theopportunity to work with EHRcoaches, do their own reporting forquality improvement, and participatein a health information exchange.Pennsylvania’s health systems receivedgrant funds for their work, and theycould also receive incentive paymentsfor using their EHRs to report andimprove their performance on certainquality measures as part of the State’sCHIPRA quality demonstration project(see Highlight 5).3Stakeholders reported that theFormat improves on existing EHRproductsIn both States, CHIPRA qualitydemonstration staff, projectmanagers, and providers involvedThe National Evaluation of theCHIPRA Quality Demonstration Grant ProgramPage 3in testing the Format said that it iscomprehensive and that it reflects asolid understanding of the delivery ofchildren’s health care. North Carolinapractices rated approximately 80percent of the requirements theyreviewed as medically relevant.Nonetheless, they also reported thatmany requirements are ambiguousor lacking in detail, and they believethat vendors might need to consultclinicians and quality experts in orderto fully understand the requirements.In addition, during the comparisonphase of their projects (describedbelow), the smaller Pennsylvaniahealth systems began to view theFormat as exceeding their EHR needsand wished it had been narrowed to amuch smaller set of core requirements.Comments from providers aboutspecific requirements (or missingrequirements) varied enough thatno common themes emerged fromtheir responses. For example, a fewproviders said they appreciatedthe Format’s decision-supportrequirements, noting that therequirements contain detailedinformation that providers needbut usually do not memorize. Twoproviders wanted the Format toinclude a way to identify siblings intheir practices so that they could betteraddress the health needs of families.The providers’ opinions about theFormat’s requirements for linkagesto school-based health data systemswere mixed. Some wanted informationabout school-based care in theirEHRs but doubted that the necessaryinteroperability would exist in thenear future. Others said that their EHRis a record of care provided by theirpractice alone and should not containexternal information.Evaluation Highlight No. 10, August 2014

How are CHIPRA quality demonstration States testing the Children’s Electronic Health Record Format?North Carolina and Pennsylvaniaused different approaches toengage vendorsNorth Carolina wants its project notonly to improve the health IT industry’sunderstanding of the role of technologyin children’s health care but also toaccomplish change at the vendorproduct level. To that end, CHIPRAquality demonstration staff asked EHRvendors to agree to (1) complete andreturn a survey that compared existingproducts to the Format, (2) train EHRcoaches to use EHR features thatpractice staff were not familiar with,and (3) indicate whether their productswill meet specific Format requirementsin the foreseeable future. By spring2014, four of six targeted vendorsagreed to participate in the NorthCarolina project. With the CHIPRAquality demonstration scheduled toend in February 2015, practices thatwork with the remaining two vendorsmay not get the training or the EHRenhancements they hoped for.EHR vendors have no formal role inthe Pennsylvania project. Some of thehealth systems involved their vendorswhen they compared the Format totheir own EHR systems. Other healthsystems did not try to involve theirvendors until they reached the stage ofdetermining how to incorporate Formatrequirements into their systems.Comparing the Format to existingEHRs was challenging but valuable“The requirements are veryspecific and may be located inhundreds of locations in differenttemplates and different types ofvisits throughout the EHR system.That probably took the most timeat first—just deciding if yoursystem does it or not.”— Pennsylvania Physician,May 2014health; asthma; and the Early andPeriodic Screening, Diagnosis, andTreatment (EPSDT) program inMedicaid. Focusing on only the 133prioritized requirements, the practicesand vendors independently completedthe survey mentioned earlier (Table 2).EHR coaches compiled and comparedthe responses of practices andvendors. Pennsylvania health systemsanswered similar questions about therequirements, but the State gave themall 568 requirements at once and didnot explicitly require responses frompractices and vendors to be collected orcompared.Over a course of several months, NorthCarolina practices, Pennsylvania healthsystems, and some vendors in eachState compared the Format to existingEHRs one requirement at a time inorder to identify gaps between the two.Tackling complexity. The director ofthe North Carolina CHIPRA qualitydemonstration project prioritized 133of the Format’s requirements that sheconsidered most relevant to the State’squality improvement goals in thefollowing areas: developmental andbehavioral health; obesity; oralTable 2. Process for Comparing the Children’s EHR Format to Existing EHRsRequirementQuestion for EHRVendorsQuestions for PracticesPossible NextStepIs the requirementmedicallyrelevant?Does yourEHR meet therequirement?If so, does yourpractice use it?Does your company’sEHR product meet therequirement?1YesYesYesYesNo change toexisting EHR2YesNoN/AYes—in ngopportunity4YesNoN/ANoIT solution5NoYes or noN/AYes or noNo change toexisting EHR;feedback forFormat refinementNotes: This table simplifies North Carolina’s approach for illustrative purposes. N/A not applicable.The National Evaluation of theCHIPRA Quality Demonstration Grant ProgramPage 4Evaluation Highlight No. 10, August 2014

How are CHIPRA quality demonstration States testing the Children’s Electronic Health Record Format?Reaching agreement. Practices andhealth systems had to resolvemany disagreements with vendorsabout whether EHRs met Formatrequirements. A Pennsylvaniaproject manager estimated that hishealth system disagreed with itsvendor on 30 to 40 percent of therequirements, and the two partiesspent 15 hours comparing the resultsof their assessments and resolvingdiscrepancies.Possible next steps varied with thereason for the disagreement. Forexample, if a practice noted that itsEHR did not meet a requirement, buta vendor indicated that an upgradedversion would meet the requirement,then the practice could consider anupgrade as a next step (Table 2, row2). In many other cases, lack of clarityover the same version of an EHR arosebecause a practice was not fully awareof all features of its EHR. In such cases,additional training of EHR users couldbe the next step for a practice or healthsystem to consider (Table 2, row 3).“The model Format becomesthe engine of explaining whata good pediatric EHR systemshould contain and what it cando for you.”— North Carolina DemonstrationStaff, May 2014Realizing benefits. Practices and healthsystems said that they benefitedfrom the comparison process. Forinstance, they learned more about thecapabilities of their EHRs. Moreover,they were not satisfied to learnthat their EHR met a requirementtechnically unless it also fit into anintuitive, efficient workflow. Thus, thecomparison process was also beneficialin that it gave practices and healthsystems the chance to use the Format todrive discussions about their needs andexpectations.Health systems and practiceshave begun incorporating FormatrequirementsAfter the comparison phase, healthsystems and practices considered howto more closely align their EHRs withthe Format requirements.Setting priorities to add requirements.One of the larger Pennsylvaniahealth systems (a major children’shospital with a network of ambulatorypractices) is working to incorporaterequirements from most of theFormat’s 21 topic areas into its EHR.Considering the 100 or so requirementsthat the system’s EHR did not meet,the system staff prioritized eachrequirement according to whether it(1) was related to patient safety, (2)was developmentally appropriateand patient focused, (3) described afunction that would be easy to use,and (4) was realistic to incorporatefrom a practical standpoint (takinginto account in-house resources, EHRvendor involvement, and costs).The health system’s project managersought input from IT staff, providers,corporate leaders, and the EHR vendor.Even after prioritizing the requirementsbased on this information, in achildren’s hospital owned by a nationalcorporation, changing an EHR is “aslow process and a long-term politicalcampaign,” said the project manager.The health systems that had to be moreselective about incorporating Formatrequirements into their EHRs becauseof resource constraints said that thewell-child visit and immunizationcategories were their highest priorities,followed by the patient portal (whichoverlaps with the CMS MedicaidEHR Incentive Program), childrenwith special health care needs, andThe National Evaluation of theCHIPRA Quality Demonstration Grant ProgramPage 5the confidentiality of informationabout minors. One health system saidthat it used the “shall” requirementsto set priorities. No systems saidthat they disagreed with the “shall”requirements, but, for whatever reason,most did not explicitly use the Format’simplied prioritization in decisionsabout EHR modification.Focusing on EHR reporting and training.Given practices’ limited IT resourcesand leverage with vendors, theNorth Carolina CHIPRA qualitydemonstration team has itself takensteps to align the Format withpractices’ ability to use their EHRs tocapture and report care processes forquality improvement. As of spring2014, the team had developed qualitymeasures written specifically forEHRs to guide vendors when, in theState’s opinion, Format requirementsprovided insufficient direction. A fewvendors had begun producing EHRreporting tools as the State envisioned.At the same time, the practices andthe EHR coaches were still involvedin the comparison process, and thecoaches were trying to arrange forEHR vendors to train them in selectedfunctionalities so that they could trainpractice staff.Struggling to involve vendors. As inearlier stages of the project, practicesand health systems had difficultyengaging their EHR vendors whenvendor assistance was needed tomodify EHRs or to train coachesin EHR functionalities. Based onthese experiences, CHIPRA qualitydemonstration staff, project managers,and EHR users concluded that mostvendors do not see a compellingbusiness reason to make their productsFormat-compliant or to meet needs forchildren’s health IT more generally.Instead, these stakeholders believethe vendors’ top priorities are theEvaluation Highlight No. 10, August 2014

How are CHIPRA quality demonstration States testing the Children’s Electronic Health Record Format?ICD-10 transition (mandatory changesin reporting medical diagnoses andinpatient procedures) and achievingcertification under the CMS MedicaidEHR Incentive Program (which greatlyaffects EHR marketability).EHR coaches and health systems hadthe attention of vendors, their testing ofthe Format helped them to identify anddiscuss their expectations for a childoriented EHR.ConclusionsThe findings from North Carolina andPennsylvania have implications forStates and other stakeholders interestedin using EHRs as a tool for measuringand improving children’s health carequality. The experiences and feedbackfrom North Carolina and Pennsylvaniacould also be useful to CMS and AHRQas they continue to refine the Format,prioritize requirements, and improvethe Format’s usability.As a result of the two States’efforts during the CHIPRA qualitydemonstration, the Format has beentested by independent primary carepractices, large children’s hospitalsand their ambulatory practice sites, anFQHC, and a small rural hospital. TheFormat has been compared with 10EHRs.State and provider stakeholdersgenerally found the Format theyreceived to be a major advance in thespecification of child-oriented EHRfunctions. The overall appreciationfor the Format’s thoroughness wasdiminished by the time-consumingprocess of comparing the Format toexisting EHRs. As they prioritizedthe Format requirements and stafftraining needs, the health systems andpractices confronted the limits of theirhealth IT resources, their leverage withEHR vendors, and the availabilityof providers to participate in thecomparison process.Lack of vendor participation impededprogress in both States. For example,North Carolina found that vendorsneeded clinical and informaticsguidance to incorporate the Formatrequirements in a way that supportsthe State’s desired improvement inchildren’s health care. However, whenImplications States should consider broadlydisseminating current and futureversions of the Format to providersthat serve children in order tostimulate discussion about and movetoward more robust, child-orientedEHRs. Because many providerscannot devote attention to 700requirements, States could considerdisseminating only the following: (1)the requirements that best align withtheir current quality improvementpriorities, (2) the Format’s prioritized“shall” requirements, or (3) thesubset of critical/core requirementsnow available through AHRQ’sUnited States Health InformationKnowledgebase (these had not beenspecified when the Format wasinitially released to North Carolinaand Pennsylvania). The Format can help EHR purchasersand frontline users prioritize theirThe National Evaluation of theCHIPRA Quality Demonstration Grant ProgramPage 6needs and develop strategies toencourage vendors to meet thoseneeds. Independent practicesand health systems that use thesame EHR product could considerapproaching EHR vendors togetherto increase their negotiating strength. The Federal government couldmotivate EHR vendors to createproducts that meet the requirementsof the Format by developing an EHRcertification module for child healththat, in turn, could create a marketfor certified child-oriented EHRs.Depending on AHRQ’s and CMS’senhancements to the Format, it maybe desirable to develop more thanone module, including one for a coreor minimum set of requirements. EHR vendors could considerdemonstrating the extent to whichtheir products already meet Formatrequirements and helping providersuse their products accordingly.Vendor-sponsored user-groupmeetings would be a suitable venuefor efficiently reaching large numbersof providers who serve children.Endnotes1. Andrew SS. We are still waiting for fullysupportive electronic health records inpediatrics. Pediatrics 2012;130(6):e1674-6.2. Andrew SS. Special requirements of electronichealth records in pediatrics. Pediatrics2007;119(3):631-7.3. The pay-for-improvement project thatPennsylvania established as part of itsCHIPRA quality demonstration wasseparate from, and not redundant with, theMedicaid Meaningful Use Incentive Programestablished by the Health InformationTechnology for Economic and Clinical Health(HITECH) Act of 2009.Evaluation Highlight No. 10, August 2014

How are CHIPRA quality demonstration States testing the Children’s Electronic Health Record Format?LEARN MOREAcknowledgmentsAdditional information about the national evaluation and the CHIPRA QualityDemonstration Grant Program is available at the tabs and information boxes on the Web page to: Find out about the 52 projects being implemented in 18 CHIPRA qualitydemonstration States. Get an overview of projects in each of the five grant categories. View reports that the national evaluation team and the State-specificevaluation teams have produced on specific evaluation topics andquestions. Learn more about the national evaluation, including the objectives,evaluation design, and methods. Sign up for email updates from the national evaluation team.The National Evaluation of theCHIPRA Quality Demonstration Grant ProgramPage 7The national evaluation of the CHIPRAQuality Demonstration Grant Program andthe Evaluation Highlights are supportedby a contract (HHSA29020090002191)from AHRQ to Mathematica PolicyResearch and its partners, the UrbanInstitute and AcademyHealth. Specialthanks are due to Cindy Brach, LindaBergofsky, and Erin Grace at AHRQ; KarenLLanos and Elizabeth Hill at CMS; StateCHIPRA quality demonstration staff; andMathematica colleagues Mynti Hossain,Dana Petersen, Joe Zickafoose, and HenryIreys. We particularly appreciate the timethat the CHIPRA quality demonstrationstaff and providers in the featured Statesspent answering our questions during sitevisits. The observations in this documentrepresent the views of the authors anddo not necessarily reflect the opinionsor perspectives of any State or Federalagency.Evaluation Highlight No. 10, August 2014

practices and EHR vendors to compare existing EHRs to the Format. The coaches also have acted as a liaison between practices and vendors in considering next steps, and they have begun training practices to use EHR functionalities that already meet Format requirements. Practices and