Originally printed in the August 2014 Edition of the Health Insurance Marketplace News, Authored by Peter B. Nichol.
In Part I of this two-part report, Nichol detailed the events of October 2012 through March 2013; this month’s installment focuses on what happened from April 1 through the official launch of Obamacare in October 2013.
Apr. 1, 2013—FirstData was selected as the Independent Verification and Validation (IV&V) vendor and almost immediately they identified areas where we needed improvement and highlighted things we were doing well. One finding of readiness uncovered that business rules were both in the rules engine product and hard coded in Java. This was fixed within the month, however had this not been identified, it undoubtedly would have resulted in additional cost for future business rule changes.
Infrastructure setup of production was absolutely on the critical path. Although the production hardware was set up, the software was not fully installed or, in some cases, not configured correctly. This quickly became the top issue. If our production environment was not ready for migration of code, a fully tested-out development environment wouldn’t save us. This turned the spotlight on defining essential infrastructure monitoring: Heartbeat (core servers, core ops, and core data), Prevention (identity management), and Detection (data loss prevention, risk mitigation).
Infrastructure configuration was critical path and so was code completion. Code completion was scheduled for May 31 and was at risk. This resulted in Connecticut communicating to CMS that any changes or requests after March 1, 2013 would simply not be in place for Oct 1. This decision, strongly supported by Kevin Counihan, CEO, unquestionably gave Connecticut a real shot of making October work. As anticipated, more requests surfaced, including multiple QHPs, which were deferred. Challenges continued with QSSI, CMS’s integrator for the FDSH, being direct when testing issues were raised. For example, initially when a state found an issue, the other states were not aware of the issues and would have to discover them independently. Months later this was in fact resolved, but resulted in a rocky first several months.
The CTDSS was legally required, starting Oct 1, to process eligibility determinations within the Integrated Eligibility System, developed by ahCT. Two programs were impacted from CTDSS, Modified Adjusted Gross Income Medicaid and the Children’s Health Insurance Program. Qualified Health Plans (QHP) were supported by ahCT involving Advance Payment of Premium Tax Credit and Cost Sharing Reductions only for QHP enrollments. There were a lot of business operational and technical decisions needed between these agencies. Therefore a Memorandum of Agreement was drafted between ahCT and CTDSS.
May. 1, 2013—Infrastructure was making progress but not the progress needed to make Oct 1. Changes and a refocus were in order. Access Health CT IT leadership quickly reestablished twice-a-week meetings to confirm critical path progress for the infrastructure build out.
Several usability testing sessions were held throughout Connecticut to observe user interaction and behavior of the system. This provided concrete examples of where systems changes were needed and where they were not. A lot of other intelligence was gained ranging from perceptions of brand to ease of use to establishing a knowledge baseline for terminology.
Meanwhile the Federal government published a new Single Streamlined application (online and paper) that defined the standard workflow and required information for enrollment. This resulted in the team reassessing the requirements, design and what had been coded to determine the degree of variance between the previous single streamlined application and the new one. There were significant changes required for Connecticut to align to the new version. These were prioritized and as a result impacted previously committed development timelines by two weeks.
Scope was officially frozen. No significant scope changes were allowed or approved unless not making that change would literally stop us from going live. This pause in continual scope changes allowed effort to shift to evaluating requirements traceability through technical components. There were gaps, but were they material or just documentation catching up to the process? Time would tell.
As testing with FDSH was underway and development was 70% complete, testing readiness was the next logical step. Everything was considered from test data, automated verse manual scripts, scrubbing of data, volume testing, timing of performance testing, and regression testing approach through tool setup. Access Health CT leveraged a Responsibility Assignment Matrix (RACI) to clarify each individual role and contribution. IT leadership stressed that focus on complete and thorough test scripts was essential to success. Connecticut had over 4,000 test scripts run on the system multiple times to ensure defects fell out of the system before users were even on the system.
Jun. 1, 2013—Plan Management was in production and successful and could compare and validate health plan benefits using the website. The first internal users are live! The scope and schedule for the Plan Management release were well known. The ITPMO and OPSPMO meetings were keeping people talking and raising issues so the teams could address them.
The release of Plan Management made it clear we needed a helpdesk solution and documentation to accompany it, fast. This initiative to track incidents was spun up quickly and within 3 weeks the helpdesk strategy and fundamental helpdesk documentation was drafted. Meanwhile, systems integrator resources for coding and testing were missing dates. Key system integrator resources left the project and critical timelines were being impacted. Yes, this is typical for longer projects, but that doesn’t help meet hard deadlines. We needed to course correct immediately. During a week three strategy discussion, Access Health CT IT leadership knew testing resources were not going to be able to support the level of testing envisioned. Access Health CT IT leadership made the decision to engaged Cognizant to augment testing resources and add a proven methodology for maturing testing processes.
After digesting the usability testing session observation data, it was evident that website behavior inconsistencies blurred the goal of a familiar user experience. Access Health CT IT leadership was very concerned with the user experience and communicated this concern to Deloitte leadership. Nagen Suriya, Deloitte Consulting LLP, engaged Deloitte Digital to focus on the user experience — specifically to create a uniform look/feel and promote consistent website behavior.
Integration with CTDSS was complicated. Existing process and procedures had to be revisited and adjusted in many cases including appeals, paper applications for programs, notifications and call center operations.
The month ended with a very good review from IV&V, “although there was a lot going on there was great IT leadership and controls in place.”
Jul. 1, 2013—Three months until open enrollment and the development was finally complete. With Plan Management development having been completed two months earlier, there were several lessons learned from Release 1. Release 1 was infinitely less complex than Release 2 supporting Oct 1. The number one lesson was that more complex test scripts were required. This meant more complete, more detailed test scripts — and simply more of them. While the scripts were being created, IT was going through a detailed planning session to create a go-live checklist. The end result was a checklist of every known step required for Oct 1 and how that step would be validated. This proved to be a valuable and insightful exercise.
Resources were at their peak across the board concentrating on integration and supporting security readiness for CMS reviews that were on-going. Integration of screen workflows involved paper application process workflows and phone application workflows for the call center. Integration of vendors was a primary concern, mainly print and notice vendors and the paper application scanning vendor. Integration with carriers providing the health plans was paramount. This integration included reconciling data formats between carriers and maintaining metrics on eligibility and enrollments and finally drafting an EDI Companion Manual to be distributed to the carriers defining the enrollment formats (834s).
All staff went through IT privacy and IT security training including multiple sessions about the Health Care Exchanges.
Wave 2 and Wave 3 testing with the FDSH services was completed and certified by the IV&V vendor. Additionally, User Acceptance Testing was on track to start on August 1, 2013.
A culture shift occurred — there was no more blame and no one was at fault. It was all “how can I lift you up and ensure your success”. The team could see the finish line and it was within reach!
Aug. 1, 2013—It was apparent that a major risk was dependency with the FDSH, aka “the Hub.” If services were unavailable — like ID Proofing (User Identification and Verification e.g. on 1st login can we validate you are who you say you are?) — the customer would not be able to start an application. Similarly, if before eligibility is determined the maximum amount of monthly Advance Payment of Premium Tax Credit (APTC) is unavailable, a customer would not be able to select a plan and therefore would be unable to complete enrollment.
To expedite testing with the Hub, an emulator was built to mirror functionality of the Hub services. This enabled Connecticut to continue testing even before the Hub was active and testing was allowed in a FDSH testing region, which was regulated.
In August, the cloud-based incident response system was deployed to track all IT and all business operational issues. This cloud-based solution provided ahCT IT executives with iPhone touch access to every issue in the environment that was impacting IT or business operations. Accompanying this deployment was the communication of a work prioritization model (defect and issue). This ensured the technical team was addressing the most impactful items first.
Challenges with changing requirements continued and IT and Business together held the line to the greatest extent possible.
After ahCT IT Leadership drafted the Memorandum of Understanding (MOU) and with light negotiation, the MOU was signed by ahCT and Connecticut Bureau of Enterprise Systems and Technology (BEST). This clarified the terms that BEST would host the data center and production infrastructure on behalf of ahCT, defined formal SLAs and support agreements.
Also, in support of a baseline and current state, ahCT IT leadership engaged FirstData (IV&V vendor) to conduct a third-party assessment of hardware and software inventory at BEST of assets purchased by ahCT. The result of this assessment was that out of $15M in spending, less than $200K was undocumented. A pretty amazing feat considering the speed in which this infrastructure was purchased and deployed.
During the last week of August, Connecticut held their Operational Readiness Review (ORR) with CMS. The ORR is a formal inspection conducted to determine if the final IT solution or automated system/application that has been developed or acquired, tested, and implemented is ready for release into the production environment for sustained operations and maintenance support. This is a very comprehensive toll gate and after this review Connecticut was informed by CMS the presentation was extremely well done and more comprehensive than states that went before Connecticut.
Sep. 1, 2013—This can only be summed up as a calculated sprint with everyone pushing to achieve the same goal. Twice-a-day stand up calls were established to ensure a steady pulse on progress was maintained and teams had every opportunity to remove roadblocks. During these calls every technical lead and functional lead reported status to ahCT IT Leadership. It’s clear we officially have a high performing team!
Late-emerging Federal guidelines attempt to impact scope. The line is held, report development is deferred, and only core reports will be available for Oct 1. The focus remains on standing up a system that is stable.
Due to Plan Management benefit templates changing from SERFF (required to load plans into the system) there is a mad scramble to resolve benefit template inconsistencies and additional rework in loading carrier plan data. Business operational staff steps up to the challenge and IT pushes hard to tackle the largest manual lifts to inject speed into the process.
Testing is running at maximum power, over 4,000 scripts are cycled and revalidated, functionality is working as expected. The 834 processing to carriers is almost finalized. To decrease the testing load another look is made towards the FDSH. Calls required are reduced from 13 to 8 Hub calls, resulting in a 40% decrease in dependency with the Hub to service customers, further decreasing testing scope. As of early September, 7 of 8 services are fully tested with FDSH and are certified as working by IV&V. A detailed walk through by ahCT IT leadership resulted in no critical defects remaining.
Despite reductions in functional testing scope, end-to-end testing with the print vendor, paper application scanning vendor and the vendor who manually keyed in scanned PDF applications remained outstanding. All pieces have been tested but not end-to-end.
Performance testing reports come in and performance benchmark tests are below target and frankly not acceptable. With 16 business days before Open Enrollment, ahCT IT leadership makes the tough decision to deploy a new technology that could help increase performance up to 200% by preventing non-US based customers and prioritizing Connecticut customers into the website over any non-Connecticut customers. During the next 10 days ahCT IT Leadership established an expedited process to procure and configure the product. Within 10 days website performance testing results had improved from a five- to seven-second average response time to less than a half-second response time.
With only one week left in September, Connecticut received their Authority to Connect (CMS Privacy requirement to connect to FDSH), IRS approval to access Federal Tax Information (FTI) and Authority to Operate (CMS Security).
Oct. 1, 2013—The expanded IT team was ready to go live on Sept 30th with a soft launch. However, the Board and executive leadership delayed that go-live until Oct. 1. The team started very early on Oct. 1 and was set to launch at 9am EST. At 9 A.M., Connecticut was up, however, there were reports of users unable to view the site and getting errors. Access Health CT IT leadership partnered with Deloitte leadership and BEST leadership and rallied hard into a conference room and brought in all the guns. Every vendor, partner and resource that was essential was on the call and the team was 100% focused in sequential triage mode, eliminating possible issues. The tone of the call was absolutely serious and all hands were on deck.
Within 5 minutes of the call Kevin Counihan, CEO, stepped in and said, “I have meeting with the Governor and Lieutenant Governor now, and when will we be up.” Peter Nichol, IT leader at ahCT, replied “Kevin we have all hands on deck, honestly we don’t know what the problem is, we have the right team on the call to solve this, and I will keep you updated.” Within seconds, another executive walked in and asked, “we have issues, when will we be up, what do you need from me.” Again, the reply was, “we have all hands on deck, don’t know the root problem, we are on this and I will text you with updates.” Understanding their concern, immediately all non-critical staff were removed from the room. Within the next 25 minutes the root cause was determined that ten IPs required for traffic routing were not provided to the hosting facility. Once the missing IPs were identified they were added in minutes and Access Health CT was live on Oct 1, 2013 at 9:30 A.M. EST.
A team member exclaimed, “WE did it! Let’s cheer on the count of three!” With relief and celebration the whole team let out an emotionally charged, “WHOOOOOOOOOO!” This yell was heard throughout the company and captured the tireless dedication of each team member!
Top Keys to Success:
1. Executive leadership empowered leaders to make informed decisions and built a culture of excellence.
2. The current state of the program was known at all times. As a leader when you are empowered with information you are able to take the correct action. This was supported by good communication by Deloitte and a strong ITPMO driving a strong vendor governance model led by KPMG
3. Exhaustive performance testing conducted early for FDSH. Seven months before go-live, Connecticut was performance testing web services. This allowed for the maximum runway to be successful. Complex and detailed test scripts were run thousands of times to ensure regression testing was comprehensive.
4. Infrastructure stabilization to ensure the user experience was consistent. Serious thought and consideration were put into contingencies and thinking through Plan A, Plan B, Plan C and then Plan D.
By week two 10,768 applications were started and 2,372 completed. Achieving business and technology excellence, Connecticut was the first state in the country to not only meet but to surpass the enrollment goal established by the Congressional Budget Office for ACA. As of March 31, 2014 we are proud to have 208,301 residents enrolled and experienced a 92% overall satisfaction rating from consumers over the enrollment process, with 70% indicating they would recommend the system to their friends or colleagues!
Access Health CT Executive Leadership went down to Pennsylvania on October 14 for three days and personally thanked each of the 130 on-site Deloitte team members that went above and beyond the call and contributed to the success. Additionally, Access Health CT hosted a Ben & Jerry’s ice cream social for the Connecticut Bureau of Enterprise Systems and Technology (BEST) and personally thanked each team member of the 89 on-site for their amazing contribution to the collective success of the Exchange.
Note: if you missed Part 1, you can find it here: