Voting System Qualification Test Report Dominion Voting Systems, Inc. Sequoia WinEDS Release , Version 1

Transcription

1 Voting System Qualification Test Report Dominion Voting Systems, Inc. Sequoia WinEDS Release , Version 1 May 2012 Florida Department of State Division of Elections R.A. Gray Building, Rm S. Bronough Street Tallahassee, FL 32399

2 Florida Department of State Page 2 of 16 Contents Introduction... 3 Background... 3 Conduct of Tests... 3 Phase I... 3 Baseline of Operating System... 4 Voting System Setup... 4 Installation/Verification of Equipment Firmware... 4 Physical Audit... 4 Functional Audit... 4 Creation/Importing of Election Definitions... 5 Phase II... 7 Election Simulations... 8 Audit Trail... 8 Security... 9 Recovery Techniques... 9 Modem capability... 9 Mass Ballot Counts... 9 Insight Precinct Scanner... 9 Edge DRE Tabulator C Central Count Scanner... 9 Mock Contest Recount Simulated Catastrophic Failure Ballot Sensitivity Sound Pressure Level Edge Audio Language Loop Force Daylight Saving Time Change Source Code Review Insight Memory Pack Engineering Change Order 781, 02/09/ Test Results and Findings of Compliance Exceptions WinEDS Release , Version

3 Florida Department of State Page 3 of 16 Introduction On November 11, 2011, Dominion Voting Systems (DVS) submitted the final documentation and source code files required for their requested certification of the WinEDS Release , Version 1 voting system. This system includes a new version of the WinEDS election management system, new firmware for Optech Insight and Insight Plus, new firmware of AVC EDGE I and II, new software for Optech 400- C, new software for upgrading uploads using Insight modem capability, and new versions for other support items. Public testing occurred from March 5 through May 22, 2012, in Tallahassee, Florida. This report is filed in summary of testing and certification activities. Background The WinEDS Release , Version 1 is a new voting system based on an existing certified system in the State of Florida. This new voting system is initiated by Palm Beach County s requirement for modeming with two or more precinct tabulators in a single precinct. That capability does not exist with the existing certified system (Release ). DVS provided this capability with their new Insight Listener software and required the new election management software (WinEDS 4.0) and Optech Insight firmware. The WinEDS Release , Version 1 system includes a new version of the WinEDS election management system, new firmware for Optech Insight and Insight Plus, new firmware of AVC EDGE I and II, new software for Optech 400-C, new software for upgrading uploads using Insight modem capability, and new versions for other support items. Thus, this voting system required a full qualification test, modem test, and mass ballot count tests. The vendor s application for Florida certification included the Sequoia WinEDS, Optech Insight and Insight Plus precinct count tabulators, AVC EDGE I and II direct recording equipment (DRE) as needed to meet Americans with Disabilities Act (ADA) specifications, and the Optech 400-C central count tabulator. This is a paper-based voting system with only one element for compliance with HAVA and ADA provisions for precinct voting: the ACV EDGE, a DRE. 1 Conduct of Tests The test objective was to verify that this voting system met the applicable requirements of Florida Statutes and Rules and Florida Voting Systems Standards (FVSS). The certification effort was conducted in two phases: Phase I consisted of establishing a baseline of the operating system environment, installation and verification of voting system applications and firmware, and creation/importing of election definitions meeting election criteria outlined below. Phase II consisted of all activities surrounding election simulation (pre-election activities, early voting and election day activities, post-election and reporting), modem testing, mass ballot counts, audit and security examination, recovery techniques, and additional tests. Phase I Phase I testing consisted of establishing a baseline of the operating system environment, installation and verification of voting system applications and firmware, and creation/importing of election definitions with previously identified parameters designed to assess the system s compliance with Florida laws, rules, and standards. 1 By 2016, ADA polling equipment must comply with Section (3), Florida Statutes that in effect requires removal of direct recording electronic (DRE) tabulator.

4 Florida Department of State Page 4 of 16 Baseline of Operating System The operating system was installed on verified clean machines. The COTS software was installed on the client and server machines. The server and client machines were then imaged to get a baseline of the operating system. Voting System Setup The election management system [WinEDS, Election Reporting Program (ERP), Ballot Printing System (BPS), WinEDS Bridge Tool (WBT), Extended Services] was installed on two client machines. The Insight Listener was installed on a separate client machine. The machines were then imaged. Installation/Verification of Equipment Firmware Firmware was upgraded, validated and verified in the following equipment: Memory Pack Reader / Writer (MPR) The MPR was upgraded with firmware version Optech Insight The Insight units were upgraded with firmware versions: o HPX L o The modem-capable units were upgraded with firmware version CPX J- L o Insight Memory Packs were upgraded with APX L AVC Edge The Edge machines were upgraded by DVS staff with firmware version Optech 400-C The 400-C was installed with WinETP software version and MS Windows XP Pro SP2. Users and workspace was configured and baseline images were created for all the election management system machines and server. Physical Audit The voting system configurations were validated by comparing the system/software identification with the application for certification descriptions and then for: System software (WinEDS, COTS) System hardware (Insight, Edge, 400-C) Hardware configuration Functional Audit A functional audit was conducted on the following: Insight validated the voting system configurations by using test scripts Edge calibrated and validated the voting system configurations by using test scripts 400-C validated the voting system configurations by using test scripts ERP Extended Services WinEDS Bridge Tool SQL Server and Election Reporting Program (ERP) In the test environment, the name of the SQL Server included a special character, a hyphen (-), which interfered with the ERP s ability to communicate with the server. The problem was solved by changing the host file that is local to each machine, which restored communication. BVSC recommends that any servers used in

5 Florida Department of State Page 5 of 16 conjunction with the WinEDS election management system follow a naming convention that excludes special characters. Creation/Importing of Election Definitions The test included four simulated elections: Presidential Preference Primary (PPP), Municipal, Primary, and General. The test plan called for using a primary and general test election based on a primary election from Indian River or Palm Beach County s 2010 database, and a general election based on Palm Beach County s 2010 database. These two elections were modified from restored election definitions, and the municipal and PPP test elections were created new. Ballots can be created using one of two methods in the WinEDS suite: ballots can be created within the WinEDS application, or they can be substantially created in the Ballot Printing System (BPS) application and ported, using the WinEDS Bridge Tool (WBT), into WinEDS for final editing. The Edge ballots are created in WinEDS, and the printed ballots can be created in either WinEDS or the BPS. This certification effort included tests utilizing both methods of ballot production. BVSC found, during the creation process, that the documentation was confusing and labyrinthine, and considerable consultation with the helpful DVS staff was necessary in order to achieve the test plan objectives. BVSC examined the version currently under test, WinEDS, Version , for the added functionality that automates the previously manual process of verifying contest order, that was in version 3.1. The four elections used in testing were defined (or modified) to meet the criteria described below. Municipal Election Election definition created Languages: English, Spanish, Haitian Creole Single sided ballot, 14" long Three or more precincts with ballots by ballot style One or more "Vote for x" contests One or more split precincts Statistical contests Precinct scanners set to: o Reject ballot for one or more overvoted contests o Reject a blank ballot o Accept a ballot that contains one or more undervoted contests Central count scanners set to: o Outstack a ballot for one or more overvoted contests o Outstack a blank ballot o Accept a ballot that contains one or more undervoted contests o Use poll header mode Tally type for early voting, absentee, provisional, and election day reported by precinct ID and Congressional district Print at least 10 ballots for each precinct Ballot definitions produced and unmarked ballots printed Minimum 4 precincts. General Election Election definition based on a general election from Palm Beach County s 2010 database. One or more two sided, three column 18 ballots for Optech Insight and 400-C Three languages (English, Spanish, and Haitian Creole) At least the maximum precincts as available in Palm Beach County Ballots by precinct ID One or more precinct only contests

6 Florida Department of State Page 6 of 16 One or more split precincts One or more "Vote for x" contests with a write-in At least one overvoted ballot will include a write-in Statistical contests Precinct scanners set to: o Reject ballot for one or more overvoted contests o Reject a blank ballot o Accept a ballot that contains one or more undervoted contests Central count scanners set to: o Outstack a ballot for one or more overvoted contests o Outstack a blank ballot o Accept a ballot that contains one or more undervoted contests o Use mixed mode Tally type (group) by early voting (EV), absentee (except for absentee on ED), absentee on ED, provisional, provisional after 7pm, overseas, and election day (ED) Summary election results for use with election night reporting Precinct-level results report as required under section (2)(a), Florida Statutes Print at least 10 ballots for each precinct Ballot definitions produced and unmarked ballots printed Setup for early voting for an election definition similar in size to Miami-Dade or Palm Beach County Minimum 20 precincts, however, EV tabulator(s) election definition must have the capability as Palm Beach precincts. Presidential Preference Primary (PPP) Election Election definition created Languages: English, Spanish, Haitian Creole Single sided ballot, 14" long Ten or more precincts with ballots by ballot style Two or more split precincts Presidential contest with the Republican and Democrat parties Three or more congressional districts At least one contest and candidate name set at the WinEDS maximum field length Statistical contests Edge configured for audio Precinct scanners set to: o Reject ballot for one or more overvoted contests o Reject a blank ballot o Accept a ballot that contains one or more undervoted contests Central count scanners set to: o Outstack a ballot for one or more overvoted contests o Outstack a blank ballot o Accept a ballot that contains one or more undervoted contests o Use either poll header mode or precinct header mode Tally types for early voting, absentee, provisional, and election day reported by precinct and Congressional District Print at least 10 ballots for each precinct Scan and tabulate single fold absentee ballots Ballot definitions produced and unmarked ballots printed Minimum 20 precincts, however, EV tabulator(s) election definition must have the capability as Palm Beach precincts.

7 Florida Department of State Page 7 of 16 Primary Election Closed Primary Election (based on an election from Indian River s or Palm Beach County s 2010 database) to include the following: o Republican, Democrat, and Nonpartisan ballots o Three languages (English, Spanish, and Haitian Creole) o At least the maximum precincts as available in Palm Beach County o One or more two sided, three column 18 ballots for Optech Insight and 400-C o One or more precinct only contests o One or more county-wide universal primary contests o One or more district-wide universal primary contests o One or more split precincts o One or more polling place with multiple precincts o Statistical contests o At least one contest and candidate name set at the WinEDS maximum field length Precinct count scanners set to: o Reject ballot for one or more overvoted contests o Reject a blank ballot o Accept a ballot that contains one or more undervoted contests Central count scanners set to: o Outstack a ballot for one or more overvoted contests o Outstack a blank ballot o Accept a ballot that contains one or more undervoted contests o Use mixed mode o Scan ballots batched by precinct with precinct header cards Early voting, absentee, provisional, and election day reported by precinct Print at least 10 ballots for each precinct Scan and tabulate tri-fold absentee ballots Ballot definitions produced and unmarked ballots printed Minimum 20 precincts, however, EV tabulator(s) election definition must have the capability as Palm Beach precincts. Additional Criteria Election officials must be able to program the AVC Edge (I or II) screen fonts from 14 points to 24 points as viewed by the voter (ref. section (f) F.S.) At least one contest and candidate name set at the WinEDS maximum field length Layout ballot and identify the WAV files needed to support the elections Setup five or more groups at least on the General election for early voting, election day, absentee to Monday, absentee on election day, provisional, provisional after 7pm, and overseas. Add/remove provisional ballot (Edge) Mark and cast paper ballots and use marked folded absentee ballots Mark ballots with approved marking devices and with the Edge touchscreen o Exercise the voting system s audio ballots and language selections Perform deck processing of absentee ballots into WinEDS Phase II Phase II consisted of all activities surrounding election simulation (pre-election activities, early voting and election day activities, post-election and reporting), audit and security examination, recovery techniques, modem testing, mass ballot counts, and additional tests.

8 Florida Department of State Page 8 of 16 Election Simulations Election simulations were comprised of pre-election activities, early voting and Election Day activities, post-election and reporting. Logic and Accuracy testing (L&A) were considered pre-election activities. Voting was simulated for four elections (municipal, PPP, primary, and general). Testing activities for L&A and elections included simulating precinct opening, casting ballots, obtaining zero reports, closing polls, and obtaining results reports. Pre-Election Pre-election activities consisted of preparing the following media for election testing: the Insight memory packs, and Edge cartridges (results, auxiliary, and voter cards). Election definitions were burned on the appropriate cards in preparation for election testing. The AVC Edge Cartridge Reader/Programmer was used to program voter access cards for early voting. (Testing of the Edge, however, was accomplished by both use of the voter card as well as the manual activation method.) Pre-audited test decks were prepared for elections including patterned machine marked and hand marked ballots with over voted, under voted, and blank ballots. L&A testing was conducted on all machines. Early Voting Early voting was simulated for all four election definitions. During early voting, each machine (Insight, Edge, 400-C) was used for ballot casting and/or tabulation. Machines were powered down during poll open state according to procedure, to simulate early voting conditions. Early voting results were tabulated, compared to test deck audits, and reported in the election management software. Election Day Election Day activities were simulated for all four election definitions. Polls were opened (with zero tapes), ballots cast, polls closed (and results tapes run), and results were reported to the central tabulator software (via modem and manual upload). Central count results were compared to precinct scanner and 400-C results, and test deck audit results. Post-Election Post-election activities included tracking all the precinct scanners and tally groups to ensure that all precincts reported in. Also, provisional reconciliation was conducted. Backup activities were conducted using the Extended Services module. Election databases were backed up to the server. Reporting Reporting activities included election night reporting and precinct-level results reporting. The WinEDS and Election Reporting Program (ERP) were used to generate the required election-night results reports and the precinct-level reports. The congressional district report was generated, for the PPP election. Machines were then imaged as part of post-testing activities. Audit Trail Audit logs were examined, as well as administrative reports. Some examples of reports that were printed: Log data The log data captured by WinEDS is a considerable improvement over that of version 3.1. Log data can now be filtered by various specifications. Users by activity Users logged in

9 Florida Department of State Page 9 of 16 Users by time/date OS event logs Voter registration Precinct listing Polling place listing Ballot style listing Security BVSC exercised the voting system s security by simulating attempts at unauthorized access to all relevant systems and sub-elements. Each component of the system is password-protected, and functions are assigned at the user/group access level. Furthermore, staff simulated using a memory card from a previous election in both the Insight and the Edge. There were no unexpected outcomes. Recovery Techniques An Insight failure / replacement on Election Day was simulated. A test of swapping the Edge cartridge during election mode was conducted. A simulation of extracting ballot images after catastrophic failure was conducted, and a review of the manual recount utilizing stored ballot images was conducted. Modem capability The modem-enabled Insights were tested with the Insight Listener modem results-reporting software. Tests included single-precinct/single-insight modeming, single-precinct/multiple-insight modeming, multiple-precinct/single-insight modeming, multiple precincts sharing ballot styles (must be reported manually, per design), catastrophic failure during modeming, and multiple Insights/simultaneous transmission. The system passed all modem-related tests. BVSC staff requested and were provided updated keypad codes to finalize modem testing which were done without incident. BVSC recommends that the vendor provide counties with the most up-to-date documentation for the Insight keypad codes, and counties train their poll workers in the use of these codes, as familiarity with the codes is necessary to operate the modem functions of the Insights. Troubleshooting instructions would be an improvement. Furthermore, due to changes in the memory pack (see section below titled Insight Memory Pack Engineering Change Order 781, 02/09/2010 ), any new installation of modem-capable Insights should be vigorously tested in pre-election activities. Mass Ballot Counts Volume testing, or mass ballot counts, was conducted on the precinct scanner (Insight), the central count scanner (400-C), and the DRE (Edge). Insight Precinct Scanner This voting device underwent a mass ballot count test, on a single unit, with the casting of 9,984 ballots (128 decks cast at 78 ballots per deck). The General Election was used for this effort. There were 504,064 vote targets. Edge DRE Tabulator This voting device underwent a mass ballot count, on a single unit, with the casting of 9,900 ballots. The voting simulator software and the Primary Election were used for this effort. 400-C Central Count Scanner A full mass ballot count was conducted on this scanner using the General election. One machine was used in Tallahassee, and two machines were used in Palm Beach County. A total of 193,440 ballots were cast,

10 Florida Department of State Page 10 of 16 with 620 vendor-produced decks cast four times, at 78 ballots per deck. The total number of vote targets was 9,766,240, with an error rate not exceeding 1 in 1,000,000 targets. The 400-C returned a Registered Voter/Turnout Check error when it reached around 65,000 ballots. This error is triggered when the number of ballots exceeds the number of registered voters, or if the number of ballots in a precinct exceeds 65,536 (2 16 ). According to DVS, the limit is a precinct limit (maximum number of ballots in a precinct). Because the ballots used in the mass ballot count were all printed in one precinct, which is highly unlikely to occur outside of test conditions, this error message was not counted as a memory register overflow. The 400-C demonstrated sensitivity to static electricity during this test. BVSC recommends that any installation of the 400-C include steps for static electricity mitigation, including but not limited to: use of the vendor-provided ground strap to ground the machine and the external ballot bin, a rubber or antistatic mat, and anti-static spray. Mock Contest Recount Recount activities were conducted on the 400-C. The General election definition was used for this activity. The 400-C demonstrated ability to recount a precinct-level race and a district-wide race on a multi-card election. Simulated Catastrophic Failure A complete failure was simulated during scanning of marked ballots to determine the system s capability to retain ballot images so that they could be extracted in the event of catastrophic failure, caused by an event such as a sudden power outage. All machines (Edge, Insight, 400-C) retained ballot information as designed after power outage. Ballot Sensitivity The test decks used to evaluate the Insight and 400-C scanners ballot sensitivity were created by marking blank ballots from the General and Primary Elections that were previously used in this initiative. The test was performed with ballots marked in such a way as to determine the capability of the scanner to read marks made by various marking devices. The table below illustrates the specific instruments utilized to mark the 48 ballots scanned for this activity. The test included a baseline scan of the ballots with selected broken arrows ( targets ) fully marked (that is, the broken arrow is fully completed ). A test deck was then run, with the same targets marked, but with a 1 millimeter line through the center of the target, rather than fully marked. Also included in the table, in the column labeled Finding, are the results obtained for each scanned ballot. The test was conducted on one Insight and one 400-C, with a separate test deck for each machine.

11 Florida Department of State Page 11 of 16 Marking Device Device Type Color Ballot Sensitivity Optech Insight Finding A=ballot accepted; UB=unvoted ballot Fully Marked Target 1 mm Horizontal Line thru [Baseline] Center of Target 1 Office Depot Pencil #2 Gray A A 2 Steadler Pencil #2B Gray A A 3 Steadler Pencil #B Gray A A 4 Steadler Pencil #HB Gray A A 5 Steadler Pencil #F Gray A A 6 Steadler Pencil #H Gray A A 7 Steadler Pencil #2H Gray A A 8 Steadler Pencil #3B Gray A A 9 Steadler Pencil #4H Gray A A 10 Ball Point K-Dent (med. pt.) Blue A A 11 Hotel Ink Pen Ball Point [Renaissance Hotel] (med. pt.) Light Blue A A 12 Ball Point Pilot G2 Ink Pen (med. pt.) Red A A 13 Ball Point Papermate (med. pt.) Green A A 14 Felt Tip EF Felt Pen (med. pt.) Black A A 15 Sharpie Highlighter Orange A UB 16 Vendor (Micron 08 Pen) Felt Pen (fine Black pt.) A A 17 Felt Tip Vendor Pen- X mark Black N/A A 18 Felt Tip Vendor Pen- mark Black N/A A 19 Felt Tip Vendor Pen- \ mark Black N/A A 20 Vendor Pen- mark on Felt Tip oval line Black N/A A 21 Steadler Pencil #4B Gray A A 22 Steadler Pencil #5B Gray A A 23 Steadler Pencil #6B Gray A A 24 Steadler Pencil #3H Gray A A

12 Florida Department of State Page 12 of 16 Marking Device Device Type Color Ballot Sensitivity 400-C Finding A=ballot accepted; UB=unvoted ballot Fully Marked Target 1 mm Horizontal Line thru [Baseline] Center of Target 1 Office Depot Pencil #2 Gray A A 2 Steadler Pencil #2B Gray A A 3 Steadler Pencil #B Gray A A 4 Steadler Pencil #HB Gray A A 5 Steadler Pencil #F Gray A A 6 Steadler Pencil #H Gray A A 7 Steadler Pencil #2H Gray A A 8 Steadler Pencil #3B Gray A A 9 Steadler Pencil #4H Gray A A 10 Ball Point K-Dent (med. pt.) Blue A A 11 Hotel Ink Pen Ball Point [Renaissance Hotel] (med. pt.) Light Blue A A 12 Ball Point Pilot G2 Ink Pen (med. pt.) Red A A 13 Ball Point Papermate (med. pt.) Green A A 14 Felt Tip EF Felt Pen (med. pt.) Black A A 15 Sharpie Highlighter Orange A A 16 Vendor (Micron 08 Pen) Felt Pen Black A A 17 Felt Tip Vendor Pen- X mark Black N/A A 18 Felt Tip Vendor Pen- mark Black N/A A 19 Felt Tip Vendor Pen- \ mark Black N/A A 20 Vendor Pen- mark on Felt Tip oval line Black N/A A 21 Steadler Pencil #4B Gray A A 22 Steadler Pencil #5B Gray A A 23 Steadler Pencil #6B Gray A A 24 Steadler Pencil #3H Gray A A NOTE: For items with N/A results, the test mark was a special mark (such as or X ) using the vendor s pen. These were compared against the baseline mark of a fully marked target using the vendor s pen.

13 Florida Department of State Page 13 of 16 This test shows that the Insight and 400-C scanners are able to detect a wide variety of marks made by several different marking devices, including ink pens, markers, highlighters, and pencils. The only tested marking device that was not accepted by the Insight was the orange Sharpie highlighter; it was, however, accepted by the 400-C. Because of the discrepancy between the two machines sensitivity to this marking device, the use of highlighters is discouraged. Sound Pressure Level This test was conducted to test conformance to section (1)(g), (h), and (i), F.S., describing the sound pressure level (SPL) standards for a voting system s audio voting features. The Edge, which incorporates the audio voting accessory (hand-held input device) and headphones, was found to conform to the standard. Voting was conducted using the touch screen and the audio voting accessory using vendor-supplied standard headphones. The test results are listed below. Sound Pressure Level Results Maximum Volume (db) Default Volume (db) Intermediate level above 12 db Right Left Edge Audio Language Loop At the request of Palm Beach County, the Edge was tested for a possible software bug in which selecting help during audio voting triggers a loop of language selection prompts. The Edge was activated for voting, and then the Help button (on the audio voting accessory) was selected at various points in the voting menus. No language prompt loop was observed, and voting was conducted as usual. During the Sound Pressure Level test (see above), however, the Edge screen presentations prior to the ballot screen were confusing. Separate testing was conducted to verify these observations, utilizing both the touch screen and the audio voting accessory. When the user selected a language at the language selection screen, the audio script proceeded to the next audio instruction, while presenting a redundant language selection screen. The audio instruction acknowledged the user s language selection, but the presented screen did not appear to acknowledge the user s selection. Upon completion of the audio instruction and without user input, the program proceeded to the ballot screen. If the user selected a language at the redundant language selection screen, the screen immediately advanced to the ballot (without waiting for the recording to end), but did not change the language selection (that was accomplished at the initial language selection screen). Although a loop was not observed, this duplicate language selection presentation can be a possible source of voter confusion, leading voters to infer that they must select a language a second time. BVSC recommends that DVS devise a more user-friendly screen that visually verifies the user s language selection while the audio instruction is playing. Poll workers, meanwhile, should be advised that, while language selection a second time is unnecessary, it will immediately advance the screen to the ballot. Force This test was conducted to test conformance to section (1)(l), F.S., in which the standard is no more than 5 pounds of force required to operate a voting system s controls. The force test was conducted on both the audio voting accessory and the Edge touch screen. A force gauge was used to measure the force necessary to activate the buttons on the audio voting accessory, and various regions of the Edge screen. Both the accessory and the Edge were found to conform to the standard. Daylight Saving Time Change This test was conducted to verify the precinct scanners ability to automatically display the correct time during a Daylight Saving time change. Neither the Edge nor the Insight was designed with this capability.

14 Florida Department of State Page 14 of 16 Once the time is set as part of the election definition on the Edge and Insight cartridges, it cannot be changed during the course of the election. Although time change capability is not an FVSS requirement, and the absence of this capability does not affect the other functionality of the scanners, counties using this voting system should make note when they deploy this system for an election that spans a time change event, as it will affect the date/time stamps on the zero tapes and results tapes. Source Code Review The core source code was reviewed by the voting system testing laboratory ibeta Quality Assurance (see report, Dominion Voting Systems Sequoia WinEDS 4.0 VSTL Certification Test Report Version 2.0 (V) Nov-001(B), November 30, 2010). The recommendation of ibeta was to certify the WinEDS voting system. Insight Memory Pack Engineering Change Order 781, 02/09/2010 DVS issued an Engineering Change Order (ECO) in February, 2010 regarding a problem with the RAM addressing inside the Insight memory pack. This ECO describes a change to the memory pack s circuit board configuration, in which a jumper is to be soldered on a chip on the circuit board. The memory packs currently in use in the Florida counties do not have the ECO configuration. The ECO was not submitted as part of this certification s technical data package. The memory packs used during this certification testing effort were of two configurations: the memory packs with the older configuration, supplied by BVSC, did not have the solder jumper and the current configuration. Unbeknownst to BVSC, the memory packs supplied by DVS 2 had the new configuration with the soldered jumper for use with the voting system tests. Both types were upgraded with the firmware as specified in the vendor s certification application. During the certification testing, BVSC did not observe any behaviors in the memory packs that might indicate differences in memory pack configurations, or any unit-specific problems. However, BVSC did not test the memory packs for this particular variable, as we were not made aware of the ECO until after testing was complete. According to documentation recently submitted by DVS, the current memory addressing does not cause problems in most cases, but some memory packs without the ECO change might exhibit problems, most likely to happen in larger election definitions. DVS has stated that the size of, and successful past history with the Insights in Indian River show that they do not need this update. The ECO has been in effect since February, Since then, no election issues have become known to BVSC surrounding the memory packs. BVSC believes the risk of implementing this ECO in the memory packs used by Florida counties is minimal. Therefore, it is with reservation that BVSC allows this ECO in the certification for WinEDS , version 1. Furthermore, BVSC recommends that counties implementing WinEDS , version 1 should vigorously test memory packs during pre-election activities, especially in election definitions that might be as large as, or larger than, definitions successfully run in the past. BVSC requires that all memory packs that have the ECO 781 change must have an external identification on the memory pack. Test Results and Findings of Compliance With the exceptions listed below, test results yielded no unexpected outcomes. However, there were items identified during testing that have prompted BVSC to make recommendations to the vendor and/or the Florida counties. None of the items below would preclude the certification of this voting system; in fact, several of the exceptions have to do with the documentation for the voting system, not the system itself. 2 Because some of the vendor-supplied memory packs went back to the vendor at the conclusion of testing and prior to this issue coming to light, it is unknown if all, or only some, of the vendor-supplied memory packs contained the modification.

15 Florida Department of State Page 15 of 16 Exceptions 1. SQL Server and ERP communication error. Due to the naming convention used for the SQL Server name, the ERP could not communicate with it. This was resolved by changing the host files. BVSC recommends that any servers used in conjunction with the WinEDS election management system follow a naming convention that excludes special characters. 2. During the coding of elections, BVSC found the documentation to be confusing and labyrinthine. 3. The modem functions of the Insight are dependent upon staff having the most updated code documentation. The vendor should provide counties with the most up-to-date documentation, with troubleshooting instructions, for the Insight keypad codes, and counties train their poll workers in the use of these codes. Furthermore, due to changes in the memory pack (see section titled Insight Memory Pack Engineering Change Order 781, 02/09/2010 ), any new installation of modem-capable Insights should be vigorously tested in pre-election activities. 4. The Sensitivity test revealed a variance in sensitivity with regards to highlighters as marking devices. BVSC discourages the use of highlighters as marking devices with this voting system. 5. The 400-C demonstrated sensitivity to static electricity during the Mass Ballot Count. BVSC recommends that any installation of the 400-C include steps for static electricity mitigation, including but not limited to: use of the vendor-provided ground strap to ground the machine and the ballot bin, a rubber or anti-static mat, and anti-static spray. 6. Memory packs with the new configuration described by ECO 781 should be vigorously tested during pre-election activities, especially in election definitions that might be as large as, or larger than, definitions successfully run in the past. BVSC requires that all memory packs that have the ECO 781 change must have an external identification on the memory pack. A copy of the firmware and software provided for this certification effort has been duplicated and has been provided to Dominion Voting Systems so that they can make certified copies [referred to by the Division as the Bronze Disks ]. As usual, DVS will create copies of the Bronze Disks and return those to the Division. The Division will verify that the contents of the media matches the firmware and software which were certified and will then forward same to a county as needed. The voting system under test is listed below. WinEDS Release , Version 1 Election Management System: WinEDS application, version o Extended Services o Election Reporting, version Optional software o Ballot Printing System II (BPS) version 3.41 N.05 o WinEDS Bridge Tool (WBT), version o Insight Listener version AVC EDGE Cartridge Reader I Programmer (Part No ) o with COTS device driver as indicated on product label o Optional Sycard PCCextend 70A (replacement/or the EDGE Cartridge Reader / Programmer) Memory Pack Reader (MPR) o with firmware version COTS software o SQL Server 2005 SP2

16 Florida Department of State Page 16 of 16 o MS Office 2003 Standard SP2 o MS Visio 2003 Standard SP2 o MS Visio 2002 Standard SP2 (Only for BPS) o.net framework 2.0 o Windows SteadyState 2.5 o Optional Adobe Acrobat Standard version 8 o MS Windows XP Pro SP2 (for workstations) o MS Windows Server 2003 R2 Standard (for servers) COTS hardware o Server and workstation(s) specifications per Sequoia WinEDS System Operations Prodecures Election Media o Sequoia Edge ATA/PCMCIA Flash Memory Cartridges for use with EDGE I and II 128MB, 256MB, 512MB,or 1,024MB o Memory pack with ECO 781 for Insight and Insight Plus APX L Precinct Count: Optech Insight and/or Insight Plus o with firmware version: HPX L o Optional MODEM with firmware version CPX v J-L o Insight Plastic Ballot Box Sequoia Insight Plastic Ballot Box or Election Works, Inc. folding ballot box ADA polling equipment 3 : o AVC EDGE I and/or EDGE II with firmware version o Auxiliary equipment/or EDGE I or II: Optional AVC EDGE Card Activator, version D with firmware version AVC EDGE Audio Component, Revision D with firmware version Optional AVC EDGE Auxiliary Power Unit Sip and Puff Interface Optional Seiko DPU-414 printer Central / Absentee Count: Optech 400-C Ballot Counter, Model 3.00P o with WinETP software version o COTS PC computer System Unit per Sequoia Voting Systems Optech 400-C System Overview with MS Windows XP Pro SP2 3 Note: By 2016, ADA polling equipment must comply with Section (3), Florida Statutes. Therefore, AVC EDGE I and EDGE II will not be acceptable for any election in 2016 and thereafter.