Improving data quality: Early lessons from the Senegal power compact

In this blog, Dr Marème Ndoye (MCA Senegal II Director of Monitoring, Evaluation, and Economic Analysis) shares how the implementation of a data validation committee has helped the understanding and communication of results.

The $550 million Senegal Power Compact (2021-2026), implemented by the Millennium Challenge Account Senegal II (MCA-Senegal II), is an important partnership between the governments of Senegal and the United States to improve electricity service for the people of Senegal. As monitoring and evaluation staff, we must be able to reliably and accurately tell these governments whether the program is improving access to quality and affordable electricity in Senegal. This is easier said than done.

While all project stakeholders approved the Monitoring and Evaluation Plan, early data collection and an external data quality review uncovered important data quality issues. The data we first received did not align with our indicator definitions. Electricity access data were based on the share of localities with at least one connection without considering connection rates within individual localities. Further, we received different values from different sources for the same indicator. The sector regulator and the Ministry of Finance had different values for the amount of government arrears owed to the national electricity company, Senelec. We also heard diverging views across and within organizations on the meaning of results and indicators. These issues hindered our team’s ability to obtain reliable data for program results.

To overcome these challenges, we established a data validation committee (ComVAD) in May 2022 in close collaboration with electricity sector actors in Senegal and with the MCA project teams. The committee comprises individuals collecting or processing performance data and stakeholders interested in project implementation status and lessons learned. The dozen or so organizations on the committee include implementing entities, project design consultants, supervisory engineers, and MCA staff. The MCA team discussed this committee during the gLocal Evaluation Week as part of the session ‘From projects to systems: How Millennium Challenge Accounts (MCAs) are linking M&E practices with government priorities’.

In May 2022, MCA   Senegal II organized the first ComVAD meeting. Project staff and stakeholders discuss important data availability and quality issues.

Our collaborative approach to the committee meetings has garnered enthusiasm, engagement, and active participation during and after meetings. The committee consults and achieves consensus with its members to facilitate timely and complete data submissions from reporting entities. It discusses challenges, recommends mitigation measures, shares lessons learned, and encourages the use of data to inform sector decision-making. The ComVAD continues to meet quarterly.

ComVAD members review the data submission dashboard at the June 2023 meeting.

The committee’s work has resulted in tangible improvements in our performance monitoring. Figures 1 and 2 show how the ComVAD improved data submission and approval rates. When MCC has both elements – sufficient data and confidence in the data – it can report results sooner and, if needed, clarify the need for mid-course corrections.

The data submission rate improved considerably following the first ComVAD meetings (Figure 1). More data means a greater ability to establish pre-project trends needed to interpret the compact’s effects. Increased data submissions did not immediately increase MCC’s approval of the data, but eventually it did (Figure 2). The initial approval rate in Q1 2022 was high because the indicators were for reform milestones achieved before the compact’s entry into force. It decreased in Q2 and Q3 due to misalignment between our indicator definitions and the data being provided and insufficient supporting documentation from reporting entities. In Q4 2022 and Q1 2023, we began to see improvements from the ComVAD and engagement with stakeholders.

Data submission rate of reporting entities (left), and MCC’s approval rate of submitted data (right). Data submission increased with the data validation committee, as did MCC’s approval rate of that submitted data.

What is driving these positive trends?

We have identified three main reasons ComVAD has proved to be such a successful model.

1. We received data we would not have otherwise received. Having multiple organizations in the same room has allowed us to clarify the appropriate data source. It also creates friendly competition and peer pressure that leads to commitment and action. Take the example of the indicator tracking the amount of sovereign guarantees used for public-private partnerships. One government ministry is responsible for having and providing data for this indicator. However, the ComVAD meeting revealed that it had never received that information from another ministry. Surfacing this issue made the first government ministry take immediate action to obtain the data and provide it to our team.
2. We improved the completeness of our indicators to better measure complex results. One key project outcome is the financial viability of the national electricity utility Senelec. While we had been only measuring this result by the timeliness and completeness of government payments to Senelec, the sector regulator (CRSE) revealed the need to also include government payments to electricity companies operating in rural areas to paint a more complete picture of financial viability of the country’s electricity companies.
3. We improved data reliability through complete supporting documentation. This documentation allows us to understand how values are being calculated and ensure they accurately reflect what we are trying to measure. This is just as important for complex financial indicators, where we are pulling from different line items in certified financial accounts, as it is for more straightforward indicators, like the number of people trained, where we are verifying that signed attendance sheets correspond to the reported values.
4. We ensured data was being calculated as described in the M&E plan. We want to track the kWh price of electricity in power purchase agreements (PPA) to see whether increased competition from reforms leads to lower prices. However, PPA prices are defined differently for thermal and renewable power plants. The former is variable based on operating expenses, and the latter is fixed. Our solution was to use the actual price paid by Senelec as an electricity purchaser regardless of whether the PPA had a fixed or variable rate. Another example is distribution system losses, which combine commercial and technical losses. However, Senelec, like many electric utilities, is unable to distinguish between the two at lower voltage levels. ComVAD discussions helped us understand how Senelec’s distribution department calculates distribution system losses and ensure its consistency with the definition in MCC’s common indicator guidance.

By creating a space for stakeholders to discuss and act upon data quality issues, discussions between the utility and regulator revealed that Senelec has the data to calculate long-run marginal cost but did not share it with the regulator, who requires it to accurately determine the cost-reflectivity of electricity tariffs. This space has allowed sector actors to better appreciate how data can inform decision-making and the importance of supporting documentation to ensure data is reliable.

We see early signs that strengthened performance reporting for the compact is improving data processes within the committee member organizations. We are now better positioned to understand and communicate whether and how this Compact had an impact on access to quality and affordable electricity in Senegal.