Harkirat Singh / Analytics
EmailPublic portfolioAll projects
Switch project

03 / Energy data quality

India renewable generation

Historical daily reports, comparable reporting levels and a traceable record of what needs investigation before use.

PythonPostgreSQLData quality

Numeric coverage does not settle source quality.

Question
Which historical generation values can be compared, and which should remain unavailable?

Fixed comparison: Selected daily reports, 30 June 2024 through 22 May 2025; state components and national comparators kept separate. Controls below do not change this summary.

Supported finding

All 34,335 main daily state-component cells are numeric, while five national daily wind comparator cells are unavailable. On 10 January 2025, the national wind cell contains the text Daily. Numeric availability does not establish accuracy.

Decision implication

Before using an affected national total, a reporting analyst should seek corrected official evidence and leave wind unavailable. Keep state-component comparisons within their own grain; a residual calculation is a diagnostic, not a replacement value.

Limitation

Historical sample and observed state roster, with unresolved anomaly causes. Main observations and reported comparators cannot be added. These checks do not certify source truth or current generation availability.

Available is not
the same as reliable.

The main population is one daily observation per report date, state label and renewable component. Reported regional and national rows remain separate comparators.

Loading the verified analytical summary…

What did the checks flag?

Findings across state, region and national levels. None establishes the cause of an anomaly.

Inspect finding counts

Inspect a quality rule

Choose a quality rule to see what it flags and what the finding does not establish.

The observed roster is not an official exhaustive list. Numeric availability does not establish source accuracy. Five unavailable national wind values sit outside the main state-component dataset.

One unavailable value.
One explicit decision.

A worked investigation from the report dated 10 January 2025.

Source observation

The numeric wind cell contains text

The national wind position contains the text Daily. The parser retains the source value and marks the numeric observation unavailable. State-component observations belong to a separate reporting population.

Analytical decision

Keep reported national wind unavailable

Diagnostic only. No replacement value enters the dataset or the quality-rule explorer.

559.76 − 355.74 − 30.24 = 173.78 MU

The residual is a diagnostic check against the reported total. It is never substituted for the unavailable reported wind generation.

Decision implication

Use comparable records and seek a correction

A reporting analyst can use the retained state-level components within their stated scope while referring the national value for source-level investigation. Neither the residual nor a parser flag identifies the cause.

A reproducible
historical pipeline

Python parses and checks the files. SQL provides an independently expressed comparison route. Versions and source checksums make the selected snapshot explicit.

  1. Select the source versionUse the recorded manifest and SHA256 checks. Reports cover 30 June 2024 through 22 May 2025, not the current generation period.
  2. Separate reporting levelsWind, solar and Other RES form the state components. Reported totals and regional/national comparators are separate. Never add these populations together.
  3. Preserve unavailable and questionable valuesRetain original text, negative values and source labels. Missing values stay unavailable, rather than becoming zero or an imputed residual.
  4. Reproduce and compareThe October portfolio replay reproduced six CSV outputs byte for byte from all 327 retained workbooks. Historical Python/PostgreSQL agreement covers 105 coverage groups, 105 generation groups and seven quality groups. Fresh PostgreSQL execution was not performed for this package.
Main population
34,335 daily state/source component observations across the observed state roster. All are numeric in this selected main table; one is negative.
Comparison population
19,293 separate comparator observations. They cannot be added to the main population to form a generation total.
Native Power BI
The current saved PBIX differs from its earlier reviewed version and needs renewed native review. It is not distributed as an accepted dashboard here.
Source-use boundary
The download contains authored code, tests and restoration metadata. Historical workbooks, processed data, screenshots and embedded models are excluded while current source redistribution conditions are unresolved.

Recreate the method

The package has two distinct routes: a deterministic synthetic smoke test and full historical reproduction with the checksum-matched source files.

Included

Code and a synthetic example included. Full historical replay needs source files.

Software

Python for the package; PostgreSQL for the database route.

Full replay

Historical CEA source files required for full replay; the synthetic example is included.

Reproducible source

Parser, daily analysis, SQL, tests, source manifest and reproduction instructions. No historical data or embedded model is bundled.

Download source ZIPZIP · 102 KB

Quick method check

Follow the package's synthetic smoke instructions to exercise missing, negative and comparator handling. Synthetic results demonstrate the mechanism only.

The extracted-package smoke and all 63 selected tests passed in the recorded check.

Full historical replay

Restore the 327 workbooks from the manifest's source locations or your retained copy, verify hashes, then run the documented full-data route.

Current downloads and a fresh dependency installation were not tested. A revised source file must not silently replace the recorded snapshot.