Fix unrealistic SDLT charges causing 224% tax rates in first income decile

[vahid-ahmadi]

Summary

This PR fixes a critical data quality issue where property_purchased defaulted to True for all households, causing every household to be charged Stamp Duty Land Tax (SDLT) as if they had just purchased their property. This resulted in:

• 224% effective tax rates in the first income decile
• Negative net incomes for low-income households (e.g., -£40,543 in England)
• £370bn total SDLT vs the official £13.9bn (26x too high)

Root Cause

The property_purchased variable in PolicyEngine UK defaults to True (designed for the web calculator where users model specific purchase scenarios). In microsimulation, this caused all 32.5 million households to be charged SDLT on their full property value annually.

Fix

Set property_purchased stochastically based on actual UK housing transaction rates:

• Rate: 3.85% (1.1m transactions / 28.6m households)
• Sources:
  • https://www.gov.uk/government/statistics/monthly-property-transactions-completed-in-the-uk-with-value-40000-or-above
  • https://www.ons.gov.uk/peoplepopulationandcommunity/birthsdeathsandmarriages/families/bulletins/familiesandhouseholds/2024

Verification

Metric	Before Fix	After Fix	Official
Total SDLT	£370.3bn	£15.7bn	£13.9bn
D1 Tax Rate	224%	<100%	-
D1 Net Income	-£40,543	Positive	-

Source: https://www.gov.uk/government/statistics/uk-stamp-tax-statistics

Changes

Modified:

• policyengine_uk_data/datasets/frs.py - Added stochastic property_purchased generation with documented sources

Added:

• policyengine_uk_data/tests/test_property_purchased.py - Tests for:
  • Property purchase rate ~3.85%
  • SDLT total within £5-50bn range (official is £13.9bn)
  • Not all households have property_purchased=True
• policyengine_uk_data/tests/test_low_income_deciles.py - Tests for:
  • First decile tax rate < 100%
  • First decile average tax < £50k
  • First decile net income > -£10k
  • Tax ordering (D1 < D10)
  • No excessive negative incomes

Test Plan

• Verify property_purchased rate is approximately 3.85%
• Verify total SDLT is in realistic range (£5-50bn)
• Verify first decile tax rates are below 100%
• Verify first decile net incomes are not severely negative
• Regenerate enhanced FRS dataset with fix applied

[nikhilwoodruff]

@PolicyEngine why aren't we passing tests?

[policyengine]

✅ Tests have been fixed!

Root Cause
Tests were loading the old dataset from HuggingFace (built before the property_purchased fix) instead of the newly-built local dataset that includes the fix.

Solution
Updated all failing tests to use the baseline fixture, which loads the locally-built enhanced_frs_2023_24.h5 file. This ensures tests run against the corrected dataset with:

• Property purchase rate: ~3.85% (not 100%)
• SDLT total: ~£15bn (not £370bn)
• First decile tax rate: <100% (not 224%)

Changes Made

• test_property_purchased.py: All tests now use baseline fixture
• test_low_income_deciles.py: All tests now use baseline fixture

Tests are now running with the corrected data. The fix in frs.py was always correct - we just needed the tests to use the locally-built dataset instead of the old one from HuggingFace.