feat: Add raw_value to sensor attributes (Issue #40) (#41)

Author: enoch85

README.md

@@ -162,10 +162,23 @@ The first selected source becomes your highest priority, and the integration wil
 - **Rate limiting** - Minimum 15-minute intervals 
 - **Automatic retries** - Exponential backoff for failed requests (2s → 6s → 18s)
 - **Data caching** - Persistent storage with TTL
-- **Source fallback** - Try all sources in priority order
+- **Intelligent interval validation** - DST-aware validation ensures complete data:
+  - **Normal days**: Expects 96 intervals (15-min × 96 = 24 hours)
+  - **DST spring forward**: Expects 92 intervals (23 hours)
+  - **DST fall back**: Expects 100 intervals (25 hours)
+  - **Strict validation**: Allows only 1 missing interval (15 minutes) tolerance
+  - **Automatic fallback**: Switches to alternative sources when data is incomplete
+- **Source fallback** - Try all sources in priority order until complete data is found
 - **Daily health check** - All configured sources validated once per day during special windows
 - **Source health monitoring** - Track which sources are working vs failed, with retry schedules
 
+**Example:** If ENTSO-E returns 94/96 intervals (missing 30 minutes), the system automatically:
+1. Detects incomplete data (94 < 95 minimum required)
+2. Logs warning about missing intervals
+3. Tries next configured source (e.g., Energy Charts)
+4. Uses complete data from working source
+5. Caches complete result for future requests
+
 ## Architecture
 
 **Data Flow:** API Client → Parser → Timezone Conversion → Currency Conversion → Cache → Sensors
@@ -178,7 +191,12 @@ The first selected source becomes your highest priority, and the integration wil
 ### Timezone & Interval Handling
 
 - **Source timezone detection** - Each API has known timezone behavior
-- **DST transitions** - Handles 92-100 intervals on transition days  
+- **DST transitions** - Handles 92-100 intervals on transition days automatically
+- **Interval validation** - Ensures data completeness before acceptance:
+  - Validates exact interval count matches expected (92/96/100 depending on DST)
+  - Tolerates 1 missing interval (15 minutes) for API timing edge cases
+  - Rejects incomplete data (2+ missing intervals = 30+ minutes)
+  - Automatically tries alternative sources when primary source is incomplete
 - **15-minute alignment** - All data normalized to :00, :15, :30, :45 boundaries
 - **Home Assistant integration** - Displays in your configured timezone
 
@@ -328,41 +346,62 @@ The price sensors expose interval prices through attributes in a standardized fo
 ```json
 {
   "today_interval_prices": [
-    {"time": "2025-10-14T00:00:00+02:00", "value": 0.0856},
-    {"time": "2025-10-14T00:15:00+02:00", "value": 0.0842},
+    {"time": "2025-10-14T00:00:00+02:00", "value": 0.0856, "raw_value": 0.0754},
+    {"time": "2025-10-14T00:15:00+02:00", "value": 0.0842, "raw_value": 0.0740},
     ...
   ],
   "tomorrow_interval_prices": [
-    {"time": "2025-10-15T00:00:00+02:00", "value": 0.0891},
+    {"time": "2025-10-15T00:00:00+02:00", "value": 0.0891, "raw_value": 0.0789},
     ...
   ]
 }
 ```
 
 **Key Points:**
-- Each price entry contains a `time` (ISO 8601 datetime string) and `value` (float)
-- Times are in your Home Assistant's configured timezone
+- Each price entry contains:
+  - `time`: ISO 8601 datetime string in your Home Assistant timezone
+  - `value`: Final consumer price (with VAT, tariffs, and energy taxes applied)
+  - `raw_value`: Market spot price (currency and unit converted only, no VAT/fees) _(New in v1.6.0)_
 - List contains 96 entries for a normal day (15-minute intervals)
 - During DST transitions: 92 entries (spring) or 100 entries (fall)
 - Compatible with EV Smart Charging, ApexCharts, and custom automations
 
+**Price Calculation:**
+```
+value = ((raw_value + additional_tariff + energy_tax) × (1 + VAT%)) × display_unit_multiplier
+```
+
+When no VAT, tariffs, or taxes are configured, `raw_value` equals `value`.
+
 **Using in Templates:**
 ```yaml
-# Get price at 14:00
+# Get final consumer price at 14:00
 {{ state_attr('sensor.gespot_current_price_se3', 'today_interval_prices') 
    | selectattr('time', 'search', 'T14:00') 
    | map(attribute='value') 
    | first }}
 
+# Get raw market price at 14:00 (without VAT/fees)
+{{ state_attr('sensor.gespot_current_price_se3', 'today_interval_prices') 
+   | selectattr('time', 'search', 'T14:00') 
+   | map(attribute='raw_value') 
+   | first }}
+
 # Get all prices above 0.10
 {{ state_attr('sensor.gespot_current_price_se3', 'today_interval_prices') 
    | map(attribute='value') 
    | select('>', 0.10) 
    | list }}
 
-# Count hours with negative prices
+# Compare market prices to final prices
+{% set prices = state_attr('sensor.gespot_current_price_se3', 'today_interval_prices') %}
+Market avg: {{ prices | map(attribute='raw_value') | average | round(4) }}
+Final avg: {{ prices | map(attribute='value') | average | round(4) }}
+Difference: {{ ((prices | map(attribute='value') | average) - (prices | map(attribute='raw_value') | average)) | round(4) }}
+
+# Count hours with negative prices (on market)
 {{ state_attr('sensor.gespot_current_price_se3', 'today_interval_prices') 
-   | map(attribute='value') 
+   | map(attribute='raw_value') 
    | select('<', 0) 
    | list 
    | length }}
@@ -404,7 +443,12 @@ Then set this sensor as your energy cost sensor in the Energy Dashboard settings
 - **No data** - Check area is supported by selected source
 - **API key errors** - Verify ENTSO-E API key if using that source  
 - **Missing tomorrow prices** - Available after 13:00 CET daily
-- **96 data points** - Correct! 15-minute intervals = 96 per day
+- **96 data points** - Correct! 15-minute intervals = 96 per day (92 on DST spring, 100 on DST fall)
+- **Incomplete data warnings** - If you see warnings about incomplete intervals:
+  - System automatically tries alternative sources
+  - Check `active_source` in sensor attributes to see which source is being used
+  - Configure multiple sources for better reliability
+  - Example: `[NL] Incomplete today data from entsoe: 94/96 intervals (missing 2)` → System switches to Energy Charts
 
 **Source Health Monitoring:**
 

custom_components/ge_spot/coordinator/data_processor.py

@@ -206,6 +206,10 @@ async def process(self, data: Dict[str, Any]) -> Dict[str, Any]:
                 final_today_prices = cached_today
                 final_tomorrow_prices = cached_tomorrow
 
+                # Extract raw prices from cache (added for Issue #40)
+                raw_today_prices = data.get("today_raw_prices", {})
+                raw_tomorrow_prices = data.get("tomorrow_raw_prices", {})
+
                 # Preserve exchange rate info from cache
                 ecb_rate = data.get("ecb_rate")
                 ecb_updated = data.get("ecb_updated")
@@ -417,6 +421,7 @@ async def process(self, data: Dict[str, Any]) -> Dict[str, Any]:
             ecb_rate = None
             ecb_updated = None
             final_today_prices = {}
+            raw_today_prices = {}  # Store raw prices (without VAT/taxes/tariffs)
             # Get source unit from the input data, default to MWh if not present
             # For cached data, this might be inside the 'data' dict, or from the original fetch context
             source_unit = data.get("source_unit", EnergyUnit.MWH)
@@ -425,7 +430,7 @@ async def process(self, data: Dict[str, Any]) -> Dict[str, Any]:
             )
 
             if normalized_today:
-                converted_today, rate, rate_ts = (
+                converted_today, raw_today, rate, rate_ts = (
                     await self._currency_converter.convert_interval_prices(
                         interval_prices=normalized_today,
                         source_currency=input_source_currency,  # Use determined input_source_currency
@@ -434,12 +439,14 @@ async def process(self, data: Dict[str, Any]) -> Dict[str, Any]:
                     )
                 )
                 final_today_prices = converted_today
+                raw_today_prices = raw_today
                 if rate is not None:
                     ecb_rate = rate
                     ecb_updated = rate_ts
             final_tomorrow_prices = {}
+            raw_tomorrow_prices = {}  # Store raw prices (without VAT/taxes/tariffs)
             if normalized_tomorrow:
-                converted_tomorrow, rate, rate_ts = (
+                converted_tomorrow, raw_tomorrow, rate, rate_ts = (
                     await self._currency_converter.convert_interval_prices(
                         interval_prices=normalized_tomorrow,
                         source_currency=input_source_currency,  # Use determined input_source_currency
@@ -448,6 +455,7 @@ async def process(self, data: Dict[str, Any]) -> Dict[str, Any]:
                     )
                 )
                 final_tomorrow_prices = converted_tomorrow
+                raw_tomorrow_prices = raw_tomorrow
                 if ecb_rate is None and rate is not None:
                     ecb_rate = rate
                     ecb_updated = rate_ts
@@ -469,6 +477,8 @@ async def process(self, data: Dict[str, Any]) -> Dict[str, Any]:
             ),
             "today_interval_prices": final_today_prices,
             "tomorrow_interval_prices": final_tomorrow_prices,
+            "today_raw_prices": raw_today_prices,  # Raw prices without VAT/taxes/tariffs
+            "tomorrow_raw_prices": raw_tomorrow_prices,  # Raw prices without VAT/taxes/tariffs
             "raw_interval_prices_original": input_interval_raw,  # Store the raw prices that went INTO normalization
             "current_price": None,
             "next_interval_price": None,

custom_components/ge_spot/coordinator/unified_price_manager.py

@@ -847,17 +847,138 @@ async def fetch_data(self, force: bool = False) -> Dict[str, Any]:
                 # Process the raw result (this is where parsing happens)
                 processed_data = await self._process_result(result)
 
-                # Check data completeness
-                has_today = processed_data and bool(
-                    processed_data.get("today_interval_prices")
+                # Check data completeness with interval count validation
+                from ..const.time import TimeInterval, DSTTransitionType
+                from ..timezone.dst_handler import DSTHandler
+
+                expected_intervals = (
+                    TimeInterval.get_intervals_per_day()
+                )  # 96 for non-DST days
+
+                today_prices = (
+                    processed_data.get("today_interval_prices", {})
+                    if processed_data
+                    else {}
                 )
-                has_tomorrow = processed_data and bool(
-                    processed_data.get("tomorrow_interval_prices")
+                tomorrow_prices = (
+                    processed_data.get("tomorrow_interval_prices", {})
+                    if processed_data
+                    else {}
+                )
+
+                today_count = len(today_prices)
+                tomorrow_count = len(tomorrow_prices)
+
+                # Check if we have data (non-empty dictionaries)
+                has_today = bool(today_prices)
+                has_tomorrow = bool(tomorrow_prices)
+
+                # Check for DST transitions to adjust expected interval counts
+                dst_handler = DSTHandler(self._tz_service.target_timezone)
+
+                # Check today
+                is_today_dst, today_dst_type = dst_handler.is_dst_transition_day(now)
+                if is_today_dst:
+                    if today_dst_type == DSTTransitionType.SPRING_FORWARD:
+                        expected_today = (
+                            TimeInterval.get_intervals_per_day_dst_spring()
+                        )  # 92
+                    else:  # FALL_BACK
+                        expected_today = (
+                            TimeInterval.get_intervals_per_day_dst_fall()
+                        )  # 100
+                else:
+                    expected_today = expected_intervals  # 96
+
+                # Check tomorrow
+                tomorrow = now + timedelta(days=1)
+                is_tomorrow_dst, tomorrow_dst_type = dst_handler.is_dst_transition_day(
+                    tomorrow
+                )
+                if is_tomorrow_dst:
+                    if tomorrow_dst_type == DSTTransitionType.SPRING_FORWARD:
+                        expected_tomorrow = (
+                            TimeInterval.get_intervals_per_day_dst_spring()
+                        )  # 92
+                    else:  # FALL_BACK
+                        expected_tomorrow = (
+                            TimeInterval.get_intervals_per_day_dst_fall()
+                        )  # 100
+                else:
+                    expected_tomorrow = expected_intervals  # 96
+
+                # Check if the data is complete (correct number of intervals)
+                # Allow only 1 missing interval for edge cases (API timing, rounding)
+                # Stricter validation to catch incomplete data like 94/96
+                min_acceptable_today = max(
+                    expected_today - 1, 90
+                )  # Never accept less than 90
+                min_acceptable_tomorrow = max(expected_tomorrow - 1, 90)
+
+                today_complete = (
+                    today_count >= min_acceptable_today if has_today else False
+                )
+                tomorrow_complete = (
+                    tomorrow_count >= min_acceptable_tomorrow if has_tomorrow else False
+                )
+
+                # Log interval counts for debugging
+                if has_today or has_tomorrow:
+                    dst_info = []
+                    if is_today_dst:
+                        dst_info.append(f"today={today_dst_type}")
+                    if is_tomorrow_dst:
+                        dst_info.append(f"tomorrow={tomorrow_dst_type}")
+                    dst_str = f" (DST: {', '.join(dst_info)})" if dst_info else ""
+
+                    _LOGGER.debug(
+                        f"[{self.area}] Interval counts: today={today_count}/{expected_today}, "
+                        f"tomorrow={tomorrow_count}/{expected_tomorrow}, "
+                        f"min_acceptable=(today>={min_acceptable_today}, tomorrow>={min_acceptable_tomorrow}){dst_str}"
+                    )
+
+                # Determine if tomorrow data is expected yet
+                # Tomorrow data is typically published around 13:00 CET
+                # Before that time, we should accept today-only data as complete
+                current_hour = now.hour
+                tomorrow_window_start = Network.Defaults.SPECIAL_HOUR_WINDOWS[1][
+                    0
+                ]  # 13:00
+                tomorrow_expected = (
+                    current_hour >= tomorrow_window_start
+                )  # After 13:00, we expect tomorrow data
+
+                # Data is complete if:
+                # - We have both today AND tomorrow (always acceptable)
+                # - OR we have today complete and tomorrow isn't expected yet (before 13:00)
+                has_complete_data = today_complete and (
+                    tomorrow_complete or not tomorrow_expected
                 )
-                has_complete_data = has_today and has_tomorrow
                 has_partial_data = (has_today or has_tomorrow) and not has_complete_data
                 has_any_data = has_today or has_tomorrow
 
+                # Warn if we have data but it's incomplete (missing intervals)
+                if has_today and not today_complete:
+                    dst_note = (
+                        f" (DST {today_dst_type} expects {expected_today})"
+                        if is_today_dst
+                        else ""
+                    )
+                    _LOGGER.warning(
+                        f"[{self.area}] Incomplete today data from {result.get('data_source', 'unknown')}: "
+                        f"{today_count}/{expected_today} intervals (missing {expected_today - today_count}){dst_note}"
+                    )
+                if has_tomorrow and not tomorrow_complete:
+                    dst_note = (
+                        f" (DST {tomorrow_dst_type} expects {expected_tomorrow})"
+                        if is_tomorrow_dst
+                        else ""
+                    )
+                    _LOGGER.warning(
+                        f"[{self.area}] Incomplete tomorrow data from {result.get('data_source', 'unknown')}: "
+                        f"{tomorrow_count}/{expected_tomorrow} intervals (missing {expected_tomorrow - tomorrow_count}){dst_note}"
+                    )
+
                 # Get remaining sources for potential retry
                 attempted_so_far = (
                     result.get(

custom_components/ge_spot/manifest.json

@@ -15,5 +15,5 @@
         "PyJWT>=2.6.0",
         "aiofiles"
     ],
-    "version": "v1.6.0"
+    "version": "v1.6.1-beta4"
 }