This line needs to be removed as it causes soiling intervals to be incorrectly determined when precip events are also considered.

[matt14muller]

rdtools/rdtools/soiling.py

Line 162 in 5b18cd7

df['clean_event'] = (df.clean_event) & (~df.clean_event.shift(-1).fillna(False))

[kandersolar]

I think this reproduces the problem:

import pandas as pd
from rdtools import soiling
import matplotlib.pyplot as plt

N = 30
idx = pd.date_range('2019-01-01', freq='d', periods=N)
data = pd.DataFrame({
    'energy_normalized_daily': 1.0,
    'insolation_daily': 1.0,
    'precipitation_daily': 0.0,
}, index=idx)
data.loc[idx[5], 'precipitation_daily'] = 0.5
data.loc[idx[10:20], 'precipitation_daily'] = 0.5

srr = soiling.SRRAnalysis(**data)
srr._calc_daily_df(clean_criterion='precip_or_shift')
srr._calc_result_df()

fig, ax = plt.subplots(3, 1, sharex=True)
data['precipitation_daily'].plot(drawstyle='steps', ax=ax[0])
srr.daily_df['clean_event'].astype(int).plot(drawstyle='steps', ax=ax[1])
srr.analyzed_daily_df['run'].plot(drawstyle='steps', ax=ax[2])
for a in ax:
    a.legend()

A string of multiple rainy days in a row only triggers a clean event on the last day of the string, meaning all the previous days get included in the previous run. But presumably the first rainy day would be sufficient for a cleaning, so the previous run should end on the first day of the rain string, not the last. Should each day in the rain string be its own run, and then it would get filtered out later for being too short?

[kandersolar]

Here is the same plot, but running RdTools with that line 162 removed:

[mdeceglie]

That line was originally intended to prevent multiple short runs from occurring when steps are detected on multiple days in a row. All but the last step would be included in the previous run and be part of the uncertainty accounted for in that run. If we remove it, we're changing the uncertainty a bit to say "when cleaning events occur on multiple days in rapid succession, treat these as part of 'invalid' soiling intervals." In effect, it shifts the uncertainty from being associated with a particular interval to being accounted for when we roll up during the monte carlo. I think the shift makes sense, and we can make that change by just removing the line. @kanderso-nrel @matt14muller does that change in uncertainty approach sound good to you?

[mdeceglie]

Like many things, this ended up being more complicated than I anticipated. When I removed that line, many of the tests failed, which was a surprise because they don't have a complicated cleaning pattern.

Using a 14 day rolling median (or any even number of days) results in values that are really the mean of two points. Thus many cleaning events result in 2-day spikes. These are eliminated in the current behavior by the line df['clean_event'] = (df.clean_event) & (~df.clean_event.shift(-1).fillna(False)). Although, they are a eliminated in a way that that causes a one day shift relative to the actual cleaning day. Simply removing the line results in two consecutive cleaning events, which is also undesirable. I propose we eliminate the line, make the default day_scale=13, and raise a warning when someone tries to use an even value for day_scale. The different behaviors as they apply to our testing case are illustrated here:

Generated with

import rdtools
import pandas as pd
import numpy as np
%matplotlib inline

# These are normally pytest fixtures
def times():
    tz = 'Etc/GMT+7'
    times = pd.date_range('2019/01/01', '2019/03/16', freq='D', tz=tz)
    return times

times = times()


def normalized_daily(times):
    interval_1 = 1 - 0.005 * np.arange(0, 25, 1)
    interval_2 = 1 - 0.002 * np.arange(0, 25, 1)
    interval_3 = 1 - 0.001 * np.arange(0, 25, 1)
    profile = np.concatenate((interval_1, interval_2, interval_3))
    np.random.seed(1977)
    noise = 0.01 * np.random.rand(75)
    normalized_daily = pd.Series(data=profile, index=times)
    normalized_daily = normalized_daily + noise

    return normalized_daily

normalized_daily = normalized_daily(times)

def insolation(times):
    insolation = np.empty((75,))
    insolation[:30] = 8000
    insolation[30:45] = 6000
    insolation[45:] = 7000

    insolation = pd.Series(data=insolation, index=times)

    return insolation

insolation = insolation(times)

# End of pytest fixtures



srr = rdtools.soiling.SRRAnalysis(normalized_daily, insolation)
srr._calc_daily_df(day_scale=14)

normalized_daily.plot()
srr.daily_df['clean_event'].astype(int).plot()

[mdeceglie]

closed with #289