Striping noise in embeddings with Sentinel-2 input

[Chaitanyapix]

Description

Observed repeating striping noise in embeddings generated using OLMoEarth
with Sentinel-2 as input. The artifact is visible both in the PCA components of the
embeddings and appears to be a systematic, repeating pattern rather than random noise.

Environment

• Model: OLMoEarth TINY/BASE
• Input: Sentinel-2 temporal data

Case 1 — Window size 256, Crop size 256

• Striping noise is visible as repeating straight lines in the Embeddings and PCA components of embeddings
• Observed within a single window (no mosaicking involved), ruling out a tile-merge artifact
• Pattern repeats consistently across all windows

Case 2 — Window size 64, Crop size 64

• Same striping artifact observed in Embeddings and embedding PCA components
• Suggests the artifact is model-level, not related to overlap blending or RasterMerger

Sample chunks

1. window size 256, Crop size 256

Embeddings:

PCA components:

2. window size 64, Crop size 64, Sentinel 2 as input.
   Embeddings:

PCA Components:

for window size 64 *64 it is not quite evident from the image, but when multiple 64 * 64 windows are loaded/mosaicked, the noise is visible as below. Note: the pink colour vertical stripes we are seeing is not at the overlapping portions.

Tested on multiple regions.
Any help in understanding and fixing this issue would be appreciated.

[Hgherzog]

@Chaitanyapix Hi, it would be helpful if you could share the code snippet that you used to produce the embeddings as well as some information about the test region you used so we can try to reproduce this behavior. Also, have you noticed worse performance on downstream tasks in areas where you see these artifacts? Thanks for engaging with our work!

[Chaitanyapix]

Hi,
thanks for the response.
i am sharing the config, model and run bash file for your reference to reproduce.

run.sh

# 
#!/bin/bash
# ============================================================
# OlmoEarth Embedding 
# ============================================================

set -e

# --- CONFIG ---------------------------------------------------
export PATH=/data/olmoearth/olmoearth_pretrain/.venv/bin:$PATH
export DATASET_PATH=/data/olmoearth/dataset
CONFIG_DIR=/data/olmoearth/configure

# --- STEP 1: Setup dataset folder -------------------------------------
rm -rf $DATASET_PATH
mkdir -p $DATASET_PATH
cp $CONFIG_DIR/config.json $DATASET_PATH/config.json

echo "=== Dataset path: $DATASET_PATH ==="

# --- STEP 2: Add windows ----------------------------------------------
rslearn dataset add_windows \
  --root $DATASET_PATH \
  --group default \
  --utm \
  --resolution 10 \
  --grid_size 256 \
  --src_crs EPSG:4326 \
  --box=3.8906525210868357,48.4682231198944606,4.3071335598031224,48.6141635830479331 \
  --start 2023-10-01T00:00:00+00:00 \
  --end 2024-05-31T00:00:00+00:00

echo "=== Windows created ==="
ls $DATASET_PATH/windows/default/

# --- STEP 3: Prepare (find matching satellite scenes) -----------------
rslearn dataset prepare \
  --root $DATASET_PATH \
  --workers 16 \
  --disabled-layers landsat \
  --retry-max-attempts 5 \
  --retry-backoff-seconds 5

echo "=== Prepare done ==="

# --- STEP 4: Ingest (download and tile the items) ---------------------
rslearn dataset ingest \
  --root $DATASET_PATH \
  --workers 8 \
  --disabled-layers landsat \
  --retry-max-attempts 5 \
  --retry-backoff-seconds 5

echo "=== Ingest done ==="
echo "=== Tiles downloaded ==="
find $DATASET_PATH -name "*.tif" | wc -l

# --- STEP 5: Materialize (crop tiles into windows) --------------------
rslearn dataset materialize \
  --root $DATASET_PATH \
  --workers 4 \
  --no-use-initial-job \
  --disabled-layers landsat \
  --retry-max-attempts 5 \
  --retry-backoff-seconds 5

echo "=== Materialize done ==="

# --- STEP 6: Verify downloaded data ------------------------------------
echo "=== Checking layers in first window ==="
FIRST_WINDOW=$(ls $DATASET_PATH/windows/default/ | head -1)
ls $DATASET_PATH/windows/default/$FIRST_WINDOW/layers/

echo "=== Sentinel-2 timesteps ==="
ls -d $DATASET_PATH/windows/default/$FIRST_WINDOW/layers/sentinel2_l2a* 2>/dev/null | wc -l

# --- STEP 7: Run embedding extraction ---------------------------------
rslearn model predict \
  --config $CONFIG_DIR/model.yaml \
  --data.init_args.path $DATASET_PATH

echo "=== Embeddings exported ==="

# --- STEP 8: Verify embeddings -----------------------------------------
echo "=== Checking embedding outputs ==="
find $DATASET_PATH/windows -path "*/layers/output/*" -name "*.tif" | head -5

echo "=== Total embedding files ==="
find $DATASET_PATH/windows -path "*/layers/output/*" -name "*.tif" | wc -l

echo "=== Done ==="

model.yaml

model:
  class_path: rslearn.train.lightning_module.RslearnLightningModule
  init_args:
    model:
      class_path: rslearn.models.singletask.SingleTaskModel
      init_args:
        encoder:
          - class_path: rslearn.models.olmoearth_pretrain.model.OlmoEarth
            init_args:
              model_id: OLMOEARTH_V1_TINY
              patch_size: 1
        decoder:
          - class_path: rslearn.train.tasks.embedding.EmbeddingHead
    optimizer:
      class_path: rslearn.train.optimizer.AdamW

data:
  class_path: rslearn.train.data_module.RslearnDataModule
  init_args:
    path: ${DATASET_PATH}
    inputs:
      sentinel2_l2a:
        data_type: raster
        layers: ["sentinel2_l2a", "sentinel2_l2a.1","sentinel2_l2a.2", "sentinel2_l2a.3", "sentinel2_l2a.4","sentinel2_l2a.5", "sentinel2_l2a.6"]
        bands: ["B02", "B03", "B04", "B08", "B05", "B06", "B07", "B8A", "B11", "B12", "B01", "B09"]
        passthrough: true
        dtype: FLOAT32
        load_all_layers: true
      
    task:
      class_path: rslearn.train.tasks.embedding.EmbeddingTask
    batch_size: 8    
    num_workers: 8  
    predict_config:
      transforms:
        - class_path: rslearn.models.olmoearth_pretrain.norm.OlmoEarthNormalize
          init_args:
            band_names:
              sentinel2_l2a: ["B02", "B03", "B04", "B08", "B05", "B06", "B07", "B8A", "B11", "B12", "B01", "B09"]
      # ADD THESE:
      crop_size: 256       # Process full window
      load_all_crops: false  # No internal cropping

trainer:
  precision: 16-mixed
  callbacks:
    - class_path: rslearn.train.prediction_writer.RslearnWriter
      init_args:
        path: ${DATASET_PATH}
        output_layer: embeddings
          # No overlap needed if no cropping

config.json

{
  "layers": {
    "sentinel2_l2a": {
      "type": "raster",
      "band_sets": [
        {
          "bands": [
            "B01",
            "B02",
            "B03",
            "B04",
            "B05",
            "B06",
            "B07",
            "B08",
            "B8A",
            "B09",
            "B11",
            "B12"
          ],
          "dtype": "uint16"
        }
      ],
      "data_source": {
        "class_path": "rslearn.data_sources.planetary_computer.Sentinel2",
        "init_args": {
          "cache_dir": "cache/planetary_computer",
          "harmonize": true,
          "query": {
            "eo:cloud_cover": {
              "lt": 20
            }
          },
          "sort_by": "eo:cloud_cover"
        },
        "query_config": {
          "max_matches": 20,
          "period_duration": "15d",
          "space_mode": "PER_PERIOD_MOSAIC"
        }
      }
    },
    "embeddings": {
      "type": "raster",
      "band_sets": [
        {
          "dtype": "float32"
        }
      ]
    }
  }
}

These scripts are for generating embeddings for the 256 cropsize and winter season. Same scripts we have tried for the 64 crop size and for full year datasets.

Yes, it is impacting the accuracies of the downstream tasks. It is creating artifacts in the final results.

[jen-abrahamson]

Hi, I wanted to comment on this thread as I think I'm experiencing a similar issue. I attempted to compute embeddings for the Florida Everglades area and am noticing some striping/spatial artifacts in the embeddings. I had separated my embedding generation using the --grid_size option to create multiple windows, each limited to 1024x1024, and each of the resulting windows had spatial artifacts. I have tested creating both annual and monthly embeddings.

Description
Observed striping/spatial artifacts in embeddings generated using OlmoEarth Tiny with
Sentinel-1 and Sentinel-2 as input.

Tested configurations (using the annual 2020 run as an example):
Note: some parts may have been slightly adapted to run in my HPC environment.

model.yaml:

model:
  class_path: rslearn.train.lightning_module.RslearnLightningModule
  init_args:
    model:
      class_path: rslearn.models.singletask.SingleTaskModel
      init_args:
        encoder:
        - class_path: rslearn.models.olmoearth_pretrain.model.OlmoEarth
          init_args:
            model_id: OLMOEARTH_V1_TINY
            patch_size: 4
        decoder:
        - class_path: rslearn.train.tasks.embedding.EmbeddingHead
    optimizer:
      class_path: rslearn.train.optimizer.AdamW
data:
  class_path: rslearn.train.data_module.RslearnDataModule
  init_args:
    path: ${DATASET_PATH}
    inputs:
      sentinel2_l2a:
        data_type: raster
        layers:
        - sentinel2_l2a
        - sentinel2_l2a.1
        - sentinel2_l2a.2
        - sentinel2_l2a.3
        - sentinel2_l2a.4
        - sentinel2_l2a.5
        - sentinel2_l2a.6
        - sentinel2_l2a.7
        - sentinel2_l2a.8
        - sentinel2_l2a.9
        - sentinel2_l2a.10
        - sentinel2_l2a.11
        bands:
        - B02
        - B03
        - B04
        - B08
        - B05
        - B06
        - B07
        - B8A
        - B11
        - B12
        - B01
        - B09
        passthrough: true
        dtype: FLOAT32
        load_all_layers: true
      sentinel1:
        data_type: raster
        layers:
        - sentinel1
        - sentinel1.1
        - sentinel1.2
        - sentinel1.3
        - sentinel1.4
        - sentinel1.5
        - sentinel1.6
        - sentinel1.7
        - sentinel1.8
        - sentinel1.9
        - sentinel1.10
        - sentinel1.11
        bands:
        - vv
        - vh
        passthrough: true
        dtype: FLOAT32
        load_all_layers: true
    task:
      class_path: rslearn.train.tasks.embedding.EmbeddingTask
    batch_size: 8
    num_workers: 8
    predict_config:
      transforms:
      - class_path: rslearn.models.olmoearth_pretrain.norm.OlmoEarthNormalize
        init_args:
          band_names:
            sentinel2_l2a:
            - B02
            - B03
            - B04
            - B08
            - B05
            - B06
            - B07
            - B8A
            - B11
            - B12
            - B01
            - B09
            sentinel1:
            - vv
            - vh
      load_all_crops: true
      crop_size: 64
      overlap_pixels: 32
trainer:
  accelerator: gpu
  devices: 1
  strategy: auto
  callbacks:
  - class_path: rslearn.train.prediction_writer.RslearnWriter
    init_args:
      output_layer: embeddings
      merger:
        class_path: rslearn.train.prediction_writer.RasterMerger
        init_args:
          overlap_pixels: 8
          downsample_factor: 4

config.json:

{
  "layers": {
    "landsat": {
      "band_sets": [
        {
          "bands": [
            "B1",
            "B2",
            "B3",
            "B4",
            "B5",
            "B6",
            "B7",
            "B8",
            "B9",
            "B10",
            "B11"
          ],
          "dtype": "uint16"
        }
      ],
      "data_source": {
        "class_path": "rslearn.data_sources.aws_landsat.LandsatOliTirs",
        "init_args": {
          "metadata_cache_dir": "/olmoearth/data_2020/cache/landsat",
          "sort_by": "cloud_cover"
        },
        "ingest": false,
        "query_config": {
          "max_matches": 12,
          "period_duration": "30d",
          "space_mode": "PER_PERIOD_MOSAIC"
        }
      },
      "type": "raster"
    },
    "sentinel1": {
      "band_sets": [
        {
          "bands": [
            "vv",
            "vh"
          ],
          "dtype": "float32",
          "nodata_vals": [
            -32768,
            -32768
          ]
        }
      ],
      "data_source": {
        "class_path": "rslearn.data_sources.planetary_computer.Sentinel1",
        "init_args": {
          "cache_dir": "/olmoearth/data_2020/cache/planetary_computer",
          "query": {
            "sar:instrument_mode": {
              "eq": "IW"
            },
            "sar:polarizations": {
              "eq": [
                "VV",
                "VH"
              ]
            }
          }
        },
        "ingest": false,
        "query_config": {
          "max_matches": 12,
          "period_duration": "30d",
          "space_mode": "PER_PERIOD_MOSAIC"
        }
      },
      "type": "raster"
    },
    "sentinel2_l2a": {
      "band_sets": [
        {
          "bands": [
            "B01",
            "B02",
            "B03",
            "B04",
            "B05",
            "B06",
            "B07",
            "B08",
            "B8A",
            "B09",
            "B11",
            "B12"
          ],
          "dtype": "uint16"
        }
      ],
      "data_source": {
        "class_path": "rslearn.data_sources.planetary_computer.Sentinel2",
        "init_args": {
          "cache_dir": "/olmoearth/data_2020/cache/planetary_computer",
          "harmonize": true,
          "sort_by": "eo:cloud_cover"
        },
        "ingest": false,
        "query_config": {
          "max_matches": 12,
          "period_duration": "30d",
          "space_mode": "PER_PERIOD_MOSAIC"
        }
      },
      "type": "raster"
    },
    "embeddings": {
      "band_sets": [
        {
          "dtype": "float32",
          "num_bands": 192
        }
      ],
      "type": "raster"
    }
  }
}

Commands that were run:

# Step 1: Add windows
rslearn dataset add_windows --root /olmoearth/data --group default --name default --utm --resolution 10 --src_crs EPSG:4326 --box=-81.36746217792019,25.15352689711134,-80.21716031743556,26.68774152852496 --start=2020-01-01T00:00:00+00:00 --end=2021-01-01T00:00:00+00:00 --grid_size=1024

# Step 2: Prepare
rslearn dataset prepare --root /olmoearth/data --workers 8 --retry-max-attempts 5 --retry-backoff-seconds 5 --disabled-layers landsat

# Step 3: Materialize
rslearn dataset materialize --root /olmoearth/data --workers 8 --no-use-initial-job --retry-max-attempts 5 --retry-backoff-seconds 5 --disabled-layers landsat

# Step 4: Predict
rslearn model predict --config /olmoearth/data/model.yaml

Observed:

Question:

• Are there some parameters I'm setting incorrectly that could cause this (thinking crop_size, overlap_pixels)?

Any help is appreciated, thanks!