Eduard Heijkoop

Getting Started with ICESat-2 ATL03 Photon Data in Python

2025-03-15T00:00:00+01:00

DISCLAIMER: THE TEXT ON THIS PAGE IS LIKELY LARGELY INCORRECT AND IS JUST A PLACEHOLDER GENERATED BY CLAUDE. THIS STATEMENT WILL BE REMOVED WHEN THIS PAGE HAS BEEN EDITED FOR ACCURACY.

ICESat-2 has revolutionized our ability to measure surface elevation from space. Its photon-counting LiDAR achieves decimeter-level vertical accuracy at a dense along-track sampling (~70 cm), making it ideal for mapping low-lying coastal terrain where traditional radar altimetry fails.

In this post, I’ll walk through the full workflow from data download to cleaned elevation profiles.

1. Data Access

ICESat-2 data is freely available through NASA’s NSIDC DAAC. I recommend using icepyx for programmatic access:

pip install icepyx h5py numpy pandas geopandas matplotlib

import icepyx as ipx

# Define your region of interest and time range
region = ipx.Query(
    dataset      = 'ATL03',
    spatial_extent = [-91.5, 29.0, -89.0, 30.5],  # Louisiana coast
    date_range   = ['2023-01-01', '2023-12-31'],
    start_time   = '00:00:00',
    end_time     = '23:59:59'
)

region.avail_granules()
region.order_granules()
region.download_granules('/data/icesat2/')

2. Reading ATL03 HDF5 Files

ATL03 stores data in a beam-organized HDF5 hierarchy. ICESat-2 has 3 pairs of beams (gt1, gt2, gt3), each with a left (l) and right (r) beam. Strong beams carry ~4× more photons than weak beams.

import h5py
import numpy as np
import pandas as pd

def read_atl03_beam(filepath, beam='gt1l'):
    """Read photon data from a single ATL03 beam."""
    with h5py.File(filepath, 'r') as f:
        # Check if beam exists
        if beam not in f:
            return None
        
        grp = f[f'{beam}/heights']
        
        df = pd.DataFrame({
            'lon':  grp['lon_ph'][:],
            'lat':  grp['lat_ph'][:],
            'h':    grp['h_ph'][:],       # WGS84 ellipsoidal height
            'conf': grp['signal_conf_ph'][:, 0],  # land conf
            'dist': grp['dist_ph_along'][:]
        })
        
        # Beam strength (needed to identify strong vs weak)
        sc_orient = f['orbit_info/sc_orient'][0]
        beam_type = f[f'{beam}/geolocation/beam_type'][0].decode()
        df['beam'] = beam
        df['beam_type'] = beam_type
        
    return df

3. Filtering Signal Photons

ATL03 provides a confidence flag (0–4) for land surface photons. For bare coastal terrain, conf >= 3 works well. In vegetated areas, you may want to use ATL08 land/veg classification.

def filter_signal_photons(df, conf_min=3, h_min=-10, h_max=50):
    """
    Filter to likely ground photons.
    
    conf_min : minimum confidence (3 = medium, 4 = high)
    h_min/h_max : rough elevation range to exclude noise
    """
    mask = (
        (df['conf'] >= conf_min) &
        (df['h'] > h_min) &
        (df['h'] < h_max)
    )
    return df[mask].copy()

4. Converting to Mean Sea Level

ATL03 heights are referenced to the WGS84 ellipsoid. For coastal studies, you need heights above a geoid (e.g., EGM2008 or GEOID18).

from pyproj import Transformer, CRS

def ellipsoid_to_msl(df, geoid_grid_path):
    """Subtract geoid undulation from ellipsoidal heights."""
    import rasterio
    from rasterio.sample import sample_gen
    
    with rasterio.open(geoid_grid_path) as src:
        coords = list(zip(df['lon'], df['lat']))
        geoid_N = np.array([val[0] for val in src.sample(coords)])
    
    df['h_msl'] = df['h'] - geoid_N
    return df

5. Visualizing the Profile

import matplotlib.pyplot as plt

fig, axes = plt.subplots(2, 1, figsize=(14, 7), sharex=True)

for beam, color in zip(['gt1l','gt1r'], ['#2ecc71','#3498db']):
    df_beam = read_atl03_beam(filepath, beam)
    df_filt = filter_signal_photons(df_beam)
    
    # All photons (noise)
    axes[0].scatter(df_beam['dist'], df_beam['h'], 
                    s=0.3, c='gray', alpha=0.3)
    # Signal photons
    axes[0].scatter(df_filt['dist'], df_filt['h'],
                    s=1, c=color, alpha=0.8, label=beam)

axes[0].set_ylabel('Height (m, WGS84)')
axes[0].legend()
axes[0].set_title('ICESat-2 ATL03 — Louisiana Coast')
plt.tight_layout()
plt.savefig('icesat2_profile.png', dpi=150)

Next Steps

In the next post, I’ll show how to use these filtered photon profiles to correct biases in existing DEMs (TanDEM-X, Copernicus) over coastal lowlands.

Code: All scripts from this post are available in the icesat2-tools repository on GitHub.

Why Your Coastal DEM Is Wrong (And How to Fix It)

2025-01-20T00:00:00+01:00

DISCLAIMER: THE TEXT ON THIS PAGE IS LIKELY LARGELY INCORRECT AND IS JUST A PLACEHOLDER GENERATED BY CLAUDE. THIS STATEMENT WILL BE REMOVED WHEN THIS PAGE HAS BEEN EDITED FOR ACCURACY.

If you’ve ever run a sea level rise inundation model using SRTM or even the Copernicus DEM, you’ve almost certainly overestimated how many people live in flood zones. Here’s why — and what to do about it.

The Problem: DEMs Measure Canopy, Not Ground

Radar-based DEMs like SRTM (C-band, 2000) and TanDEM-X (X-band, 2010–2015) measure the first radar return — which over vegetated terrain is the top of the canopy, not the ground.

In flat coastal lowlands dominated by mangroves, rice paddies, and coastal scrub, this canopy bias can reach +0.5 to +2.5 m. Since we’re trying to map areas below 2 m elevation, a 1 m positive bias means:

We think a field is 1.8 m above sea level.
It’s actually 0.8 m above sea level.
Under a 1 m SLR scenario, we say it’s safe. It floods.

This isn’t a small rounding error — it changes how many people fall in the flood zone by 20–40% in heavily vegetated deltas.

Where the Bias Is Largest

We can quantify this bias by comparing DEMs to ICESat-2 ground photons (which measure actual ground elevation):

Land Cover	SRTM Bias	Copernicus Bias	TanDEM-X Bias
Bare soil / beach	+0.1 m	+0.05 m	+0.08 m
Low crops	+0.4 m	+0.3 m	+0.2 m
Mangrove	+2.1 m	+1.8 m	+1.4 m
Rice paddy	+0.7 m	+0.5 m	+0.35 m
Urban (low-rise)	+0.3 m	+0.2 m	+0.15 m

X-band (TanDEM-X) penetrates vegetation better than C-band (SRTM), hence lower bias — but it’s still significant.

The Fix: ICESat-2-Based Bias Correction

The cleanest solution is to use ICESat-2 ATL08 ground photons as a reference surface and correct the DEM locally.

Step 1: Extract DEM values at ICESat-2 locations

import rasterio
import numpy as np
import pandas as pd
from rasterio.sample import sample_gen

def sample_dem_at_icesat2(df_icesat2, dem_path):
    """Sample DEM elevation at ICESat-2 photon locations."""
    with rasterio.open(dem_path) as src:
        coords = list(zip(df_icesat2['lon'], df_icesat2['lat']))
        dem_vals = np.array([v[0] for v in sample_gen(src, coords)])
    
    df_icesat2['h_dem'] = dem_vals
    df_icesat2['residual'] = df_icesat2['h_msl'] - dem_vals  # ICESat-2 minus DEM
    return df_icesat2

Step 2: Stratify by land cover

import geopandas as gpd

def stratify_by_landcover(df, landcover_path):
    """Assign ESA WorldCover class to each ICESat-2 point."""
    with rasterio.open(landcover_path) as src:
        coords = list(zip(df['lon'], df['lat']))
        lc = np.array([v[0] for v in sample_gen(src, coords)])
    df['landcover'] = lc
    return df

# Compute bias per land cover class
bias_by_lc = (
    df.groupby('landcover')['residual']
    .agg(['mean', 'std', 'count'])
    .rename(columns={'mean': 'bias'})
)
print(bias_by_lc)

Step 3: Apply spatially varying correction

Rather than a single global offset, we apply a kriged correction surface — essentially spatial interpolation of the residuals:

from pykrige.ok import OrdinaryKriging
import numpy as np

def kriging_correction(df_residuals, target_lon, target_lat):
    """
    Spatially interpolate DEM bias using ordinary kriging.
    """
    OK = OrdinaryKriging(
        df_residuals['lon'].values,
        df_residuals['lat'].values,
        df_residuals['residual'].values,
        variogram_model='spherical',
        verbose=False,
        enable_plotting=False
    )
    
    z_correction, ss = OK.execute(
        'grid', target_lon, target_lat
    )
    return z_correction

Step 4: Apply to raster

import rasterio
import numpy as np

def apply_dem_correction(dem_path, correction_grid, output_path):
    """Add correction surface to DEM raster."""
    with rasterio.open(dem_path) as src:
        dem = src.read(1).astype(float)
        profile = src.profile
        nodata = src.nodata
    
    dem_corrected = dem + correction_grid
    dem_corrected[dem == nodata] = nodata
    
    profile.update(dtype='float32')
    with rasterio.open(output_path, 'w', **profile) as dst:
        dst.write(dem_corrected.astype('float32'), 1)

Validation Results

After correction, RMSE against independent airborne LiDAR drops from:

SRTM: 0.89 m → 0.21 m RMSE
Copernicus DEM: 0.63 m → 0.17 m RMSE

And flood exposure changes substantially:

In the Mekong Delta at 1 m SLR:

Uncorrected Copernicus: 14.2 million exposed

Corrected DEM: 9.8 million exposed

−31% difference

Takeaway

If you’re using off-the-shelf DEMs for coastal flood modeling without vegetation bias correction, your results are likely significantly overestimating exposure in vegetated delta regions. The good news: with ICESat-2 data (free, global coverage) and a few hundred lines of Python, you can substantially improve accuracy.

Next post: I’ll show how to build a production pipeline that processes all ICESat-2 tracks over a region automatically and generates a corrected DEM tile mosaic.

Questions? Open an issue or discussion on the GitHub repo.

Interactive Coastal Maps with Python Folium

2024-11-05T00:00:00+01:00

DISCLAIMER: THE TEXT ON THIS PAGE IS LIKELY LARGELY INCORRECT AND IS JUST A PLACEHOLDER GENERATED BY CLAUDE. THIS STATEMENT WILL BE REMOVED WHEN THIS PAGE HAS BEEN EDITED FOR ACCURACY.

Static maps are great for publications, but when you’re exploring coastal flood extents or satellite coverage, interactive maps are a game-changer. In this post I’ll share my Folium workflow for creating the kinds of embedded maps you see throughout this portfolio.

Why Folium?

Pure Python (no JavaScript knowledge needed)
Outputs self-contained HTML — easy to embed anywhere
Built on Leaflet.js — smooth, professional maps
Works seamlessly with GeoPandas / Shapely geometries
Great tile options: OpenStreetMap, CartoDB, Stamen, Esri

Basic Setup

pip install folium geopandas branca

1. A Simple DEM Coverage Map

import folium
from folium import plugins

# Base map — dark tiles look great for scientific data
m = folium.Map(
    location=[10.5, 106.0],   # Mekong Delta center
    zoom_start=8,
    tiles='CartoDB dark_matter',
    width='100%',
    height='100%'
)

# Add a scale bar
plugins.MeasureControl(position='bottomleft').add_to(m)

# Add fullscreen button
plugins.Fullscreen().add_to(m)

m.save('mekong_base.html')

2. Adding ICESat-2 Ground Tracks

import geopandas as gpd
import pandas as pd

def add_icesat2_tracks(m, tracks_gdf, colormap='YlOrRd'):
    """Add ICESat-2 ground tracks colored by acquisition date."""
    from branca.colormap import linear
    
    # Create date colormap
    dates = pd.to_datetime(tracks_gdf['date'])
    date_num = (dates - dates.min()).dt.days
    
    cmap = linear.YlOrRd_09.scale(date_num.min(), date_num.max())
    cmap.caption = 'ICESat-2 Acquisition Date'
    
    for _, row in tracks_gdf.iterrows():
        d_num = (pd.to_datetime(row['date']) - dates.min()).days
        color = cmap(d_num)
        
        folium.GeoJson(
            row.geometry,
            style_function=lambda x, c=color: {
                'color': c,
                'weight': 1.5,
                'opacity': 0.8
            },
            tooltip=f"Date: {row['date']}
Beam: {row['beam']}"
        ).add_to(m)
    
    cmap.add_to(m)
    return m

3. Choropleth: Exposed Population by Province

import json

def add_population_choropleth(m, provinces_gdf, pop_column):
    """Add population exposure choropleth."""
    import branca.colormap as cm
    
    colormap = cm.LinearColormap(
        colors=['#ffffcc', '#fd8d3c', '#800026'],
        vmin=provinces_gdf[pop_column].min(),
        vmax=provinces_gdf[pop_column].max(),
        caption=f'Exposed Population ({pop_column})'
    )
    
    folium.GeoJson(
        provinces_gdf,
        style_function=lambda feature: {
            'fillColor': colormap(
                feature['properties'][pop_column] or 0
            ),
            'color': 'white',
            'weight': 0.5,
            'fillOpacity': 0.7
        },
        tooltip=folium.GeoJsonTooltip(
            fields=['province', pop_column],
            aliases=['Province', 'Exposed Population'],
            localize=True
        )
    ).add_to(m)
    
    colormap.add_to(m)
    return m

4. Layer Control for SLR Scenarios

The key to a good SLR comparison map is FeatureGroup + LayerControl:

import folium
from folium import FeatureGroup, LayerControl

def create_slr_comparison_map(flood_polygons_dict, center):
    """
    flood_polygons_dict: {'0.5m': GeoDataFrame, '1.0m': GDF, '2.0m': GDF}
    """
    m = folium.Map(location=center, zoom_start=9,
                   tiles='CartoDB positron')
    
    scenario_colors = {
        '0.5m': {'fill': '#3498db', 'label': 'SLR +0.5 m'},
        '1.0m': {'fill': '#e67e22', 'label': 'SLR +1.0 m'},
        '2.0m': {'fill': '#e74c3c', 'label': 'SLR +2.0 m'},
    }
    
    for scenario, gdf in flood_polygons_dict.items():
        cfg = scenario_colors[scenario]
        fg = FeatureGroup(name=cfg['label'], show=(scenario == '1.0m'))
        
        folium.GeoJson(
            gdf.__geo_interface__,
            style_function=lambda x, c=cfg['fill']: {
                'fillColor': c,
                'color': c,
                'fillOpacity': 0.45,
                'weight': 0.3
            }
        ).add_to(fg)
        
        fg.add_to(m)
    
    LayerControl(collapsed=False).add_to(m)
    return m

5. Embedding in Jekyll / GitHub Pages

Save the Folium map as HTML and put it in assets/maps/:

m.save('assets/maps/slr_impact.html')

Then in your Markdown:

 
  src="/assets/maps/slr_impact.html" 
  width="100%" 
  height="500" 
  frameborder="0">

One gotcha: Folium generates full HTML pages with and tags. This works fine in iframes, but if you want inline embedding you’ll need branca.element.Figure to generate just the map div. For most use cases, iframes are the simplest approach.

Pro Tips

Use prefer_canvas=True in folium.Map() for large point datasets (>10k points)
Convert large GeoDataFrames to GeoJSON strings first — it’s faster than passing the GDF directly
Use folium.plugins.MarkerCluster for many point markers
CartoDB dark_matter and CartoDB positron are ideal for scientific maps — clean, no clutter

All map generation scripts for this portfolio are available on GitHub. Feel free to fork and adapt for your own coastal research.