drtk.source_features#

`source_feature_stencil`	Compute bilinear sampling indices and visibility weights for mesh points.
`sample_feature_stencil`	Sample image features using a visibility-weighted bilinear stencil.
`source_feature_stencil_ref`	Reference for `source_feature_stencil()`, with smooth gradients only.
`sample_feature_stencil_ref`	Reference for `sample_feature_stencil()`, supporting CPU and float64.

Sample image features at points on a mesh, weighted by their visibility from the source camera. The outputs are a feature vector and visibility confidence for each point. Gradients propagate to features, surface positions, and occluding geometry.

See Sampling features with source visibility for an example.

Sampling features#

source_feature_stencil() computes the four source texels and weights for each surface query. sample_feature_stencil() gathers their features:

indices, weights, _, _ = source_feature_stencil(
    v_pix, faces, query_faces, query_barycentrics, height, width,
    vertex_opacity=opacity,
)
sampled, confidence = sample_feature_stencil(features, indices, weights)

A query is specified by a face ID and 3D barycentric coordinates. Each bilinear tap must lie on the query’s local surface. The surface continues through shared manifold edges and ends at cuts, projected folds, and nonmanifold junctions. Occlusion is evaluated at each tap, including self-occlusion within a connected mesh.

The same indices and weights can sample several feature maps with matching batch and spatial dimensions. Recompute them when the geometry, topology, material opacity, or source camera changes.

Visibility and confidence#

For a receiving face \(r\), visibility at source pixel \(p\) is

\[T_r(p)=\prod_{f\prec r}\left(1-\alpha_f(p)\right).\]

Faces are ordered by mean camera-space vertex depth, with face index breaking ties, as in drtk.transparent_render. This approximates visibility for small triangles; intersecting or depth-crossing triangles can require a different order at different pixels. The receiving face’s own opacity is excluded from the product.

With bilinear coefficients \(b_{qj}\) and source features \(F_j\), the sampling weights, feature sum, and confidence are

\[w_{qj}=b_{qj}T_{qj},\qquad S_q=\sum_{j=1}^{4}w_{qj}F_j,\qquad c_q=\sum_{j=1}^{4}w_{qj}.\]

Invalid taps have zero weight. The feature sum is unnormalized: a fully hidden query returns zero features and zero confidence. For example, an existing surface feature \(A_q\) can fill the missing contribution as \(S_q+(1-c_q)A_q\).

Use geometric or material opacity for visibility. If opacity is predicted from these same sampled features, the dependency is circular. Partial visibility supplies a confidence value; features already mixed by transparency or a neural receptive field remain mixed.

Gradients#

Smooth derivatives propagate through projection, bilinear interpolation, material opacity, and feature values. Coverage boundaries use DRTK’s edge estimator, enabled by edge_gradients=True. For upstream feature and confidence gradients \(g_S,g_c\), its weight derivative is

\[\frac{\partial L}{\partial w_{qj}} =\langle g_{S_q},F_j\rangle+g_{c_q}.\]

The estimator combines this derivative with the visibility change across an edge to obtain a geometry gradient. In-image taps retain their indices even at zero weight, allowing this change to account for features revealed by boundary motion. Ordinary shared tessellation edges, zero-opacity occluders, and fully hidden edges contribute zero boundary jump.

Call loss.backward() on the downstream loss. Applying edge_grad_estimator() again would count the source boundary contribution twice. A target-view renderer computes its own boundary contribution.

Quantity	Gradient
Vertices, query barycentrics, source features, material opacity	Smooth derivatives for fixed coverage and ordering.
Occlusion and receiving-surface boundaries	Estimated geometry derivatives from boundary motion.
Face IDs, connectivity, ordering, and surface-continuation decisions	Discrete. Use `bernoulli_render_loss()` for sampled topology, rebuilding the stencil for each sample.
Second derivatives	Unsupported.

The reference functions support CPU tensors and smooth derivatives. The boundary estimator is available in source_feature_stencil().

Coordinates and data types#

v_pix contains source pixel XY and positive camera Z. Integer XY denotes pixel centers. For 3D barycentrics \(\beta\), a query projects to

\[z_q=\sum_i\beta_i z_i,\qquad p_q=\frac{\sum_i\beta_i z_i p_i}{z_q}.\]

For a feature map resized from \(W\times H\) to \(W_f\times H_f\), transform pixel coordinates as x_f = (x + 0.5) * W_f / W - 0.5 and similarly for Y. return_queries=True returns normalized UV coordinates compatible with grid_sample(..., align_corners=False) and camera Z.

The stencil requires float32 CUDA geometry and produces float32 weights. Sampling accepts float16, bfloat16, and float32 CUDA features, with arbitrary channel counts and strides. Accumulation uses float32. Backward may be nondeterministic. source_feature_stencil does not support CUDA graph capture.

Functions#

drtk.source_feature_stencil(v_pix, vi, query_face_ids, query_barycentrics, height, width, *, vertex_opacity=None, edge_gradients=True, return_queries=False)[source]#

Compute bilinear sampling indices and visibility weights for mesh points.

Each query is attached to a face by 3D barycentric coordinates. Its four image taps follow the receiving surface through shared manifold edges, stopping at cuts, projected folds, or nonmanifold junctions. Each valid tap is weighted by transmittance from faces preceding its receiver in (mean triangle camera Z, face index) order.

Apply the result to a feature image with sample_feature_stencil(). Feature maps with matching batch and spatial dimensions can share the same stencil. Recompute it when geometry, topology, material opacity, or the source camera changes.

Parameters:

v_pix (Tensor) – Float32 CUDA vertices, (N, V, 3). XY are source pixel coordinates, with integer values at pixel centers. Z is positive camera depth. Use the feature image’s pixel grid.
vi (Tensor) – Int32 CUDA faces, (F, 3) or (N, F, 3). Faces with invalid indices, nonfinite vertices, nonpositive depth, or zero projected area are ignored. Shared indices define connectivity; coincident vertices with separate indices remain separate.
query_face_ids (Tensor) – Int32 CUDA receiving-face IDs, (N, Q). -1 denotes an unused query. At a shared vertex, choose an incident face on the intended side of the surface.
query_barycentrics (Tensor) – Float32 CUDA 3D barycentrics, (N, Q, 3). Entries must be nonnegative and sum to one. Invalid queries have zero weights and diagnostics, with tap indices set to -1.
height (int) – Source feature-image height.
width (int) – Source feature-image width.
vertex_opacity (Optional[Tensor]) – Float32 CUDA material opacity, (N, V), clamped to [0, 1]. Defaults to one. The receiving face’s own opacity is excluded from visibility. Opacity must be available before sampling; predicting it from these same sampled features creates a circular dependency.
edge_gradients (bool) – Estimate geometry gradients at visibility and receiving-surface boundaries. Default: True.
return_queries (bool) – Return each query’s projected UV and camera Z. Default: False.

Returns:

(tap_indices, tap_weights, query_uv, query_depth).

tap_indices is int32 (N, Q, 4), ordered top-left, top-right, bottom-left, bottom-right. Each entry is flattened as y * width + x. Out-of-image and unused-query taps have index -1. In-image indices remain valid at zero weight to support boundary gradients.

tap_weights is float32 (N, Q, 4): bilinear coefficients times source transmittance. Taps outside the receiving surface have zero weight. The weights are unnormalized.

query_uv is (N, Q, 2) in normalized align_corners=False coordinates; query_depth is (N, Q) camera Z. Both are None unless requested. Unused queries have zero diagnostics.

Return type:

Tuple[Tensor, Tensor, Optional[Tensor], Optional[Tensor]]

Gradients:

Smooth gradients reach vertices, query barycentrics, and material opacity. Coverage boundaries use DRTK’s edge-motion estimator. Ordinary tessellation edges and zero-opacity occluders have zero boundary jump. An additional edge_grad_estimator() call would count this contribution twice.

Face IDs, topology, and ordering are discrete. To optimize topology logits, use bernoulli_render_loss() with a fresh stencil for each sampled mesh. Only first derivatives are supported. Backward may be nondeterministic.

Note

Geometry and weights remain float32 under autocast. CUDA graph capture is unsupported. source_feature_stencil_ref() supports CPU inputs and smooth derivatives.

Visibility weights describe geometric access to the source image. Features mixed by transparency or a feature extractor’s receptive field remain mixed.

Example:

indices, weights, _, _ = source_feature_stencil(
    v_pix, faces, query_faces, query_barycentrics, H, W
)
sampled, confidence = sample_feature_stencil(features, indices, weights)
prediction = model(anchor_features, sampled, confidence)
loss_fn(prediction, target).backward()

drtk.sample_feature_stencil(feature_image, tap_indices, tap_weights)[source]#

Sample image features using a visibility-weighted bilinear stencil.

For texel features \(F_j\) and weights \(w_{qj}\), returns \(S_q=\sum_j w_{qj}F_j\) and confidence \(c_q=\sum_j w_{qj}\). The feature sum is unnormalized. For example, sampled + (1 - confidence) * anchor blends the observations with an existing surface feature where visibility is incomplete.

Parameters:

feature_image (Tensor) – CUDA (N, C, H, W), float16, bfloat16, or float32. Arbitrary nonnegative strides, including channels-last and sliced images, are supported. The grid and batch must match the stencil.
tap_indices (Tensor) – Int32 CUDA (N, Q, 4) from source_feature_stencil(). -1 and out-of-range indices contribute zero. Indices have no gradient.
tap_weights (Tensor) – Float32 CUDA (N, Q, 4) from source_feature_stencil().

Returns:

(sampled_features, confidence). Samples have shape (N, C, Q) and the image dtype, with float32 accumulation. Confidence is float32 (N, 1, Q). Gradients reach the image and weights, including zero weights with valid indices.

Return type:

Tuple[Tensor, Tensor]

Note

Only first derivatives are supported. Backward may be nondeterministic. sample_feature_stencil_ref() supports CPU and float64 inputs.

drtk.source_feature_stencil_ref(v_pix, vi, query_face_ids, query_barycentrics, height, width, *, vertex_opacity=None, edge_gradients=False, return_queries=False)[source]#: Reference for source_feature_stencil(), with smooth gradients only.

drtk.sample_feature_stencil_ref(feature_image, tap_indices, tap_weights)[source]#: Reference for sample_feature_stencil(), supporting CPU and float64.