# Source tracking

!!! abstract ""

    How `source_proxy` preserves the link between input identity and output when data is transformed through a pipeline.

## The problem

When you call `apply_to()` or `as_completed()` on a data store, each member is fed through the pipeline independently. The pipeline may transform the data into something completely different — a new object with no reference back to the input that produced it. But the writer at the end of the pipeline needs to know *which input* produced *which output* so it can assign the correct unique ID in the output data store.

For example, if a loader reads `"gene_001.fa"` and the pipeline returns a translated protein sequence, the writer needs to store that result under the key `"gene_001"`. Without a mechanism to carry the input identity forward, this link is lost.

## How `source_proxy` solves it

`source_proxy` is a transparent wrapper that carries two extra pieces of state alongside the wrapped object:

- **`.source`** — the original input (or its identifier), preserved across transformations
- **`.uuid`** — a unique identifier for this proxy instance, used for hashing

When `as_completed()` or `apply_to()` processes a data store, each member is wrapped in a `source_proxy` before entering the pipeline. Because `source_proxy` delegates attribute access to the wrapped object via `__getattr__`, downstream apps see the original object and do not need to know about the proxy.

```python { notest }
from scinexus.composable import source_proxy

proxy = source_proxy(some_data)
proxy.source      # the original input
proxy.uuid        # unique identifier for this proxy
proxy.any_attr    # delegates to some_data.any_attr
```

## How `propagate_source` preserves the link

After each pipeline step, the result needs to be re-associated with the original source. `propagate_source` handles this:

1. If the result already has a `.source` attribute (e.g. it is a `DataMember` or another object that natively tracks its origin), the proxy is **unwrapped** — the result stands on its own.
2. Otherwise, the proxy's wrapped object is **updated** to the new result via `set_obj()`, and the proxy (still carrying the original `.source`) is returned.

This means the source identity survives an arbitrary number of pipeline steps, even when intermediate apps return entirely new objects.

## Why this matters for writers

`WriterApp.apply_to()` uses the source to derive unique IDs for output records. This enables **append-only semantics**: on a subsequent run against the same data store, records that already exist in the output are skipped. The unique ID comes from the original input's identity (via `get_data_source()`), which is only available because `source_proxy` carried it through the pipeline.

Without source tracking, the writer would have no way to determine whether a result corresponds to an input that has already been processed.
