> ## Documentation Index
> Fetch the complete documentation index at: https://docs.corsa.finance/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Pipeline Skill

> AI skill for building production data pipelines to Corsa — historical backfill, real-time sync, entity mapping for exchanges and payment companies.

## Overview

The **corsa-data-pipeline** skill adds Corsa data pipeline knowledge to your coding tool (Cursor, Claude Code, VS Code, etc.) so it can help you build production-grade pipelines that sync your existing data into Corsa. It's designed for exchanges, payment processors, and fintechs that already have users, transactions, and accounts in their own systems and need to push everything into Corsa for compliance monitoring.

### What It Helps With

* Mapping your existing data model (users, transactions, accounts) to Corsa entities
* Building historical backfill pipelines for millions of records
* Setting up real-time event-driven sync after the initial backfill
* Managing rate limits (500 req/60s) with throttled queues
* Maintaining ID mappings between your system and Corsa
* Handling retries, resumability, and error recovery

***

## Quick Start

After installing the skill, ask your coding tool questions like:

* *"Build a pipeline to sync all our users into Corsa as clients"*
* *"Backfill our historical transactions into Corsa deposits and withdrawals"*
* *"Set up real-time sync when a new payment is completed"*
* *"Map our exchange order model to Corsa trades"*
* *"How do I handle rate limits during a large backfill?"*

***

## What the Skill Knows

### Data Model Mapping

Complete mapping from common fintech entity types to Corsa entities:

| Your System                            | Corsa Entity                                         |
| -------------------------------------- | ---------------------------------------------------- |
| Users / Customers (individuals)        | `IndividualClient`                                   |
| Users / Customers (businesses)         | `CorporateClient`                                    |
| Corporate officers, UBOs               | `IndividualMember` / `CorporateMember`               |
| Bank accounts                          | `BankAccount` + client association                   |
| Crypto wallets                         | `BlockchainWallet` + client association              |
| PIX keys, CLABE, mobile money accounts | `PaymentAccount` + client association                |
| Fiat deposits / incoming payments      | `Deposit` operation                                  |
| Fiat withdrawals / outgoing payments   | `Withdrawal` operation                               |
| Crypto receives                        | `Deposit` with `txHash` and `blockchainNetworkId`    |
| Crypto sends                           | `Withdrawal` with `txHash` and `blockchainNetworkId` |
| Trades / swaps / conversions           | `Trade` operation with fill transactions             |
| Compliance alerts                      | `Alert` (batch up to 50)                             |
| Investigation cases                    | `Case`                                               |

### Ingestion Dependency Order

The skill enforces the strict entity dependency order: Clients first, then members, accounts, wallets, sessions, operations, alerts, cases, and finally verifications (KYC/KYB results). It knows that `initiatedBy` on operations expects a Corsa-generated UUID, not your internal user ID.

### Historical Backfill

* Throttled queue pattern that respects 500 req/60s
* Resumable cursor-based pagination with progress checkpoints
* Time estimates: 100K records takes \~3.5 hours, 1M+ takes \~35 hours
* Alert batch endpoint for faster alert ingestion (50 per call)

### Real-Time Sync

* Event-driven handler pattern for domain events (user created, transaction completed)
* Retry with exponential backoff for 429s and 5xx errors
* Bidirectional sync via Corsa webhooks to receive events back

### ID Mapping

SQL schema and repository pattern for maintaining `internal_id` to `corsa_id` mappings, with fallback to `referenceId` lookups.

***

## Production Gotchas It Catches

| Mistake                                        | What the skill does                                          |
| ---------------------------------------------- | ------------------------------------------------------------ |
| Using internal user ID as `initiatedBy`        | Reminds that operations expect a Corsa-generated client UUID |
| Forgetting to link accounts/wallets to clients | Includes the `associateWithClients` call after creation      |
| Expecting batch endpoints for all entities     | Clarifies that only alerts have a batch endpoint             |
| Not normalizing `referenceId`                  | Warns that upsert uses exact string match                    |
| Not handling rate limits during backfill       | Provides a throttled queue with token bucket and retry logic |

***

## Source

<Card title="GitHub Repository" icon="github" href="https://github.com/corsa-labs/corsa-skills/tree/main/corsa-data-pipeline">
  View the full skill source, including production code templates for throttled queues, backfill pipelines, and entity mapping.
</Card>