# Batch Processing

> Technical overview of Egon Batch Processing (PPN): high-throughput asynchronous address normalization and geocoding of large datasets via SFTP.

High-throughput asynchronous address normalization and geocoding via SFTP for massive enterprise datasets.

## Architecture & Integration Model

Egon **Polling Process Normalization (PPN)** is an automated, file-based engine optimized for processing millions of records without per-row API overhead. Compare the integration models to choose the right approach for your workload:

| Architectural Dimension | Batch PPN Engine                                    | Real-Time REST API                             |
| ----------------------- | --------------------------------------------------- | ---------------------------------------------- |
| **Transport**           | SFTP (paired `.zip` + `.trk` files)                 | HTTPS REST (JSON / GET / POST)                 |
| **Primary Use Case**    | Database cleanups, ERP migrations, data warehouses  | Checkout forms, CRM single lookups, web apps   |
| **Execution**           | Asynchronous daemon polling the input folder        | Synchronous request-response with low latency  |
| **Throughput**          | Hundreds of thousands to millions of rows per batch | Optimized for individual and small burst calls |
| **Data Preservation**   | Zero data loss: source columns `0..N` kept intact   | Normalized JSON payload structure              |

---

## The 3-Step Lifecycle

Processing follows an automated lifecycle between your integration and the Egon server:

| Phase           | Actor / Location     | Action                                                                                                                       |
| --------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
| **1. Deposit**  | Client → `./input/`  | Upload `<filename>.zip` (containing `<filename>.csv`) and its `<filename>.trk`, which can stay on the server for later runs. |
| **2. Process**  | Daemon Engine        | A background worker polls `./input/`, picks up each archive with its descriptor, and enriches the records.                   |
| **3. Delivery** | Client ← `./output/` | Engine deposits `<filename>_output_<timestamp>.gz` with 32 appended fields (27 with `NOR;`) ready for retrieval and cleanup. |

> **Validate Your .trk**
>
> Want to check your files before configuring SFTP? Use the [Batch
> Playground](https://documentation.egon.com/docs/batch/playground.md) to map your CSV columns, generate the
> `.trk` descriptor, and validate it before upload.

---

## Documentation Guide

Explore the technical specifications and operational workflows:

### Processing Architecture

#### Core Modules & Capabilities

- **[SFTP & Workflow](https://documentation.egon.com/docs/batch/sftp-workflow.md)**: Connect over SFTP, deposit paired input archives, and retrieve enriched datasets automatically.

- **[Input & TRK Mapping](https://documentation.egon.com/docs/batch/input-specifications.md)**: Delimited data is compressed into `<filename>.zip` accompanied by a `.trk` configuration defining country ISO, delimiters, and column indices.

- **[Output & Enrichments](https://documentation.egon.com/docs/batch/output-specifications.md)**: The engine standardizes postal addresses across 250+ countries, adds WGS84 coordinates with `GEO;`, and generates enriched `.gz` archives.

- **[Batch Playground](https://documentation.egon.com/docs/batch/playground.md)**: Map your CSV columns, generate the `.trk` descriptor, and validate it before upload.