n8n - Jazz ETL

This workflow is a Jazz OMS ETL (Extract, Transform, Load) pipeline that runs on weekdays at 7 AM to extract order and inventory data from Jazz OMS API and load it into Google Cloud Storage.

Workflow Overview
- Trigger: Schedule (7 AM, Monday-Friday)

Main Process Flow:
1) Authentication - Obtains API token from Jazz OMS
2) Date Range Generation - Creates a 14-day lookback window (current date - 14 days to current date - 1 day). This can be modified as needed.
3) Three Parallel Data Extraction Branches:

Branch A: Order Status & Line Items
- Loops through each date in the range
- Fetches order status data via API
- Extracts and deduplicates order status records
- Extracts and deduplicates order line items (detail_set)
- Transforms to NDJSON format
- Uploads to GCS: jazz/order_status/order_status__YYYYMMDD.jsonl and jazz/order_line_items/order_line_items__YYYYMMDD.jsonl

Branch B: Shipments
- Loops through each date in the range
- Fetches shipment data via API
- Deduplicates by shipment_number
- Transforms to NDJSON format
- Uploads to GCS: jazz/shipments/shipments__YYYYMMDD.jsonl

Branch C: Inventory Levels
- Fetches current inventory levels (single snapshot, not date-based)
- Processes inventory records with facility and quantity details
- Transforms to NDJSON format
- Uploads to GCS: jazz/inventory/inventory.jsonl

4 )File Management - Each upload includes:
- Delete existing file (if present)
- Create new file
- 2-second wait for consistency
- Merge to confirm completion
- Notifications - Sends success messages to Slack after completion

Key Features:
- Pagination handling for large API responses
- Deduplication logic to prevent duplicate records
- Parallel processing of multiple date ranges
- Error handling with retry logic on GCS uploads
- Timestamps added to all records (__loaded_at)