commit cce7b03ca951a96aa2b34bbfc1ad6219c130a108 Author: Wild Karl-Heinz Date: Thu Oct 8 15:29:15 2026 +0200 datapipeline 2.0.0: Gem aus dem Projekt etl herausgelöst diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..2b8e831 --- /dev/null +++ b/.gitignore @@ -0,0 +1,8 @@ +*~ +.DS_* +.bundle +vendor +*.gem + +# Ausgabe der Beispiele (examples/*.rb, Default --out=tmp) +/tmp/ diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 0000000..92d671a --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,123 @@ +# Changelog + +Framework `datapipeline` (Gem). Neueste Version oben. + +## 2.0.0 – 2026-10-08 + +- Gem `datapipeline`: `datapipeline.gemspec`, Executable `etl` (exe/etl), Code unter `lib/datapipeline/` +- eigenes Projekt `datapipeline` (vorher Teil von `etl`, dessen Historie die frühere Entwicklung enthält); die + Anwendungen in `etl` binden das Gem mit `path: '../datapipeline'` ein +- **Breaking:** alle Klassen im Namensraum `DataPipeline`: `DataPipeline::StoragePlugin`, `DataPipeline::CsvStorage`, + `…::PostgresStorage`, `…::JsonStorage`, `…::SqliteStorage`, `…::ExcelStorage`, `…::RestStorage`, `DataPipeline::Slack`. + `container :csv` usw. bleibt gleich; eigene Plugins dürfen weiter auf oberster Ebene (`Storage`) liegen +- `DataPipeline::VERSION`; Versionsverlauf hier statt in `version.rb` +- Postgres-Plugin lädt `pg` über `require_gem` (Hinweis, wenn das Gem fehlt) + +## 1.9.0 – 2026-10-08 + +- rest: call(method, path, json:, params:, retries:) for rpc style apis, Error#status, ssl_verify: false +- json: a single object is read as one row +- dsl: dry_run? for steps with side effects +- application: rechnung_anlegen.rb (pds API: verify or create customer, invoice to a project file) + +## 1.8.2 – 2026-10-08 + +- rest: write_url as lambda for one url per row (e.g. PUT /archiv/) +- REST examples (read with paging/retry/lookup, write in batches, API to API) with a local demo API + +## 1.8.1 – 2026-10-08 + +- sqlite: database opened on first use, create: true and truncate: true; postgres: truncate: true +- log line "start" without trailing blank when no version is set +- examples with data in examples/, documented in DOKUMENTATION.md section 1 and tested + +## 1.8.0 – 2026-10-08 + +- DataPipeline::Layout: field list per record type (type, length, required); csv option layout: writes in field order and rejects unknown fields; validate layout checks required, length, format +- validate warn: true (warning, row kept, stats.warnings) and ignore: fields; warning log capped at 100 +- DataPipeline.params reads --name=value / --param key=value from ARGV when not started by bin/etl +- bin/etl passes unknown --name=value options to the script and to params + +## 1.7.0 – 2026-10-08 + +- new plugins: json (array and ndjson), sqlite (:sqlite, :sqlite3), excel (:excel, :xlsx), rest +- gems of a plugin are loaded on first use (StoragePlugin.require_gem with a hint when missing) + +## 1.6.0 – 2026-10-08 + +- plugins loaded on first use (autoload): pg only for postgres, http only for slack +- DataPipeline.register for own plugins, optionally loaded from a path on first use +- defaults are frozen and replaced instead of changed; DataPipeline.new(**settings) per pipeline, reset_settings! +- write can be called several times, only: lambda routes rows to a target +- clear error when read or write is missing + +## 1.5.0 – 2026-10-08 + +- on_error :abort | :skip | :collect (to: storage) for row errors in body steps +- validate step with rules (required, max, min, format, in, type) or block +- run returns stats (read, written, output, rejected, errors, duration), on_finish hooks, logger (default warn on stderr) +- watermark: highest value of a column saved after a successful run (json file) +- dry_run (nothing written, preview of the first rows) and limit +- bin/etl run PIPELINE.rb with --from/--till/--param/--dry-run/--limit and exit codes, Rakefile with arguments through bin/etl + +## 1.4.0 – 2026-10-08 + +- streamed works: rows go one by one from source.read { } through the chain to target.write_row, header/trailer/prepend/append and conditions included +- batch and streamed share one engine for chained steps +- parallel mode: every step gets its own copy of the row +- lookup: load a table once as hash (grouped or unique) instead of one query per row +- postgres: read with block uses a server side cursor (fetch_size), queries on the same connection stay possible while reading + +## 1.3.0 – 2026-10-08 + +- chained mode (chained true): every body step gets the output of the previous step +- new dsl steps map, filter, header, trailer with context (read, written, index); map and filter turn chaining on +- all prepend and append transformations run once, stable order of registration +- condition also as lambda +- without any transformation the source is copied unchanged + +## 1.2.1 – 2026-10-08 + +- transformations returning nil (or nil inside an array) produce no row +- explicit dsl methods version, verbose, streamed; unknown calls raise NoMethodError (method_missing removed, unused setting processing removed) +- container only reports a missing plugin class as unsupported, plugin errors pass through + +## 1.2.0 – 2026-10-08 + +- csv: read works with ruby 3, defaults merged, option columns for fixed column order, streaming read/write_row, string keys like postgres +- postgres: bind parameters (params:), write with table: option, nil as NULL, one transaction, quoted identifiers, optional typed results, close +- storage plugin: common interface with set, query, write_row, close +- pipeline closes source and target after run +- slack notification fixed +- removed unused lib/source.rb and lib/target.rb + +## 1.1.2 – 2025-02-06 + +- split data and output data to be able to insert results to output nut comming from db +- added query to Datapipline to execute subqueries if nessessary +- added prepend, append and normal to transform. we are now be able to order processing + +## 1.1.1 – 2025-01-10 + +- merge options in csv +- verbose and counter + +## 1.1.0 – 2024-12-21 + +- change to plugins, postgres and csv for read and write + +## 1.0.3 – 2024-10-18 + +- added slack as plugin +- added settings to source +- added verbose in processing non stream +- added transformation in blocks +- added transformation condition + +## 1.0.2 – 2024-10-15 + +- handle data streamed + +## 1.0.1 – 2024-10-10 + +- handle missing or wrong parameter diff --git a/Gemfile b/Gemfile new file mode 100644 index 0000000..e0d090c --- /dev/null +++ b/Gemfile @@ -0,0 +1,19 @@ +source 'https://rubygems.org' + +ruby '~> 3.4' + +gemspec + +# optional plugin gems: needed for the plugin tests, loaded only when a plugin is used +group :plugins do + gem 'caxlsx', require: false # excel write + gem 'http', require: false # rest, slack + gem 'pg', require: false # postgres + gem 'roo', require: false # excel read + gem 'sqlite3', require: false # sqlite +end + +group :test do + gem 'minitest', '~> 5.25' + gem 'rake' +end diff --git a/Gemfile.lock b/Gemfile.lock new file mode 100644 index 0000000..ac649ff --- /dev/null +++ b/Gemfile.lock @@ -0,0 +1,96 @@ +PATH + remote: . + specs: + datapipeline (2.0.0) + csv (~> 3.3) + logger (~> 1.6) + +GEM + remote: https://rubygems.org/ + specs: + base64 (0.3.0) + caxlsx (4.5.0) + htmlentities (~> 4.3, >= 4.3.4) + marcel (~> 1.0) + nokogiri (~> 1.10, >= 1.10.4) + rubyzip (>= 2.4, < 4) + csv (3.3.6) + domain_name (0.6.20260921) + htmlentities (4.4.2) + http (6.0.4) + http-cookie (~> 1.0) + llhttp (~> 0.6.1) + http-cookie (1.1.6) + domain_name (~> 0.5) + llhttp (0.6.2) + logger (1.7.0) + marcel (1.2.1) + minitest (5.27.0) + nokogiri (1.19.4-aarch64-linux-gnu) + racc (~> 1.4) + nokogiri (1.19.4-aarch64-linux-musl) + racc (~> 1.4) + nokogiri (1.19.4-arm-linux-gnu) + racc (~> 1.4) + nokogiri (1.19.4-arm-linux-musl) + racc (~> 1.4) + nokogiri (1.19.4-arm64-darwin) + racc (~> 1.4) + nokogiri (1.19.4-x86_64-darwin) + racc (~> 1.4) + nokogiri (1.19.4-x86_64-linux-gnu) + racc (~> 1.4) + nokogiri (1.19.4-x86_64-linux-musl) + racc (~> 1.4) + pg (1.7.0) + pg (1.7.0-aarch64-linux) + pg (1.7.0-aarch64-linux-musl) + pg (1.7.0-arm64-darwin) + pg (1.7.0-x86_64-darwin) + pg (1.7.0-x86_64-linux) + pg (1.7.0-x86_64-linux-musl) + racc (1.8.1) + rake (13.4.2) + roo (3.0.0) + base64 (~> 0.2) + csv (~> 3) + logger (~> 1) + nokogiri (~> 1) + rubyzip (>= 3.0.0, < 4.0.0) + rubyzip (3.7.0) + sqlite3 (2.9.6-aarch64-linux-gnu) + sqlite3 (2.9.6-aarch64-linux-musl) + sqlite3 (2.9.6-arm-linux-gnu) + sqlite3 (2.9.6-arm-linux-musl) + sqlite3 (2.9.6-arm64-darwin) + sqlite3 (2.9.6-x86_64-darwin) + sqlite3 (2.9.6-x86_64-linux-gnu) + sqlite3 (2.9.6-x86_64-linux-musl) + +PLATFORMS + aarch64-linux + aarch64-linux-gnu + aarch64-linux-musl + arm-linux-gnu + arm-linux-musl + arm64-darwin + x86_64-darwin + x86_64-linux + x86_64-linux-gnu + x86_64-linux-musl + +DEPENDENCIES + caxlsx + datapipeline! + http + minitest (~> 5.25) + pg + rake + roo + sqlite3 + +RUBY VERSION + ruby 3.4.9p82 + +BUNDLED WITH + 2.6.9 diff --git a/README.md b/README.md new file mode 100644 index 0000000..2a6bb07 --- /dev/null +++ b/README.md @@ -0,0 +1,912 @@ +# DataPipeline – generisches ETL-Framework in Ruby + +Stand: 2026-10-08 · Gem `datapipeline` 2.0.0 ([CHANGELOG.md](CHANGELOG.md)) · Ruby 3.4 + +--- + +## 1. Beschreibung + +`DataPipeline` ist ein schlankes ETL-Framework (Extract – Transform – Load) mit einer Ruby-DSL. Eine Pipeline +liest Daten aus einer Quelle, schickt jede Zeile durch eine Kette von Schritten (prüfen, filtern, umbauen, +anreichern) und schreibt das Ergebnis in ein oder mehrere Ziele. Quellen und Ziele sind austauschbare +**Storage-Plugins**: PostgreSQL, SQLite, CSV, JSON/NDJSON, Excel und REST-APIs. + +Die Pipeline beschreibt *was* passiert, die Plugins kapseln *wie* gelesen und geschrieben wird. Für den Betrieb +gibt es Fehlerdateien, Prüfregeln, Dry-Run, inkrementelle Läufe, Streaming für grosse Datenmengen und eine +Kommandozeile mit Exit-Codes. + +**Anwendungen:** Das Projekt `etl` nutzt das Gem für die PDS → SAP-Schnittstelle (CUDO) und für das Anlegen von +Rechnungen über die pds API; dort liegen auch die zugehörige Doku und Tests. + +### 1.1 Schnellstart + +```sh +mise install # Ruby 3.4.9 aus mise.toml +bundle install +bundle exec rake test # Tests, lokales Postgres optional +bin/etl run examples/kunden_import.rb # Beispiel unten, Ergebnis in tmp/ +bin/etl run examples/kunden_import.rb --dry-run # alles prüfen, nichts schreiben +``` + +Das Framework ist das Gem `datapipeline`. In einer anderen Anwendung wird es so eingebunden: + +```ruby +# Gemfile +gem 'datapipeline', path: '../datapipeline' # oder git: '…' +gem 'pg' # nur die Gems der benötigten Plugins, siehe 2.6 +``` + +```sh +bundle install +bundle exec etl pipeline.rb --dry-run # das Kommando etl kommt mit dem Gem +``` + +Eine Pipeline ist ein Ruby-Skript. Das kleinste sinnvolle Beispiel kopiert eine CSV-Datei nach JSON: + +```ruby +require 'datapipeline' + +DataPipeline.new do + read container(:csv, file: 'kunden.csv') + write container(:json, file: 'kunden.json') +end.run +``` + +Zwischen `read` und `write` stehen die Schritte; sie laufen in der Reihenfolge, in der sie im Skript stehen: + +| Schritt | Wirkung | +| --- | --- | +| `validate regeln` | prüft jede Zeile; Fehler gehen an `on_error` | +| `filter { \|row\| … }` | Zeile bleibt, wenn der Block wahr ist | +| `map { \|row, ctx\| … }` | gibt eine Zeile, mehrere (Array) oder keine (`nil`) zurück | +| `header { \|ctx\| … }`, `trailer { \|ctx\| … }` | Kopf- und Schlusssatz mit den Zählern `ctx.read`, `ctx.written` | + +### 1.2 Ausführliches Beispiel: Kundenimport + +Eine Kundenliste (`examples/data/kunden.csv`, 9 Zeilen) soll geprüft, mit Ländernamen angereichert und in eine +SQLite-Datenbank geschrieben werden; EU-Kunden kommen zusätzlich in eine Excel-Liste. Fehlerhafte Zeilen sollen den +Lauf nicht abbrechen, sondern mit Grund in einer eigenen Datei landen. + +```mermaid +flowchart LR + CSV[kunden.csv] --> V1[validate
Pflicht, Format, Typ] + LJ[laender.json] -. lookup .-> V2 + V1 --> V2[validate
Land bekannt] + V2 --> F[filter
nur aktiv] + F --> M[map
anreichern, umbauen] + LJ -. lookup .-> M + M --> DB[(kunden.sqlite3
alle)] + M -->|only: EU| XL[eu_kunden.xlsx] + V1 -. on_error .-> ERR[kunden_fehler.csv] + V2 -. on_error .-> ERR +``` + + +```ruby +# +# Kundenliste aus CSV prüfen, mit Ländern anreichern, nach SQLite und Excel schreiben +# +# bin/etl run examples/kunden_import.rb [--out=tmp] [--dry-run] +# +require 'fileutils' +require 'datapipeline' + +DATA = File.expand_path('data', __dir__) +out = DataPipeline.params.fetch(:out, 'tmp') +FileUtils.mkdir_p(out) + +DataPipeline.new do + version 'kunden_import 1.0' + + # Nachschlagetabelle, einmal geladen: { 'DE' => { 'code' => 'DE', 'name' => 'Deutschland', 'eu' => true }, ... } + laender = lookup container(:json, file: "#{DATA}/laender.json"), key: 'code', unique: true + + read container(:csv, file: "#{DATA}/kunden.csv", csv: { col_sep: ';' }) + + # fehlerhafte Zeilen mit Grund in eine eigene Datei, der Lauf geht weiter + on_error :collect, to: container(:csv, file: "#{out}/kunden_fehler.csv") + + # 1. prüfen + validate 'kundennr' => { required: true, format: /\A\d{6}\z/ }, + 'name' => { required: true, max: 40 }, + 'umsatz' => { type: :decimal } + validate { |row| "Land #{row['land']} unbekannt" unless laender.key?(row['land']) } + + # 2. nur aktive Kunden + filter { |row| row['status'] == 'aktiv' } + + # 3. anreichern und umbauen + map do |row| + land = laender.fetch(row['land']) + { + 'kundennr' => row['kundennr'], + 'name' => row['name'].strip, + 'land' => land['name'], + 'eu' => land['eu'] ? 'ja' : 'nein', + 'umsatz' => row['umsatz'].to_f.round(2) + } + end + + # 4. schreiben: alle Kunden nach SQLite, EU-Kunden zusätzlich als Excel-Liste + write container(:sqlite, database: "#{out}/kunden.sqlite3", table: 'kunden', create: true, truncate: true) + write container(:excel, file: "#{out}/eu_kunden.xlsx", sheet: 'EU-Kunden'), only: ->(row) { row['eu'] == 'ja' } + + on_finish { |stats| puts "kunden_import: #{stats}" } +end.run +``` + +Was die einzelnen Teile tun: + +| Zeile | Erklärung | +| --- | --- | +| `DataPipeline.params.fetch(:out, 'tmp')` | Parameter von der Kommandozeile: `--out=verzeichnis` | +| `lookup …, key: 'code', unique: true` | lädt die Länder einmal als Hash nach Ländercode; kein Zugriff je Zeile | +| `on_error :collect, to: …` | jede Zeile, die in einem Schritt scheitert, kommt mit `_step` und `_error` in die Fehlerdatei | +| `validate 'kundennr' => { … }` | Regeln je Spalte: `required`, `format`, `max`, `min`, `in`, `type` | +| `validate { \|row\| … }` | eigene Regel: ein Text (oder ein Array) ist ein Fehler, `nil` heisst in Ordnung | +| `filter` | inaktive Kunden fallen weg, ohne Fehler | +| `map` | baut die Zielzeile; `laender.fetch` kann nicht scheitern, weil `validate` das Land schon geprüft hat | +| `write …` zweimal | jedes Ziel bekommt die Zeilen; `only:` wählt die Zeilen für die Excel-Liste | +| `create: true, truncate: true` | SQLite legt die Tabelle beim ersten Schreiben an und leert sie; im Dry-Run passiert nichts | +| `on_finish` | läuft nach jedem Lauf, auch nach einem Abbruch; `stats` siehe 2.7 | + +Ergebnis von `bin/etl run examples/kunden_import.rb`: + +```text +INFO start kunden_import 1.0 +WARN rejected by validate#1: name: missing +WARN rejected by validate#1: kundennr: format /\A\d{6}\z/ +WARN rejected by validate#2: Land XX unbekannt +WARN rejected by validate#1: umsatz: not decimal +INFO finish read=9 written=4 rejected=4 0.097s +kunden_import: read=9 written=4 rejected=4 0.097s +``` + +| Datei | Inhalt | +| --- | --- | +| `tmp/kunden.sqlite3`, Tabelle `kunden` | 4 aktive, gültige Kunden (DE, AT, CH, IT) mit Ländername, `eu`, `umsatz` als Zahl | +| `tmp/eu_kunden.xlsx`, Blatt `EU-Kunden` | 3 Kunden (ohne CH) | +| `tmp/kunden_fehler.csv` | 4 Zeilen: leerer Name, Kundennummer mit 5 Stellen, Land `XX`, Umsatz `abc` – je mit `_step` und `_error` | + +Der Exit-Code ist `2`, weil Zeilen verworfen wurden; ein Abbruch ergäbe `1`, ein fehlerfreier Lauf `0`. Mit +`--dry-run` läuft alles gleich, es entstehen aber weder Datenbank noch Excel- oder Fehlerdatei; stattdessen stehen +die ersten Zeilen auf dem Bildschirm. + +### 1.3 Weitere Beispiele + +Alle Beispiele liegen in [examples/](examples/), arbeiten mit den Daten in `examples/data/` und schreiben nach `tmp/`. +`test/examples/doc_examples_test.rb` führt sie aus und prüft, dass der Code hier mit den Dateien übereinstimmt. + +**Datei mit Satzarten** – Rechnungen aus verschachteltem JSON werden zu einem Kopfsatz mit Positionssätzen; Datei- +und Schlusssatz zählen die Datensätze. Das Layout legt Reihenfolge, Typ und Länge der Felder je Satzart fest (2.9). + + +```ruby +# +# Rechnungen aus verschachteltem JSON in eine Datei mit Satzarten (Datei-, Kopf-, Positions-, Schlusssatz) +# +# bin/etl run examples/rechnungen_satzarten.rb [--out=tmp] +# +require 'fileutils' +require 'datapipeline' + +DATA = File.expand_path('data', __dir__) +out = DataPipeline.params.fetch(:out, 'tmp') +FileUtils.mkdir_p(out) + +# Feld: [Name, Typ, Länge, Pflicht]; Typ AN Text, N Zahl (11.2 = 9 Vor-, 2 Nachkommastellen), DAT YYYYMMDD +LAYOUT = DataPipeline::Layout.new do + record 'DATEI', [['SATYP', 'AN', 5, true], ['ERSTELLT', 'DAT', 8, true], ['ANZAHL', 'N', 6, true]] + record 'KOPF', [['SATYP', 'AN', 5, true], ['LFDNR', 'N', 6, true], ['BELNR', 'AN', 10, true], + ['DATUM', 'DAT', 8, true], ['KUNNR', 'AN', 10, true], ['SUMME', 'N', '11.2', true]] + record 'POS', [['SATYP', 'AN', 5, true], ['LFDNR', 'N', 6, true], ['BELNR', 'AN', 10, true], + ['POSNR', 'N', 3, true], ['TEXT', 'AN', 30], ['BETRAG', 'N', '11.2', true]] + record 'ENDE', [['SATYP', 'AN', 5, true], ['ANZAHL', 'N', 6, true]] +end + +DataPipeline.new do + read container(:json, file: "#{DATA}/rechnungen.json") + + # 1. eine Rechnung -> Kopfsatz + n Positionssätze + map do |rechnung| + positionen = rechnung['positionen'].each_with_index.map do |pos, i| + { 'satyp' => 'POS', 'belnr' => rechnung['nr'], 'posnr' => format('%03d', i + 1), + 'text' => pos['text'], 'betrag' => format('%.2f', pos['menge'] * pos['preis']) } + end + kopf = { 'satyp' => 'KOPF', 'belnr' => rechnung['nr'], 'datum' => rechnung['datum'].delete('-'), + 'kunnr' => rechnung['kunde'], 'summe' => format('%.2f', positionen.sum { |p| p['betrag'].to_f }) } + [kopf, *positionen] + end + + # 2. laufende Nummer über alle Kopf- und Positionssätze + map { |row, ctx| row.merge('lfdnr' => format('%06d', ctx.index + 1)) } + + # 3. gegen das Layout prüfen: zu langer Text wird gemeldet, die Zeile bleibt + validate LAYOUT, warn: true + + # Datei- und Schlusssatz kennen die Anzahl der Datensätze + header { |ctx| { 'satyp' => 'DATEI', 'erstellt' => Date.today.strftime('%Y%m%d'), 'anzahl' => format('%06d', ctx.written) } } + trailer { |ctx| { 'satyp' => 'ENDE', 'anzahl' => format('%06d', ctx.written) } } + + # Felder in der Reihenfolge des Layouts; Positionen zusätzlich als NDJSON + write container(:csv, file: "#{out}/rechnungen.txt", layout: LAYOUT, csv: { headers: false, col_sep: '|' }) + write container(:json, file: "#{out}/positionen.ndjson"), only: ->(row) { row['satyp'] == 'POS' } +end.run +``` + +Ergebnis `tmp/rechnungen.txt` (die Warnung meldet den Positionstext mit mehr als 30 Zeichen): + +```text +DATEI|20261008|000005 +KOPF|000001|R-1001|20251103|100001|380.50 +POS|000002|R-1001|001|Beratung|300.00 +POS|000003|R-1001|002|Reisekosten|80.50 +KOPF|000004|R-1002|20251105|100002|1200.00 +POS|000005|R-1002|001|Jahreslizenz Software mit Wartung und Support|1200.00 +ENDE|000005 +``` + +**Inkrementell und gestreamt** – nur Bewegungen, die seit dem letzten erfolgreichen Lauf dazugekommen sind; jede +Zeile wird sofort geschrieben, der Speicherbedarf bleibt auch bei Millionen Zeilen klein. + + +```ruby +# +# Nur neue Buchungsbewegungen exportieren, Zeile für Zeile (Streaming) +# Ein zweiter Lauf ohne neue Daten schreibt nichts. +# +# bin/etl run examples/bewegungen_inkrementell.rb [--out=tmp] +# +require 'fileutils' +require 'datapipeline' + +DATA = File.expand_path('data', __dir__) +out = DataPipeline.params.fetch(:out, 'tmp') +FileUtils.mkdir_p(out) + +DataPipeline.new do + streamed + + # höchste id des letzten erfolgreichen Laufs, gespeichert in tmp/.state.json + stand = watermark :bewegungen, column: 'id', file: "#{out}/.state.json", initial: 0 + + # bei einer Datenbank gehört der Filter ins SQL: where id > $1, params: [stand.value] + read container(:json, file: "#{DATA}/bewegungen.ndjson") + filter { |row| row['id'] > stand.value.to_i } + map { |row| row.merge('soll_haben' => row['betrag'].negative? ? 'H' : 'S', 'betrag' => row['betrag'].abs) } + + write container(:json, + file: "#{out}/bewegungen_#{Time.now.strftime('%Y%m%d_%H%M%S')}.ndjson" + ) + + on_finish { |stats| puts "neu: #{stats.written}, Stand jetzt: #{stand.current}" } +end.run +``` + +Der erste Lauf schreibt 4 Bewegungen und merkt sich `4`; der zweite schreibt nichts (`neu: 0`) und legt auch keine +Datei an. Bricht ein Lauf ab, bleibt der alte Stand stehen. + +**Eigenes Plugin** – eine Quelle, die alle CSV-Dateien eines Verzeichnisses liest. Ein Plugin braucht nur die +Methoden, die es anbietet (hier `read`); die Schnittstelle steht in 2.4. + + +```ruby +# +# Eigenes Plugin: liest alle CSV-Dateien eines Verzeichnisses als eine Quelle +# +# bin/etl run examples/eigenes_plugin.rb [--out=tmp] +# +require 'fileutils' +require 'datapipeline' + +DATA = File.expand_path('data', __dir__) +out = DataPipeline.params.fetch(:out, 'tmp') +FileUtils.mkdir_p(out) + +class VerzeichnisStorage < DataPipeline::StoragePlugin + def initialize dir:, pattern: '*.csv', **csv_options + @dir = dir + @pattern = pattern + @csv_options = csv_options + end + + # jede Zeile bekommt den Dateinamen dazu; mit Block zeilenweise (für streamed) + def read(&block) + rows = Dir.glob(File.join(@dir, @pattern)).sort.flat_map do |file| + DataPipeline::CsvStorage.new(file:, **@csv_options).read.map { |row| row.merge('datei' => File.basename(file, '.*')) } + end + return rows unless block + + rows.each(&block) + nil + end +end + +DataPipeline.register :verzeichnis, VerzeichnisStorage + +DataPipeline.new do + read container(:verzeichnis, dir: "#{DATA}/filialen") + map { |row| row.merge('menge' => row['menge'].to_i) } + write container(:csv, file: "#{out}/alle_filialen.csv", columns: %w[datei artikel menge]) +end.run +``` + +Ergebnis `tmp/alle_filialen.csv`: + +```text +datei,artikel,menge +vaduz,A-100,5 +vaduz,B-200,2 +zuerich,A-100,7 +zuerich,C-300,1 +``` + +Weitere Ausschnitte: REST-API nach Excel in 2.5 (REST ausführlich in 1.4), Benachrichtigung über Slack in 2.7. + +### 1.4 Beispiele mit REST-APIs + +Die drei Beispiele laufen gegen eine kleine Demo-API ([examples/demo_api.rb](examples/demo_api.rb)), die ohne Internet +und ohne weitere Gems auskommt. Sie verlangt ein Token, liefert Seiten und Folge-Links, nimmt Daten per POST und PUT an +und antwortet einmal absichtlich mit `503`, damit die Wiederholung sichtbar wird. + +```sh +ruby examples/demo_api.rb & # http://127.0.0.1:4567, Token demo-token +bin/etl run examples/rest_lesen.rb +bin/etl run examples/rest_schreiben.rb +bin/etl run examples/rest_zu_rest.rb +``` + +Gegen eine echte API ändern sich nur `--url=…` und das Token (`DEMO_API_TOKEN`, in der Praxis aus der `.env`). + +| Endpunkt der Demo-API | Antwort | +| --- | --- | +| `GET /api/kunden?page=1&per_page=3` | `{"data": [...], "page": 1}`; Seite 2 antwortet einmal mit `503` | +| `GET /api/laender` | `[{"code": "DE", "name": "Deutschland"}, ...]` | +| `GET /api/bestellungen?cursor=n` | `{"items": [...], "next": "/api/bestellungen?cursor=n+2"}`, auf der letzten Seite `"next": null` | +| `POST /api/import` | ein Objekt oder Array; `422`, wenn `kundennr` oder `name` fehlt | +| `PUT /api/archiv/` | legt das Objekt unter der id ab | + +**Lesen: seitenweise, mit Wiederholung und Nachschlagen** – `page_param` holt Seite für Seite, bis eine leer ist; +`root` zeigt auf das Array in der Antwort. Ein zweiter Container auf denselben Server dient mit `query` als Lookup. + + +```ruby +# +# Kunden seitenweise aus einer REST-API lesen, mit Ländern aus einem zweiten Endpunkt anreichern, +# nach CSV und JSON schreiben +# +# ruby examples/demo_api.rb & # Demo-API starten +# bin/etl run examples/rest_lesen.rb [--url=http://127.0.0.1:4567] [--out=tmp] +# +require 'fileutils' +require 'datapipeline' + +API = DataPipeline.params.fetch(:url, 'http://127.0.0.1:4567') +AUTH = "Bearer #{ENV.fetch('DEMO_API_TOKEN', 'demo-token')}" +out = DataPipeline.params.fetch(:out, 'tmp') +FileUtils.mkdir_p(out) + +DataPipeline.new do + version 'rest_lesen 1.0' + + # GET /api/kunden?page=1&per_page=3, dann page=2, ... bis eine Seite leer ist + # die Zeilen stehen in der Antwort unter "data"; 429/5xx und Netzfehler werden bis zu 3-mal wiederholt + kunden = container :rest, url: "#{API}/api/kunden", auth: AUTH, + root: 'data', page_param: 'page', params: { 'per_page' => 3 }, + retries: 3, retry_wait: 0.2 + + # derselbe Server, anderer Endpunkt: query hängt den Pfad an die url an (GET /api/laender) + stamm = container :rest, url: "#{API}/api", auth: AUTH + laender = lookup stamm, query: 'laender', key: 'code', unique: true + + read kunden + + on_error :collect, to: container(:csv, file: "#{out}/api_fehler.csv") + validate 'id' => { required: true }, 'umsatz' => { type: :decimal } + + map do |kunde| + { + 'id' => kunde['id'], + 'name' => kunde['name'], + 'land' => laender.dig(kunde['land'], 'name') || kunde['land'], + 'umsatz' => kunde['umsatz'].to_f + } + end + + write container(:csv, file: "#{out}/api_kunden.csv", columns: %w[id name land umsatz]) + write container(:json, file: "#{out}/api_kunden.json", pretty: true) + + on_finish { |stats| puts "rest_lesen: #{stats}" } +end.run +``` + +```text +INFO start rest_lesen 1.0 +WARN rejected by validate#1: umsatz: not decimal +INFO finish read=7 written=6 rejected=1 0.213s +``` + +Gelesen werden 7 Kunden in 3 Seiten (plus eine leere Seite als Ende); die `503` auf Seite 2 bemerkt man nur an der +Laufzeit. Kunde 5 hat den Umsatz `n/a` und steht in `tmp/api_fehler.csv`. Ergebnis `tmp/api_kunden.csv`: + +```text +id,name,land,umsatz +1,Müller GmbH,Deutschland,12500.5 +2,Huber & Partner,Österreich,8300.0 +3,Swiss Trade AG,Schweiz,22000.75 +4,Rossi Srl,Italien,4100.1 +6,Nordic AB,SE,990.0 +7,Vaduz Treuhand,Liechtenstein,15000.0 +``` + +Schweden fehlt in `/api/laender`, deshalb bleibt bei Kunde 6 der Code `SE` stehen. + +**Schreiben: prüfen, dann in Paketen senden** – `batch_size: 2` schickt je 2 Zeilen als JSON-Array. Was die API +ablehnen würde, wird vorher geprüft: Lehnt die API ein Paket trotzdem ab (`4xx`), bricht der Lauf ab, weil ein Fehler +beim Schreiben nicht einer einzelnen Zeile zugeordnet werden kann. + + +```ruby +# +# Kunden aus CSV prüfen und in Paketen per POST an eine REST-API senden +# +# ruby examples/demo_api.rb & +# bin/etl run examples/rest_schreiben.rb [--url=http://127.0.0.1:4567] [--dry-run] +# +require 'fileutils' +require 'datapipeline' + +API = DataPipeline.params.fetch(:url, 'http://127.0.0.1:4567') +AUTH = "Bearer #{ENV.fetch('DEMO_API_TOKEN', 'demo-token')}" +DATA = File.expand_path('data', __dir__) +out = DataPipeline.params.fetch(:out, 'tmp') +FileUtils.mkdir_p(out) + +DataPipeline.new do + version 'rest_schreiben 1.0' + + read container(:csv, file: "#{DATA}/kunden.csv", csv: { col_sep: ';' }) + + # vor dem Senden prüfen: was die API ablehnen würde, landet hier in der Fehlerdatei + on_error :collect, to: container(:csv, file: "#{out}/import_fehler.csv") + validate 'kundennr' => { required: true, format: /\A\d{6}\z/ }, 'name' => { required: true } + + filter { |row| row['status'] == 'aktiv' } + map do |row| + { 'kundennr' => row['kundennr'], 'name' => row['name'].strip, 'land' => row['land'], 'umsatz' => row['umsatz'].to_f } + end + + # POST /api/import mit je 2 Kunden als JSON-Array; im Dry-Run wird nichts gesendet + write container(:rest, url: "#{API}/api/import", auth: AUTH, batch_size: 2) + + on_finish do |stats| + puts stats.dry_run ? 'Dry-Run: nichts gesendet' : "gesendet: #{stats.written} Kunden in #{(stats.written / 2.0).ceil} Paketen" + end +end.run +``` + +Ergebnis: 6 aktive Kunden in 3 Anfragen an `POST /api/import`, 2 Kunden in `tmp/import_fehler.csv` (leerer Name, +Kundennummer mit 5 Stellen). Mit `--dry-run` wird nichts gesendet; auf dem Bildschirm stehen die Daten, die gesendet +würden: + +```text +DRY RUN – nothing written: read=9 written=6 rejected=2 0.0s (dry run) +{"kundennr":"100001","name":"Müller GmbH","land":"DE","umsatz":12500.5} +... +``` + +**Von API zu API: Folge-Links, Streaming, eine URL je Datensatz** – Die Antwort nennt die nächste Seite selbst; das +Lambda in `next:` macht daraus eine vollständige URL. `write_url` als Lambda schickt jede Bestellung an ihre eigene +Adresse. Mit dem Wasserzeichen archiviert ein zweiter Lauf nur, was neu dazugekommen ist. + + +```ruby +# +# Von API zu API: neue Bestellungen über Folge-Links lesen und einzeln per PUT archivieren +# Ein zweiter Lauf ohne neue Bestellungen sendet nichts. +# +# ruby examples/demo_api.rb & +# bin/etl run examples/rest_zu_rest.rb [--url=http://127.0.0.1:4567] [--out=tmp] +# +require 'fileutils' +require 'datapipeline' + +API = DataPipeline.params.fetch(:url, 'http://127.0.0.1:4567') +AUTH = "Bearer #{ENV.fetch('DEMO_API_TOKEN', 'demo-token')}" +out = DataPipeline.params.fetch(:out, 'tmp') +FileUtils.mkdir_p(out) + +DataPipeline.new do + streamed + + # höchste Bestellnummer des letzten erfolgreichen Laufs + stand = watermark :bestellungen, column: 'id', file: "#{out}/.state.json", initial: 0 + + # die Antwort nennt die nächste Seite selbst: {"items": [...], "next": "/api/bestellungen?cursor=2"} + read container(:rest, url: "#{API}/api/bestellungen", auth: AUTH, root: 'items', + next: ->(body) { body['next'] && "#{API}#{body['next']}" }) + + filter { |row| row['id'] > stand.value.to_i } + map { |row| row.merge('archiviert' => Date.today.iso8601) } + + # eine url je Bestellung: PUT /api/archiv/ + write container(:rest, url: "#{API}/api/archiv", auth: AUTH, write_method: :put, + write_url: ->(row) { "#{API}/api/archiv/#{row['id']}" }) + + on_finish { |stats| puts "archiviert: #{stats.written}, Stand jetzt: #{stand.current}" } +end.run +``` + +```text +archiviert: 5, Stand jetzt: 5 # erster Lauf: PUT /api/archiv/1 … /5 +archiviert: 0, Stand jetzt: 5 # zweiter Lauf: keine Anfrage +``` + + +--- + +## 2. Technische Dokumentation + +### 2.1 Aufbau + +```mermaid +flowchart LR + subgraph Pipeline-Skript + DSL[DataPipeline.new do ... end] + end + DSL -->|container :typ| REG[Registry / autoload
<Typ>Storage] + REG --> SP[DataPipeline::StoragePlugin
read · write · write_row · query · close] + SP --- PL[postgres · sqlite · csv · json
excel · rest · eigene] + DSL -->|lookup| LK[(Nachschlage-Hash)] + SRC((Quelle)) --> STEPS[Schritte
validate · filter · map · transform] + LK -.-> STEPS + STEPS --> EDGE[header · trailer
ctx.read · ctx.written] + EDGE --> DST1((Ziel 1)) + EDGE -->|only:| DST2((Ziel 2)) + STEPS -. on_error .-> REJ((Fehlerdatei)) + DSL -->|run| ST[Stats · on_finish
Exit-Code über bin/etl] +``` + +| Datei | Inhalt | +| --- | --- | +| [lib/datapipeline.rb](lib/datapipeline.rb) | DSL und Ausführung (`run`), `Stats`, `Context` | +| [lib/datapipeline/validation.rb](lib/datapipeline/validation.rb) | Regeln für `validate` | +| [lib/datapipeline/watermark.rb](lib/datapipeline/watermark.rb) | Wasserzeichen für inkrementelle Läufe | +| [lib/datapipeline/layout.rb](lib/datapipeline/layout.rb) | Satzarten-Layouts (Feldliste je Satzart, siehe 2.9) | +| [exe/etl](exe/etl) | Kommando `etl` des Gems mit Exit-Codes (siehe 2.7); [bin/etl](bin/etl) startet es im Repository mit dem Bundle | +| [datapipeline.gemspec](datapipeline.gemspec), [CHANGELOG.md](CHANGELOG.md) | Gem-Beschreibung, Versionsverlauf | +| [examples/](examples/) | Beispiele aus Abschnitt 1 mit Daten in `examples/data/`, Demo-API für die REST-Beispiele | +| [lib/datapipeline/storage/storage_plugin.rb](lib/datapipeline/storage/storage_plugin.rb) | Basisklasse der Storage-Plugins | +| [lib/datapipeline/storage/postgres.rb](lib/datapipeline/storage/postgres.rb) | Quelle/Ziel PostgreSQL (`pg`) | +| [lib/datapipeline/storage/csv.rb](lib/datapipeline/storage/csv.rb) | Quelle/Ziel CSV | +| [lib/datapipeline/storage/json.rb](lib/datapipeline/storage/json.rb) | Quelle/Ziel JSON und NDJSON | +| [lib/datapipeline/storage/sqlite.rb](lib/datapipeline/storage/sqlite.rb) | Quelle/Ziel SQLite (`sqlite3`) | +| [lib/datapipeline/storage/excel.rb](lib/datapipeline/storage/excel.rb) | Quelle/Ziel Excel xlsx (`roo`, `caxlsx`) | +| [lib/datapipeline/storage/rest.rb](lib/datapipeline/storage/rest.rb) | Quelle/Ziel REST-API (`http`) | +| [lib/datapipeline/notification/slack.rb](lib/datapipeline/notification/slack.rb) | Benachrichtigung, z. B. aus `on_finish` | +| [lib/datapipeline/version.rb](lib/datapipeline/version.rb) | `DataPipeline::VERSION` | + +### 2.2 DSL-Referenz + +| Aufruf | Wirkung | +| --- | --- | +| `DataPipeline.settings(opts)` | Defaults für alle folgenden Pipelines; der Hash wird ersetzt, nicht verändert (laufende Pipelines behalten ihre Kopie); `reset_settings!` setzt zurück | +| `DataPipeline.register(:typ, Klasse \| 'Klassenname', path:)` | eigenes Plugin; mit `path:` erst bei der ersten Nutzung geladen | +| `DataPipeline.new(**settings) { ... }` | führt den Block per `instance_eval` aus; `settings` gelten nur für diese Pipeline, z. B. `new(limit: 10)` | +| `configure(opts)` | Einstellungen dieser Pipeline als Hash: `verbose`, `streamed`, `chained`, `version` | +| `version(str)`, `verbose(flag = true)`, `streamed(flag = true)`, `chained(flag = true)` | dasselbe als DSL-Aufruf | +| `container(:typ, **opts)` | erzeugt das registrierte Plugin, sonst `Storage.new(**opts)` (z. B. `:postgres` → `DataPipeline::PostgresStorage`); fehlt es: „Unsupported storage type“; fehlende Gems, Verbindungs- und Optionsfehler kommen unverändert durch | +| `read(storage, opts = {})` | legt die Quelle fest; `opts` gehen an `storage.set` | +| `transform(opts = {}) { \|row, index\| ... }` | Transformation; `processing: :prepend \| :normal \| :append`, `condition:` (siehe 2.3) | +| `map(opts = {}) { \|row, ctx\| ... }` | Schritt in der Kette: Hash, Array oder `nil` zurückgeben; schaltet Verkettung ein | +| `filter(opts = {}) { \|row, ctx\| ... }` | Schritt in der Kette: Zeile bleibt, wenn der Block wahr ist; schaltet Verkettung ein | +| `header { \|ctx\| ... }` | Zeile(n) vor den Daten; wird nach den Daten berechnet, kennt also `ctx.written` | +| `trailer { \|ctx\| ... }` | Zeile(n) nach den Daten | +| `query(storage, query: sql, params: [])` | führt sofort eine Abfrage aus (Subquery in einem Schritt) | +| `lookup(storage, key:, unique: false, query:, params:)` | lädt eine Tabelle einmal als Hash: `key => [Zeilen]`, mit `unique: true` `key => Zeile`; `key` als Spaltenname oder Lambda; ohne `query:` wird `storage.read` genutzt | +| `write(storage, only: ->(row) { ... })` | Ziel; mehrfach möglich, jedes Ziel bekommt alle Zeilen oder mit `only:` die passenden (auch Kopf- und Schlusssätze) | +| `validate(rules \| layout, warn: false, ignore: []) { \|row\| ... }` | Prüfschritt (siehe 2.7, 2.9); `warn: true` nur Warnung, Zeile bleibt; schaltet Verkettung ein | +| `on_error(:abort \| :skip \| :collect, to: storage)` | Umgang mit Fehlern in Datenschritten (siehe 2.7) | +| `on_finish { \|stats\| ... }` | nach jedem Lauf, auch nach Abbruch | +| `watermark(name, column:, file:, initial:)` | letzter verarbeiteter Wert für inkrementelle Läufe (siehe 2.7) | +| `dry_run(flag = true)`, `limit(n)` | nichts schreiben bzw. nur die ersten n Quellzeilen | +| `dry_run?` | in Schritten mit Nebenwirkungen (API-Aufrufe): `true` im Dry-Run | +| `logger(logger)` | eigener Ruby-`Logger`; Default: Warnungen auf `$stderr` | +| `params` | Parameter von der Kommandozeile: über `bin/etl` oder direkt aus `ARGV` (`--name=wert`, `--param key=wert`) | +| `run` | führt die Pipeline aus und liefert `DataPipeline::Stats` | +| jeder andere Aufruf | `NoMethodError` (Tippfehler fallen sofort auf) | + +### 2.3 Ausführungsmodell (`run`) + +1. Batch (Default): `data = source.read` lädt die gesamte Quelle in den Speicher. Streaming: siehe unten. +2. Die Schritte werden in drei Gruppen geteilt, innerhalb jeder Gruppe gilt die Reihenfolge der Registrierung: + **vorne** (`:prepend`, `header`), **Daten** (`:normal`, `map`, `filter`), **hinten** (`:append`, `trailer`). +3. Die Daten-Schritte laufen in einem von zwei Modi: + + | Modus | Wann | Eingabe eines Schritts | + | --- | --- | --- | + | parallel | Default | immer eine eigene Kopie der Quellzeilen; die Ergebnisse aller Schritte werden hintereinander angehängt | + | verkettet | `chained`, oder sobald `map`/`filter` vorkommt, oder ganz ohne Schritte | die Ausgabe des vorherigen Schritts | + + Beispiel mit 2 Quellzeilen und `transform { |r| r.merge('b'=>1) }` + `transform { |r| r.merge('c'=>2) }`: + parallel 4 Zeilen (2× mit `b`, dann 2× mit `c`), verkettet 2 Zeilen mit `b` und `c`. +4. Rückgabe eines Blocks: ein Hash ergibt eine Zeile, ein Array mehrere Zeilen (1:n, z. B. Kopf + Positionen), + `nil` oder `[]` keine Zeile; `nil` innerhalb eines Arrays wird übersprungen. +5. `condition:` an `transform` oder `map`: String (per `eval`, Variable `row` sichtbar) oder Lambda `->(row) { ... }`. + Zeilen, die die Bedingung nicht erfüllen, fallen in diesem Schritt weg. +6. Vorne und hinten: + + | Schritt | Aufruf | Zeitpunkt | + | --- | --- | --- | + | `transform(processing: :prepend)` | `(nil, 0)` | vor den Daten | + | `header` | `(ctx)` mit `read`, `written` | nach den Daten, steht aber vorne | + | `transform(processing: :append)` | `(nil, n)`: parallel n = Quellzeilen, verkettet n = Datenzeilen | nach den Daten | + | `trailer` | `(ctx)` mit `read`, `written` | nach den Daten | + +7. Ohne Schritte wird die Quelle unverändert ins Ziel kopiert. +8. Batch: `target.write(ausgabe)` schreibt alles auf einmal. + +Der Kontext `ctx` (`DataPipeline::Context`) enthält `read` (Anzahl Quellzeilen), `written` (Anzahl Datenzeilen nach allen +Schritten) und bei `map`/`filter` `index` (Position der Zeile im aktuellen Schritt). + +Mit `verbose` werden im Batch Quelle, Daten nach allen Schritten und Ziel als JSON ausgegeben. + +Nach dem Lauf schliesst `run` Quelle und Ziel (`close`), auch im Fehlerfall. + +**Streaming-Modus** (`streamed`): Jede Zeile geht einzeln von `source.read { |row| ... }` durch die Kette und sofort +an `target.write_row`; am Ende `target.finalize`. Der Speicherbedarf bleibt konstant (getestet: 500 000 CSV-Zeilen +mit 30 MB statt 614 MB im Batch). Die Datenzeilen sind dieselben wie im Batch, mit diesen Unterschieden: + +| Thema | Batch | Streaming | +| --- | --- | --- | +| `header` | kennt `read` und `written` | wird zuerst geschrieben, `read` und `written` sind `nil` | +| `ctx.read` in `map`/`filter` | Anzahl Quellzeilen | `nil` | +| mehrere `transform` im Parallelmodus | erlaubt | `ArgumentError`, dafür `chained` setzen | +| Postgres als Ziel | eine Transaktion für alles | je Zeile ein `INSERT` ohne gemeinsame Transaktion | + +Postgres als Quelle liest im Streaming über einen serverseitigen Cursor (`fetch_size`, Default 1000). Abfragen und +Lookups auf derselben Verbindung bleiben währenddessen möglich. + +### 2.4 Plugin-Schnittstelle + +Ein Storage-Plugin ist eine Klasse `Storage < DataPipeline::StoragePlugin` mit: + +| Methode | Pflicht | Verwendung | +| --- | --- | --- | +| `initialize(**opts, &block)` | ja | Verbindung/Datei vorbereiten | +| `read` → `Array>` | als Quelle | Batch-Lesen | +| `read { \|row\| }` | als Quelle | zeilenweises Lesen | +| `write(rows)` | als Ziel | Batch-Schreiben | +| `write_row(row)` | als Ziel | zeilenweises Schreiben | +| `query(query: sql, params: [])` → `Array` | optional | Ad-hoc-Abfragen aus Transformationen | +| `set(opts)` | Basisklasse | Optionen nachreichen, wird von `read(storage, opts)` genutzt | +| `close` / `finalize` | Basisklasse | Verbindung/Datei freigeben; `run` ruft `close` für Quelle und Ziel am Ende auf | + +Alle Plugins liefern Zeilen mit **String-Schlüsseln**. Die mitgelieferten Plugins liegen im Namensraum `DataPipeline` (`DataPipeline::CsvStorage` usw.) und werden per `autoload` erst bei der ersten Nutzung geladen. Neues Plugin: Klasse `Storage < DataPipeline::StoragePlugin` in der Anwendung schreiben und mit `DataPipeline.register :name, NameStorage` (oder `'NameStorage', path: 'datei.rb'` für Laden bei Bedarf) bekannt machen – danach ist `container :name` verfügbar. Ohne `register` findet `container :name` eine Klasse `Storage` auf oberster Ebene. + +### 2.5 Mitgelieferte Plugins + +**DataPipeline::PostgresStorage** – Optionen `host`, `port` (Default 5432), `database`, `username`, `password`, `query`, `params`, +`table`, `typed`, `fetch_size`, `truncate`. +- `read` / `query` mit Platzhaltern `$1, $2, …` und `params: [...]`; Ergebnis als `Hash` + (`NULL` = `nil`), mit `typed: true` als Ruby-Typen (Integer, Float, Date, …). +- `read { |row| }` liest über einen Cursor in einer Transaktion, je `fetch_size` Zeilen (Default 1000). +- `write(rows)` schreibt in `table:` (oder alte Form `write(table, rows)`), alle Zeilen in **einer Transaktion**, + Werte als Bind-Parameter, `nil` = `NULL`, Tabellen-/Spaltennamen gequotet (`schema.tabelle` erlaubt). +- `truncate: true` leert die Tabelle vor dem ersten Schreiben (bei `write` in derselben Transaktion). + +```ruby +pg = container :postgres, host: 'localhost', database: 'crm', username: 'etl', table: 'kunde_import' +read pg, query: 'select * from kunde where geaendert >= $1', params: [Date.today - 7] +``` + +**DataPipeline::CsvStorage** – Optionen `file` (Pflicht), `csv:` (an Rubys `CSV`, gemergt mit den Defaults `headers: true`, +`col_sep: ','`), `columns`, `counter`, `verbose`. +- `columns: %w[a b c]` legt die Spaltenreihenfolge fest: fehlende Felder bleiben leer, String- und Symbol-Schlüssel + werden gefunden, die Kopfzeile kommt aus `columns`. Ohne `columns` gilt die Hash-Reihenfolge (nötig für Dateien mit + mehreren Satzarten wie CUDO). +- Lesen ohne Kopfzeile (`headers: false`): Schlüssel aus `columns`, sonst `"0"`, `"1"`, … +- Leere Daten: ohne `columns` leere Datei, mit `columns` nur die Kopfzeile. +- `layout: DataPipeline::Layout` schreibt Dateien mit mehreren Satzarten in der Feldreihenfolge je Satzart, ohne + Kopfzeile; ein Feld, das nicht im Layout steht, bricht ab (siehe 2.9). + +Übersicht der übrigen Plugins (alle mit `read`, `read { }`, `write`, `write_row`, `close`): + +| `container` | Klasse | Gem (bei Nutzung geladen) | Wichtige Optionen | +| --- | --- | --- | --- | +| `:json` | `DataPipeline::JsonStorage` | – | `file`, `format: :array \| :lines` (Default aus Endung: `.ndjson`/`.jsonl` → Zeilen), `root: 'data.items'`, `pretty`; ein einzelnes Objekt wird als eine Zeile gelesen | +| `:sqlite`, `:sqlite3` | `DataPipeline::SqliteStorage` | `sqlite3` | `database` (Datei oder `':memory:'`), `query`, `params` (Array für `?`, Hash für `:name`), `table`, `create`, `truncate` | +| `:excel`, `:xlsx` | `DataPipeline::ExcelStorage` | `roo` (lesen), `caxlsx` (schreiben) | `file`, `sheet` (Name/Index), `headers`, `columns`, `skip` (Titelzeilen) | +| `:rest` | `DataPipeline::RestStorage` | `http` | `url`, `auth`, `headers`, `params`, `root`, Paging `page_param` oder `next: ->(body) { … }`, `retries`, `timeout`, `batch_size`, `write_method`, `write_url` (auch als Lambda je Zeile), `ssl_verify` | + +**DataPipeline::JsonStorage** – NDJSON wird zeilenweise gelesen und geschrieben. Ein JSON-Array wird beim Streaming als +`[`…`]` geschrieben und in `close` abgeschlossen, die Datei ist also auch beim Streaming gültiges JSON. + +**DataPipeline::SqliteStorage** – Verhält sich wie Postgres: Bind-Parameter, Schreiben in einer Transaktion, Tabellen- und +Spaltennamen gequotet, `nil` = `NULL`, Datum als ISO-Text. Werte kommen mit SQLite-Typen zurück (Integer, Float, String). +`create: true` legt die Tabelle beim ersten Schreiben aus der ersten Zeile an (Integer → `integer`, Float → `real`, +sonst `text`), `truncate: true` leert sie vorher. Die Datei wird erst bei der ersten Nutzung geöffnet; ein Dry-Run +legt keine an. + +**DataPipeline::ExcelStorage** – Lesen von xlsx zeilenweise (`each_row_streaming`), leere Zeilen werden übersprungen. Beim +Schreiben bleibt Text immer Text: `'0000000001'` behält die Nullen, `'=1+1'` wird keine Formel. Zahlen und Datumswerte +behalten ihren Typ. Die Datei entsteht in `close` (caxlsx baut sie im Speicher auf). + +**DataPipeline::RestStorage** – Liest JSON, der Pfad zum Array kommt aus `root`. Paging über eine Seitennummer (bis eine Seite +leer ist) oder über eine Folge-URL aus der Antwort; `max_pages` (Default 1000) stoppt endlose Schleifen. Bei 429, 5xx +und Netzwerkfehlern wird `retries`-mal wiederholt (Wartezeit `retry_wait` × Versuch), 4xx bricht sofort mit +`DataPipeline::RestStorage::Error` ab. Schreiben je Zeile oder mit `batch_size` als JSON-Array; `write_url: ->(row) { … }` baut +je Zeile eine eigene URL (z. B. `PUT /kunden/`, nicht zusammen mit `batch_size`). `query(query: 'pfad', params:)` +hängt den Pfad an `url` an und erlaubt `lookup` gegen die API. Ein Fehler beim Schreiben bricht den Lauf ab (siehe 1.4). + +Für APIs im RPC-Stil (Suchen und Anlegen per POST mit Body, z. B. die pds API) gibt es einzelne Aufrufe: +`call(:post, 'person/listkunden', json: { … })` liefert das geparste JSON. Mit `retries: 0` wird ein anlegender Aufruf +bei `5xx` nicht wiederholt. Ein Fehler wirft `DataPipeline::RestStorage::Error` mit `status` (HTTP-Code, `nil` bei Netzwerkfehlern). +`ssl_verify: false` akzeptiert jedes Server-Zertifikat (eigene CA, Zugriff über IP-Adresse). + +```ruby +api = container :rest, url: 'https://api.example.com/v1/kunden', auth: "Bearer #{ENV['API_TOKEN']}", + root: 'data', page_param: 'page' +xlsx = container :excel, file: 'data/kunden.xlsx', sheet: 'Kunden', columns: %w[id name land] +read api +map { |r| r.slice('id', 'name', 'land') } +write xlsx +``` + +### 2.6 Abhängigkeiten + +Das Gem selbst braucht nur `csv` und `logger` (Ruby `>= 3.4`). Die Gems der Plugins sind optional und gehören ins +Gemfile der Anwendung, die das Plugin nutzt: + +| Plugin | Gem | +| --- | --- | +| `:postgres` | `pg` | +| `:sqlite` | `sqlite3` | +| `:excel` | `roo` (lesen), `caxlsx` (schreiben) | +| `:rest`, `DataPipeline::Slack` | `http` | +| `:csv`, `:json` | – | + +Im Repository bindet das `Gemfile` das Gem über `gemspec` ein und ergänzt die Gruppe `plugins` (`pg`, `sqlite3`, +`roo`, `caxlsx`, `http`) für die Plugin-Tests und die Gruppe `test` (`minitest`, `rake`). `Gemfile.lock` ist versioniert. +Gem bauen: `gem build datapipeline.gemspec`. + +Plugins und ihre Gems werden erst bei der ersten Nutzung geladen (`autoload` und `DataPipeline::StoragePlugin.require_gem`): eine +CSV- oder JSON-Pipeline lädt keines dieser Gems. Wer ein Plugin nicht braucht, lässt die Gruppe weg +(`bundle config set without plugins`); bei Nutzung meldet das Plugin dann, welches Gem fehlt. + +### 2.7 Betrieb + +**Fehler je Zeile** – `on_error` legt fest, was passiert, wenn ein Datenschritt (`transform`, `map`, `filter`, +`validate`) für eine Zeile eine Exception wirft. Fehler in `header`, `trailer`, prepend und append brechen immer ab. + +| Modus | Wirkung | +| --- | --- | +| `:abort` (Default) | Lauf bricht ab, Exception wird weitergereicht | +| `:skip` | Zeile fällt weg, Warnung im Log, Zähler `rejected` | +| `:collect, to: storage` | wie `:skip`, zusätzlich die Eingangszeile mit `_step` und `_error` per `write_row` ins Ziel `storage` | + +**Validierung** – `validate` prüft Spalten nach Regeln, nach einem Layout (2.9) und/oder mit einem Block. Eine +verletzte Regel löst `DataPipeline::ValidationError` aus und läuft damit über `on_error`. Leere Werte prüft nur +`required`. Mit `warn: true` bleibt die Zeile drin, es gibt eine Warnung und `stats.warnings` zählt mit (ins Log +kommen höchstens 100); `ignore: %w[FELD]` lässt bekannte Lücken aus. + +```ruby +on_error :collect, to: container(:csv, file: 'data/fehler.csv', columns: %w[belnr konto dmbtr _step _error]) +validate 'konto' => { required: true, format: /\A\d+\z/ }, + 'shkzg' => { in: %w[S H] }, + 'dmbtr' => { type: :decimal }, + 'name1' => { max: 35 } +validate { |row| 'Betrag 0' if row['dmbtr'].to_f.zero? } +``` + +**Ergebnis, Log, Hooks** – `run` liefert `DataPipeline::Stats`: + +| Feld | Inhalt | +| --- | --- | +| `read` | gelesene Quellzeilen | +| `written` | Datenzeilen nach allen Schritten (ohne Kopf- und Schlusssätze) | +| `output` | an die Ziele gegebene Zeilen inkl. Kopf- und Schlusssätzen | +| `rejected`, `errors` | verworfene Zeilen; die ersten 100 mit Schritt, Meldung und Zeile | +| `warnings` | Zeilen mit Warnung aus `validate …, warn: true` | +| `duration`, `started_at`, `finished_at` | Laufzeit | +| `error` | Exception eines abgebrochenen Laufs | +| `ok?` | kein Abbruch und nichts verworfen | +| `to_s` | `read=9 written=4 rejected=4 0.097s` | + +Der Logger schreibt `start` und `finish` (Info), jede verworfene Zeile (Warnung) und einen Abbruch (Fehler). +`on_finish` läuft nach jedem Lauf, auch nach einem Abbruch; ein Fehler im Hook wird nur geloggt. + +```ruby +on_finish { |stats| DataPipeline::Slack.new(channel: '#etl').send_message("Buchungen: #{stats}") unless stats.ok? } +``` + +**Inkrementelle Läufe** – `watermark` merkt sich je Name den höchsten Wert einer Spalte in einer JSON-Datei. Die +Pipeline verfolgt jede Quellzeile und speichert den neuen Wert nur nach einem erfolgreichen Lauf (nicht nach Abbruch, +nicht im Dry-Run). Zahlen als Text werden numerisch verglichen. Mit `limit` sollte die Quelle nach der Spalte +sortiert sein; verworfene Zeilen schieben das Wasserzeichen mit (sie liegen bei `:collect` in der Fehlerdatei). + +```ruby +wm = watermark :rechnungen, column: 'rechnung_id', file: 'data/.state.json', initial: 0 +read pg, query: 'select * from rechnung where rechnung_id > $1 order by rechnung_id', params: [wm.value] +``` + +**Dry-Run und Limit** – Im Dry-Run läuft alles, aber nichts wird geschrieben: kein Ziel, keine Fehlerdatei, kein +Wasserzeichen; eine CSV-Datei wird nicht angelegt. Am Ende stehen die Zusammenfassung und die ersten Zeilen +(`preview`, Default 10) als JSON auf `$stdout`. `limit n` verarbeitet nur die ersten n Quellzeilen, im Streaming +wird das Lesen dann beendet. + +**Kommandozeile** – `bin/etl` startet ein Pipeline-Skript; `run` ist optional (`bin/etl skript.rb` genügt): + +```sh +bin/etl run export.rb --from=2025-11-01 --till=2025-11-30 +bin/etl run examples/kunden_import.rb --dry-run --limit=20 +bin/etl run pipeline.rb --param land=DE # im Skript: params[:land] +``` + +| Option | Wirkung | +| --- | --- | +| `--from`, `--till` | in `params[:from]`/`params[:till]` und unverändert an das Skript weitergereicht (eigener `OptionParser` der Importer) | +| `--param KEY=VALUE` | `params[:key]`, mehrfach möglich | +| `--dry-run`, `--limit=N` | für alle Pipelines des Skripts | +| `--verbose`, `--quiet` | Log-Level Debug bzw. Warnung (Default über `bin/etl`: Info) | +| jede andere `--name=wert` | `params[:name]` und unverändert an das Skript, z. B. `--out=data`, `--incremental=1` | + +Exit-Code: `0` ok, `1` Abbruch, `2` fertig, aber mit verworfenen Zeilen. Das `Rakefile` ruft `bin/etl` auf: +`rake process:posting[2025-11-01,2025-11-30]`, `rake process:all[2025-11-01,2025-11-30,dry]`. Ein fehlgeschlagener Lauf stoppt Rake. + +### 2.8 Tests + +`rake test` (oder `bundle exec rake`) startet die Minitest-Suite in [test/](test/): rund 165 Fälle für Plugins, Kern, +Streaming, Betrieb, Kommandozeile und die Beispiele. + +| Ordner | Inhalt | +| --- | --- | +| `test/plugins` | je Plugin das gemeinsame Testkit `test/support/storage_contract.rb` (Schreiben und Zurücklesen, zeilenweise, als Quelle und Ziel einer Pipeline) plus eigene Fälle; REST gegen einen lokalen Testserver; Laden der Gems erst bei Nutzung in einem frischen Prozess | +| `test/core` | DSL, Einstellungen, Plugin-Registry, Parallel- und Verkettungsmodus, Streaming, Lookups, mehrere Ziele, Betrieb; `compat_test.rb` hält die Ausgabe des `transform`-Modells von v1.2.1 fest | +| `test/cli` | `bin/etl` als Prozess: Weitergabe von `--from`/`--till`, Parameter, Dry-Run, Exit-Codes | +| `test/examples` | die Beispiele aus Abschnitt 1 (Code in der Doku = Datei, Lauf und Ergebnis; REST gegen die Demo-API) | + +Die `.env` wird in Tests nicht geladen; kein Test erreicht die Produktivdatenbank. Postgres-Tests legen je Test eine +Wegwerf-Datenbank auf einem lokalen Server an (`ETL_TEST_PG_HOST`, `_PORT`, `_USER`, `_PASSWORD`; Default +`localhost`, `$USER`). Ohne erreichbaren Server werden sie übersprungen (21 Tests). Ein eigenes Plugin lässt sich mit +`MemoryStorage` aus `test/test_helper.rb` als Quelle oder Ziel testen; mit `include StorageContract` und einer +Methode `contract_storage(env, :write | :read)` bekommt es die gemeinsamen Tests. + +### 2.9 Satzarten-Layouts + +Dateien mit mehreren Satzarten (Kopf, Positionen, Schluss) haben je Satzart eine feste Feldliste. `DataPipeline::Layout` +beschreibt sie; die Satzart einer Zeile ist ihr erster Wert (oder `type_field:`), Schlüssel werden ohne Rücksicht auf +Gross-/Kleinschreibung zugeordnet. + +```ruby +LAYOUT = DataPipeline::Layout.new do + record 'DOHD', [['SATYP', 'AN', 4, true], ['LFDNR', 'N', 10, true], ['BKTXT', 'AN', 25, true], ...] + record 'LARC', [['SATYP', 'AN', 4, true], ['SUMME', 'N', 10, true]] +end + +validate LAYOUT, warn: true # Pflicht, Länge, Format je Feld +write container(:csv, file: 'x.ok', layout: LAYOUT, csv: { headers: false, col_sep: '|' }) +``` + +| Typ | Prüfung | +| --- | --- | +| `AN` | Länge als Text | +| `N` | Zahl; Länge `10` = Stellen, `13.2` = 11 Vor- und 2 Nachkommastellen | +| `DAT` | `YYYYMMDD`, gültiges Datum | +| `TIM` | `HHMMSS` | + +Beim Schreiben stehen die Werte in der Reihenfolge des Layouts, fehlende Felder bleiben leer, ein unbekanntes Feld +bricht mit `DataPipeline::LayoutError` ab. Eine falsch benannte SQL-Spalte verschiebt also nicht mehr still alle +folgenden Felder. Ein vollständiges Beispiel sind die CUDO-Satzarten der SAP-Schnittstelle im Projekt `etl` (`cudo/layout.rb`), +erzeugt aus der Schnittstellenbeschreibung. + +--- + +## 3. Features und Verbesserungen + +Ziel: aus dem Exportwerkzeug ein generisches, testbares ETL-Framework machen. Reihenfolge = empfohlene Umsetzung. + +### 3.1 Erweiterbarkeit + +1. **Weitere Plugins:** Festbreiten-Format (feste Spaltenbreiten, nutzt die Längen aus dem Layout), SFTP-Ablage. +2. **Veröffentlichung:** eigenes Remote-Repository für das Gem, Lizenz festlegen, Versionen taggen. diff --git a/Rakefile b/Rakefile new file mode 100644 index 0000000..39ddc08 --- /dev/null +++ b/Rakefile @@ -0,0 +1,11 @@ +require 'rake/testtask' + +# rake test all tests (postgres tests need a local server, see test/test_helper.rb) +# rake test TEST=test/core/chained_test.rb +Rake::TestTask.new(:test) do |t| + t.libs << 'lib' << 'test' + t.test_files = FileList['test/**/*_test.rb'] + t.warning = false +end + +task default: :test diff --git a/bin/etl b/bin/etl new file mode 100755 index 0000000..c1723ce --- /dev/null +++ b/bin/etl @@ -0,0 +1,7 @@ +#!/usr/bin/env ruby +# +# binstub for this repository: loads the bundle (optional plugin gems) and runs the gem executable +# exe/etl. Installed as a gem, the command is just `etl`. +# +require 'bundler/setup' +load File.expand_path('../exe/etl', __dir__) diff --git a/datapipeline.gemspec b/datapipeline.gemspec new file mode 100644 index 0000000..0cf47b6 --- /dev/null +++ b/datapipeline.gemspec @@ -0,0 +1,32 @@ +require_relative 'lib/datapipeline/version' + +Gem::Specification.new do |spec| + spec.name = 'datapipeline' + spec.version = DataPipeline::VERSION + spec.authors = ['Karl-Heinz Wild'] + spec.email = ['kh.wild@wicom.li'] + spec.summary = 'Schlankes ETL-Framework mit Ruby-DSL: lesen, prüfen, umbauen, schreiben' + spec.description = <<~TEXT + Pipelines als Ruby-DSL mit austauschbaren Storage-Plugins (PostgreSQL, SQLite, CSV, JSON/NDJSON, Excel, REST), + verketteten Schritten (map, filter, validate, header, trailer), Lookups, Streaming, Satzarten-Layouts, + Fehlerdateien, Dry-Run, inkrementellen Läufen und dem Kommando etl mit Exit-Codes. + TEXT + spec.homepage = 'https://git.wicom.li/wkh/datapipeline' + spec.license = 'Nonstandard' + spec.required_ruby_version = '>= 3.4' + + spec.metadata['changelog_uri'] = "#{spec.homepage}/src/branch/main/CHANGELOG.md" + spec.metadata['rubygems_mfa_required'] = 'true' + + spec.files = Dir['lib/**/*.rb', 'exe/*', 'README.md', 'CHANGELOG.md'] + spec.bindir = 'exe' + spec.executables = ['etl'] + spec.require_paths = ['lib'] + + # csv and logger are no default gems any more from ruby 3.4 / 4.0 + spec.add_dependency 'csv', '~> 3.3' + spec.add_dependency 'logger', '~> 1.6' + + # optional, loaded only when the plugin is used (add them to the Gemfile of the application): + # pg (postgres), sqlite3 (sqlite), roo + caxlsx (excel), http (rest, slack) +end diff --git a/examples/bewegungen_inkrementell.rb b/examples/bewegungen_inkrementell.rb new file mode 100644 index 0000000..8a6f188 --- /dev/null +++ b/examples/bewegungen_inkrementell.rb @@ -0,0 +1,30 @@ +# +# Nur neue Buchungsbewegungen exportieren, Zeile für Zeile (Streaming) +# Ein zweiter Lauf ohne neue Daten schreibt nichts. +# +# bin/etl run examples/bewegungen_inkrementell.rb [--out=tmp] +# +require 'fileutils' +require 'datapipeline' + +DATA = File.expand_path('data', __dir__) +out = DataPipeline.params.fetch(:out, 'tmp') +FileUtils.mkdir_p(out) + +DataPipeline.new do + streamed + + # höchste id des letzten erfolgreichen Laufs, gespeichert in tmp/.state.json + stand = watermark :bewegungen, column: 'id', file: "#{out}/.state.json", initial: 0 + + # bei einer Datenbank gehört der Filter ins SQL: where id > $1, params: [stand.value] + read container(:json, file: "#{DATA}/bewegungen.ndjson") + filter { |row| row['id'] > stand.value.to_i } + map { |row| row.merge('soll_haben' => row['betrag'].negative? ? 'H' : 'S', 'betrag' => row['betrag'].abs) } + + write container(:json, + file: "#{out}/bewegungen_#{Time.now.strftime('%Y%m%d_%H%M%S')}.ndjson" + ) + + on_finish { |stats| puts "neu: #{stats.written}, Stand jetzt: #{stand.current}" } +end.run diff --git a/examples/data/bewegungen.ndjson b/examples/data/bewegungen.ndjson new file mode 100644 index 0000000..867c473 --- /dev/null +++ b/examples/data/bewegungen.ndjson @@ -0,0 +1,4 @@ +{"id":1,"konto":"4000","betrag":120.0} +{"id":2,"konto":"4000","betrag":-20.0} +{"id":3,"konto":"4400","betrag":75.5} +{"id":4,"konto":"4000","betrag":10.0} diff --git a/examples/data/filialen/vaduz.csv b/examples/data/filialen/vaduz.csv new file mode 100644 index 0000000..d3dfa28 --- /dev/null +++ b/examples/data/filialen/vaduz.csv @@ -0,0 +1,3 @@ +artikel,menge +A-100,5 +B-200,2 diff --git a/examples/data/filialen/zuerich.csv b/examples/data/filialen/zuerich.csv new file mode 100644 index 0000000..301db7d --- /dev/null +++ b/examples/data/filialen/zuerich.csv @@ -0,0 +1,3 @@ +artikel,menge +A-100,7 +C-300,1 diff --git a/examples/data/kunden.csv b/examples/data/kunden.csv new file mode 100644 index 0000000..38d76c4 --- /dev/null +++ b/examples/data/kunden.csv @@ -0,0 +1,10 @@ +kundennr;name;land;status;umsatz +100001;Müller GmbH;DE;aktiv;12500.50 +100002;Huber & Partner;AT;aktiv;8300 +100003;Swiss Trade AG;CH;aktiv;22000.75 +100004;Alte Firma KG;DE;inaktiv;0 +100005; ;DE;aktiv;100 +10006;Kurz;DE;aktiv;50 +100007;Nowhere Ltd;XX;aktiv;10 +100008;Dupont SARL;FR;aktiv;abc +100009;Rossi Srl;IT;aktiv;4100.10 diff --git a/examples/data/laender.json b/examples/data/laender.json new file mode 100644 index 0000000..d807d63 --- /dev/null +++ b/examples/data/laender.json @@ -0,0 +1,7 @@ +[ + { "code": "DE", "name": "Deutschland", "eu": true }, + { "code": "AT", "name": "Österreich", "eu": true }, + { "code": "CH", "name": "Schweiz", "eu": false }, + { "code": "FR", "name": "Frankreich", "eu": true }, + { "code": "IT", "name": "Italien", "eu": true } +] diff --git a/examples/data/rechnungen.json b/examples/data/rechnungen.json new file mode 100644 index 0000000..2df62f4 --- /dev/null +++ b/examples/data/rechnungen.json @@ -0,0 +1,7 @@ +[ + { "nr": "R-1001", "datum": "2025-11-03", "kunde": "100001", + "positionen": [{ "text": "Beratung", "menge": 2, "preis": 150.0 }, + { "text": "Reisekosten", "menge": 1, "preis": 80.5 }] }, + { "nr": "R-1002", "datum": "2025-11-05", "kunde": "100002", + "positionen": [{ "text": "Jahreslizenz Software mit Wartung und Support", "menge": 1, "preis": 1200 }] } +] diff --git a/examples/demo_api.rb b/examples/demo_api.rb new file mode 100644 index 0000000..c1c6ddd --- /dev/null +++ b/examples/demo_api.rb @@ -0,0 +1,137 @@ +# +# Kleine Demo-API für die REST-Beispiele, ohne Internet und ohne weitere Gems +# +# ruby examples/demo_api.rb # http://127.0.0.1:4567, Token: demo-token +# +# Endpunkte (alle mit Authorization: Bearer demo-token, sonst 401) +# GET /api/kunden?page=1&per_page=3 {"data": [...], "page": 1} die 2. Seite antwortet einmal mit 503 +# GET /api/laender [{"code": "DE", "name": "Deutschland"}, ...] +# GET /api/bestellungen[?cursor=n] {"items": [...], "next": "/api/bestellungen?cursor=3"} oder "next": null +# POST /api/import ein Objekt oder ein Array; 422 ohne kundennr/name +# GET /api/import alles, was angenommen wurde +# PUT /api/archiv/ ein Objekt ablegen +# GET /api/archiv alles im Archiv +# +require 'json' +require 'socket' +require 'uri' + +class DemoApi + TOKEN = 'demo-token'.freeze + + KUNDEN = [ + { 'id' => 1, 'name' => 'Müller GmbH', 'land' => 'DE', 'umsatz' => '12500.50' }, + { 'id' => 2, 'name' => 'Huber & Partner', 'land' => 'AT', 'umsatz' => '8300' }, + { 'id' => 3, 'name' => 'Swiss Trade AG', 'land' => 'CH', 'umsatz' => '22000.75' }, + { 'id' => 4, 'name' => 'Rossi Srl', 'land' => 'IT', 'umsatz' => '4100.10' }, + { 'id' => 5, 'name' => 'Dupont SARL', 'land' => 'FR', 'umsatz' => 'n/a' }, + { 'id' => 6, 'name' => 'Nordic AB', 'land' => 'SE', 'umsatz' => '990' }, + { 'id' => 7, 'name' => 'Vaduz Treuhand', 'land' => 'LI', 'umsatz' => '15000' } + ].freeze + + LAENDER = [ + { 'code' => 'DE', 'name' => 'Deutschland' }, { 'code' => 'AT', 'name' => 'Österreich' }, + { 'code' => 'CH', 'name' => 'Schweiz' }, { 'code' => 'IT', 'name' => 'Italien' }, + { 'code' => 'FR', 'name' => 'Frankreich' }, { 'code' => 'LI', 'name' => 'Liechtenstein' } + ].freeze + + BESTELLUNGEN = (1..5).map { |i| { 'id' => i, 'kunde' => (i % 3) + 1, 'betrag' => i * 100.0 } }.freeze + + attr_reader :imported, :archive + + def self.start(port: 4567) = new(port).tap(&:start) + + def initialize port + @server = TCPServer.new('127.0.0.1', port) + @imported = [] + @archive = {} + @failed_once = false + end + + def url = "http://127.0.0.1:#{@server.addr[1]}" + + def start + @thread = Thread.new { loop { Thread.new(@server.accept) { |client| serve(client) } } } + self + end + + def stop + @thread&.kill + @server.close + end + + private + + def serve client + method, target = client.gets.to_s.split + return unless method + + headers = {} + while (line = client.gets) && line != "\r\n" + key, value = line.split(':', 2) + headers[key.downcase] = value.strip + end + body = headers['content-length'] ? client.read(headers['content-length'].to_i) : '' + path, query = target.split('?', 2) + params = URI.decode_www_form(query.to_s).to_h + + status, payload = headers['authorization'] == "Bearer #{TOKEN}" ? route(method, path, params, body) : [401, { 'error' => 'token' }] + text = JSON.generate(payload) + client.write "HTTP/1.1 #{status} X\r\nContent-Type: application/json\r\nContent-Length: #{text.bytesize}\r\n" \ + "Connection: close\r\n\r\n#{text}" + rescue StandardError => e + warn "DemoApi: #{e.class}: #{e.message}" + ensure + client.close + end + + def route method, path, params, body + case [method, path] + in ['GET', '/api/kunden'] then kunden(params) + in ['GET', '/api/laender'] then [200, LAENDER] + in ['GET', '/api/bestellungen'] then bestellungen(params) + in ['POST', '/api/import'] then import(JSON.parse(body)) + in ['GET', '/api/import'] then [200, @imported] + in ['PUT', %r{\A/api/archiv/\w+\z}] then archivieren(path.split('/').last, JSON.parse(body)) + in ['GET', '/api/archiv'] then [200, @archive.values] + else [404, { 'error' => "#{method} #{path}" }] + end + end + + def kunden params + page = params.fetch('page', '1').to_i + per_page = params.fetch('per_page', '3').to_i + if page == 2 && !@failed_once + @failed_once = true + return [503, { 'error' => 'kurz nicht verfügbar' }] + end + [200, { 'data' => KUNDEN.drop((page - 1) * per_page).first(per_page), 'page' => page }] + end + + def bestellungen params + cursor = params.fetch('cursor', '0').to_i + items = BESTELLUNGEN.drop(cursor).first(2) + following = cursor + 2 < BESTELLUNGEN.size ? "/api/bestellungen?cursor=#{cursor + 2}" : nil + [200, { 'items' => items, 'next' => following }] + end + + def archivieren id, data + @archive[id] = data + [200, { 'ok' => true }] + end + + def import data + rows = data.is_a?(Array) ? data : [data] + invalid = rows.reject { |r| r['kundennr'] && r['name'] } + return [422, { 'error' => 'kundennr und name sind Pflicht', 'zeilen' => invalid }] if invalid.any? + + @imported.concat(rows) + [201, { 'angenommen' => rows.size }] + end +end + +if $PROGRAM_NAME == __FILE__ + api = DemoApi.start(port: ENV.fetch('PORT', 4567).to_i) + puts "Demo-API auf #{api.url}, Token #{DemoApi::TOKEN} (Ctrl-C beendet)" + sleep +end diff --git a/examples/eigenes_plugin.rb b/examples/eigenes_plugin.rb new file mode 100644 index 0000000..63abed0 --- /dev/null +++ b/examples/eigenes_plugin.rb @@ -0,0 +1,38 @@ +# +# Eigenes Plugin: liest alle CSV-Dateien eines Verzeichnisses als eine Quelle +# +# bin/etl run examples/eigenes_plugin.rb [--out=tmp] +# +require 'fileutils' +require 'datapipeline' + +DATA = File.expand_path('data', __dir__) +out = DataPipeline.params.fetch(:out, 'tmp') +FileUtils.mkdir_p(out) + +class VerzeichnisStorage < DataPipeline::StoragePlugin + def initialize dir:, pattern: '*.csv', **csv_options + @dir = dir + @pattern = pattern + @csv_options = csv_options + end + + # jede Zeile bekommt den Dateinamen dazu; mit Block zeilenweise (für streamed) + def read(&block) + rows = Dir.glob(File.join(@dir, @pattern)).sort.flat_map do |file| + DataPipeline::CsvStorage.new(file:, **@csv_options).read.map { |row| row.merge('datei' => File.basename(file, '.*')) } + end + return rows unless block + + rows.each(&block) + nil + end +end + +DataPipeline.register :verzeichnis, VerzeichnisStorage + +DataPipeline.new do + read container(:verzeichnis, dir: "#{DATA}/filialen") + map { |row| row.merge('menge' => row['menge'].to_i) } + write container(:csv, file: "#{out}/alle_filialen.csv", columns: %w[datei artikel menge]) +end.run diff --git a/examples/kunden_import.rb b/examples/kunden_import.rb new file mode 100644 index 0000000..da369f9 --- /dev/null +++ b/examples/kunden_import.rb @@ -0,0 +1,50 @@ +# +# Kundenliste aus CSV prüfen, mit Ländern anreichern, nach SQLite und Excel schreiben +# +# bin/etl run examples/kunden_import.rb [--out=tmp] [--dry-run] +# +require 'fileutils' +require 'datapipeline' + +DATA = File.expand_path('data', __dir__) +out = DataPipeline.params.fetch(:out, 'tmp') +FileUtils.mkdir_p(out) + +DataPipeline.new do + version 'kunden_import 1.0' + + # Nachschlagetabelle, einmal geladen: { 'DE' => { 'code' => 'DE', 'name' => 'Deutschland', 'eu' => true }, ... } + laender = lookup container(:json, file: "#{DATA}/laender.json"), key: 'code', unique: true + + read container(:csv, file: "#{DATA}/kunden.csv", csv: { col_sep: ';' }) + + # fehlerhafte Zeilen mit Grund in eine eigene Datei, der Lauf geht weiter + on_error :collect, to: container(:csv, file: "#{out}/kunden_fehler.csv") + + # 1. prüfen + validate 'kundennr' => { required: true, format: /\A\d{6}\z/ }, + 'name' => { required: true, max: 40 }, + 'umsatz' => { type: :decimal } + validate { |row| "Land #{row['land']} unbekannt" unless laender.key?(row['land']) } + + # 2. nur aktive Kunden + filter { |row| row['status'] == 'aktiv' } + + # 3. anreichern und umbauen + map do |row| + land = laender.fetch(row['land']) + { + 'kundennr' => row['kundennr'], + 'name' => row['name'].strip, + 'land' => land['name'], + 'eu' => land['eu'] ? 'ja' : 'nein', + 'umsatz' => row['umsatz'].to_f.round(2) + } + end + + # 4. schreiben: alle Kunden nach SQLite, EU-Kunden zusätzlich als Excel-Liste + write container(:sqlite, database: "#{out}/kunden.sqlite3", table: 'kunden', create: true, truncate: true) + write container(:excel, file: "#{out}/eu_kunden.xlsx", sheet: 'EU-Kunden'), only: ->(row) { row['eu'] == 'ja' } + + on_finish { |stats| puts "kunden_import: #{stats}" } +end.run diff --git a/examples/rechnungen_satzarten.rb b/examples/rechnungen_satzarten.rb new file mode 100644 index 0000000..007277d --- /dev/null +++ b/examples/rechnungen_satzarten.rb @@ -0,0 +1,50 @@ +# +# Rechnungen aus verschachteltem JSON in eine Datei mit Satzarten (Datei-, Kopf-, Positions-, Schlusssatz) +# +# bin/etl run examples/rechnungen_satzarten.rb [--out=tmp] +# +require 'fileutils' +require 'datapipeline' + +DATA = File.expand_path('data', __dir__) +out = DataPipeline.params.fetch(:out, 'tmp') +FileUtils.mkdir_p(out) + +# Feld: [Name, Typ, Länge, Pflicht]; Typ AN Text, N Zahl (11.2 = 9 Vor-, 2 Nachkommastellen), DAT YYYYMMDD +LAYOUT = DataPipeline::Layout.new do + record 'DATEI', [['SATYP', 'AN', 5, true], ['ERSTELLT', 'DAT', 8, true], ['ANZAHL', 'N', 6, true]] + record 'KOPF', [['SATYP', 'AN', 5, true], ['LFDNR', 'N', 6, true], ['BELNR', 'AN', 10, true], + ['DATUM', 'DAT', 8, true], ['KUNNR', 'AN', 10, true], ['SUMME', 'N', '11.2', true]] + record 'POS', [['SATYP', 'AN', 5, true], ['LFDNR', 'N', 6, true], ['BELNR', 'AN', 10, true], + ['POSNR', 'N', 3, true], ['TEXT', 'AN', 30], ['BETRAG', 'N', '11.2', true]] + record 'ENDE', [['SATYP', 'AN', 5, true], ['ANZAHL', 'N', 6, true]] +end + +DataPipeline.new do + read container(:json, file: "#{DATA}/rechnungen.json") + + # 1. eine Rechnung -> Kopfsatz + n Positionssätze + map do |rechnung| + positionen = rechnung['positionen'].each_with_index.map do |pos, i| + { 'satyp' => 'POS', 'belnr' => rechnung['nr'], 'posnr' => format('%03d', i + 1), + 'text' => pos['text'], 'betrag' => format('%.2f', pos['menge'] * pos['preis']) } + end + kopf = { 'satyp' => 'KOPF', 'belnr' => rechnung['nr'], 'datum' => rechnung['datum'].delete('-'), + 'kunnr' => rechnung['kunde'], 'summe' => format('%.2f', positionen.sum { |p| p['betrag'].to_f }) } + [kopf, *positionen] + end + + # 2. laufende Nummer über alle Kopf- und Positionssätze + map { |row, ctx| row.merge('lfdnr' => format('%06d', ctx.index + 1)) } + + # 3. gegen das Layout prüfen: zu langer Text wird gemeldet, die Zeile bleibt + validate LAYOUT, warn: true + + # Datei- und Schlusssatz kennen die Anzahl der Datensätze + header { |ctx| { 'satyp' => 'DATEI', 'erstellt' => Date.today.strftime('%Y%m%d'), 'anzahl' => format('%06d', ctx.written) } } + trailer { |ctx| { 'satyp' => 'ENDE', 'anzahl' => format('%06d', ctx.written) } } + + # Felder in der Reihenfolge des Layouts; Positionen zusätzlich als NDJSON + write container(:csv, file: "#{out}/rechnungen.txt", layout: LAYOUT, csv: { headers: false, col_sep: '|' }) + write container(:json, file: "#{out}/positionen.ndjson"), only: ->(row) { row['satyp'] == 'POS' } +end.run diff --git a/examples/rest_lesen.rb b/examples/rest_lesen.rb new file mode 100644 index 0000000..99c8485 --- /dev/null +++ b/examples/rest_lesen.rb @@ -0,0 +1,47 @@ +# +# Kunden seitenweise aus einer REST-API lesen, mit Ländern aus einem zweiten Endpunkt anreichern, +# nach CSV und JSON schreiben +# +# ruby examples/demo_api.rb & # Demo-API starten +# bin/etl run examples/rest_lesen.rb [--url=http://127.0.0.1:4567] [--out=tmp] +# +require 'fileutils' +require 'datapipeline' + +API = DataPipeline.params.fetch(:url, 'http://127.0.0.1:4567') +AUTH = "Bearer #{ENV.fetch('DEMO_API_TOKEN', 'demo-token')}" +out = DataPipeline.params.fetch(:out, 'tmp') +FileUtils.mkdir_p(out) + +DataPipeline.new do + version 'rest_lesen 1.0' + + # GET /api/kunden?page=1&per_page=3, dann page=2, ... bis eine Seite leer ist + # die Zeilen stehen in der Antwort unter "data"; 429/5xx und Netzfehler werden bis zu 3-mal wiederholt + kunden = container :rest, url: "#{API}/api/kunden", auth: AUTH, + root: 'data', page_param: 'page', params: { 'per_page' => 3 }, + retries: 3, retry_wait: 0.2 + + # derselbe Server, anderer Endpunkt: query hängt den Pfad an die url an (GET /api/laender) + stamm = container :rest, url: "#{API}/api", auth: AUTH + laender = lookup stamm, query: 'laender', key: 'code', unique: true + + read kunden + + on_error :collect, to: container(:csv, file: "#{out}/api_fehler.csv") + validate 'id' => { required: true }, 'umsatz' => { type: :decimal } + + map do |kunde| + { + 'id' => kunde['id'], + 'name' => kunde['name'], + 'land' => laender.dig(kunde['land'], 'name') || kunde['land'], + 'umsatz' => kunde['umsatz'].to_f + } + end + + write container(:csv, file: "#{out}/api_kunden.csv", columns: %w[id name land umsatz]) + write container(:json, file: "#{out}/api_kunden.json", pretty: true) + + on_finish { |stats| puts "rest_lesen: #{stats}" } +end.run diff --git a/examples/rest_schreiben.rb b/examples/rest_schreiben.rb new file mode 100644 index 0000000..26176e7 --- /dev/null +++ b/examples/rest_schreiben.rb @@ -0,0 +1,36 @@ +# +# Kunden aus CSV prüfen und in Paketen per POST an eine REST-API senden +# +# ruby examples/demo_api.rb & +# bin/etl run examples/rest_schreiben.rb [--url=http://127.0.0.1:4567] [--dry-run] +# +require 'fileutils' +require 'datapipeline' + +API = DataPipeline.params.fetch(:url, 'http://127.0.0.1:4567') +AUTH = "Bearer #{ENV.fetch('DEMO_API_TOKEN', 'demo-token')}" +DATA = File.expand_path('data', __dir__) +out = DataPipeline.params.fetch(:out, 'tmp') +FileUtils.mkdir_p(out) + +DataPipeline.new do + version 'rest_schreiben 1.0' + + read container(:csv, file: "#{DATA}/kunden.csv", csv: { col_sep: ';' }) + + # vor dem Senden prüfen: was die API ablehnen würde, landet hier in der Fehlerdatei + on_error :collect, to: container(:csv, file: "#{out}/import_fehler.csv") + validate 'kundennr' => { required: true, format: /\A\d{6}\z/ }, 'name' => { required: true } + + filter { |row| row['status'] == 'aktiv' } + map do |row| + { 'kundennr' => row['kundennr'], 'name' => row['name'].strip, 'land' => row['land'], 'umsatz' => row['umsatz'].to_f } + end + + # POST /api/import mit je 2 Kunden als JSON-Array; im Dry-Run wird nichts gesendet + write container(:rest, url: "#{API}/api/import", auth: AUTH, batch_size: 2) + + on_finish do |stats| + puts stats.dry_run ? 'Dry-Run: nichts gesendet' : "gesendet: #{stats.written} Kunden in #{(stats.written / 2.0).ceil} Paketen" + end +end.run diff --git a/examples/rest_zu_rest.rb b/examples/rest_zu_rest.rb new file mode 100644 index 0000000..742b451 --- /dev/null +++ b/examples/rest_zu_rest.rb @@ -0,0 +1,34 @@ +# +# Von API zu API: neue Bestellungen über Folge-Links lesen und einzeln per PUT archivieren +# Ein zweiter Lauf ohne neue Bestellungen sendet nichts. +# +# ruby examples/demo_api.rb & +# bin/etl run examples/rest_zu_rest.rb [--url=http://127.0.0.1:4567] [--out=tmp] +# +require 'fileutils' +require 'datapipeline' + +API = DataPipeline.params.fetch(:url, 'http://127.0.0.1:4567') +AUTH = "Bearer #{ENV.fetch('DEMO_API_TOKEN', 'demo-token')}" +out = DataPipeline.params.fetch(:out, 'tmp') +FileUtils.mkdir_p(out) + +DataPipeline.new do + streamed + + # höchste Bestellnummer des letzten erfolgreichen Laufs + stand = watermark :bestellungen, column: 'id', file: "#{out}/.state.json", initial: 0 + + # die Antwort nennt die nächste Seite selbst: {"items": [...], "next": "/api/bestellungen?cursor=2"} + read container(:rest, url: "#{API}/api/bestellungen", auth: AUTH, root: 'items', + next: ->(body) { body['next'] && "#{API}#{body['next']}" }) + + filter { |row| row['id'] > stand.value.to_i } + map { |row| row.merge('archiviert' => Date.today.iso8601) } + + # eine url je Bestellung: PUT /api/archiv/ + write container(:rest, url: "#{API}/api/archiv", auth: AUTH, write_method: :put, + write_url: ->(row) { "#{API}/api/archiv/#{row['id']}" }) + + on_finish { |stats| puts "archiviert: #{stats.written}, Stand jetzt: #{stand.current}" } +end.run diff --git a/exe/etl b/exe/etl new file mode 100755 index 0000000..8f7f909 --- /dev/null +++ b/exe/etl @@ -0,0 +1,70 @@ +#!/usr/bin/env ruby +# +# etl [run] [options] ('run' is optional) +# +# executable of the gem datapipeline; in this repository bin/etl loads the bundle first +# +# --from=DATE --till=DATE in DataPipeline.params, also passed on to the pipeline script +# --param key=value in DataPipeline.params[:key], repeatable +# --dry-run run everything, write nothing, print the first rows +# --limit=N only the first N source rows +# --verbose log level debug, --quiet only warnings (default info) +# --name=value any other option: DataPipeline.params[:name] and passed on to the script +# +# exit code: 0 ok, 1 aborted, 2 finished with rejected rows +# +require 'optparse' +require 'logger' +require_relative '../lib/datapipeline' + +options = { settings: { log_level: Logger::INFO }, passthrough: [] } +DataPipeline.params = {} +parser = OptionParser.new do |opts| + opts.banner = "Usage: etl [run] PIPELINE.rb [options]\n\n etl examples/kunden_import.rb --dry-run\n" + opts.on('--from=DATE', String) { |v| DataPipeline.params[:from] = v; options[:passthrough] << "--from=#{v}" } + opts.on('--till=DATE', String) { |v| DataPipeline.params[:till] = v; options[:passthrough] << "--till=#{v}" } + opts.on('--param KEY=VALUE', String) do |v| + key, value = v.split('=', 2) + DataPipeline.params[key.to_sym] = value + end + opts.on('--dry-run') { options[:settings][:dry_run] = true } + opts.on('--limit=N', Integer) { |v| options[:settings][:limit] = v } + opts.on('--verbose') { options[:settings][:log_level] = Logger::DEBUG } + opts.on('--quiet') { options[:settings][:log_level] = Logger::WARN } +end + +# own options for the parser, everything else (also --name=value of the script) goes to the script +own = [] +args = [] +argv = ARGV.dup +while (arg = argv.shift) + if arg.match?(/\A--(from|till|param|dry-run|limit|verbose|quiet)(=|\z)/) + own << arg + own << argv.shift if %w[--param --limit --from --till].include?(arg) + else + args << arg + end +end +parser.parse(own) +args.shift if args.first == 'run' +script = args.shift +DataPipeline.params.merge!(DataPipeline.parse_params(args)) { |_key, own_value, _| own_value } +abort parser.help if script.nil? +abort "etl: #{script} not found\n\n#{parser.help}" unless File.exist?(script) + +DataPipeline.settings(options[:settings]) + +# the pipeline script sees only its own arguments +ARGV.replace(options[:passthrough] + args) + +begin + load File.expand_path(script) +rescue StandardError => e + warn "#{File.basename(script)}: #{e.class}: #{e.message}" + exit 1 +end + +runs = DataPipeline.runs +exit 1 if runs.any?(&:error) +exit 2 if runs.any? { |stats| stats.rejected.positive? } +exit 0 diff --git a/lib/datapipeline.rb b/lib/datapipeline.rb new file mode 100644 index 0000000..7a970b2 --- /dev/null +++ b/lib/datapipeline.rb @@ -0,0 +1,553 @@ +require 'json' +require 'logger' + +require_relative 'datapipeline/version' +require_relative 'datapipeline/storage/storage_plugin' +require_relative 'datapipeline/validation' +require_relative 'datapipeline/layout' +require_relative 'datapipeline/watermark' + +class DataPipeline + # plugins are loaded on first use, their gems too: pg (postgres), sqlite3 (sqlite), roo/caxlsx (excel), + # http (rest, slack) + autoload :CsvStorage, File.expand_path('datapipeline/storage/csv', __dir__) + autoload :PostgresStorage, File.expand_path('datapipeline/storage/postgres', __dir__) + autoload :JsonStorage, File.expand_path('datapipeline/storage/json', __dir__) + autoload :SqliteStorage, File.expand_path('datapipeline/storage/sqlite', __dir__) + autoload :ExcelStorage, File.expand_path('datapipeline/storage/excel', __dir__) + autoload :RestStorage, File.expand_path('datapipeline/storage/rest', __dir__) + autoload :Slack, File.expand_path('datapipeline/notification/slack', __dir__) + + # passed to header, trailer, map and filter + # read: number of source rows (nil while streaming, until the trailer) + # written: number of body rows after all steps (nil while the body is processed) + # index: index of the current row within the current step + Context = Struct.new(:read, :written, :index, keyword_init: true) + + # result of a run, returned by run and passed to on_finish + # read: source rows, written: body rows, output: rows to the target incl. header/trailer, + # rejected: rows dropped by on_error :skip/:collect, errors: first 100 of them, + # warnings: rows that failed a validate step with warn: true (kept), error: exception of an aborted run + Stats = Struct.new(:version, :read, :written, :output, :rejected, :errors, :warnings, :started_at, :finished_at, + :dry_run, :error, keyword_init: true) do + def duration = finished_at && started_at ? (finished_at - started_at).round(3) : nil + def ok? = error.nil? && rejected.zero? + + def to_s + "read=#{read} written=#{written} rejected=#{rejected}#{warnings.to_i.positive? ? " warnings=#{warnings}" : ''} " \ + "#{duration}s#{dry_run ? ' (dry run)' : ''}" \ + "#{error ? " ABORTED: #{error.class}: #{error.message}" : ''}" + end + end + + # stops reading at the limit, not an error + class LimitReached < StandardError; end + + EDGE_BEFORE = %i[prepend header].freeze + EDGE_AFTER = %i[append trailer].freeze + CHAINING = %i[map filter validate].freeze + ON_ERROR = %i[abort skip collect].freeze + MAX_ERRORS = 100 + + DEFAULT_SETTINGS = { + version: nil, + streamed: false, + chained: false, + verbose: false, + dry_run: false, + limit: nil, + preview: 10, + on_error: :abort, + log_level: Logger::WARN + }.freeze + + SETTINGS_LOCK = Mutex.new + @defaults = DEFAULT_SETTINGS + @plugins = {} + + # + # defaults for every following DataPipeline.new, e.g. from bin/etl + # the hash is replaced, never changed: a running pipeline keeps its own copy + # + def self.settings(options = {}) + SETTINGS_LOCK.synchronize { @defaults = @defaults.merge(options).freeze } if options.any? + @defaults + end + + def self.reset_settings! + SETTINGS_LOCK.synchronize { @defaults = DEFAULT_SETTINGS } + end + + # + # container :name finds a registered plugin first, then Storage by convention + # (DataPipeline::Storage of the gem, else a top level Storage of the application) + # DataPipeline.register :mem, MemStorage + # DataPipeline.register :sftp, 'SftpStorage', path: '/path/to/sftp_storage' # loaded on first use + # + def self.register type, klass, path: nil + Object.autoload(klass.to_sym, File.expand_path(path)) if path && klass.is_a?(String) + SETTINGS_LOCK.synchronize { @plugins = @plugins.merge(type.to_sym => klass).freeze } + end + + def self.plugin type + klass = @plugins[type.to_sym] + klass = Object.const_get(klass) if klass.is_a?(String) + return klass if klass + + class_name = type.to_s.capitalize + 'Storage' + return const_get(class_name, false) if const_defined?(class_name, false) + + Object.const_get(class_name) if Object.const_defined?(class_name) + end + + # + # parameters from the command line: bin/etl sets them; a script started with ruby directly + # reads them from ARGV on first use: --name=value -> params[:name], --param key=value -> params[:key] + # + def self.params + @params ||= parse_params(ARGV) + end + + def self.params= hash + @params = hash + end + + def self.parse_params args + params = {} + args.each_with_index do |arg, i| + case arg + when /\A--param=(\w+)=(.*)\z/ then params[$1.to_sym] = $2 + when '--param' then key, value = args[i + 1].to_s.split('=', 2); params[key.to_sym] = value if value + when /\A--([a-z][\w-]*)=(.*)\z/ then params[$1.tr('-', '_').to_sym] = $2 + end + end + params + end + + # stats of every run in this process, used by bin/etl for the exit code + def self.runs + @runs ||= [] + end + + # DataPipeline.new(verbose: true) { ... } overrides the defaults for this pipeline only + def initialize(**settings, &block) + @settings = DataPipeline.settings.merge(settings) + @transformations = [] + @targets = [] + @on_finish = [] + @logger = Logger.new($stderr, level: @settings[:log_level], progname: 'etl') + @logger.formatter = proc { |severity, time, _prog, msg| "#{time.strftime('%F %T')} #{severity} #{msg}\n" } + instance_eval(&block) + end + + def params + self.class.params + end + + def configure(options = {}) + @settings.merge!(options) + end + + def version value + @settings[:version] = value + end + + def verbose flag = true + @settings[:verbose] = flag + end + + def streamed flag = true + @settings[:streamed] = flag + end + + # each body step gets the output of the previous step instead of the source rows + def chained flag = true + @settings[:chained] = flag + end + + # run everything, write nothing (target, rejects, watermark); prints the first rows + def dry_run flag = true + @settings[:dry_run] = flag + end + + # for steps with side effects (api calls): true in a dry run + def dry_run? + @settings[:dry_run] ? true : false + end + + # process only the first n source rows + def limit count + @settings[:limit] = count + end + + def logger logger = nil + @logger = logger if logger + @logger + end + + # + # what happens when a body step raises (or a validate step fails) for a row + # on_error :abort -> stop the run (default) + # on_error :skip -> drop the row, log a warning, count it + # on_error :collect, to: storage -> like skip, plus the row with _step and _error into storage + # + def on_error mode, to: nil + raise ArgumentError, "on_error: #{mode} unknown, use #{ON_ERROR.join(', ')}" unless ON_ERROR.include?(mode) + raise ArgumentError, 'on_error :collect needs to: storage' if mode == :collect && to.nil? + + @settings[:on_error] = mode + @reject_target = to + end + + # on_finish { |stats| ... } -> after every run, also after an abort (stats.error) + def on_finish(&block) + @on_finish << block + end + + # see DataPipeline::Watermark + def watermark name, column:, file:, initial: nil + @watermark = Watermark.new(name, column:, file:, initial:) + end + + # only a missing plugin is reported as unsupported, errors while loading it + # (missing gem) or from the plugin itself (connection, options) are passed through + def container type, **options, &block + klass = DataPipeline.plugin(type) + raise "Unsupported storage type or class missing: #{type.to_s.capitalize}Storage" unless klass + + klass.new(**options, &block) + end + + # + # transform(options) { |row, index| ... } + # processing: :prepend | :normal (default) | :append + # condition: ruby expression as string (row is visible) or lambda { |row| ... } + # + def transform(options = {}, &block) + kind = options.fetch(:processing, :normal) + kind = :normal unless %i[prepend append].include?(kind) + add_step kind, options, &block + end + + # map { |row, ctx| ... } -> hash, array of hashes or nil; turns chaining on + def map(options = {}, &block) + add_step :map, options, &block + end + + # filter { |row, ctx| ... } -> keeps the row when the block is truthy; turns chaining on + def filter(options = {}, &block) + add_step :filter, options, &block + end + + # header { |ctx| ... } -> rows in front of the body, called after the body with full counters + def header(&block) + add_step :header, {}, &block + end + + # trailer { |ctx| ... } -> rows after the body + def trailer(&block) + add_step :trailer, {}, &block + end + + # + # validate 'kostl' => { required: true, max: 10 }, 'shkzg' => { in: %w[S H] } + # validate layout -> fields of a DataPipeline::Layout + # validate { |row| 'soll ungleich haben' if ... } -> nil, message or array of messages + # + # a failing row raises ValidationError, handled by on_error; turns chaining on + # warn: true -> only a warning in the log and stats.warnings, the row is kept + # ignore: %w[A B] -> messages for these fields are left out (known gaps) + # + def validate(rules = nil, **options, &block) + warn = options.delete(:warn) || false + ignore = Array(options.delete(:ignore)).map(&:to_s) + rules ||= options # validate 'konto' => { ... } arrives as keywords + @transformations << { kind: :validate, block:, options: { rules:, warn:, ignore: } } + end + + def read storage, options = {} + @source = storage + @source.set options if options.any? + end + + # + # write storage -> all rows + # write storage, only: ->(row) { ... } -> only matching rows (also header/trailer rows) + # + # can be called several times, every target gets its rows + # + def write storage, only: nil + @targets << { storage:, only: } + end + + def query storage, options = {} + storage.query options + end + + # + # loads a table once, for lookups in map/transform instead of one query per row + # + # lookup pg, query: 'select ...', params: [...], key: 'rechnung_dbid' -> { key => [rows] } + # lookup pg, query: 'select ...', key: 'id', unique: true -> { key => row } + # lookup csv, key: ->(row) { row['land'] } -> reads the whole storage + # + def lookup storage, key:, unique: false, **options + rows = options[:query] ? storage.query(options) : storage.read + key_of = key.respond_to?(:call) ? key : ->(row) { row[key] } + unique ? rows.to_h { |row| [key_of.call(row), row] } : rows.group_by(&key_of) + end + + def run + @stats = Stats.new(version: @settings[:version], read: 0, written: 0, output: 0, rejected: 0, errors: [], warnings: 0, + started_at: Time.now, dry_run: @settings[:dry_run]) + @logger.info "start#{" #{@settings[:version]}" if @settings[:version]}#{@settings[:dry_run] ? ' (dry run)' : ''}" + @preview = [] + raise ArgumentError, 'read storage missing' unless @source + raise ArgumentError, 'write storage missing' if @targets.empty? + + before = @transformations.select { |t| EDGE_BEFORE.include?(t[:kind]) } + after = @transformations.select { |t| EDGE_AFTER.include?(t[:kind]) } + body = @transformations - before - after + + @settings[:streamed] ? run_streamed(before, body, after) : run_batch(before, body, after) + + @watermark&.save unless @settings[:dry_run] + @stats + rescue StandardError => e + @stats.error = e + @logger.error "abort: #{e.class}: #{e.message}" + raise + ensure + finish + end + + private + + def add_step kind, options, &block + raise ArgumentError, "#{kind}: block missing" unless block + + @transformations << { kind:, block:, options: } + end + + # without any transformation the source is copied unchanged + def chained? + @settings[:chained] || @transformations.empty? || + @transformations.any? { |t| CHAINING.include?(t[:kind]) } + end + + def write_row row + @targets.each { |t| t[:storage].write_row(row) if t[:only].nil? || t[:only].call(row) } + end + + def finish + @stats.finished_at = Time.now + [@source, *@targets.map { |t| t[:storage] }, @reject_target].compact.uniq.each(&:close) + self.class.runs << @stats + print_preview if @stats.dry_run && @stats.error.nil? + @logger.info "finish #{@stats}" + @on_finish.each do |hook| + hook.call(@stats) + rescue StandardError => e + @logger.error "on_finish: #{e.class}: #{e.message}" + end + end + + def source_rows + data = @source.read + data = data.first(@settings[:limit]) if @settings[:limit] + data + end + + def output rows + @stats.output += rows.size + @preview.concat(rows.first(@settings[:preview] - @preview.size)) if @preview.size < @settings[:preview] + return if @settings[:dry_run] + + yield rows + end + + def print_preview + puts "DRY RUN – nothing written: #{@stats}" + @preview.each { |row| puts JSON.generate(row) } + end + + # whole source in memory, one target.write at the end + def run_batch before, body, after + data = source_rows + @stats.read = data.length + data.each { |row| @watermark.track(row) } if @watermark + dump 'Source#data', data + + # prepend keeps its old timing: called before the body + results = {}.compare_by_identity + before.select { |t| t[:kind] == :prepend }.each { |t| results[t] = edge(t, nil, 0) } + + rows = [] + if chained? + counters = Array.new(body.size, 0) + data.each { |row| push(row, body, counters, data.length) { |out| rows << out } } + else + body.each { |step| rows.concat apply(step, data) } + end + dump 'Transformation#data', rows + + ctx = Context.new(read: data.length, written: rows.length) + before.select { |t| t[:kind] == :header }.each { |t| results[t] = edge(t, ctx, 0) } + + # parallel mode keeps the old count of source rows for append + count = chained? ? rows.length : data.length + new_data = [] + before.each { |t| add_rows new_data, results[t] } + new_data.concat rows + after.each { |t| add_rows new_data, edge(t, ctx, count) } + + @stats.written = rows.length + dump 'Target#data', new_data + output(new_data) do |all| + @targets.each { |t| t[:storage].write(t[:only] ? all.select(&t[:only]) : all) } + end + end + + # row by row from source.read { } to target.write_row, header only knows nothing yet + def run_streamed before, body, after + if !chained? && body.size > 1 + raise ArgumentError, 'streamed: several transformations need chained (or map/filter)' + end + + emit = ->(result) { output(add_rows([], result)) { |rows| rows.each { |row| write_row(row) } } } + + before.each { |t| emit.call edge(t, Context.new, 0) } + + read = written = 0 + counters = Array.new(body.size, 0) + begin + @source.read do |row| + raise LimitReached if @settings[:limit] && read >= @settings[:limit] + + read += 1 + @watermark&.track(row) + push(row, body, counters, nil) do |out| + written += 1 + emit.call out + end + end + rescue LimitReached + @logger.info "limit #{@settings[:limit]} reached" + end + @stats.read = read + @stats.written = written + + ctx = Context.new(read:, written:) + after.each { |t| emit.call edge(t, ctx, chained? ? written : read) } + @targets.each { |t| t[:storage].finalize } + end + + # one row through the chain, depth first; counters hold the index per step + def push row, body, counters, read, level = 0, &emit + return emit.call(row) if level == body.size + + index = counters[level] + counters[level] += 1 + safe_step_rows(body[level], row, index, read).each do |out| + push(out, body, counters, read, level + 1, &emit) + end + end + + # parallel mode: every step reads its own copy of the source rows + def apply step, rows + rows.each_with_index.flat_map { |row, index| safe_step_rows(step, row.dup, index, rows.length) } + end + + # errors of a step for one row go to on_error + def safe_step_rows step, row, index, read + step_rows(step, row, index, read) + rescue StandardError => e + reject(step, row, e) + [] + end + + def reject step, row, error + raise error if @settings[:on_error] == :abort + + name = step_name(step) + @stats.rejected += 1 + @stats.errors << { step: name, error: error.message, row: } if @stats.errors.size < MAX_ERRORS + @logger.warn "rejected by #{name}: #{error.message}" + return if @settings[:dry_run] || @reject_target.nil? + + @reject_target.write_row(row.merge('_step' => name, '_error' => error.message)) + end + + def step_name step + "#{step[:kind]}##{@transformations.index(step) + 1}" + end + + # rows produced by one step for one input row + def step_rows step, row, index, read + return [] unless condition?(step, row) + + case step[:kind] + when :validate + rules = step[:options][:rules] + messages = rules.is_a?(Layout) ? rules.errors(row) : Validation.errors(row, rules) + messages += Array(step[:block].call(row)).compact if step[:block] + ignore = step[:options][:ignore] + messages = messages.reject { |m| ignore.any? { |field| m.start_with?("#{field}:") } } if ignore.any? + return [row] if messages.empty? + raise ValidationError, messages.join('; ') unless step[:options][:warn] + + @stats.warnings += 1 + if @stats.warnings <= MAX_ERRORS + @logger.warn "#{step_name(step)}: #{messages.join('; ')}" + elsif @stats.warnings == MAX_ERRORS + 1 + @logger.warn "more than #{MAX_ERRORS} warnings, further ones only counted" + end + [row] + when :filter + step[:block].call(row, Context.new(read:, index:)) ? [row] : [] + when :map + add_rows [], step[:block].call(row, Context.new(read:, index:)) + else + add_rows [], step[:block].call(row, index) + end + end + + def edge step, ctx, count + case step[:kind] + when :prepend then step[:block].call(nil, 0) + when :append then step[:block].call(nil, count) + else step[:block].call(ctx) + end + end + + # string conditions are evaluated with the local variable row + def condition? step, row + condition = step[:options].fetch(:condition, true) + return true if condition == true + return condition.call(row) if condition.respond_to?(:call) + + eval(condition) + end + + # a hash is one row, an array are several rows, nil is no row + def add_rows new_data, result + if result.is_a?(Array) + new_data.concat(result.compact) + elsif result + new_data << result + end + new_data + end + + def dump title, data + return unless @settings[:verbose] + + puts "#{title}".ljust(28, '-') + puts JSON.pretty_generate data + puts "----------------------------" + end +end + +# short names besides the Storage convention +DataPipeline.register :sqlite3, 'DataPipeline::SqliteStorage' +DataPipeline.register :xlsx, 'DataPipeline::ExcelStorage' diff --git a/lib/datapipeline/layout.rb b/lib/datapipeline/layout.rb new file mode 100644 index 0000000..cdb4832 --- /dev/null +++ b/lib/datapipeline/layout.rb @@ -0,0 +1,113 @@ +require 'date' + +class DataPipeline + class LayoutError < StandardError; end + + # + # fixed field list per record type, for files with several record types (e.g. SAP CUDO) + # + # layout = DataPipeline::Layout.new do + # record 'DOHD', [['SATYP', 'AN', 4, true], ['LFDNR', 'N', 10, true], ['BELNR', 'AN', 10], ...] + # record 'LARC', [['SATYP', 'AN', 4, true], ['SUMME', 'N', 10, true]] + # end + # + # field: [name, type, length, required] + # type AN (text), N (number, length 10 or 13.2 = digits.decimals), DAT (YYYYMMDD), TIM (HHMMSS) + # + # the record type of a row is the value of type_field (default: the first value of the row). + # keys of a row are matched without regard to case, string or symbol. + # + # write container(:csv, file: ..., layout: layout) -> values in field order, unknown keys raise + # validate layout -> required, length, format per field + # + class Layout + Field = Struct.new(:name, :type, :length, :required) do + def key = name.downcase + + # 13.2 -> [11, 2] digits before and after the point, 10 -> [10, 0] + def digits + before, after = length.to_s.split('.').map(&:to_i) + after ||= 0 + [before - after, after] + end + end + + attr_reader :records + + def initialize type_field: nil, strict: true, &block + @type_field = type_field&.to_s&.downcase + @strict = strict + @records = {} + instance_eval(&block) if block + end + + def record type, fields + @records[type.to_s] = fields.map do |name, kind, length, required| + Field.new(name.to_s, (kind || 'AN').to_s.upcase, length, required ? true : false) + end + end + + def fields_for row + type = record_type(row) + @records.fetch(type.to_s) { raise LayoutError, "record type #{type.inspect} not in layout (#{@records.keys.join(', ')})" } + end + + # values in the order of the layout; unknown keys raise (strict), missing keys are nil + def values row + fields = fields_for(row) + normalized = normalize(row) + if @strict + unknown = normalized.keys - fields.map(&:key) + raise LayoutError, "#{record_type(row)}: fields not in layout: #{unknown.join(', ')}" if unknown.any? + end + fields.map { |field| normalized[field.key] } + end + + def errors row + fields = fields_for(row) + normalized = normalize(row) + fields.flat_map { |field| check(field, normalized[field.key]) } + rescue LayoutError => e + [e.message] + end + + private + + def record_type row + @type_field ? normalize(row)[@type_field] : row.values.first + end + + def normalize row + row.to_h { |key, value| [key.to_s.downcase, value] } + end + + def check field, value + text = value.to_s + return(field.required ? ["#{field.name}: missing"] : []) if text.strip.empty? + + case field.type + when 'N' then check_number(field, text) + when 'DAT' then valid_date?(text) ? [] : ["#{field.name}: not a date YYYYMMDD"] + when 'TIM' then text.match?(/\A([01]\d|2[0-3])[0-5]\d[0-5]\d\z/) ? [] : ["#{field.name}: not a time HHMMSS"] + else + field.length && text.length > field.length.to_i ? ["#{field.name}: longer than #{field.length}"] : [] + end + end + + def check_number field, text + return ["#{field.name}: not a number"] unless text.match?(/\A[+-]?\d+(\.\d+)?\z/) + return [] unless field.length + + before, after = field.digits + int, dec = text.delete('+-').split('.') + return ["#{field.name}: more than #{before} digits"] if int.length > before + return ["#{field.name}: more than #{after} decimals"] if dec && dec.length > after + + [] + end + + def valid_date? text + text.match?(/\A\d{8}\z/) && Date.valid_date?(text[0, 4].to_i, text[4, 2].to_i, text[6, 2].to_i) + end + end +end diff --git a/lib/datapipeline/notification/slack.rb b/lib/datapipeline/notification/slack.rb new file mode 100644 index 0000000..405d786 --- /dev/null +++ b/lib/datapipeline/notification/slack.rb @@ -0,0 +1,36 @@ +require 'http' +require 'json' + +class DataPipeline + # + # Slack.new(channel: '#etl').send_message('posting export done') + # + # options + # token: bot token, default ENV['SLACK_TOKEN'] + # channel: default '#general' + # + class Slack + API_URL = 'https://slack.com/api/chat.postMessage'.freeze + + DEFAULTS = { + token: nil, + channel: '#general' + }.freeze + + def initialize options = {} + @options = DEFAULTS.merge(token: ENV['SLACK_TOKEN']).merge(options) + raise ArgumentError, 'Slack: token missing (option token: or ENV SLACK_TOKEN)' unless @options[:token] + end + + # returns the parsed slack response, raises when slack reports an error + def send_message(content, verbose: false) + response = HTTP.auth("Bearer #{@options[:token]}") + .post(API_URL, json: { channel: @options[:channel], text: content }) + body = JSON.parse(response.body.to_s) + puts JSON.pretty_generate(body) if verbose + raise "Slack#send_message failed: #{body['error']}" unless body['ok'] + + body + end + end +end diff --git a/lib/datapipeline/storage/csv.rb b/lib/datapipeline/storage/csv.rb new file mode 100644 index 0000000..af5de55 --- /dev/null +++ b/lib/datapipeline/storage/csv.rb @@ -0,0 +1,121 @@ +require 'csv' +require 'json' +require_relative 'storage_plugin' + +class DataPipeline + # + # options + # file: path of the csv file + # csv: options passed to ruby CSV, merged with DEFAULTS + # columns: fixed column order for writing (and keys for reading without headers) + # without columns the values are written in hash order + # layout: DataPipeline::Layout, field order per record type (several record types in one file), + # unknown fields raise; no header line + # counter: print a dot per written row + # verbose: print settings + # + class CsvStorage < StoragePlugin + DEFAULTS = { + verbose: false, + counter: false, + columns: nil, + csv: { + headers: true, + col_sep: ',', + empty_value: '' + } + }.freeze + + def initialize **options, &block + @options = DEFAULTS.merge(options) do |_key, default, value| + default.is_a?(Hash) && value.is_a?(Hash) ? default.merge(value) : value + end + @file_name = @options.fetch(:file) { raise ArgumentError, 'CsvStorage: option file missing' } + @verbose = @options[:verbose] + + if @verbose + puts "CsvStorage#options--------" + puts JSON.pretty_generate @options + puts "---------------------------" + end + + yield if block_given? + end + + def read(&block) + if block_given? + CSV.foreach(@file_name, **csv_options) { |row| yield to_hash(row) } + return + end + + CSV.read(@file_name, **csv_options).map { |row| to_hash(row) } + end + + def write data + puts "> CsvStorage#write" if @verbose + open_file + data.each { |row| write_row row } + ensure + close + end + + def write_row row + open_file + if @options[:layout] + @file << @options[:layout].values(row) + print '.' if @options[:counter] + return + end + + if !@header_written && csv_options[:headers] + header = columns || row.keys + @file << header.map(&:to_s) if header + end + @header_written = true + + @file << (columns ? columns.map { |col| fetch(row, col) } : row.values) + print "." if @options[:counter] + end + + def close + @file&.close + @file = nil + @header_written = false + end + + private + + def csv_options + @options[:csv] + end + + def columns + @options[:columns] + end + + def open_file + return if @file + + @file = CSV.open(@file_name, 'w', **csv_options) + @header_written = false + return unless columns && csv_options[:headers] && !@options[:layout] + + @file << columns.map(&:to_s) + @header_written = true + end + + # rows with headers are CSV::Row, without headers plain arrays + def to_hash row + return row.to_h if row.is_a?(CSV::Row) + + keys = columns&.map(&:to_s) || (0...row.length).map(&:to_s) + keys.zip(row).to_h + end + + def fetch row, col + return row[col] if row.key?(col) + + row.key?(col.to_s) ? row[col.to_s] : row[col.to_s.to_sym] + end + end +end diff --git a/lib/datapipeline/storage/excel.rb b/lib/datapipeline/storage/excel.rb new file mode 100644 index 0000000..e21abd4 --- /dev/null +++ b/lib/datapipeline/storage/excel.rb @@ -0,0 +1,108 @@ +require_relative 'storage_plugin' + +class DataPipeline + # + # options + # file: path of the .xlsx file + # sheet: read: name or index (0 = first, default), write: name (default 'Daten') + # headers: first row holds the column names (default true) + # columns: column names when there is no header row, fixed column order for writing + # skip: read: rows before the header row (title lines), default 0 + # + # read needs the gem roo, write the gem caxlsx; both are loaded on first use. + # text stays text when writing (no number conversion, leading zeros are kept). + # + class ExcelStorage < StoragePlugin + def initialize **options, &block + @options = { headers: true, skip: 0 }.merge(options) + @file_name = @options.fetch(:file) { raise ArgumentError, 'ExcelStorage: option file missing' } + + yield if block_given? + end + + # without block all rows, with block row by row (xlsx is streamed) + def read(&block) + return enum_for(:each_row).to_a unless block + + each_row(&block) + nil + end + + def write rows + open_sheet + rows.each { |row| write_row row } + ensure + close + end + + def write_row row + open_sheet + if @columns.nil? + @columns = (@options[:columns] || row.keys).map(&:to_s) + add_row @columns, header: true if @options[:headers] + end + add_row(@columns.map { |col| fetch(row, col) }) + end + + def close + return unless @package + + @package.serialize(@file_name) + @package = @sheet = @columns = nil + end + + private + + def each_row + self.class.require_gem 'roo' + book = Roo::Spreadsheet.open(@file_name, extension: File.extname(@file_name).delete('.').to_sym) + sheet = book.sheet(@options.fetch(:sheet, 0)) + keys = @options[:headers] ? nil : @options[:columns]&.map(&:to_s) + skip = @options[:skip] + + rows_of(sheet).each do |values| + next if (skip -= 1) >= 0 + next if values.all? { |v| v.nil? || v.to_s.strip.empty? } + + if keys.nil? + keys = values.each_with_index.map { |v, i| v.nil? ? i.to_s : v.to_s } + next + end + yield keys.zip(values).to_h + end + ensure + book&.close + end + + # xlsx row by row without loading the sheet, other formats over the row api + def rows_of sheet + if sheet.respond_to?(:each_row_streaming) + Enumerator.new do |y| + sheet.each_row_streaming(pad_cells: true) { |cells| y << cells.map { |c| c&.value } } + end + else + (sheet.first_row.to_i..sheet.last_row.to_i).lazy.map { |i| sheet.row(i) } + end + end + + def open_sheet + return if @package + + self.class.require_gem 'axlsx', 'caxlsx' + @package = Axlsx::Package.new + @sheet = @package.workbook.add_worksheet(name: @options.fetch(:sheet, 'Daten').to_s[0, 31]) + @bold = @package.workbook.styles.add_style(b: true) + end + + def add_row values, header: false + types = values.map { |v| v.is_a?(String) ? :string : nil } + @sheet.add_row(values, types:, style: header ? @bold : nil) + end + + def fetch row, col + return row[col] if row.key?(col) + + row.key?(col.to_sym) ? row[col.to_sym] : nil + end + end +end diff --git a/lib/datapipeline/storage/json.rb b/lib/datapipeline/storage/json.rb new file mode 100644 index 0000000..1507e69 --- /dev/null +++ b/lib/datapipeline/storage/json.rb @@ -0,0 +1,79 @@ +require 'json' +require_relative 'storage_plugin' + +class DataPipeline + # + # options + # file: path of the file + # format: :array (one json array) or :lines (one object per line, ndjson) + # default from the extension: .ndjson / .jsonl -> :lines, else :array + # root: read only: path to the array inside the document, e.g. 'data.items' + # a single object (instead of an array) is read as one row + # pretty: write :array pretty printed (default false) + # + class JsonStorage < StoragePlugin + LINE_EXTENSIONS = %w[.ndjson .jsonl].freeze + + def initialize **options, &block + @options = options + @file_name = options.fetch(:file) { raise ArgumentError, 'JsonStorage: option file missing' } + @format = options.fetch(:format) { LINE_EXTENSIONS.include?(File.extname(@file_name)) ? :lines : :array } + raise ArgumentError, "JsonStorage: format #{@format} unknown, use :array or :lines" unless %i[array lines].include?(@format) + + yield if block_given? + end + + # without block all rows, with block row by row (ndjson is read line by line) + def read(&block) + rows = @format == :lines ? read_lines : read_array + return rows.to_a unless block + + rows.each(&block) + nil + end + + def write rows + if @format == :lines + File.open(@file_name, 'w') { |f| rows.each { |row| f.puts JSON.generate(row) } } + else + File.write(@file_name, @options[:pretty] ? JSON.pretty_generate(rows) : JSON.generate(rows)) + end + end + + # streaming: ndjson line by line, array as "[row,row,...]" closed in close + def write_row row + @file ||= File.open(@file_name, 'w').tap { |f| f.write('[') if @format == :array } + if @format == :array + @file.write(',') if @rows_written + @file.write(JSON.generate(row)) + else + @file.puts JSON.generate(row) + end + @rows_written = true + end + + def close + return unless @file + + @file.write(']') if @format == :array + @file.close + @file = nil + @rows_written = false + end + + private + + def read_lines + File.foreach(@file_name).lazy.map(&:strip).reject(&:empty?).map { |line| JSON.parse(line) } + end + + def read_array + data = JSON.parse(File.read(@file_name)) + @options[:root].to_s.split('.').each { |key| data = data.fetch(key) } if @options[:root] + data = [data] if data.is_a?(Hash) # a single object is one row + raise ArgumentError, "JsonStorage: #{@file_name} does not contain an array" unless data.is_a?(Array) + + data + end + end +end diff --git a/lib/datapipeline/storage/postgres.rb b/lib/datapipeline/storage/postgres.rb new file mode 100644 index 0000000..d39016a --- /dev/null +++ b/lib/datapipeline/storage/postgres.rb @@ -0,0 +1,125 @@ +require_relative 'storage_plugin' + +class DataPipeline + # + # options + # host, port, database, username, password + # query: sql for read, placeholders $1, $2, ... + # params: values for the placeholders of query + # table: target table for write + # truncate: true -> delete all rows of the table before the first write (same transaction) + # typed: true -> values come back as ruby types (Integer, Float, Date, ...) + # false (default) -> all values as String, NULL as nil + # fetch_size: rows per fetch when reading with a block (default 1000) + # + class PostgresStorage < StoragePlugin + def initialize **options, &block + self.class.require_gem 'pg' + @options = options + + begin + @conn = PG.connect( + host: options[:host], + port: options[:port] || '5432', + dbname: options[:database], + user: options[:username], + password: options[:password] + ) + rescue PG::ConnectionBad => e + raise PG::ConnectionBad, "Error connection to database: #{e.message}" + end + + @conn.type_map_for_queries = PG::BasicTypeMapForQueries.new(@conn) + @conn.type_map_for_results = PG::BasicTypeMapForResults.new(@conn) if options[:typed] + + yield if block_given? + end + + # without block all rows at once, with block row by row through a server side cursor + def read(&block) + raise "Query for PostgresStorage missing" unless @options[:query] + + params = @options.fetch(:params, []) + return query(query: @options[:query], params:) unless block_given? + + each_row(@options[:query], params, &block) + end + + def query options = {} + raise "Query for PostgresStorage:query missing" unless options[:query] + + result = @conn.exec_params(options[:query], options.fetch(:params, [])) + return result.to_a unless block_given? + + result.each { |row| yield row } + nil + end + + # + # write rows -> table from option table: + # write table_name, rows -> explicit table (old signature) + # + # all rows in one transaction, values as bind parameters, nil -> NULL + # + def write table_or_data, data = nil + table_name, data = data.nil? ? [@options[:table], table_or_data] : [table_or_data, data] + raise "Table for PostgresStorage#write missing" unless table_name + + @conn.transaction do + truncate_once table_name + data.each { |row| insert table_name, row } + end + end + + def write_row row + raise "Table for PostgresStorage#write_row missing" unless @options[:table] + + truncate_once @options[:table] + insert @options[:table], row + end + + def close + @conn.close if @conn && !@conn.finished? + end + + private + + # rows are fetched in batches inside a transaction; other queries on the + # same connection (lookups, subqueries) are possible while reading + def each_row sql, params + size = Integer(@options.fetch(:fetch_size, 1000)) + @conn.transaction do + @conn.exec_params("declare etl_read no scroll cursor for #{sql.strip.chomp(';')}", params) + loop do + result = @conn.exec("fetch #{size} from etl_read") + break if result.ntuples.zero? + + result.each { |row| yield row } + end + @conn.exec('close etl_read') + end + nil + end + + def truncate_once table_name + return unless @options[:truncate] && !(@truncated ||= {})[table_name] + + @conn.exec("delete from #{quote_table(table_name)}") + @truncated[table_name] = true + end + + def insert table_name, row + columns = row.keys.map { |col| @conn.quote_ident(col.to_s) }.join(',') + placeholders = (1..row.size).map { |i| "$#{i}" }.join(',') + @conn.exec_params( + "insert into #{quote_table(table_name)} (#{columns}) values (#{placeholders})", + row.values + ) + end + + # schema.table -> "schema"."table" + def quote_table name + name.to_s.split('.').map { |part| @conn.quote_ident(part) }.join('.') + end + end +end diff --git a/lib/datapipeline/storage/rest.rb b/lib/datapipeline/storage/rest.rb new file mode 100644 index 0000000..0bafae5 --- /dev/null +++ b/lib/datapipeline/storage/rest.rb @@ -0,0 +1,180 @@ +require 'json' +require_relative 'storage_plugin' + +class DataPipeline + # + # options + # url: base url (read and write) + # headers: hash of request headers + # auth: value of the Authorization header, e.g. "Bearer #{ENV['API_TOKEN']}" + # params: query parameters for read + # root: path to the array in the response, e.g. 'data.items' (default: the response itself) + # timeout: seconds (default 30) + # ssl_verify: false -> accept any server certificate (own CA, IP address); default true + # retries: repeats on 5xx, 429 and network errors (default 2), wait retry_wait * attempt seconds + # + # paging for read (one of) + # page_param: 'page' -> page_start (default 1), page_start + 1, ... until a page returns no rows + # next: ->(body) { body['next'] } -> url of the next page or nil + # max_pages: safety stop (default 1000) + # + # write + # write_url: default url; or ->(row) { "#{base}/kunden/#{row['id']}" } for one url per row + # write_method: :post (default), :put, :patch + # batch_size: nil -> one request per row, n -> json array of n rows per request + # + # query(query: 'path', params: {...}) -> GET url + path, rows like read (for lookups) + # + # call(:post, 'person/listkunden', json: {...}) -> one request, parsed json back (for rpc style apis) + # retries: 0 for requests that create something, so a 5xx is not sent twice + # a failed request raises RestStorage::Error with #status (nil for network errors) + # + class RestStorage < StoragePlugin + class Error < StandardError + attr_reader :status + + def initialize message = nil, status: nil + super(message) + @status = status + end + end + + class RetryableError < Error; end + + RETRY_STATUS = [429, 500, 502, 503, 504].freeze + + def initialize **options, &block + self.class.require_gem 'http' + @options = { timeout: 30, retries: 2, retry_wait: 1, max_pages: 1000, write_method: :post }.merge(options) + @url = @options.fetch(:url) { raise ArgumentError, 'RestStorage: option url missing' } + if @options[:batch_size] && @options[:write_url].respond_to?(:call) + raise ArgumentError, 'RestStorage: write_url per row does not work with batch_size' + end + @buffer = [] + + yield if block_given? + end + + def read(&block) + rows = enum_for(:each_page_row, @url, @options.fetch(:params, {})) + return rows.to_a unless block + + rows.each(&block) + nil + end + + def query options = {} + path = options[:query].to_s + url = path.start_with?('http') ? path : "#{@url.chomp('/')}/#{path.delete_prefix('/')}" + rows = enum_for(:each_page_row, url, options.fetch(:params, {})) + return rows.to_a unless block_given? + + rows.each { |row| yield row } + nil + end + + # one request against url + path; json body and/or query params + def call method, path, json: nil, params: nil, retries: nil + url = path.to_s.start_with?('http') ? path.to_s : "#{@url.chomp('/')}/#{path.to_s.delete_prefix('/')}" + args = { json:, params: }.compact + request(method, url, retries:, **args) + end + + def write rows + rows.each { |row| write_row row } + ensure + close + end + + def write_row row + if @options[:batch_size] + @buffer << row + flush if @buffer.size >= @options[:batch_size] + else + send_rows row + end + end + + def close + flush + end + + private + + def flush + return if @buffer.empty? + + send_rows @buffer + @buffer = [] + end + + def send_rows payload + url = @options.fetch(:write_url, @url) + url = url.call(payload) if url.respond_to?(:call) + request(@options[:write_method], url, json: payload) + end + + def each_page_row url, params + page = @options.fetch(:page_start, 1) + @options[:max_pages].times do + query = @options[:page_param] ? params.merge(@options[:page_param] => page) : params + body = request(:get, url, params: query) + rows = rows_of(body) + rows.each { |row| yield row } + + if @options[:next] + url = @options[:next].call(body) + params = {} + break if url.nil? || url.to_s.empty? + elsif @options[:page_param] + break if rows.empty? + + page += 1 + else + break + end + end + end + + def rows_of body + data = body + @options[:root].to_s.split('.').each { |key| data = data.fetch(key) } if @options[:root] + data = [] if data.nil? + raise Error, "RestStorage: response is not an array (root: #{@options[:root].inspect})" unless data.is_a?(Array) + + data + end + + def request method, url, retries: nil, **args + retries ||= @options[:retries] + args[:ssl_context] = ssl_context if @options[:ssl_verify] == false && url.start_with?('https') + attempt = 0 + begin + attempt += 1 + response = client.request(method, url, **args) + status = response.status.code + text = response.body.to_s + raise RetryableError.new("HTTP #{status}", status:) if RETRY_STATUS.include?(status) + raise Error.new("#{method.upcase} #{url}: HTTP #{status} #{text[0, 200]}", status:) unless response.status.success? + + text.strip.empty? ? nil : JSON.parse(text) + rescue RetryableError, HTTP::Error, SystemCallError => e + raise Error.new("#{method.upcase} #{url}: #{e.message}", status: e.respond_to?(:status) ? e.status : nil) if attempt > retries + + sleep(@options[:retry_wait] * attempt) + retry + end + end + + def ssl_context + require 'openssl' + OpenSSL::SSL::SSLContext.new.tap { |ctx| ctx.verify_mode = OpenSSL::SSL::VERIFY_NONE } + end + + def client + headers = { 'Accept' => 'application/json' }.merge(@options.fetch(:headers, {})) + headers['Authorization'] = @options[:auth] if @options[:auth] + HTTP.timeout(@options[:timeout]).headers(headers) + end + end +end diff --git a/lib/datapipeline/storage/sqlite.rb b/lib/datapipeline/storage/sqlite.rb new file mode 100644 index 0000000..c92f1d0 --- /dev/null +++ b/lib/datapipeline/storage/sqlite.rb @@ -0,0 +1,122 @@ +require_relative 'storage_plugin' + +class DataPipeline + # + # options + # database: path of the database file or ':memory:' + # query: sql for read, placeholders ? or $1 / :name + # params: values for the placeholders (array, or hash for :name) + # table: target table for write + # create: true -> create the table on the first write, columns from the first row + # (Integer -> integer, Float -> real, else text) + # truncate: true -> delete all rows of the table before the first write + # + # rows come back as Hash with sqlite types (Integer, Float, String, nil). + # the database file is opened on first use: a dry run creates no file. + # + class SqliteStorage < StoragePlugin + def initialize **options, &block + self.class.require_gem 'sqlite3' + @options = options + @database = options.fetch(:database) { raise ArgumentError, 'SqliteStorage: option database missing' }.to_s + @prepared = {} + + yield if block_given? + end + + # without block all rows at once, with block row by row + def read(&block) + raise 'Query for SqliteStorage missing' unless @options[:query] + + query(query: @options[:query], params: @options.fetch(:params, []), &block) + end + + def query options = {} + raise 'Query for SqliteStorage:query missing' unless options[:query] + + statement = db.prepare(options[:query]) + result = statement.execute(*bind_args(options.fetch(:params, []))) + return result.map(&:to_h) unless block_given? + + result.each { |row| yield row.to_h } + nil + ensure + statement&.close + end + + # + # write rows -> table from option table: + # write table_name, rows -> explicit table + # + # all rows in one transaction, values as bind parameters, nil -> NULL + # + def write table_or_data, data = nil + table_name, data = data.nil? ? [@options[:table], table_or_data] : [table_or_data, data] + raise 'Table for SqliteStorage#write missing' unless table_name + + return if data.empty? + + db.transaction do + prepare_table table_name, data.first + data.each { |row| insert table_name, row } + end + end + + def write_row row + raise 'Table for SqliteStorage#write_row missing' unless @options[:table] + + prepare_table @options[:table], row + insert @options[:table], row + end + + def close + @db.close if @db && !@db.closed? + end + + private + + def db + @db ||= SQLite3::Database.new(@database).tap { |d| d.results_as_hash = true } + end + + # create / truncate once per table, before its first row + def prepare_table table_name, row + return if @prepared[table_name] + + if @options[:create] + columns = row.map { |key, value| "#{quote(key)} #{column_type(value)}" }.join(', ') + db.execute("create table if not exists #{quote_table(table_name)} (#{columns})") + end + db.execute("delete from #{quote_table(table_name)}") if @options[:truncate] + @prepared[table_name] = true + end + + def column_type value + case value + when Integer then 'integer' + when Float then 'real' + else 'text' + end + end + + # a hash binds named parameters, an array positional ones + def bind_args params + params.is_a?(Hash) ? [params] : Array(params) + end + + def insert table_name, row + columns = row.keys.map { |col| quote(col) }.join(',') + placeholders = (['?'] * row.size).join(',') + values = row.values.map { |v| v.is_a?(Date) || v.is_a?(Time) ? v.iso8601 : v } + db.execute("insert into #{quote_table(table_name)} (#{columns}) values (#{placeholders})", values) + end + + def quote name + %("#{name.to_s.gsub('"', '""')}") + end + + def quote_table name + name.to_s.split('.').map { |part| quote(part) }.join('.') + end + end +end diff --git a/lib/datapipeline/storage/storage_plugin.rb b/lib/datapipeline/storage/storage_plugin.rb new file mode 100644 index 0000000..c60eeee --- /dev/null +++ b/lib/datapipeline/storage/storage_plugin.rb @@ -0,0 +1,58 @@ + + +class DataPipeline + # + # common interface for all storage plugins + # + # read -> Array>, with block: yields each row + # write rows -> writes Array + # write_row row -> writes a single row (streaming) + # query opts -> Array, ad hoc queries from transformations + # set opts -> merge options after creation (e.g. read pg, query: ...) + # close -> release connections and files, safe to call more than once + # + class StoragePlugin + # loads the gem of a plugin on first use, with a hint when it is missing + def self.require_gem feature, gem_name = feature + require feature + rescue LoadError + raise LoadError, "#{name} needs the gem '#{gem_name}': add `gem '#{gem_name}'` to the Gemfile and run bundle install" + end + + def initialize **options + @options = options + end + + def connect + raise NotImplementedError, "#{self.class}#connect is missing" + end + + def read + raise NotImplementedError, "#{self.class}#read is missing" + end + + def write data = nil + raise NotImplementedError, "#{self.class}#write is missing" + end + + def write_row row + raise NotImplementedError, "#{self.class}#write_row is missing" + end + + def query options = {} + raise NotImplementedError, "#{self.class}#query is missing" + end + + def set options + (@options ||= {}).merge! options + self + end + + def close + end + + def finalize + close + end + end +end diff --git a/lib/datapipeline/validation.rb b/lib/datapipeline/validation.rb new file mode 100644 index 0000000..f066010 --- /dev/null +++ b/lib/datapipeline/validation.rb @@ -0,0 +1,53 @@ +require 'date' + +class DataPipeline + # raised by a validate step, handled like any other row error (on_error) + class ValidationError < StandardError; end + + # + # rules per column + # required: true value present and not empty + # max: 35, min: 1 length of the value as text + # format: /\A\d+\z/ regex on the value as text + # in: %w[S H] allowed values + # type: :integer | :decimal | :date + # + # empty values are only checked by required + # + module Validation + TYPES = { + integer: ->(v) { Integer(v.to_s, 10) }, + decimal: ->(v) { Float(v.to_s) }, + date: ->(v) { v.is_a?(Date) ? v : Date.iso8601(v.to_s) } + }.freeze + + module_function + + def errors row, rules + rules.flat_map do |column, rule| + value = row.key?(column) ? row[column] : row[column.to_s] + check(column, value, rule) + end + end + + def check column, value, rule + empty = value.nil? || value.to_s.strip.empty? + return(rule[:required] ? ["#{column}: missing"] : []) if empty + + text = value.to_s + messages = [] + messages << "#{column}: longer than #{rule[:max]}" if rule[:max] && text.length > rule[:max] + messages << "#{column}: shorter than #{rule[:min]}" if rule[:min] && text.length < rule[:min] + messages << "#{column}: format #{rule[:format].inspect}" if rule[:format] && !text.match?(rule[:format]) + messages << "#{column}: not in #{rule[:in].inspect}" if rule[:in] && !rule[:in].include?(value) + if rule[:type] + begin + TYPES.fetch(rule[:type]).call(value) + rescue ArgumentError, TypeError, Date::Error + messages << "#{column}: not #{rule[:type]}" + end + end + messages + end + end +end diff --git a/lib/datapipeline/version.rb b/lib/datapipeline/version.rb new file mode 100644 index 0000000..ea97cc7 --- /dev/null +++ b/lib/datapipeline/version.rb @@ -0,0 +1,8 @@ +class DataPipeline + VERSION = '2.0.0'.freeze + + # kept for scripts that print the version (rake info:info); the history is in CHANGELOG.md + class Version + def self.version = "datapipeline #{VERSION}" + end +end diff --git a/lib/datapipeline/watermark.rb b/lib/datapipeline/watermark.rb new file mode 100644 index 0000000..e3558ed --- /dev/null +++ b/lib/datapipeline/watermark.rb @@ -0,0 +1,60 @@ +require 'json' +require 'fileutils' + +class DataPipeline + # + # remembers the highest value of a column between runs (json file, one entry per name) + # + # wm = watermark :posting, column: 'rechnung_dbid', file: 'data/.state.json', initial: 0 + # read pg, query: 'select ... where rechnung_dbid > $1', params: [wm.value] + # + # the pipeline tracks every source row and saves the new value after a successful run + # (not after an abort, not in dry run). numeric texts are compared as numbers. + # + class Watermark + attr_reader :name, :column, :file, :value, :current + + def initialize name, column:, file:, initial: nil + @name = name.to_s + @column = column + @file = file + @value = load.fetch(@name, initial) + @current = @value + end + + def track row + candidate = row.key?(@column) ? row[@column] : row[@column.to_s] + return if candidate.nil? + + @current = candidate if @current.nil? || greater?(candidate, @current) + end + + def changed? + @current != @value + end + + def save + return unless changed? + + state = load.merge(@name => @current) + FileUtils.mkdir_p(File.dirname(@file)) + File.write("#{@file}.tmp", JSON.pretty_generate(state)) + File.rename("#{@file}.tmp", @file) + @value = @current + end + + private + + def load + File.exist?(@file) ? JSON.parse(File.read(@file)) : {} + end + + def greater? a, b + numeric?(a) && numeric?(b) ? a.to_s.to_r > b.to_s.to_r : a.to_s > b.to_s + end + + def numeric? value + value.is_a?(Numeric) || value.to_s.match?(/\A-?\d+(\.\d+)?\z/) + end + end +end diff --git a/mise.toml b/mise.toml new file mode 100644 index 0000000..c502692 --- /dev/null +++ b/mise.toml @@ -0,0 +1,2 @@ +[tools] +ruby = "3.4.9" diff --git a/test/cli/etl_cli_test.rb b/test/cli/etl_cli_test.rb new file mode 100644 index 0000000..add3399 --- /dev/null +++ b/test/cli/etl_cli_test.rb @@ -0,0 +1,89 @@ +require_relative '../test_helper' +require 'open3' + +# bin/etl as a subprocess: arguments, params, dry run, exit codes +class EtlCliTest < PipelineTest + def etl dir, *args + env = { 'BUNDLE_GEMFILE' => File.join(ROOT, 'Gemfile') } + Open3.capture3(env, RbConfig.ruby, File.join(ROOT, 'bin/etl'), *args, chdir: dir) + end + + def script dir, name, body + File.write("#{dir}/in.csv", "a,b\n1,x\n2,y\nz,q\n") + File.write("#{dir}/#{name}", "require #{File.join(ROOT, 'lib/datapipeline').inspect}\n#{body}") + "#{dir}/#{name}" + end + + # like import.*.rb: own OptionParser for --from/--till, transform api + def test_legacy_script_gets_from_till_and_limit + with_tmpdir do |dir| + path = script(dir, 'legacy.rb', <<~RUBY) + require 'optparse/date' + opts = {} + OptionParser.new { |p| p.on('--from=DATE', Date) { |v| opts[:from] = v }; p.on('--till=DATE', Date) { |v| opts[:till] = v } }.parse! + File.write('args.txt', "\#{opts[:from]} \#{opts[:till]}") + DataPipeline.new do + read container(:csv, file: 'in.csv') + transform(processing: :prepend) { |_, _| { 'h' => 'HEAD' } } + transform { |r, i| r.merge('i' => i) } + write container(:csv, file: 'out.csv', csv: { headers: false, col_sep: '|' }) + end.run + RUBY + _out, err, status = etl(dir, 'run', path, '--from=2025-11-01', '--till=2025-11-30', '--limit=2') + assert_equal 0, status.exitstatus, err + assert_equal '2025-11-01 2025-11-30', File.read("#{dir}/args.txt") + assert_equal "HEAD\n1|x|0\n2|y|1\n", File.read("#{dir}/out.csv") + assert_includes err, 'INFO finish read=2 written=2 rejected=0' + end + end + + def test_params_and_exit_code_2_for_rejected_rows + with_tmpdir do |dir| + path = script(dir, 'neu.rb', <<~RUBY) + DataPipeline.new do + on_error :skip + read container(:csv, file: 'in.csv') + validate 'a' => { format: /\\A\\d+\\z/ } + map { |r| r.merge('land' => params[:land]) } + write container(:csv, file: 'out.csv') + end.run + RUBY + _out, err, status = etl(dir, 'run', path, '--param', 'land=DE') + assert_equal 2, status.exitstatus, err + assert_equal "a,b,land\n1,x,DE\n2,y,DE\n", File.read("#{dir}/out.csv") + + File.delete("#{dir}/out.csv") + out, _err, status = etl(dir, 'run', path, '--param', 'land=AT', '--dry-run', '--quiet') + assert_equal 2, status.exitstatus + refute File.exist?("#{dir}/out.csv") + assert out.start_with?('DRY RUN') + assert_includes out, '"land":"AT"' + end + end + + def test_run_is_optional + with_tmpdir do |dir| + path = script(dir, 'kopie.rb', "DataPipeline.new { read container(:csv, file: 'in.csv'); write container(:csv, file: 'out.csv') }.run\n") + _out, err, status = etl(dir, path, '--quiet') + assert_equal 0, status.exitstatus, err + assert_equal File.read("#{dir}/in.csv"), File.read("#{dir}/out.csv") + end + end + + def test_exit_code_1_on_abort_and_usage + with_tmpdir do |dir| + path = script(dir, 'kaputt.rb', "DataPipeline.new { read container(:csv, file: 'in.csv'); map { raise 'boom' }; write container(:csv, file: 'k.csv') }.run\n") + _out, err, status = etl(dir, 'run', path, '--quiet') + assert_equal 1, status.exitstatus + assert_includes err, 'RuntimeError: boom' + + _out, err, status = etl(dir, 'gibtsnicht.rb') + assert_equal 1, status.exitstatus + assert_includes err, 'gibtsnicht.rb not found' + + _out, err, status = etl(dir) + assert_equal 1, status.exitstatus + assert_includes err, 'Usage: etl [run] PIPELINE.rb' + end + end +end diff --git a/test/core/chained_test.rb b/test/core/chained_test.rb new file mode 100644 index 0000000..820b201 --- /dev/null +++ b/test/core/chained_test.rb @@ -0,0 +1,75 @@ +require_relative '../test_helper' + +# chained steps: map, filter, header, trailer, context +class ChainedTest < PipelineTest + ROWS = [{ 'a' => 1 }, { 'a' => 2 }, { 'a' => 3 }].freeze + + def test_chained_transform_gets_previous_output + out = output(ROWS) { chained; transform { |r| r.merge('b' => 1) }; transform { |r| r.merge('c' => 2) } } + assert_equal ROWS.map { |r| r.merge('b' => 1, 'c' => 2) }, out + end + + def test_one_to_many_then_index + out = output(ROWS) { chained; transform { |r| [r, r] }; transform { |r, i| r.merge('i' => i) } } + assert_equal (0...6).to_a, out.map { |r| r['i'] } + end + + def test_map_and_filter + out = output(ROWS) do + map { |r| r.merge('d' => r['a'] * 2) } + filter { |r| r['d'] > 2 } + map { |r, ctx| r.merge('i' => ctx.index) } + end + assert_equal [{ 'a' => 2, 'd' => 4, 'i' => 0 }, { 'a' => 3, 'd' => 6, 'i' => 1 }], out + end + + def test_map_turns_chaining_on_for_transform + out = output(ROWS) { transform { |r| r.merge('b' => 1) }; map { |r| r.merge('c' => 1) } } + assert_equal 3, out.size + assert(out.all? { |r| r['b'] && r['c'] }) + end + + def test_header_and_trailer_with_context + out = output(ROWS) do + header { |c| { 'H' => c.read, 'W' => c.written } } + filter { |r| r['a'] > 1 } + map { |r| [r, r] } + trailer { |c| { 'T' => c.written } } + end + assert_equal({ 'H' => 3, 'W' => 4 }, out.first) + assert_equal({ 'T' => 4 }, out.last) + assert_equal 6, out.size + end + + def test_edge_order_follows_registration + out = output(ROWS) { header { { 'h' => 1 } }; transform(processing: :prepend) { { 'p' => 1 } }; map { |r| r } } + assert_equal [{ 'h' => 1 }, { 'p' => 1 }], out.first(2) + end + + def test_header_and_trailer_nil_and_arrays + out = output(ROWS) { header { nil }; trailer { [{ 'x' => 1 }, nil, { 'y' => 2 }] }; map { |r| r } } + assert_equal ROWS + [{ 'x' => 1 }, { 'y' => 2 }], out + end + + def test_condition_on_map + assert_equal [{ 'a' => 1 }], output(ROWS) { map(condition: ->(r) { r['a'] == 1 }) { |r| r } } + end + + def test_chained_false + assert_equal 6, output(ROWS) { chained false; transform { |r| r }; transform { |r| r } }.size + end + + def test_csv_end_to_end + with_tmpdir do |dir| + DataPipeline.new do + logger PipelineHelper::QUIET + read MemoryStorage.new(rows: ROWS) + filter { |r| r['a'].odd? } + map { |r| { 'id' => r['a'], 'x' => 'y' } } + trailer { |c| { 'id' => 'TOTAL', 'x' => c.written } } + write container(:csv, file: "#{dir}/c.csv", columns: %w[id x]) + end.run + assert_equal "id,x\n1,y\n3,y\nTOTAL,2\n", File.read("#{dir}/c.csv") + end + end +end diff --git a/test/core/compat_test.rb b/test/core/compat_test.rb new file mode 100644 index 0000000..baeb4e4 --- /dev/null +++ b/test/core/compat_test.rb @@ -0,0 +1,81 @@ +require_relative '../test_helper' + +# +# parallel mode (transform without chained/map/filter) must keep the output of v1.2.1, +# the version the CUDO importers were built on. expected values were recorded with v1.2.1. +# +class CompatTest < PipelineTest + ROWS = (1..6).map { |i| { 'id' => i, 'land' => i.even? ? 'DE' : 'AT' } } + + def assert_output expected_json, &block + assert_equal expected_json, JSON.generate(output(ROWS, &block)) + end + + # pattern of import.customer.rb + def test_customer_pattern + assert_output '[{"SATYP":"FIRC","LFDNR":0},{"id":1,"land":"AT","lfdnr":"0000000001"},' \ + '{"id":2,"land":"DE","lfdnr":"0000000002"},{"id":3,"land":"AT","lfdnr":"0000000003"},' \ + '{"id":4,"land":"DE","lfdnr":"0000000004"},{"id":5,"land":"AT","lfdnr":"0000000005"},' \ + '{"id":6,"land":"DE","lfdnr":"0000000006"},{"topic":"LARC","counter":"0000000006"}]' do + transform(processing: :prepend) { |_row, n| { 'SATYP': 'FIRC', 'LFDNR': n } } + transform(processing: :append) { |_row, n| { 'topic': 'LARC', 'counter': format('%010d', n) } } + transform { |row, _i| row.merge('lfdnr' => format('%010d', row['id'])) } + end + end + + # pattern of import.posting.rb: closure counter, 1:n + def test_posting_pattern + expected = [{ SATYP: 'FIRC', LFDNR: 0 }] + lfdnr = 0 + ROWS.each do |row| + expected << row.merge('lfdnr' => lfdnr += 1) + row['id'].times { |k| expected << { 'pos' => k, 'lfdnr' => lfdnr += 1 } } + end + expected << { SATYPE: 'LARC', SUMME: '0000000027' } + + assert_output JSON.generate(expected) do + cnt = 1 + transform(processing: :prepend) { |_row, _n| { 'SATYP': 'FIRC', 'LFDNR': 0 } } + transform do |row| + out = [row.merge('lfdnr' => cnt)] + cnt += 1 + row['id'].times do |k| + out << { 'pos' => k, 'lfdnr' => cnt } + cnt += 1 + end + out + end + transform(processing: :append) { |_row| { 'SATYPE': 'LARC', 'SUMME': format('%010d', cnt - 1) } } + end + end + + def test_append_registered_before_prepend + assert_output '[{"h":0},{"id":1,"land":"AT"},{"id":2,"land":"DE"},{"id":3,"land":"AT"},' \ + '{"id":4,"land":"DE"},{"id":5,"land":"AT"},{"id":6,"land":"DE"},{"t":6}]' do + transform(processing: :append) { |_, n| { 't' => n } } + transform { |r| r } + transform(processing: :prepend) { |_, n| { 'h' => n } } + end + end + + def test_condition_string + assert_output '[{"id":2,"land":"DE","i":1},{"id":4,"land":"DE","i":3},{"id":6,"land":"DE","i":5}]' do + transform(condition: "row['land'] == 'DE'") { |r, i| r.merge('i' => i) } + end + end + + def test_two_parallel_transforms + expected = ROWS.map { |r| r.merge('a' => 1) } + ROWS.map { |r| r.merge('b' => 1) } + assert_output JSON.generate(expected) do + transform { |r| r.merge('a' => 1) } + transform { |r| [r.merge('b' => 1), nil] } + end + end + + def test_nil_and_empty_array + assert_output '[{"id":1,"land":"AT"}]' do + transform { |r| r['id'] > 3 ? nil : [] } + transform { |r| r['id'] == 1 ? r : nil } + end + end +end diff --git a/test/core/layout_test.rb b/test/core/layout_test.rb new file mode 100644 index 0000000..95b7343 --- /dev/null +++ b/test/core/layout_test.rb @@ -0,0 +1,74 @@ +require_relative '../test_helper' + +class LayoutTest < PipelineTest + LAYOUT = DataPipeline::Layout.new do + record 'HEAD', [['SATYP', 'AN', 4, true], ['DATUM', 'DAT', 8, true], ['ZEIT', 'TIM', 6]] + record 'POS', [['SATYP', 'AN', 4, true], ['NR', 'N', 3, true], ['TEXT', 'AN', 5], ['BETRAG', 'N', '13.2']] + end + + def test_values_in_field_order_any_key_case + assert_equal ['POS', '1', 'abc', nil], LAYOUT.values({ SATYP: 'POS', betrag: nil, 'text' => 'abc', 'NR' => '1' }) + assert_equal ['HEAD', '20251103', nil], LAYOUT.values({ 'satyp' => 'HEAD', 'datum' => '20251103' }) + end + + def test_unknown_field_and_record_type + assert_includes assert_raises(DataPipeline::LayoutError) { LAYOUT.values({ 'satyp' => 'POS', 'extra' => 1 }) }.message, 'extra' + assert_includes assert_raises(DataPipeline::LayoutError) { LAYOUT.values({ 'satyp' => 'XXXX' }) }.message, 'XXXX' + lenient = DataPipeline::Layout.new(strict: false) { record 'A', [['SATYP']] } + assert_equal ['A'], lenient.values({ 'satyp' => 'A', 'extra' => 1 }) + end + + def test_type_field + layout = DataPipeline::Layout.new(type_field: 'art') { record 'X', [['ID'], ['ART']] } + assert_equal %w[7 X], layout.values({ 'id' => '7', 'art' => 'X' }) + end + + def test_errors + assert_empty LAYOUT.errors({ 'satyp' => 'POS', 'nr' => '001', 'text' => 'abcde', 'betrag' => '-123.45' }) + assert_equal ['NR: missing', 'TEXT: longer than 5', 'BETRAG: more than 2 decimals'], + LAYOUT.errors({ 'satyp' => 'POS', 'text' => 'abcdef', 'betrag' => '1.234' }) + assert_equal ['NR: more than 3 digits', 'BETRAG: not a number'], LAYOUT.errors({ 'satyp' => 'POS', 'nr' => '1000', 'betrag' => '1,5' }) + assert_equal ['DATUM: not a date YYYYMMDD', 'ZEIT: not a time HHMMSS'], LAYOUT.errors({ 'satyp' => 'HEAD', 'datum' => '20251331', 'zeit' => '250000' }) + assert_equal ['record type "?" not in layout (HEAD, POS)'], LAYOUT.errors({ 'satyp' => '?' }) + end + + def test_csv_with_layout + with_tmpdir do |dir| + rows = [{ SATYP: 'HEAD', DATUM: '20251103' }, { 'satyp' => 'POS', 'betrag' => '1.50', 'nr' => '1' }] + DataPipeline::CsvStorage.new(file: "#{dir}/l.ok", layout: LAYOUT, csv: { col_sep: '|', quote_char: '' }).write(rows) + assert_equal "HEAD|20251103|\nPOS|1||1.50\n", File.read("#{dir}/l.ok") + end + end + + def test_validate_layout_abort_warn_and_ignore + rows = [{ 'satyp' => 'POS', 'nr' => '1' }, { 'satyp' => 'POS', 'text' => 'zu lang' }] + assert_raises(DataPipeline::ValidationError) { run_pipeline(rows) { validate LAYOUT } } + + target, stats = run_pipeline(rows) { validate LAYOUT, warn: true } + assert_equal 2, target.output.size + assert_equal 1, stats.warnings + assert_match(/warnings=1/, stats.to_s) + + _target, stats = run_pipeline(rows) { validate LAYOUT, warn: true, ignore: %w[NR TEXT] } + assert_equal 0, stats.warnings + end + + def test_warning_log_is_capped + log = StringIO.new + stats = DataPipeline.new do + logger Logger.new(log) + read MemoryStorage.new(rows: Array.new(150) { { 'satyp' => 'POS' } }) + write MemoryStorage.new + validate LAYOUT, warn: true + end.run + assert_equal 150, stats.warnings + assert_equal 101, log.string.lines.count { |l| l.include?('WARN') } + assert_includes log.string, 'more than 100 warnings' + end + + def test_validate_hash_without_braces + target, stats = run_pipeline([{ 'a' => 'x' }, { 'a' => '1' }]) { on_error :skip; validate 'a' => { format: /\A\d\z/ } } + assert_equal [{ 'a' => '1' }], target.output + assert_equal 1, stats.rejected + end +end diff --git a/test/core/operations_test.rb b/test/core/operations_test.rb new file mode 100644 index 0000000..65878e6 --- /dev/null +++ b/test/core/operations_test.rb @@ -0,0 +1,178 @@ +require_relative '../test_helper' + +# on_error, validate, stats, hooks, watermark, dry run, limit +class OperationsTest < PipelineTest + ROWS = (1..5).map { |i| { 'id' => i.to_s, 'shkzg' => i == 3 ? 'X' : 'S', 'betrag' => i == 4 ? 'abc' : "#{i}.50" } }.freeze + RULES = { 'id' => { required: true, format: /\A\d+\z/ }, 'shkzg' => { in: %w[S H] }, 'betrag' => { type: :decimal } }.freeze + + def failing_map = proc { map { |r| r['id'] == '2' ? raise('kaputt') : r } } + + def test_abort_is_default + error = assert_raises(RuntimeError) { run_pipeline(ROWS, &failing_map) } + assert_equal 'kaputt', error.message + assert_equal 'kaputt', DataPipeline.runs.last.error.message + end + + def test_skip + map = failing_map + target, stats = run_pipeline(ROWS) { on_error :skip; instance_eval(&map) } + assert_equal 4, target.output.size + assert_equal 1, stats.rejected + assert_equal 'map#1', stats.errors.first[:step] + refute stats.ok? + end + + def test_collect_batch_and_stream + [false, true].each do |streamed| + rejects = MemoryStorage.new + map = failing_map + target, = run_pipeline(ROWS, streamed:) { on_error :collect, to: rejects; instance_eval(&map) } + assert_equal [ROWS[1].merge('_step' => 'map#1', '_error' => 'kaputt')], rejects.streamed_rows + assert_equal 4, target.output.size + end + end + + def test_on_error_arguments + assert_raises(ArgumentError) { DataPipeline.new { on_error :collect } } + assert_raises(ArgumentError) { DataPipeline.new { on_error :ignore } } + end + + def test_edge_errors_always_abort + assert_raises(RuntimeError) { run_pipeline(ROWS) { on_error :skip; header { raise 'h' }; map { |r| r } } } + end + + def test_skip_in_parallel_mode + target, stats = run_pipeline(ROWS) { on_error :skip; transform { |r| r['id'] == '1' ? raise('x') : r } } + assert_equal 4, target.output.size + assert_equal 1, stats.rejected + end + + def test_validate_rules + assert_raises(DataPipeline::ValidationError) { run_pipeline(ROWS) { validate RULES } } + target, stats = run_pipeline(ROWS) { on_error :skip; validate RULES } + assert_equal %w[1 2 5], target.output.map { |r| r['id'] } + assert_equal 2, stats.rejected + end + + def test_validate_block + target, stats = run_pipeline(ROWS) { on_error :skip; validate { |r| 'zu gross' if r['id'].to_i > 3 } } + assert_equal 3, target.output.size + assert_equal 'zu gross', stats.errors.first[:error] + end + + def test_validation_messages + errors = DataPipeline::Validation.errors({ 'a' => '12345', 'd' => '2025-13-01' }, + 'a' => { max: 3, format: /x/ }, 'd' => { type: :date }, 'b' => { required: true }) + assert_equal ['a: longer than 3', 'a: format /x/', 'd: not date', 'b: missing'], errors + assert_empty DataPipeline::Validation.errors({ 'a' => '' }, 'a' => { max: 1, type: :integer }) + end + + def test_validate_turns_chaining_on + target, = run_pipeline(ROWS) { on_error :skip; transform { |r| r.merge('x' => 1) }; validate RULES } + assert_equal 3, target.output.size + assert(target.output.all? { |r| r['x'] }) + end + + def test_stats + _target, stats = run_pipeline(ROWS) { map { |r| [r, r] } } + assert_equal [5, 10, 10], [stats.read, stats.written, stats.output] + assert stats.ok? + assert_kind_of Float, stats.duration + map = failing_map + _target, stats = run_pipeline(ROWS) { on_error :skip; instance_eval(&map) } + assert_match(/\Aread=5 written=4 rejected=1/, stats.to_s) + end + + def test_on_finish_after_success_and_abort + seen = [] + run_pipeline(ROWS) { on_finish { |s| seen << s.error } } + assert_raises(RuntimeError) { run_pipeline(ROWS) { on_finish { |s| seen << s.error.class }; map { raise 'x' } } } + assert_equal [nil, RuntimeError], seen + end + + def test_hook_error_is_only_logged + log = StringIO.new + stats = DataPipeline.new do + logger Logger.new(log) + read MemoryStorage.new(rows: ROWS) + write MemoryStorage.new + on_finish { raise 'hook' } + end.run + assert stats.ok? + assert_includes log.string, 'on_finish: RuntimeError: hook' + end + + def test_logger_messages + log = StringIO.new + DataPipeline.new do + logger Logger.new(log) + on_error :skip + read MemoryStorage.new(rows: ROWS) + write MemoryStorage.new + map { |r| r['id'] == '1' ? raise('weg') : r } + end.run + assert_includes log.string, 'rejected by map#1: weg' + assert_includes log.string, 'finish read=5 written=4 rejected=1' + end + + def test_watermark + with_tmpdir do |dir| + file = "#{dir}/state.json" + wm = nil + run_pipeline(ROWS) { wm = watermark :t, column: 'id', file:, initial: 0 } + assert_equal '5', wm.value + + run_pipeline([{ 'id' => '9' }, { 'id' => '10' }]) { watermark :t, column: 'id', file: } + assert_equal '10', JSON.parse(File.read(file))['t'], 'numeric texts compared as numbers' + + assert_raises(RuntimeError) { run_pipeline([{ 'id' => '99' }]) { watermark :t, column: 'id', file:; map { raise 'x' } } } + assert_equal '10', JSON.parse(File.read(file))['t'], 'not saved after abort' + + run_pipeline(ROWS, streamed: true) { watermark :s, column: 'id', file: } + assert_equal({ 't' => '10', 's' => '5' }, JSON.parse(File.read(file))) + end + end + + def test_dry_run + with_tmpdir do |dir| + rejects = MemoryStorage.new + target = nil + text = capture_stdout do + target, = run_pipeline(ROWS, dry_run: true, preview: 2) do + watermark :d, column: 'id', file: "#{dir}/state.json" + on_error :collect, to: rejects + map { |r| r['id'] == '1' ? raise('x') : r } + end + end + assert_nil target.written + assert_empty target.streamed_rows + assert_empty rejects.streamed_rows + refute File.exist?("#{dir}/state.json") + assert text.start_with?('DRY RUN') + assert_equal 3, text.lines.size + assert DataPipeline.runs.last.dry_run + end + end + + def test_dry_run_creates_no_csv_file + with_tmpdir do |dir| + capture_stdout do + DataPipeline.new(dry_run: true) do + logger PipelineHelper::QUIET + read MemoryStorage.new(rows: ROWS) + write container(:csv, file: "#{dir}/dry.csv") + end.run + end + refute File.exist?("#{dir}/dry.csv") + end + end + + def test_limit + target, stats = run_pipeline(ROWS, limit: 2) { map { |r| r } } + assert_equal [2, 2], [target.output.size, stats.read] + + target, stats = run_pipeline(ROWS, streamed: true, limit: 3) { trailer { |c| { 'n' => c.read } } } + assert_equal({ 'n' => 3 }, target.output.last) + assert_equal 3, stats.read + end +end diff --git a/test/core/pipeline_test.rb b/test/core/pipeline_test.rb new file mode 100644 index 0000000..63040a3 --- /dev/null +++ b/test/core/pipeline_test.rb @@ -0,0 +1,131 @@ +require_relative '../test_helper' + +# dsl, settings, plugin lookup, parallel mode +class PipelineCoreTest < PipelineTest + ROWS = [{ 'a' => 1 }, { 'a' => 2 }, { 'a' => 3 }].freeze + + def settings_of(pipeline) = pipeline.instance_variable_get(:@settings) + + def test_nil_filters_rows + assert_equal [{ 'a' => 1 }], output(ROWS) { transform { |r| r['a'] == 1 ? r : nil } } + assert_equal ROWS, output(ROWS) { transform { |r| [r, nil] } } + assert_equal ROWS, output(ROWS) { transform(processing: :prepend) { nil }; transform { |r| r }; transform(processing: :append) { nil } } + end + + def test_dsl_switches + assert_equal false, settings_of(DataPipeline.new { verbose false })[:verbose] + assert_equal true, settings_of(DataPipeline.new { verbose })[:verbose] + assert_equal false, settings_of(DataPipeline.new { streamed false })[:streamed] + assert_equal '1.0', settings_of(DataPipeline.new { version '1.0' })[:version] + assert_equal true, settings_of(DataPipeline.new { configure verbose: true })[:verbose] + end + + def test_typo_raises + assert_raises(NoMethodError) { DataPipeline.new { verbsoe true } } + end + + def test_unknown_plugin + error = assert_raises(RuntimeError) { DataPipeline.new { container :gibtsnicht } } + assert_includes error.message, 'GibtsnichtStorage' + end + + def test_plugin_errors_pass_through + assert_raises(ArgumentError) { DataPipeline.new { container :csv } } + broken = Class.new(DataPipeline::StoragePlugin) { def initialize(**) = undefined_thing } + DataPipeline.register :broken, broken + assert_includes assert_raises(NameError) { DataPipeline.new { container :broken } }.message, 'undefined_thing' + end + + def test_register_lazy_path + with_tmpdir do |dir| + File.write("#{dir}/lazy_storage.rb", "class LazyStorage < MemoryStorage; end\n") + DataPipeline.register :lazy, 'LazyStorage', path: "#{dir}/lazy_storage.rb" + assert_empty $LOADED_FEATURES.grep(/lazy_storage/) + storage = nil + DataPipeline.new { storage = container :lazy } + assert_equal 'LazyStorage', storage.class.name + refute_empty $LOADED_FEATURES.grep(/lazy_storage/) + end + end + + def test_csv_plugin_by_convention + assert_equal DataPipeline::CsvStorage, DataPipeline.plugin(:csv) + end + + def test_settings_are_frozen_and_replaced + assert DataPipeline.settings.frozen? + pipeline = DataPipeline.new { logger PipelineHelper::QUIET } + DataPipeline.settings(verbose: true) + assert_equal false, settings_of(pipeline)[:verbose] + assert_equal true, DataPipeline.settings[:verbose] + DataPipeline.reset_settings! + assert_equal DataPipeline::DEFAULT_SETTINGS, DataPipeline.settings + end + + def test_settings_per_pipeline + _target, stats = run_pipeline(ROWS, limit: 1) + assert_equal 1, stats.read + assert_nil DataPipeline.settings[:limit] + end + + def test_parallel_threads + reads = Array.new(8) { |i| Thread.new { run_pipeline(ROWS, limit: (i % 3) + 1).last.read } }.map(&:value) + assert_equal Array.new(8) { |i| (i % 3) + 1 }, reads + end + + def test_parallel_mode_is_side_by_side + out = output(ROWS) { transform { |r| r.merge('b' => 1) }; transform { |r| r.merge('c' => 2) } } + assert_equal 6, out.size + end + + def test_parallel_mode_copies_rows_per_step + out = output(ROWS) { transform { |r| r['x'] = 1; r }; transform { |r| r.key?('x') ? nil : r } } + assert_equal 6, out.size + assert_equal [{ 'a' => 10 }, { 'a' => 20 }, { 'a' => 30 }], output(ROWS) { transform { |r| [r.transform_values! { |v| v * 10 }] } } + end + + def test_stable_order_and_all_prepends_appends + out = output(ROWS) { 20.times { |k| transform { |r| r['a'] == 1 ? { 'k' => k } : nil } } } + assert_equal (0...20).to_a, out.map { |r| r['k'] } + + out = output(ROWS) do + transform(processing: :prepend) { { 'h' => 1 } } + transform(processing: :prepend) { { 'h' => 2 } } + transform { |r| r } + transform(processing: :append) { { 't' => 1 } } + transform(processing: :append) { { 't' => 2 } } + end + assert_equal [{ 'h' => 1 }, { 'h' => 2 }, *ROWS, { 't' => 1 }, { 't' => 2 }], out + end + + def test_append_count_parallel_vs_chained + assert_equal({ 'n' => 3 }, output(ROWS) { transform { |r| [r, r] }; transform(processing: :append) { |_, n| { 'n' => n } } }.last) + assert_equal({ 'n' => 6 }, output(ROWS) { chained; transform { |r| [r, r] }; transform(processing: :append) { |_, n| { 'n' => n } } }.last) + end + + def test_conditions + assert_equal [{ 'a' => 1 }, { 'a' => 3 }], output(ROWS) { transform(condition: ->(r) { r['a'] != 2 }) { |r| r } } + assert_equal [{ 'a' => 2 }], output(ROWS) { transform(condition: "row['a'] == 2") { |r| r } } + end + + def test_copy_without_steps + assert_equal ROWS, output(ROWS) + end + + def test_closes_source_and_target + source = MemoryStorage.new(rows: ROWS) + target = MemoryStorage.new + DataPipeline.new { logger PipelineHelper::QUIET; read source; write target }.run + assert source.closed? + assert target.closed? + end + + def test_read_and_write_required + assert_includes assert_raises(ArgumentError) { DataPipeline.new { logger PipelineHelper::QUIET; read MemoryStorage.new }.run }.message, 'write' + assert_includes assert_raises(ArgumentError) { DataPipeline.new { logger PipelineHelper::QUIET; write MemoryStorage.new }.run }.message, 'read' + end + + def test_block_required + assert_raises(ArgumentError) { DataPipeline.new { map } } + end +end diff --git a/test/core/streaming_test.rb b/test/core/streaming_test.rb new file mode 100644 index 0000000..850f7ea --- /dev/null +++ b/test/core/streaming_test.rb @@ -0,0 +1,153 @@ +require_relative '../test_helper' + +# streamed mode, lookups, several targets +class StreamingTest < PipelineTest + ROWS = [{ 'a' => 1 }, { 'a' => 2 }, { 'a' => 3 }].freeze + + CHAIN = proc do + header { |c| { 'H' => c.written } } + filter { |r| r['a'] > 1 } + map { |r, c| [r, r.merge('k' => c.index)] } + map { |r, c| r.merge('i' => c.index) } + trailer { |c| { 'T' => c.written, 'R' => c.read } } + end + + def test_batch_and_stream_same_body + batch = output(ROWS, &CHAIN) + stream = output(ROWS, streamed: true, &CHAIN) + assert_equal batch[1..-2], stream[1..-2] + assert_equal({ 'H' => 4 }, batch.first) + assert_equal({ 'H' => nil }, stream.first) + assert_equal({ 'T' => 4, 'R' => 3 }, stream.last) + end + + def test_stream_prepend_append + out = output(ROWS, streamed: true) do + chained + transform(processing: :prepend) { |_, n| { 'p' => n } } + transform { |r| [r, r] } + transform(processing: :append) { |_, n| { 'n' => n } } + end + assert_equal({ 'p' => 0 }, out.first) + assert_equal({ 'n' => 6 }, out.last) + assert_equal 8, out.size + end + + def test_stream_parallel_single_step_and_error_for_several + out = output(ROWS, streamed: true) { transform { |r| [r, r] }; transform(processing: :append) { |_, n| { 'n' => n } } } + assert_equal({ 'n' => 3 }, out.last) + error = assert_raises(ArgumentError) { output(ROWS, streamed: true) { transform { |r| r }; transform { |r| r } } } + assert_includes error.message, 'chained' + end + + def test_stream_condition_and_copy + assert_equal [{ 'a' => 1 }, { 'a' => 3 }], output(ROWS, streamed: true) { map(condition: ->(r) { r['a'] != 2 }) { |r| r } } + assert_equal ROWS, output(ROWS, streamed: true) + end + + def test_stream_csv_to_csv + with_tmpdir do |dir| + File.write("#{dir}/in.csv", "a,b\n1,x\n2,y\n3,z\n") + DataPipeline.new do + logger PipelineHelper::QUIET + streamed + read container(:csv, file: "#{dir}/in.csv") + filter { |r| r['a'] != '2' } + trailer { |c| { 'a' => 'T', 'b' => c.written } } + write container(:csv, file: "#{dir}/out.csv") + end.run + assert_equal "a,b\n1,x\n3,z\nT,2\n", File.read("#{dir}/out.csv") + end + end + + def test_stream_postgres_with_query_on_same_connection + with_pg_database do |db| + pg_exec db, "create table k as select i as id, case when i % 2 = 0 then 'DE' else 'AT' end as land from generate_series(1, 5) i; + create table l (land text, name text); insert into l values ('DE','Deutschland'),('AT','Oesterreich')" + with_tmpdir do |dir| + DataPipeline.new do + logger PipelineHelper::QUIET + streamed + pg = container :postgres, **db + read pg, query: 'select id, land from k where id <= $1 order by id', params: [5], fetch_size: 2 + map { |r| r.merge('name' => query(pg, query: 'select name from l where land = $1', params: [r['land']]).first['name']) } + write container(:csv, file: "#{dir}/o.csv", columns: %w[id name]) + end.run + assert_equal 6, File.readlines("#{dir}/o.csv").size + assert_includes File.read("#{dir}/o.csv"), '2,Deutschland' + end + end + end + + def test_lookup_grouped_unique_and_lambda + with_tmpdir do |dir| + File.write("#{dir}/lk.csv", "id,grp\n1,a\n2,b\n3,a\n") + grouped = unique = by_lambda = nil + DataPipeline.new do + csv = container(:csv, file: "#{dir}/lk.csv") + grouped = lookup csv, key: 'grp' + unique = lookup csv, key: 'id', unique: true + by_lambda = lookup csv, key: ->(r) { r['grp'].upcase } + end + assert_equal 2, grouped['a'].size + assert_equal 'b', unique['2']['grp'] + assert_equal %w[A B], by_lambda.keys.sort + end + end + + def test_lookup_postgres_join_without_n_plus_one + with_pg_database do |db| + pg_exec db, "create table k as select i as id, case when i % 2 = 0 then 'DE' else 'AT' end as land from generate_series(1, 3) i; + create table l (land text, name text); insert into l values ('DE','Deutschland'),('AT','Oesterreich')" + out = nil + DataPipeline.new do + logger PipelineHelper::QUIET + pg = container :postgres, **db + names = lookup pg, query: 'select * from l where land like $1', params: ['%'], key: 'land', unique: true + read pg, query: 'select id, land from k order by id' + map { |r| r.merge('name' => names.dig(r['land'], 'name')) } + write(out = MemoryStorage.new) + end.run + assert_equal %w[Oesterreich Deutschland Oesterreich], out.output.map { |r| r['name'] } + end + end + + def test_several_targets_with_only + rows = [{ 'satyp' => 'DOHD', 'n' => 1 }, { 'satyp' => 'DOPS', 'n' => 2 }, { 'satyp' => 'DOPS', 'n' => 3 }] + [false, true].each do |streamed| + all = MemoryStorage.new + heads = MemoryStorage.new + positions = MemoryStorage.new + trailers = MemoryStorage.new + DataPipeline.new(streamed:) do + logger PipelineHelper::QUIET + read MemoryStorage.new(rows:) + header { { 'satyp' => 'FIRC' } } + map { |r| r } + trailer { |c| { 'satyp' => 'LARC', 'n' => c.written } } + write all + write heads, only: ->(r) { r['satyp'] == 'DOHD' } + write positions, only: ->(r) { r['satyp'] == 'DOPS' } + write trailers, only: ->(r) { r['satyp'] == 'LARC' } + end.run + assert_equal 5, all.output.size, "streamed=#{streamed}" + assert_equal [1], heads.output.map { |r| r['n'] } + assert_equal 2, positions.output.size + assert_equal [{ 'satyp' => 'LARC', 'n' => 3 }], trailers.output + end + end + + def test_memory_stays_flat_when_streaming + with_tmpdir do |dir| + File.open("#{dir}/big.csv", 'w') { |f| f.puts 'id,text'; 50_000.times { |i| f.puts "#{i},#{'x' * 80}" } } + stats = DataPipeline.new(streamed: true) do + logger PipelineHelper::QUIET + read container(:csv, file: "#{dir}/big.csv") + map { |r| r.merge('len' => r['text'].size) } + write container(:csv, file: "#{dir}/out.csv", columns: %w[id len]) + end.run + assert_equal 50_000, stats.written + assert_equal 50_001, File.foreach("#{dir}/out.csv").count + end + end +end diff --git a/test/examples/doc_examples_test.rb b/test/examples/doc_examples_test.rb new file mode 100644 index 0000000..42ee237 --- /dev/null +++ b/test/examples/doc_examples_test.rb @@ -0,0 +1,155 @@ +require_relative '../test_helper' +require_relative '../../examples/demo_api' +require 'open3' +require 'fileutils' + +# +# the examples of README.md section 1: the code in the documentation is the file, +# and every example runs and produces the documented result +# +class DocExamplesTest < PipelineTest + DOC = File.read(File.join(ROOT, 'README.md')) + EXAMPLES = DOC.scan(/\n```ruby\n(.*?)\n```/m) + + def run_example dir, name, *args + env = { 'BUNDLE_GEMFILE' => File.join(ROOT, 'Gemfile') } + Open3.capture3(env, RbConfig.ruby, File.join(ROOT, 'bin/etl'), 'run', File.join(ROOT, name), "--out=#{dir}", *args, + chdir: dir) + end + + def test_documented_code_equals_files + assert_equal %w[examples/kunden_import.rb examples/rechnungen_satzarten.rb examples/bewegungen_inkrementell.rb + examples/eigenes_plugin.rb examples/rest_lesen.rb examples/rest_schreiben.rb + examples/rest_zu_rest.rb], EXAMPLES.map(&:first) + EXAMPLES.each do |file, code| + assert_equal File.read(File.join(ROOT, file)).rstrip, code.rstrip, "#{file} differs from README.md" + end + end + + def test_kunden_import + with_tmpdir do |dir| + out, err, status = run_example(dir, 'examples/kunden_import.rb') + assert_equal 2, status.exitstatus, err + assert_includes out, 'kunden_import: read=9 written=4 rejected=4' + + db = DataPipeline::SqliteStorage.new(database: "#{dir}/kunden.sqlite3", query: 'select * from kunden order by kundennr') + rows = db.read + db.close + assert_equal %w[100001 100002 100003 100009], rows.map { |r| r['kundennr'] } + assert_equal({ 'kundennr' => '100002', 'name' => 'Huber & Partner', 'land' => 'Österreich', 'eu' => 'ja', 'umsatz' => 8300.0 }, rows[1]) + + excel = DataPipeline::ExcelStorage.new(file: "#{dir}/eu_kunden.xlsx", sheet: 'EU-Kunden').read + assert_equal %w[100001 100002 100009], excel.map { |r| r['kundennr'] } + + errors = DataPipeline::CsvStorage.new(file: "#{dir}/kunden_fehler.csv").read + assert_equal ['name: missing', 'kundennr: format /\A\d{6}\z/', 'Land XX unbekannt', 'umsatz: not decimal'], + errors.map { |r| r['_error'] } + + # a second run replaces the table content (truncate) + run_example(dir, 'examples/kunden_import.rb') + db = DataPipeline::SqliteStorage.new(database: "#{dir}/kunden.sqlite3", query: 'select count(*) as n from kunden') + assert_equal 4, db.read.first['n'] + db.close + end + end + + def test_kunden_import_dry_run_creates_no_files + with_tmpdir do |dir| + out, _err, status = run_example(dir, 'examples/kunden_import.rb', '--dry-run') + assert_equal 2, status.exitstatus + assert_includes out, 'DRY RUN' + assert_empty Dir.children(dir) + end + end + + def test_rechnungen_satzarten + with_tmpdir do |dir| + _out, err, status = run_example(dir, 'examples/rechnungen_satzarten.rb') + assert status.success?, err + assert_includes err, 'TEXT: longer than 30' + documented = DOC[/Ergebnis `tmp\/rechnungen.txt`.*?```text\n(.*?)```/m, 1] + expected = documented.sub(/\ADATEI\|\d{8}/, "DATEI|#{Date.today.strftime('%Y%m%d')}") + assert_equal expected, File.read("#{dir}/rechnungen.txt") + assert_equal 3, File.readlines("#{dir}/positionen.ndjson").size + end + end + + def test_bewegungen_inkrementell + with_tmpdir do |dir| + out, err, status = run_example(dir, 'examples/bewegungen_inkrementell.rb') + assert status.success?, err + assert_includes out, 'neu: 4, Stand jetzt: 4' + file = Dir["#{dir}/bewegungen_*.ndjson"].first + assert_equal({ 'id' => 2, 'konto' => '4000', 'betrag' => 20.0, 'soll_haben' => 'H' }, JSON.parse(File.readlines(file)[1])) + + FileUtils.rm(file) + out, = run_example(dir, 'examples/bewegungen_inkrementell.rb') + assert_includes out, 'neu: 0, Stand jetzt: 4' + assert_empty Dir["#{dir}/bewegungen_*.ndjson"], 'no file without new rows' + end + end + + def test_eigenes_plugin + with_tmpdir do |dir| + _out, err, status = run_example(dir, 'examples/eigenes_plugin.rb') + assert status.success?, err + documented = DOC[/Ergebnis `tmp\/alle_filialen.csv`:\n\n```text\n(.*?)```/m, 1] + assert_equal documented, File.read("#{dir}/alle_filialen.csv") + end + end + + def with_demo_api + api = DemoApi.start(port: 0) + yield api + ensure + api&.stop + end + + def test_rest_lesen + with_demo_api do |api| + with_tmpdir do |dir| + out, err, status = run_example(dir, 'examples/rest_lesen.rb', "--url=#{api.url}") + assert_equal 2, status.exitstatus, err + assert_includes out, 'rest_lesen: read=7 written=6 rejected=1' + documented = DOC[/Ergebnis `tmp\/api_kunden.csv`:\n\n```text\n(.*?)```/m, 1] + assert_equal documented, File.read("#{dir}/api_kunden.csv") + assert_equal 6, JSON.parse(File.read("#{dir}/api_kunden.json")).size + assert_equal ['umsatz: not decimal'], DataPipeline::CsvStorage.new(file: "#{dir}/api_fehler.csv").read.map { |r| r['_error'] } + end + end + end + + def test_rest_schreiben_and_dry_run + with_demo_api do |api| + with_tmpdir do |dir| + out, _err, status = run_example(dir, 'examples/rest_schreiben.rb', "--url=#{api.url}", '--dry-run') + assert_equal 2, status.exitstatus + assert_includes out, 'DRY RUN' + assert_empty api.imported, 'dry run sends nothing' + + out, err, status = run_example(dir, 'examples/rest_schreiben.rb', "--url=#{api.url}") + assert_equal 2, status.exitstatus, err + assert_includes out, 'gesendet: 6 Kunden in 3 Paketen' + assert_equal %w[100001 100002 100003 100007 100008 100009], api.imported.map { |r| r['kundennr'] } + assert_equal 2, DataPipeline::CsvStorage.new(file: "#{dir}/import_fehler.csv").read.size + end + end + end + + def test_rest_zu_rest + with_demo_api do |api| + with_tmpdir do |dir| + out, err, status = run_example(dir, 'examples/rest_zu_rest.rb', "--url=#{api.url}") + assert status.success?, err + assert_includes out, 'archiviert: 5, Stand jetzt: 5' + assert_equal %w[1 2 3 4 5], api.archive.keys + assert_equal Date.today.iso8601, api.archive['3']['archiviert'] + + api.archive.clear + out, = run_example(dir, 'examples/rest_zu_rest.rb', "--url=#{api.url}") + assert_includes out, 'archiviert: 0, Stand jetzt: 5' + assert_empty api.archive + end + end + end +end diff --git a/test/plugins/csv_storage_test.rb b/test/plugins/csv_storage_test.rb new file mode 100644 index 0000000..643557b --- /dev/null +++ b/test/plugins/csv_storage_test.rb @@ -0,0 +1,89 @@ +require_relative '../test_helper' +require_relative '../support/storage_contract' + +class CsvStorageTest < PipelineTest + include StorageContract + + def contract_storage(dir, _purpose) = DataPipeline::CsvStorage.new(file: "#{dir}/data.csv") + + def test_read_with_defaults + with_tmpdir do |dir| + File.write("#{dir}/in.csv", "a,b\n1,x\n2,\"\"\n") + assert_equal [{ 'a' => '1', 'b' => 'x' }, { 'a' => '2', 'b' => '' }], DataPipeline::CsvStorage.new(file: "#{dir}/in.csv").read + end + end + + def test_read_without_header_uses_columns_or_index + with_tmpdir do |dir| + File.write("#{dir}/in.csv", "1|x\n2|y\n") + csv = { headers: false, col_sep: '|' } + assert_equal [{ 'id' => '1', 'txt' => 'x' }, { 'id' => '2', 'txt' => 'y' }], + DataPipeline::CsvStorage.new(file: "#{dir}/in.csv", columns: %w[id txt], csv:).read + assert_equal({ '0' => '1', '1' => 'x' }, DataPipeline::CsvStorage.new(file: "#{dir}/in.csv", csv:).read.first) + end + end + + def test_read_streaming_with_block + with_tmpdir do |dir| + File.write("#{dir}/in.csv", "a\n1\n2\n") + rows = [] + DataPipeline::CsvStorage.new(file: "#{dir}/in.csv").read { |row| rows << row } + assert_equal %w[1 2], rows.map { |r| r['a'] } + end + end + + def test_defaults_are_merged + options = DataPipeline::CsvStorage.new(file: 'x', csv: { col_sep: ';' }).instance_variable_get(:@options)[:csv] + assert_equal ';', options[:col_sep] + assert options[:headers] + end + + def test_file_is_required + error = assert_raises(ArgumentError) { DataPipeline::CsvStorage.new } + assert_includes error.message, 'file' + end + + def test_write_without_csv_options + with_tmpdir do |dir| + DataPipeline::CsvStorage.new(file: "#{dir}/o.csv").write([{ 'a' => 1, 'b' => 2 }]) + assert_equal "a,b\n1,2\n", File.read("#{dir}/o.csv") + end + end + + def test_columns_fix_order_missing_fields_and_key_types + with_tmpdir do |dir| + DataPipeline::CsvStorage.new(file: "#{dir}/o.csv", columns: %w[a b c]).write([{ 'b' => 2, 'a' => 1 }, { c: 3, a: 4 }]) + assert_equal "a,b,c\n1,2,\n4,,3\n", File.read("#{dir}/o.csv") + end + end + + def test_empty_data + with_tmpdir do |dir| + DataPipeline::CsvStorage.new(file: "#{dir}/a.csv").write([]) + DataPipeline::CsvStorage.new(file: "#{dir}/b.csv", columns: %w[a b]).write([]) + assert_equal '', File.read("#{dir}/a.csv") + assert_equal "a,b\n", File.read("#{dir}/b.csv") + end + end + + def test_write_row_and_close + with_tmpdir do |dir| + csv = DataPipeline::CsvStorage.new(file: "#{dir}/o.csv") + csv.write_row({ 'a' => 1 }) + csv.write_row({ 'a' => 2 }) + csv.close + assert_equal "a\n1\n2\n", File.read("#{dir}/o.csv") + end + end + + # options of the CUDO exports: no header, pipe, no quotes, mixed record types and key types + def test_cudo_format_mixed_record_types + with_tmpdir do |dir| + rows = [{ SATYP: 'FIRC', LFDNR: 0 }, { 'satyp' => 'DOHD', 'belnr' => 'AR1', 'wwert' => nil }, + { 'satyp' => 'DOPS', 'dmbtr' => '1.50' }, { 'SATYPE' => 'LARC', 'SUMME' => '0000000002' }] + DataPipeline::CsvStorage.new(file: "#{dir}/c.ok", csv: { headers: false, col_sep: '|', quote_char: '', encoding: 'utf-8' }) + .write(rows) + assert_equal "FIRC|0\nDOHD|AR1|\nDOPS|1.50\nLARC|0000000002\n", File.read("#{dir}/c.ok") + end + end +end diff --git a/test/plugins/excel_storage_test.rb b/test/plugins/excel_storage_test.rb new file mode 100644 index 0000000..0fbd617 --- /dev/null +++ b/test/plugins/excel_storage_test.rb @@ -0,0 +1,57 @@ +require_relative '../test_helper' +require_relative '../support/storage_contract' + +class ExcelStorageTest < PipelineTest + include StorageContract + + def contract_storage(dir, _purpose) = DataPipeline::ExcelStorage.new(file: "#{dir}/data.xlsx") + + # an empty text is an empty cell in excel + def contract_rows(rows) = rows.map { |row| row.transform_values { |v| v == '' ? nil : v } } + + def test_text_stays_text_numbers_and_dates_keep_type + with_tmpdir do |dir| + rows = [{ 'lfdnr' => '0000000001', 'betrag' => 1.5, 'menge' => 3, 'datum' => Date.new(2025, 11, 3), 'text' => '=1+1' }] + DataPipeline::ExcelStorage.new(file: "#{dir}/t.xlsx").write(rows) + row = DataPipeline::ExcelStorage.new(file: "#{dir}/t.xlsx").read.first + assert_equal '0000000001', row['lfdnr'] + assert_in_delta 1.5, row['betrag'] + assert_equal 3, row['menge'].to_i + assert_equal Date.new(2025, 11, 3), row['datum'].to_date + assert_equal '=1+1', row['text'] + end + end + + def test_columns_sheet_name_and_symbol_keys + with_tmpdir do |dir| + DataPipeline::ExcelStorage.new(file: "#{dir}/c.xlsx", sheet: 'Buchungen', columns: %w[b a]) + .write([{ a: '1', b: '2' }, { 'b' => '4' }]) + assert_equal [{ 'b' => '2', 'a' => '1' }, { 'b' => '4', 'a' => nil }], + DataPipeline::ExcelStorage.new(file: "#{dir}/c.xlsx", sheet: 'Buchungen').read + end + end + + def test_without_header_and_skip_title_rows + with_tmpdir do |dir| + DataPipeline::ExcelStorage.new(file: "#{dir}/h.xlsx", headers: false).write([{ 'x' => 'Titel' }, { 'x' => 'Kopf' }, { 'x' => 'Wert' }]) + assert_equal [{ 'name' => 'Titel' }, { 'name' => 'Kopf' }, { 'name' => 'Wert' }], + DataPipeline::ExcelStorage.new(file: "#{dir}/h.xlsx", headers: false, columns: %w[name]).read + assert_equal [{ 'Kopf' => 'Wert' }], DataPipeline::ExcelStorage.new(file: "#{dir}/h.xlsx", skip: 1).read + end + end + + def test_empty_rows_are_skipped_and_short_name + with_tmpdir do |dir| + DataPipeline.new do + logger PipelineHelper::QUIET + read MemoryStorage.new(rows: [{ 'a' => '1' }, { 'a' => nil }, { 'a' => '3' }]) + write container(:xlsx, file: "#{dir}/e.xlsx") + end.run + assert_equal [{ 'a' => '1' }, { 'a' => '3' }], DataPipeline::ExcelStorage.new(file: "#{dir}/e.xlsx").read + end + end + + def test_file_required + assert_raises(ArgumentError) { DataPipeline::ExcelStorage.new } + end +end diff --git a/test/plugins/json_storage_test.rb b/test/plugins/json_storage_test.rb new file mode 100644 index 0000000..8603fbe --- /dev/null +++ b/test/plugins/json_storage_test.rb @@ -0,0 +1,56 @@ +require_relative '../test_helper' +require_relative '../support/storage_contract' + +class JsonArrayStorageTest < PipelineTest + include StorageContract + + def contract_storage(dir, _purpose) = DataPipeline::JsonStorage.new(file: "#{dir}/data.json") + + def test_streamed_array_is_valid_json + with_tmpdir do |dir| + json = DataPipeline::JsonStorage.new(file: "#{dir}/a.json") + json.write_row({ 'a' => 1 }) + json.write_row({ 'a' => 2 }) + json.close + assert_equal [{ 'a' => 1 }, { 'a' => 2 }], JSON.parse(File.read("#{dir}/a.json")) + end + end + + def test_root_path_and_pretty + with_tmpdir do |dir| + File.write("#{dir}/api.json", JSON.generate('data' => { 'items' => [{ 'id' => 1 }] })) + assert_equal [{ 'id' => 1 }], DataPipeline::JsonStorage.new(file: "#{dir}/api.json", root: 'data.items').read + DataPipeline::JsonStorage.new(file: "#{dir}/p.json", pretty: true).write([{ 'a' => 1 }]) + assert_includes File.read("#{dir}/p.json"), "\n" + end + end + + def test_single_object_is_one_row_and_scalar_raises + with_tmpdir do |dir| + File.write("#{dir}/o.json", '{"a":1}') + assert_equal [{ 'a' => 1 }], DataPipeline::JsonStorage.new(file: "#{dir}/o.json").read + File.write("#{dir}/s.json", '42') + assert_raises(ArgumentError) { DataPipeline::JsonStorage.new(file: "#{dir}/s.json").read } + end + end + + def test_options + assert_raises(ArgumentError) { DataPipeline::JsonStorage.new } + assert_raises(ArgumentError) { DataPipeline::JsonStorage.new(file: 'x.json', format: :xml) } + end +end + +class JsonLinesStorageTest < PipelineTest + include StorageContract + + def contract_storage(dir, _purpose) = DataPipeline::JsonStorage.new(file: "#{dir}/data.ndjson") + + def test_format_from_extension_and_blank_lines + with_tmpdir do |dir| + File.write("#{dir}/d.jsonl", "{\"a\":1}\n\n{\"a\":2}\n") + assert_equal [{ 'a' => 1 }, { 'a' => 2 }], DataPipeline::JsonStorage.new(file: "#{dir}/d.jsonl").read + DataPipeline::JsonStorage.new(file: "#{dir}/x.txt", format: :lines).write([{ 'a' => 1 }, { 'a' => 2 }]) + assert_equal "{\"a\":1}\n{\"a\":2}\n", File.read("#{dir}/x.txt") + end + end +end diff --git a/test/plugins/lazy_loading_test.rb b/test/plugins/lazy_loading_test.rb new file mode 100644 index 0000000..63d5c47 --- /dev/null +++ b/test/plugins/lazy_loading_test.rb @@ -0,0 +1,45 @@ +require_relative '../test_helper' +require 'open3' + +# in a fresh process: plugins and their gems are loaded only when used +class LazyLoadingTest < PipelineTest + def loaded_after code + script = <<~RUBY + require 'bundler/setup' + require #{File.join(ROOT, 'lib/datapipeline').inspect} + #{code} + features = $LOADED_FEATURES.join(' ') + puts %w[pg sqlite3 roo caxlsx http].select { |gem| features.match?(%r{/gems/\#{gem}-\\d}) }.join(',') + RUBY + out, err, status = Open3.capture3({ 'BUNDLE_GEMFILE' => File.join(ROOT, 'Gemfile') }, RbConfig.ruby, '-e', script) + assert status.success?, err + out.strip.split(',') + end + + def test_nothing_loaded_at_start + assert_empty loaded_after('') + end + + def test_json_and_csv_need_no_gem + with_tmpdir do |dir| + code = "DataPipeline.new { logger Logger.new(IO::NULL); read container(:json, file: '#{dir}/x.json').tap { File.write('#{dir}/x.json', '[]') }; write container(:csv, file: '#{dir}/y.csv') }.run" + assert_empty loaded_after(code) + end + end + + def test_each_plugin_loads_only_its_gem + with_tmpdir do |dir| + assert_equal %w[sqlite3], loaded_after("DataPipeline::SqliteStorage.new(database: ':memory:').close") + assert_equal %w[caxlsx], loaded_after("DataPipeline::ExcelStorage.new(file: '#{dir}/a.xlsx').write([{ 'a' => 1 }])") + assert_equal %w[roo caxlsx].sort, loaded_after("DataPipeline::ExcelStorage.new(file: '#{dir}/b.xlsx').write([{ 'a' => 1 }]); DataPipeline::ExcelStorage.new(file: '#{dir}/b.xlsx').read").sort + assert_equal %w[http], loaded_after("DataPipeline::RestStorage.new(url: 'http://localhost')") + end + end + + def test_missing_gem_message + plugin = Class.new(DataPipeline::StoragePlugin) { def initialize(**) = self.class.require_gem('gibtsnicht_gem') } + Object.const_set(:GibtsNichtStorage, plugin) unless defined?(GibtsNichtStorage) + error = assert_raises(LoadError) { GibtsNichtStorage.new } + assert_includes error.message, "gem 'gibtsnicht_gem'" + end +end diff --git a/test/plugins/postgres_storage_test.rb b/test/plugins/postgres_storage_test.rb new file mode 100644 index 0000000..2d30226 --- /dev/null +++ b/test/plugins/postgres_storage_test.rb @@ -0,0 +1,133 @@ +require_relative '../test_helper' +require_relative '../support/storage_contract' + +class PostgresStorageTest < PipelineTest + include StorageContract + + def with_contract_env + with_pg_database do |db| + pg_exec db, 'create table rows (id text, code text, name text, note text)' + yield db + end + end + + def contract_storage db, purpose + purpose == :write ? DataPipeline::PostgresStorage.new(**db, table: 'rows') : DataPipeline::PostgresStorage.new(**db, query: 'select * from rows order by id') + end + + def with_table + with_pg_database do |db| + pg_exec db, 'create table t (id int, name text, note text); create schema s; create table s."Mixed Case" ("Col A" text)' + storage = DataPipeline::PostgresStorage.new(**db) + yield db, storage + ensure + storage&.close + end + end + + def test_write_rows_with_table_option_quotes_and_null + with_table do |db, pg| + DataPipeline::PostgresStorage.new(**db, table: 't').tap do |w| + w.write([{ 'id' => 1, 'name' => "O'Brien", 'note' => nil }, { id: 2, name: "x'); drop table t; --" }]) + w.close + end + assert_equal [{ 'id' => '1', 'name' => "O'Brien", 'note' => nil }, + { 'id' => '2', 'name' => "x'); drop table t; --", 'note' => nil }], + pg.query(query: 'select id, name, note from t order by id') + end + end + + def test_write_old_signature_and_schema_quoting + with_table do |_db, pg| + pg.write('t', [{ 'id' => 3 }]) + pg.write('s.Mixed Case', [{ 'Col A' => 'v' }]) + assert_equal '1', pg.query(query: 'select count(*) from t').first['count'] + assert_equal [{ 'Col A' => 'v' }], pg.query(query: 'select * from s."Mixed Case"') + end + end + + def test_truncate_before_first_write + with_table do |db, pg| + pg.write('t', [{ 'id' => 1 }, { 'id' => 2 }]) + writer = DataPipeline::PostgresStorage.new(**db, table: 't', truncate: true) + writer.write([{ 'id' => 3 }]) + writer.write_row({ 'id' => 4 }) + writer.close + assert_equal %w[3 4], pg.query(query: 'select id from t order by id').map { |r| r['id'] } + end + end + + def test_write_is_one_transaction + with_table do |_db, pg| + assert_raises(PG::Error) { pg.write('t', [{ 'id' => 10 }, { 'id' => 'kein int' }]) } + assert_equal '0', pg.query(query: 'select count(*) from t').first['count'] + end + end + + def test_write_without_table + with_table do |_db, pg| + assert_includes assert_raises(RuntimeError) { pg.write([{ 'id' => 1 }]) }.message, 'Table' + end + end + + def test_query_and_read_with_params + with_table do |db, pg| + pg.write('t', [{ 'id' => 1, 'name' => 'a' }, { 'id' => 2, 'name' => 'b' }, { 'id' => 3 }]) + assert_equal [{ 'name' => 'a' }], pg.query(query: 'select name from t where id = $1', params: [1]) + + reader = DataPipeline::PostgresStorage.new(**db) + reader.set(query: 'select id from t where id > $1 order by id', params: [1]) + assert_equal [{ 'id' => '2' }, { 'id' => '3' }], reader.read + reader.close + end + end + + def test_typed_and_default_string_values + with_table do |db, pg| + typed = DataPipeline::PostgresStorage.new(**db, typed: true) + row = typed.query(query: 'select 1::int as i, 1.5::float8 as f, current_date as d').first + typed.close + assert_equal 1, row['i'] + assert_in_delta 1.5, row['f'] + assert_kind_of Date, row['d'] + assert_equal '1', pg.query(query: 'select 1::int as i').first['i'] + end + end + + def test_cursor_read_in_batches + with_pg_database do |db| + pg_exec db, 'create table k as select i as id from generate_series(1, 2500) i' + reader = DataPipeline::PostgresStorage.new(**db, query: "select id from k where id > $1 order by id;\n", params: [500], + fetch_size: 100) + count = 0 + reader.read { count += 1 } + reader.close + assert_equal 2000, count + end + end + + def test_cursor_rollback_on_error_keeps_connection_usable + with_pg_database do |db| + pg_exec db, 'create table k as select 1 as id' + pg = DataPipeline::PostgresStorage.new(**db, query: 'select id from k') + assert_raises(RuntimeError) { pg.read { raise 'x' } } + assert_equal [{ 'x' => '1' }], pg.query(query: 'select 1 as x') + pg.close + end + end + + def test_close_twice + with_pg_database do |db| + pg = DataPipeline::PostgresStorage.new(**db) + pg.close + pg.close + assert pg.instance_variable_get(:@conn).finished? + end + end + + def test_connection_error_keeps_cause + skip 'no local postgres' unless PgHelper.available? + error = assert_raises(PG::ConnectionBad) { DataPipeline::PostgresStorage.new(host: 'localhost', port: 1, database: 'x') } + assert_includes error.message, 'Error connection to database:' + end +end diff --git a/test/plugins/rest_storage_test.rb b/test/plugins/rest_storage_test.rb new file mode 100644 index 0000000..49a8336 --- /dev/null +++ b/test/plugins/rest_storage_test.rb @@ -0,0 +1,151 @@ +require_relative '../test_helper' +require_relative '../support/storage_contract' +require_relative '../support/test_http_server' + +class RestStorageTest < PipelineTest + include StorageContract + + # a tiny api: POST /items stores one row or an array, GET /items returns all + def with_contract_env + store = [] + server = TestHttpServer.new do |req| + if req[:method] == 'POST' + Array(JSON.parse(req[:body]).then { |b| b.is_a?(Array) ? b : [b] }).each { |row| store << row } + [201, { 'ok' => true }] + else + [200, { 'data' => { 'items' => store } }] + end + end + yield server + ensure + server&.stop + end + + def contract_storage(server, _purpose) = DataPipeline::RestStorage.new(url: "#{server.url}/items", root: 'data.items') + + # handler: ->(request) { [status, body] } + def server handler + s = TestHttpServer.new(&handler) + yield s + ensure + s&.stop + end + + def test_page_param_paging + pages = { '1' => [{ 'id' => 1 }, { 'id' => 2 }], '2' => [{ 'id' => 3 }] } + server(->(req) { [200, pages.fetch(req[:query]['page'], [])] }) do |s| + rows = DataPipeline::RestStorage.new(url: "#{s.url}/kunden", page_param: 'page', params: { 'land' => 'DE' }).read + assert_equal [1, 2, 3], rows.map { |r| r['id'] } + assert_equal %w[1 2 3], s.requests.map { |r| r[:query]['page'] } + assert(s.requests.all? { |r| r[:query]['land'] == 'DE' }) + end + end + + def test_next_url_paging + base = nil + handler = lambda do |req| + if req[:path] == '/a' + [200, { 'rows' => [{ 'n' => 1 }], 'next' => "#{base}/b" }] + else + [200, { 'rows' => [{ 'n' => 2 }], 'next' => nil }] + end + end + server(handler) do |s| + base = s.url + rows = DataPipeline::RestStorage.new(url: "#{s.url}/a", root: 'rows', next: ->(body) { body['next'] }).read + assert_equal [1, 2], rows.map { |r| r['n'] } + end + end + + def test_auth_and_headers + server(->(_req) { [200, []] }) do |s| + DataPipeline::RestStorage.new(url: s.url, auth: 'Bearer geheim', headers: { 'X-Mandant' => '001' }).read + assert_equal 'Bearer geheim', s.requests.first[:headers]['authorization'] + assert_equal '001', s.requests.first[:headers]['x-mandant'] + end + end + + def test_retry_on_503_then_success + calls = 0 + server(->(_req) { (calls += 1) < 3 ? [503, 'busy'] : [200, [{ 'ok' => 1 }]] }) do |s| + assert_equal [{ 'ok' => 1 }], DataPipeline::RestStorage.new(url: s.url, retries: 2, retry_wait: 0).read + assert_equal 3, calls + end + end + + def test_no_retry_on_404_and_error_message + server(->(_req) { [404, { 'error' => 'nicht da' }] }) do |s| + error = assert_raises(DataPipeline::RestStorage::Error) { DataPipeline::RestStorage.new(url: "#{s.url}/x", retry_wait: 0).read } + assert_includes error.message, 'HTTP 404' + assert_includes error.message, 'nicht da' + assert_equal 1, s.requests.size + end + end + + def test_retries_exhausted_and_connection_refused + server(->(_req) { [500, 'kaputt'] }) do |s| + assert_includes assert_raises(DataPipeline::RestStorage::Error) { DataPipeline::RestStorage.new(url: s.url, retries: 1, retry_wait: 0).read }.message, 'HTTP 500' + assert_equal 2, s.requests.size + end + closed = TCPServer.new('127.0.0.1', 0).then { |t| port = t.addr[1]; t.close; port } + assert_raises(DataPipeline::RestStorage::Error) { DataPipeline::RestStorage.new(url: "http://127.0.0.1:#{closed}", retries: 1, retry_wait: 0).read } + end + + def test_batch_write_and_write_method + server(->(_req) { [200, ''] }) do |s| + rest = DataPipeline::RestStorage.new(url: "#{s.url}/import", batch_size: 2, write_method: :put) + rest.write([{ 'a' => 1 }, { 'a' => 2 }, { 'a' => 3 }]) + assert_equal [[{ 'a' => 1 }, { 'a' => 2 }], [{ 'a' => 3 }]], s.requests.map { |r| JSON.parse(r[:body]) } + assert(s.requests.all? { |r| r[:method] == 'PUT' }) + end + end + + def test_write_url_per_row + server(->(_req) { [200, ''] }) do |s| + DataPipeline::RestStorage.new(url: s.url, write_method: :put, write_url: ->(row) { "#{s.url}/kunden/#{row['id']}" }) + .write([{ 'id' => 7 }, { 'id' => 8 }]) + assert_equal %w[/kunden/7 /kunden/8], s.requests.map { |r| r[:path] } + assert_equal({ 'id' => 8 }, JSON.parse(s.requests.last[:body])) + end + assert_raises(ArgumentError) { DataPipeline::RestStorage.new(url: 'http://x', batch_size: 2, write_url: ->(_row) { 'http://x' }) } + end + + def test_call_json_status_and_retries + calls = 0 + server(->(req) { calls += 1; req[:path] == '/api/fail' ? [503, ''] : [201, { 'echo' => JSON.parse(req[:body]) }] }) do |s| + rest = DataPipeline::RestStorage.new(url: "#{s.url}/api/", retries: 2, retry_wait: 0) + assert_equal({ 'echo' => { 'a' => 1 } }, rest.call(:post, 'person/create', json: { 'a' => 1 })) + assert_equal '/api/person/create', s.requests.last[:path] + + calls = 0 + error = assert_raises(DataPipeline::RestStorage::Error) { rest.call(:post, 'fail', json: {}, retries: 0) } + assert_equal 503, error.status + assert_equal 1, calls, 'retries: 0 sends once' + end + server(->(_req) { [404, { 'message' => 'weg' }] }) do |s| + assert_equal 404, assert_raises(DataPipeline::RestStorage::Error) { DataPipeline::RestStorage.new(url: s.url).call(:get, 'x') }.status + end + end + + def test_ssl_verify_false_context + context = DataPipeline::RestStorage.new(url: 'https://192.168.178.251:50443/pds/rest/api', ssl_verify: false).send(:ssl_context) + assert_equal OpenSSL::SSL::VERIFY_NONE, context.verify_mode + end + + def test_query_for_lookup + server(->(req) { [200, [{ 'land' => req[:query]['land'], 'pfad' => req[:path] }]] }) do |s| + names = nil + DataPipeline.new do + api = container :rest, url: "#{s.url}/api/" + names = lookup api, query: '/laender', params: { 'land' => 'DE' }, key: 'land', unique: true + end + assert_equal '/api/laender', names['DE']['pfad'] + end + end + + def test_not_an_array + server(->(_req) { [200, { 'a' => 1 }] }) do |s| + assert_raises(DataPipeline::RestStorage::Error) { DataPipeline::RestStorage.new(url: s.url).read } + end + end +end diff --git a/test/plugins/slack_test.rb b/test/plugins/slack_test.rb new file mode 100644 index 0000000..a450dc6 --- /dev/null +++ b/test/plugins/slack_test.rb @@ -0,0 +1,18 @@ +require_relative '../test_helper' + +# no network: only construction and options +class SlackTest < PipelineTest + def test_token_required + token = ENV.delete('SLACK_TOKEN') + assert_raises(ArgumentError) { DataPipeline::Slack.new } + ensure + ENV['SLACK_TOKEN'] = token if token + end + + def test_options_per_instance + a = DataPipeline::Slack.new(token: 'x', channel: '#a') + b = DataPipeline::Slack.new(token: 'y') + assert_equal '#a', a.instance_variable_get(:@options)[:channel] + assert_equal '#general', b.instance_variable_get(:@options)[:channel] + end +end diff --git a/test/plugins/sqlite_storage_test.rb b/test/plugins/sqlite_storage_test.rb new file mode 100644 index 0000000..632430a --- /dev/null +++ b/test/plugins/sqlite_storage_test.rb @@ -0,0 +1,95 @@ +require_relative '../test_helper' +require_relative '../support/storage_contract' + +class SqliteStorageTest < PipelineTest + include StorageContract + + def contract_storage dir, purpose + db = "#{dir}/test.sqlite3" + unless File.exist?(db) + DataPipeline::SqliteStorage.new(database: db).tap do |s| + s.query(query: 'create table rows (id text, code text, name text, note text)') + s.close + end + end + purpose == :write ? DataPipeline::SqliteStorage.new(database: db, table: 'rows') : DataPipeline::SqliteStorage.new(database: db, query: 'select * from rows order by id') + end + + def memory_db + DataPipeline::SqliteStorage.new(database: ':memory:').tap do |s| + s.query(query: 'create table t (id integer, name text, betrag real, datum text)') + end + end + + def test_types_null_and_quotes + db = memory_db + db.write('t', [{ 'id' => 1, 'name' => "O'Brien", 'betrag' => 1.5, 'datum' => Date.new(2025, 11, 3) }, { id: 2, name: nil }]) + assert_equal [{ 'id' => 1, 'name' => "O'Brien", 'betrag' => 1.5, 'datum' => '2025-11-03' }, + { 'id' => 2, 'name' => nil, 'betrag' => nil, 'datum' => nil }], + db.query(query: 'select * from t order by id') + db.close + end + + def test_positional_and_named_params + db = memory_db + db.write('t', [{ 'id' => 1, 'name' => 'a' }, { 'id' => 2, 'name' => 'b' }]) + assert_equal [{ 'name' => 'b' }], db.query(query: 'select name from t where id = ?', params: [2]) + assert_equal [{ 'name' => 'a' }], db.query(query: 'select name from t where id = :id', params: { id: 1 }) + db.close + end + + def test_write_is_one_transaction + db = memory_db + db.query(query: 'create table u (id integer not null)') + assert_raises(SQLite3::Exception) { db.write('u', [{ 'id' => 1 }, { 'id' => nil }]) } + assert_equal 0, db.query(query: 'select count(*) as n from u').first['n'] + db.close + end + + def test_sql_injection_in_values_and_identifiers + db = memory_db + db.query(query: 'create table "Mixed ""Name""" ("Col A" text)') + db.write('Mixed "Name"', [{ 'Col A' => "x'); drop table t; --" }]) + assert_equal [{ 'Col A' => "x'); drop table t; --" }], db.query(query: 'select * from "Mixed ""Name"""') + assert_equal 0, db.query(query: 'select count(*) as n from t').first['n'] + db.close + end + + def test_create_truncate_and_lazy_open + with_tmpdir do |dir| + path = "#{dir}/neu.sqlite3" + storage = DataPipeline::SqliteStorage.new(database: path, table: 'k', create: true, truncate: true) + refute File.exist?(path), 'opened only on first use' + storage.write([{ 'id' => 1, 'betrag' => 1.5, 'name' => 'a' }, { 'id' => 2, 'betrag' => 2.0, 'name' => 'b' }]) + storage.close + + again = DataPipeline::SqliteStorage.new(database: path, table: 'k', create: true, truncate: true) + again.write_row({ 'id' => 3, 'betrag' => 3.0, 'name' => 'c' }) + assert_equal [{ 'id' => 3, 'betrag' => 3.0, 'name' => 'c' }], again.query(query: 'select * from k') + assert_equal %w[integer real text], again.query(query: 'pragma table_info(k)').map { |c| c['type'].downcase } + again.close + end + end + + def test_missing_options + assert_raises(ArgumentError) { DataPipeline::SqliteStorage.new } + db = memory_db + assert_raises(RuntimeError) { db.write([{ 'id' => 1 }]) } + assert_raises(RuntimeError) { db.read } + db.close + end + + def test_short_name_and_lookup + with_tmpdir do |dir| + path = "#{dir}/l.sqlite3" + names = nil + DataPipeline.new do + db = container :sqlite3, database: path + db.query(query: 'create table land (code text, name text)') + db.write('land', [{ 'code' => 'DE', 'name' => 'Deutschland' }]) + names = lookup db, query: 'select * from land where code like ?', params: ['%'], key: 'code', unique: true + end + assert_equal 'Deutschland', names['DE']['name'] + end + end +end diff --git a/test/support/storage_contract.rb b/test/support/storage_contract.rb new file mode 100644 index 0000000..3d9dec7 --- /dev/null +++ b/test/support/storage_contract.rb @@ -0,0 +1,73 @@ +# +# shared tests for every storage plugin that can write and read back +# +# a test class includes StorageContract and implements +# contract_storage(env, purpose) -> a new storage; purpose is :write or :read +# optionally +# with_contract_env { |env| ... } -> environment per test (default: a tmp dir) +# contract_rows(rows) -> rows as the plugin reads them back (default: unchanged) +# +module StorageContract + ROWS = [ + { 'id' => '1', 'code' => '007', 'name' => "O'Brien", 'note' => nil }, + { 'id' => '2', 'code' => '010', 'name' => 'Zürich | "ä", ß', 'note' => 'zwei' }, + { 'id' => '3', 'code' => '100', 'name' => 'drei', 'note' => '' } + ].freeze + + def with_contract_env(&block) = with_tmpdir(&block) + def contract_rows(rows) = rows + + def test_contract_write_and_read_back + with_contract_env do |env| + contract_storage(env, :write).tap { |s| s.write(ROWS) }.close + reader = contract_storage(env, :read) + assert_equal contract_rows(ROWS), reader.read + reader.close + end + end + + def test_contract_write_row_and_read_with_block + with_contract_env do |env| + writer = contract_storage(env, :write) + ROWS.each { |row| writer.write_row(row) } + writer.close + rows = [] + reader = contract_storage(env, :read) + assert_nil reader.read { |row| rows << row } + reader.close + assert_equal contract_rows(ROWS), rows + end + end + + def test_contract_as_pipeline_source_and_target + [false, true].each do |streamed| + with_contract_env do |env| + target = contract_storage(env, :write) + stats = DataPipeline.new(streamed:) do + logger PipelineHelper::QUIET + read MemoryStorage.new(rows: ROWS) + write target + end.run + assert_equal 3, stats.written + + source = contract_storage(env, :read) + copy = MemoryStorage.new + DataPipeline.new(streamed:) do + logger PipelineHelper::QUIET + read source + write copy + end.run + assert_equal contract_rows(ROWS), copy.output, "streamed=#{streamed}" + end + end + end + + def test_contract_close_twice + with_contract_env do |env| + storage = contract_storage(env, :write) + storage.close + storage.close + pass + end + end +end diff --git a/test/support/test_http_server.rb b/test/support/test_http_server.rb new file mode 100644 index 0000000..ff95a38 --- /dev/null +++ b/test/support/test_http_server.rb @@ -0,0 +1,52 @@ +require 'socket' +require 'uri' +require 'json' + +# +# minimal http server for DataPipeline::RestStorage tests, one request per connection +# server = TestHttpServer.new { |req| [200, { 'items' => [] }] } +# req: { method:, path:, query:, headers:, body: } -> [status, hash/array or string] +# +class TestHttpServer + attr_reader :requests + + def initialize(&handler) + @handler = handler + @requests = [] + @server = TCPServer.new('127.0.0.1', 0) + @thread = Thread.new { loop { serve(@server.accept) } } + end + + def url = "http://127.0.0.1:#{@server.addr[1]}" + + def stop + @thread.kill + @server.close + end + + private + + def serve client + method, target = client.gets.to_s.split + return unless method + + headers = {} + while (line = client.gets) && line != "\r\n" + key, value = line.split(':', 2) + headers[key.downcase] = value.strip + end + body = headers['content-length'] ? client.read(headers['content-length'].to_i) : '' + path, query = target.split('?', 2) + request = { method:, path:, query: URI.decode_www_form(query.to_s).to_h, headers:, body: } + @requests << request + + status, payload = @handler.call(request) + text = payload.is_a?(String) ? payload : JSON.generate(payload) + client.write "HTTP/1.1 #{status} X\r\nContent-Type: application/json\r\n" \ + "Content-Length: #{text.bytesize}\r\nConnection: close\r\n\r\n#{text}" + rescue StandardError => e + warn "TestHttpServer: #{e.class}: #{e.message}" + ensure + client.close + end +end diff --git a/test/test_helper.rb b/test/test_helper.rb new file mode 100644 index 0000000..175d92c --- /dev/null +++ b/test/test_helper.rb @@ -0,0 +1,148 @@ +# +# rake test or ruby -Itest test/core/chained_test.rb +# +# .env is NOT loaded: no test may reach the production database. +# postgres tests use a local server and a throw-away database per test: +# ETL_TEST_PG_HOST (default localhost), ETL_TEST_PG_PORT, ETL_TEST_PG_USER (default $USER), ETL_TEST_PG_PASSWORD +# without a reachable server they are skipped. +# +require 'bundler/setup' +require 'minitest/autorun' +require 'json' +require 'logger' +require 'stringio' +require 'tmpdir' + +require 'datapipeline' + +ROOT = File.expand_path('..', __dir__) + +# source and target in memory +class MemoryStorage < DataPipeline::StoragePlugin + attr_reader :rows, :written, :streamed_rows + + def initialize rows: [], **options + @options = options + @rows = rows + @written = nil + @streamed_rows = [] + @closed = false + end + + def read(&block) + return @rows.map(&:dup) unless block + + @rows.map(&:dup).each(&block) + nil + end + + def write(rows) = (@written = rows) + def write_row(row) = (@streamed_rows << row) + def close = (@closed = true) + def closed? = @closed + + # rows of the last batch write or of the streamed rows + def output = @written || @streamed_rows +end + +DataPipeline.register :memory, MemoryStorage + +module PipelineHelper + QUIET = Logger.new(IO::NULL) + + def setup + super + DataPipeline.reset_settings! + DataPipeline.runs.clear + DataPipeline.params.clear + end + + # runs a pipeline from rows into a MemoryStorage, returns [target, stats] + def run_pipeline rows = [], **settings, &block + source = MemoryStorage.new(rows:) + target = MemoryStorage.new + stats = DataPipeline.new(**settings) do + logger QUIET + read source + write target + instance_eval(&block) if block + end.run + [target, stats] + end + + def output rows = [], **settings, &block + run_pipeline(rows, **settings, &block).first.output + end + + def with_tmpdir(&block) + Dir.mktmpdir('etl-test', &block) + end + + def capture_stdout + old = $stdout + $stdout = StringIO.new + yield + $stdout.string + ensure + $stdout = old + end +end + +module PgHelper + def self.config + { + host: ENV.fetch('ETL_TEST_PG_HOST', 'localhost'), + port: ENV.fetch('ETL_TEST_PG_PORT', '5432'), + username: ENV.fetch('ETL_TEST_PG_USER', ENV['USER']), + password: ENV.fetch('ETL_TEST_PG_PASSWORD', nil) + } + end + + def self.admin + require 'pg' + c = config + PG.connect(host: c[:host], port: c[:port], user: c[:username], password: c[:password], dbname: 'postgres', + connect_timeout: 3) + end + + def self.available? + return @available unless @available.nil? + + @available = begin + admin.close + true + rescue StandardError, LoadError + false + end + end + + # yields connection options for a fresh database, dropped afterwards + def with_pg_database + skip 'no local postgres (ETL_TEST_PG_HOST)' unless PgHelper.available? + + name = "etl_test_#{Process.pid}_#{rand(1_000_000)}" + admin = PgHelper.admin + admin.exec("create database #{name}") + yield PgHelper.config.merge(database: name) + ensure + if admin + admin.exec("select pg_terminate_backend(pid) from pg_stat_activity where datname = '#{name}' and pid <> pg_backend_pid()") + admin.exec("drop database if exists #{name}") + admin.close + end + end + + # plain sql, several statements allowed (setup of test tables) + def pg_exec options, sql + conn = PG.connect(host: options[:host], port: options[:port], user: options[:username], + password: options[:password], dbname: options[:database]) + conn.exec(sql).to_a + ensure + conn&.close + end +end + +class PipelineTest < Minitest::Test + include PipelineHelper + include PgHelper +end