datapipeline 2.0.0: Gem aus dem Projekt etl herausgelöst
This commit is contained in:
@@ -0,0 +1,8 @@
|
|||||||
|
*~
|
||||||
|
.DS_*
|
||||||
|
.bundle
|
||||||
|
vendor
|
||||||
|
*.gem
|
||||||
|
|
||||||
|
# Ausgabe der Beispiele (examples/*.rb, Default --out=tmp)
|
||||||
|
/tmp/
|
||||||
+123
@@ -0,0 +1,123 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
Framework `datapipeline` (Gem). Neueste Version oben.
|
||||||
|
|
||||||
|
## 2.0.0 – 2026-10-08
|
||||||
|
|
||||||
|
- Gem `datapipeline`: `datapipeline.gemspec`, Executable `etl` (exe/etl), Code unter `lib/datapipeline/`
|
||||||
|
- eigenes Projekt `datapipeline` (vorher Teil von `etl`, dessen Historie die frühere Entwicklung enthält); die
|
||||||
|
Anwendungen in `etl` binden das Gem mit `path: '../datapipeline'` ein
|
||||||
|
- **Breaking:** alle Klassen im Namensraum `DataPipeline`: `DataPipeline::StoragePlugin`, `DataPipeline::CsvStorage`,
|
||||||
|
`…::PostgresStorage`, `…::JsonStorage`, `…::SqliteStorage`, `…::ExcelStorage`, `…::RestStorage`, `DataPipeline::Slack`.
|
||||||
|
`container :csv` usw. bleibt gleich; eigene Plugins dürfen weiter auf oberster Ebene (`<Name>Storage`) liegen
|
||||||
|
- `DataPipeline::VERSION`; Versionsverlauf hier statt in `version.rb`
|
||||||
|
- Postgres-Plugin lädt `pg` über `require_gem` (Hinweis, wenn das Gem fehlt)
|
||||||
|
|
||||||
|
## 1.9.0 – 2026-10-08
|
||||||
|
|
||||||
|
- rest: call(method, path, json:, params:, retries:) for rpc style apis, Error#status, ssl_verify: false
|
||||||
|
- json: a single object is read as one row
|
||||||
|
- dsl: dry_run? for steps with side effects
|
||||||
|
- application: rechnung_anlegen.rb (pds API: verify or create customer, invoice to a project file)
|
||||||
|
|
||||||
|
## 1.8.2 – 2026-10-08
|
||||||
|
|
||||||
|
- rest: write_url as lambda for one url per row (e.g. PUT /archiv/<id>)
|
||||||
|
- REST examples (read with paging/retry/lookup, write in batches, API to API) with a local demo API
|
||||||
|
|
||||||
|
## 1.8.1 – 2026-10-08
|
||||||
|
|
||||||
|
- sqlite: database opened on first use, create: true and truncate: true; postgres: truncate: true
|
||||||
|
- log line "start" without trailing blank when no version is set
|
||||||
|
- examples with data in examples/, documented in DOKUMENTATION.md section 1 and tested
|
||||||
|
|
||||||
|
## 1.8.0 – 2026-10-08
|
||||||
|
|
||||||
|
- DataPipeline::Layout: field list per record type (type, length, required); csv option layout: writes in field order and rejects unknown fields; validate layout checks required, length, format
|
||||||
|
- validate warn: true (warning, row kept, stats.warnings) and ignore: fields; warning log capped at 100
|
||||||
|
- DataPipeline.params reads --name=value / --param key=value from ARGV when not started by bin/etl
|
||||||
|
- bin/etl passes unknown --name=value options to the script and to params
|
||||||
|
|
||||||
|
## 1.7.0 – 2026-10-08
|
||||||
|
|
||||||
|
- new plugins: json (array and ndjson), sqlite (:sqlite, :sqlite3), excel (:excel, :xlsx), rest
|
||||||
|
- gems of a plugin are loaded on first use (StoragePlugin.require_gem with a hint when missing)
|
||||||
|
|
||||||
|
## 1.6.0 – 2026-10-08
|
||||||
|
|
||||||
|
- plugins loaded on first use (autoload): pg only for postgres, http only for slack
|
||||||
|
- DataPipeline.register for own plugins, optionally loaded from a path on first use
|
||||||
|
- defaults are frozen and replaced instead of changed; DataPipeline.new(**settings) per pipeline, reset_settings!
|
||||||
|
- write can be called several times, only: lambda routes rows to a target
|
||||||
|
- clear error when read or write is missing
|
||||||
|
|
||||||
|
## 1.5.0 – 2026-10-08
|
||||||
|
|
||||||
|
- on_error :abort | :skip | :collect (to: storage) for row errors in body steps
|
||||||
|
- validate step with rules (required, max, min, format, in, type) or block
|
||||||
|
- run returns stats (read, written, output, rejected, errors, duration), on_finish hooks, logger (default warn on stderr)
|
||||||
|
- watermark: highest value of a column saved after a successful run (json file)
|
||||||
|
- dry_run (nothing written, preview of the first rows) and limit
|
||||||
|
- bin/etl run PIPELINE.rb with --from/--till/--param/--dry-run/--limit and exit codes, Rakefile with arguments through bin/etl
|
||||||
|
|
||||||
|
## 1.4.0 – 2026-10-08
|
||||||
|
|
||||||
|
- streamed works: rows go one by one from source.read { } through the chain to target.write_row, header/trailer/prepend/append and conditions included
|
||||||
|
- batch and streamed share one engine for chained steps
|
||||||
|
- parallel mode: every step gets its own copy of the row
|
||||||
|
- lookup: load a table once as hash (grouped or unique) instead of one query per row
|
||||||
|
- postgres: read with block uses a server side cursor (fetch_size), queries on the same connection stay possible while reading
|
||||||
|
|
||||||
|
## 1.3.0 – 2026-10-08
|
||||||
|
|
||||||
|
- chained mode (chained true): every body step gets the output of the previous step
|
||||||
|
- new dsl steps map, filter, header, trailer with context (read, written, index); map and filter turn chaining on
|
||||||
|
- all prepend and append transformations run once, stable order of registration
|
||||||
|
- condition also as lambda
|
||||||
|
- without any transformation the source is copied unchanged
|
||||||
|
|
||||||
|
## 1.2.1 – 2026-10-08
|
||||||
|
|
||||||
|
- transformations returning nil (or nil inside an array) produce no row
|
||||||
|
- explicit dsl methods version, verbose, streamed; unknown calls raise NoMethodError (method_missing removed, unused setting processing removed)
|
||||||
|
- container only reports a missing plugin class as unsupported, plugin errors pass through
|
||||||
|
|
||||||
|
## 1.2.0 – 2026-10-08
|
||||||
|
|
||||||
|
- csv: read works with ruby 3, defaults merged, option columns for fixed column order, streaming read/write_row, string keys like postgres
|
||||||
|
- postgres: bind parameters (params:), write with table: option, nil as NULL, one transaction, quoted identifiers, optional typed results, close
|
||||||
|
- storage plugin: common interface with set, query, write_row, close
|
||||||
|
- pipeline closes source and target after run
|
||||||
|
- slack notification fixed
|
||||||
|
- removed unused lib/source.rb and lib/target.rb
|
||||||
|
|
||||||
|
## 1.1.2 – 2025-02-06
|
||||||
|
|
||||||
|
- split data and output data to be able to insert results to output nut comming from db
|
||||||
|
- added query to Datapipline to execute subqueries if nessessary
|
||||||
|
- added prepend, append and normal to transform. we are now be able to order processing
|
||||||
|
|
||||||
|
## 1.1.1 – 2025-01-10
|
||||||
|
|
||||||
|
- merge options in csv
|
||||||
|
- verbose and counter
|
||||||
|
|
||||||
|
## 1.1.0 – 2024-12-21
|
||||||
|
|
||||||
|
- change to plugins, postgres and csv for read and write
|
||||||
|
|
||||||
|
## 1.0.3 – 2024-10-18
|
||||||
|
|
||||||
|
- added slack as plugin
|
||||||
|
- added settings to source
|
||||||
|
- added verbose in processing non stream
|
||||||
|
- added transformation in blocks
|
||||||
|
- added transformation condition
|
||||||
|
|
||||||
|
## 1.0.2 – 2024-10-15
|
||||||
|
|
||||||
|
- handle data streamed
|
||||||
|
|
||||||
|
## 1.0.1 – 2024-10-10
|
||||||
|
|
||||||
|
- handle missing or wrong parameter
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
source 'https://rubygems.org'
|
||||||
|
|
||||||
|
ruby '~> 3.4'
|
||||||
|
|
||||||
|
gemspec
|
||||||
|
|
||||||
|
# optional plugin gems: needed for the plugin tests, loaded only when a plugin is used
|
||||||
|
group :plugins do
|
||||||
|
gem 'caxlsx', require: false # excel write
|
||||||
|
gem 'http', require: false # rest, slack
|
||||||
|
gem 'pg', require: false # postgres
|
||||||
|
gem 'roo', require: false # excel read
|
||||||
|
gem 'sqlite3', require: false # sqlite
|
||||||
|
end
|
||||||
|
|
||||||
|
group :test do
|
||||||
|
gem 'minitest', '~> 5.25'
|
||||||
|
gem 'rake'
|
||||||
|
end
|
||||||
@@ -0,0 +1,96 @@
|
|||||||
|
PATH
|
||||||
|
remote: .
|
||||||
|
specs:
|
||||||
|
datapipeline (2.0.0)
|
||||||
|
csv (~> 3.3)
|
||||||
|
logger (~> 1.6)
|
||||||
|
|
||||||
|
GEM
|
||||||
|
remote: https://rubygems.org/
|
||||||
|
specs:
|
||||||
|
base64 (0.3.0)
|
||||||
|
caxlsx (4.5.0)
|
||||||
|
htmlentities (~> 4.3, >= 4.3.4)
|
||||||
|
marcel (~> 1.0)
|
||||||
|
nokogiri (~> 1.10, >= 1.10.4)
|
||||||
|
rubyzip (>= 2.4, < 4)
|
||||||
|
csv (3.3.6)
|
||||||
|
domain_name (0.6.20260921)
|
||||||
|
htmlentities (4.4.2)
|
||||||
|
http (6.0.4)
|
||||||
|
http-cookie (~> 1.0)
|
||||||
|
llhttp (~> 0.6.1)
|
||||||
|
http-cookie (1.1.6)
|
||||||
|
domain_name (~> 0.5)
|
||||||
|
llhttp (0.6.2)
|
||||||
|
logger (1.7.0)
|
||||||
|
marcel (1.2.1)
|
||||||
|
minitest (5.27.0)
|
||||||
|
nokogiri (1.19.4-aarch64-linux-gnu)
|
||||||
|
racc (~> 1.4)
|
||||||
|
nokogiri (1.19.4-aarch64-linux-musl)
|
||||||
|
racc (~> 1.4)
|
||||||
|
nokogiri (1.19.4-arm-linux-gnu)
|
||||||
|
racc (~> 1.4)
|
||||||
|
nokogiri (1.19.4-arm-linux-musl)
|
||||||
|
racc (~> 1.4)
|
||||||
|
nokogiri (1.19.4-arm64-darwin)
|
||||||
|
racc (~> 1.4)
|
||||||
|
nokogiri (1.19.4-x86_64-darwin)
|
||||||
|
racc (~> 1.4)
|
||||||
|
nokogiri (1.19.4-x86_64-linux-gnu)
|
||||||
|
racc (~> 1.4)
|
||||||
|
nokogiri (1.19.4-x86_64-linux-musl)
|
||||||
|
racc (~> 1.4)
|
||||||
|
pg (1.7.0)
|
||||||
|
pg (1.7.0-aarch64-linux)
|
||||||
|
pg (1.7.0-aarch64-linux-musl)
|
||||||
|
pg (1.7.0-arm64-darwin)
|
||||||
|
pg (1.7.0-x86_64-darwin)
|
||||||
|
pg (1.7.0-x86_64-linux)
|
||||||
|
pg (1.7.0-x86_64-linux-musl)
|
||||||
|
racc (1.8.1)
|
||||||
|
rake (13.4.2)
|
||||||
|
roo (3.0.0)
|
||||||
|
base64 (~> 0.2)
|
||||||
|
csv (~> 3)
|
||||||
|
logger (~> 1)
|
||||||
|
nokogiri (~> 1)
|
||||||
|
rubyzip (>= 3.0.0, < 4.0.0)
|
||||||
|
rubyzip (3.7.0)
|
||||||
|
sqlite3 (2.9.6-aarch64-linux-gnu)
|
||||||
|
sqlite3 (2.9.6-aarch64-linux-musl)
|
||||||
|
sqlite3 (2.9.6-arm-linux-gnu)
|
||||||
|
sqlite3 (2.9.6-arm-linux-musl)
|
||||||
|
sqlite3 (2.9.6-arm64-darwin)
|
||||||
|
sqlite3 (2.9.6-x86_64-darwin)
|
||||||
|
sqlite3 (2.9.6-x86_64-linux-gnu)
|
||||||
|
sqlite3 (2.9.6-x86_64-linux-musl)
|
||||||
|
|
||||||
|
PLATFORMS
|
||||||
|
aarch64-linux
|
||||||
|
aarch64-linux-gnu
|
||||||
|
aarch64-linux-musl
|
||||||
|
arm-linux-gnu
|
||||||
|
arm-linux-musl
|
||||||
|
arm64-darwin
|
||||||
|
x86_64-darwin
|
||||||
|
x86_64-linux
|
||||||
|
x86_64-linux-gnu
|
||||||
|
x86_64-linux-musl
|
||||||
|
|
||||||
|
DEPENDENCIES
|
||||||
|
caxlsx
|
||||||
|
datapipeline!
|
||||||
|
http
|
||||||
|
minitest (~> 5.25)
|
||||||
|
pg
|
||||||
|
rake
|
||||||
|
roo
|
||||||
|
sqlite3
|
||||||
|
|
||||||
|
RUBY VERSION
|
||||||
|
ruby 3.4.9p82
|
||||||
|
|
||||||
|
BUNDLED WITH
|
||||||
|
2.6.9
|
||||||
@@ -0,0 +1,912 @@
|
|||||||
|
# DataPipeline – generisches ETL-Framework in Ruby
|
||||||
|
|
||||||
|
Stand: 2026-10-08 · Gem `datapipeline` 2.0.0 ([CHANGELOG.md](CHANGELOG.md)) · Ruby 3.4
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Beschreibung
|
||||||
|
|
||||||
|
`DataPipeline` ist ein schlankes ETL-Framework (Extract – Transform – Load) mit einer Ruby-DSL. Eine Pipeline
|
||||||
|
liest Daten aus einer Quelle, schickt jede Zeile durch eine Kette von Schritten (prüfen, filtern, umbauen,
|
||||||
|
anreichern) und schreibt das Ergebnis in ein oder mehrere Ziele. Quellen und Ziele sind austauschbare
|
||||||
|
**Storage-Plugins**: PostgreSQL, SQLite, CSV, JSON/NDJSON, Excel und REST-APIs.
|
||||||
|
|
||||||
|
Die Pipeline beschreibt *was* passiert, die Plugins kapseln *wie* gelesen und geschrieben wird. Für den Betrieb
|
||||||
|
gibt es Fehlerdateien, Prüfregeln, Dry-Run, inkrementelle Läufe, Streaming für grosse Datenmengen und eine
|
||||||
|
Kommandozeile mit Exit-Codes.
|
||||||
|
|
||||||
|
**Anwendungen:** Das Projekt `etl` nutzt das Gem für die PDS → SAP-Schnittstelle (CUDO) und für das Anlegen von
|
||||||
|
Rechnungen über die pds API; dort liegen auch die zugehörige Doku und Tests.
|
||||||
|
|
||||||
|
### 1.1 Schnellstart
|
||||||
|
|
||||||
|
```sh
|
||||||
|
mise install # Ruby 3.4.9 aus mise.toml
|
||||||
|
bundle install
|
||||||
|
bundle exec rake test # Tests, lokales Postgres optional
|
||||||
|
bin/etl run examples/kunden_import.rb # Beispiel unten, Ergebnis in tmp/
|
||||||
|
bin/etl run examples/kunden_import.rb --dry-run # alles prüfen, nichts schreiben
|
||||||
|
```
|
||||||
|
|
||||||
|
Das Framework ist das Gem `datapipeline`. In einer anderen Anwendung wird es so eingebunden:
|
||||||
|
|
||||||
|
```ruby
|
||||||
|
# Gemfile
|
||||||
|
gem 'datapipeline', path: '../datapipeline' # oder git: '…'
|
||||||
|
gem 'pg' # nur die Gems der benötigten Plugins, siehe 2.6
|
||||||
|
```
|
||||||
|
|
||||||
|
```sh
|
||||||
|
bundle install
|
||||||
|
bundle exec etl pipeline.rb --dry-run # das Kommando etl kommt mit dem Gem
|
||||||
|
```
|
||||||
|
|
||||||
|
Eine Pipeline ist ein Ruby-Skript. Das kleinste sinnvolle Beispiel kopiert eine CSV-Datei nach JSON:
|
||||||
|
|
||||||
|
```ruby
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
read container(:csv, file: 'kunden.csv')
|
||||||
|
write container(:json, file: 'kunden.json')
|
||||||
|
end.run
|
||||||
|
```
|
||||||
|
|
||||||
|
Zwischen `read` und `write` stehen die Schritte; sie laufen in der Reihenfolge, in der sie im Skript stehen:
|
||||||
|
|
||||||
|
| Schritt | Wirkung |
|
||||||
|
| --- | --- |
|
||||||
|
| `validate regeln` | prüft jede Zeile; Fehler gehen an `on_error` |
|
||||||
|
| `filter { \|row\| … }` | Zeile bleibt, wenn der Block wahr ist |
|
||||||
|
| `map { \|row, ctx\| … }` | gibt eine Zeile, mehrere (Array) oder keine (`nil`) zurück |
|
||||||
|
| `header { \|ctx\| … }`, `trailer { \|ctx\| … }` | Kopf- und Schlusssatz mit den Zählern `ctx.read`, `ctx.written` |
|
||||||
|
|
||||||
|
### 1.2 Ausführliches Beispiel: Kundenimport
|
||||||
|
|
||||||
|
Eine Kundenliste (`examples/data/kunden.csv`, 9 Zeilen) soll geprüft, mit Ländernamen angereichert und in eine
|
||||||
|
SQLite-Datenbank geschrieben werden; EU-Kunden kommen zusätzlich in eine Excel-Liste. Fehlerhafte Zeilen sollen den
|
||||||
|
Lauf nicht abbrechen, sondern mit Grund in einer eigenen Datei landen.
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
CSV[kunden.csv] --> V1[validate<br/>Pflicht, Format, Typ]
|
||||||
|
LJ[laender.json] -. lookup .-> V2
|
||||||
|
V1 --> V2[validate<br/>Land bekannt]
|
||||||
|
V2 --> F[filter<br/>nur aktiv]
|
||||||
|
F --> M[map<br/>anreichern, umbauen]
|
||||||
|
LJ -. lookup .-> M
|
||||||
|
M --> DB[(kunden.sqlite3<br/>alle)]
|
||||||
|
M -->|only: EU| XL[eu_kunden.xlsx]
|
||||||
|
V1 -. on_error .-> ERR[kunden_fehler.csv]
|
||||||
|
V2 -. on_error .-> ERR
|
||||||
|
```
|
||||||
|
|
||||||
|
<!-- beispiel: examples/kunden_import.rb -->
|
||||||
|
```ruby
|
||||||
|
#
|
||||||
|
# Kundenliste aus CSV prüfen, mit Ländern anreichern, nach SQLite und Excel schreiben
|
||||||
|
#
|
||||||
|
# bin/etl run examples/kunden_import.rb [--out=tmp] [--dry-run]
|
||||||
|
#
|
||||||
|
require 'fileutils'
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
DATA = File.expand_path('data', __dir__)
|
||||||
|
out = DataPipeline.params.fetch(:out, 'tmp')
|
||||||
|
FileUtils.mkdir_p(out)
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
version 'kunden_import 1.0'
|
||||||
|
|
||||||
|
# Nachschlagetabelle, einmal geladen: { 'DE' => { 'code' => 'DE', 'name' => 'Deutschland', 'eu' => true }, ... }
|
||||||
|
laender = lookup container(:json, file: "#{DATA}/laender.json"), key: 'code', unique: true
|
||||||
|
|
||||||
|
read container(:csv, file: "#{DATA}/kunden.csv", csv: { col_sep: ';' })
|
||||||
|
|
||||||
|
# fehlerhafte Zeilen mit Grund in eine eigene Datei, der Lauf geht weiter
|
||||||
|
on_error :collect, to: container(:csv, file: "#{out}/kunden_fehler.csv")
|
||||||
|
|
||||||
|
# 1. prüfen
|
||||||
|
validate 'kundennr' => { required: true, format: /\A\d{6}\z/ },
|
||||||
|
'name' => { required: true, max: 40 },
|
||||||
|
'umsatz' => { type: :decimal }
|
||||||
|
validate { |row| "Land #{row['land']} unbekannt" unless laender.key?(row['land']) }
|
||||||
|
|
||||||
|
# 2. nur aktive Kunden
|
||||||
|
filter { |row| row['status'] == 'aktiv' }
|
||||||
|
|
||||||
|
# 3. anreichern und umbauen
|
||||||
|
map do |row|
|
||||||
|
land = laender.fetch(row['land'])
|
||||||
|
{
|
||||||
|
'kundennr' => row['kundennr'],
|
||||||
|
'name' => row['name'].strip,
|
||||||
|
'land' => land['name'],
|
||||||
|
'eu' => land['eu'] ? 'ja' : 'nein',
|
||||||
|
'umsatz' => row['umsatz'].to_f.round(2)
|
||||||
|
}
|
||||||
|
end
|
||||||
|
|
||||||
|
# 4. schreiben: alle Kunden nach SQLite, EU-Kunden zusätzlich als Excel-Liste
|
||||||
|
write container(:sqlite, database: "#{out}/kunden.sqlite3", table: 'kunden', create: true, truncate: true)
|
||||||
|
write container(:excel, file: "#{out}/eu_kunden.xlsx", sheet: 'EU-Kunden'), only: ->(row) { row['eu'] == 'ja' }
|
||||||
|
|
||||||
|
on_finish { |stats| puts "kunden_import: #{stats}" }
|
||||||
|
end.run
|
||||||
|
```
|
||||||
|
|
||||||
|
Was die einzelnen Teile tun:
|
||||||
|
|
||||||
|
| Zeile | Erklärung |
|
||||||
|
| --- | --- |
|
||||||
|
| `DataPipeline.params.fetch(:out, 'tmp')` | Parameter von der Kommandozeile: `--out=verzeichnis` |
|
||||||
|
| `lookup …, key: 'code', unique: true` | lädt die Länder einmal als Hash nach Ländercode; kein Zugriff je Zeile |
|
||||||
|
| `on_error :collect, to: …` | jede Zeile, die in einem Schritt scheitert, kommt mit `_step` und `_error` in die Fehlerdatei |
|
||||||
|
| `validate 'kundennr' => { … }` | Regeln je Spalte: `required`, `format`, `max`, `min`, `in`, `type` |
|
||||||
|
| `validate { \|row\| … }` | eigene Regel: ein Text (oder ein Array) ist ein Fehler, `nil` heisst in Ordnung |
|
||||||
|
| `filter` | inaktive Kunden fallen weg, ohne Fehler |
|
||||||
|
| `map` | baut die Zielzeile; `laender.fetch` kann nicht scheitern, weil `validate` das Land schon geprüft hat |
|
||||||
|
| `write …` zweimal | jedes Ziel bekommt die Zeilen; `only:` wählt die Zeilen für die Excel-Liste |
|
||||||
|
| `create: true, truncate: true` | SQLite legt die Tabelle beim ersten Schreiben an und leert sie; im Dry-Run passiert nichts |
|
||||||
|
| `on_finish` | läuft nach jedem Lauf, auch nach einem Abbruch; `stats` siehe 2.7 |
|
||||||
|
|
||||||
|
Ergebnis von `bin/etl run examples/kunden_import.rb`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
INFO start kunden_import 1.0
|
||||||
|
WARN rejected by validate#1: name: missing
|
||||||
|
WARN rejected by validate#1: kundennr: format /\A\d{6}\z/
|
||||||
|
WARN rejected by validate#2: Land XX unbekannt
|
||||||
|
WARN rejected by validate#1: umsatz: not decimal
|
||||||
|
INFO finish read=9 written=4 rejected=4 0.097s
|
||||||
|
kunden_import: read=9 written=4 rejected=4 0.097s
|
||||||
|
```
|
||||||
|
|
||||||
|
| Datei | Inhalt |
|
||||||
|
| --- | --- |
|
||||||
|
| `tmp/kunden.sqlite3`, Tabelle `kunden` | 4 aktive, gültige Kunden (DE, AT, CH, IT) mit Ländername, `eu`, `umsatz` als Zahl |
|
||||||
|
| `tmp/eu_kunden.xlsx`, Blatt `EU-Kunden` | 3 Kunden (ohne CH) |
|
||||||
|
| `tmp/kunden_fehler.csv` | 4 Zeilen: leerer Name, Kundennummer mit 5 Stellen, Land `XX`, Umsatz `abc` – je mit `_step` und `_error` |
|
||||||
|
|
||||||
|
Der Exit-Code ist `2`, weil Zeilen verworfen wurden; ein Abbruch ergäbe `1`, ein fehlerfreier Lauf `0`. Mit
|
||||||
|
`--dry-run` läuft alles gleich, es entstehen aber weder Datenbank noch Excel- oder Fehlerdatei; stattdessen stehen
|
||||||
|
die ersten Zeilen auf dem Bildschirm.
|
||||||
|
|
||||||
|
### 1.3 Weitere Beispiele
|
||||||
|
|
||||||
|
Alle Beispiele liegen in [examples/](examples/), arbeiten mit den Daten in `examples/data/` und schreiben nach `tmp/`.
|
||||||
|
`test/examples/doc_examples_test.rb` führt sie aus und prüft, dass der Code hier mit den Dateien übereinstimmt.
|
||||||
|
|
||||||
|
**Datei mit Satzarten** – Rechnungen aus verschachteltem JSON werden zu einem Kopfsatz mit Positionssätzen; Datei-
|
||||||
|
und Schlusssatz zählen die Datensätze. Das Layout legt Reihenfolge, Typ und Länge der Felder je Satzart fest (2.9).
|
||||||
|
|
||||||
|
<!-- beispiel: examples/rechnungen_satzarten.rb -->
|
||||||
|
```ruby
|
||||||
|
#
|
||||||
|
# Rechnungen aus verschachteltem JSON in eine Datei mit Satzarten (Datei-, Kopf-, Positions-, Schlusssatz)
|
||||||
|
#
|
||||||
|
# bin/etl run examples/rechnungen_satzarten.rb [--out=tmp]
|
||||||
|
#
|
||||||
|
require 'fileutils'
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
DATA = File.expand_path('data', __dir__)
|
||||||
|
out = DataPipeline.params.fetch(:out, 'tmp')
|
||||||
|
FileUtils.mkdir_p(out)
|
||||||
|
|
||||||
|
# Feld: [Name, Typ, Länge, Pflicht]; Typ AN Text, N Zahl (11.2 = 9 Vor-, 2 Nachkommastellen), DAT YYYYMMDD
|
||||||
|
LAYOUT = DataPipeline::Layout.new do
|
||||||
|
record 'DATEI', [['SATYP', 'AN', 5, true], ['ERSTELLT', 'DAT', 8, true], ['ANZAHL', 'N', 6, true]]
|
||||||
|
record 'KOPF', [['SATYP', 'AN', 5, true], ['LFDNR', 'N', 6, true], ['BELNR', 'AN', 10, true],
|
||||||
|
['DATUM', 'DAT', 8, true], ['KUNNR', 'AN', 10, true], ['SUMME', 'N', '11.2', true]]
|
||||||
|
record 'POS', [['SATYP', 'AN', 5, true], ['LFDNR', 'N', 6, true], ['BELNR', 'AN', 10, true],
|
||||||
|
['POSNR', 'N', 3, true], ['TEXT', 'AN', 30], ['BETRAG', 'N', '11.2', true]]
|
||||||
|
record 'ENDE', [['SATYP', 'AN', 5, true], ['ANZAHL', 'N', 6, true]]
|
||||||
|
end
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
read container(:json, file: "#{DATA}/rechnungen.json")
|
||||||
|
|
||||||
|
# 1. eine Rechnung -> Kopfsatz + n Positionssätze
|
||||||
|
map do |rechnung|
|
||||||
|
positionen = rechnung['positionen'].each_with_index.map do |pos, i|
|
||||||
|
{ 'satyp' => 'POS', 'belnr' => rechnung['nr'], 'posnr' => format('%03d', i + 1),
|
||||||
|
'text' => pos['text'], 'betrag' => format('%.2f', pos['menge'] * pos['preis']) }
|
||||||
|
end
|
||||||
|
kopf = { 'satyp' => 'KOPF', 'belnr' => rechnung['nr'], 'datum' => rechnung['datum'].delete('-'),
|
||||||
|
'kunnr' => rechnung['kunde'], 'summe' => format('%.2f', positionen.sum { |p| p['betrag'].to_f }) }
|
||||||
|
[kopf, *positionen]
|
||||||
|
end
|
||||||
|
|
||||||
|
# 2. laufende Nummer über alle Kopf- und Positionssätze
|
||||||
|
map { |row, ctx| row.merge('lfdnr' => format('%06d', ctx.index + 1)) }
|
||||||
|
|
||||||
|
# 3. gegen das Layout prüfen: zu langer Text wird gemeldet, die Zeile bleibt
|
||||||
|
validate LAYOUT, warn: true
|
||||||
|
|
||||||
|
# Datei- und Schlusssatz kennen die Anzahl der Datensätze
|
||||||
|
header { |ctx| { 'satyp' => 'DATEI', 'erstellt' => Date.today.strftime('%Y%m%d'), 'anzahl' => format('%06d', ctx.written) } }
|
||||||
|
trailer { |ctx| { 'satyp' => 'ENDE', 'anzahl' => format('%06d', ctx.written) } }
|
||||||
|
|
||||||
|
# Felder in der Reihenfolge des Layouts; Positionen zusätzlich als NDJSON
|
||||||
|
write container(:csv, file: "#{out}/rechnungen.txt", layout: LAYOUT, csv: { headers: false, col_sep: '|' })
|
||||||
|
write container(:json, file: "#{out}/positionen.ndjson"), only: ->(row) { row['satyp'] == 'POS' }
|
||||||
|
end.run
|
||||||
|
```
|
||||||
|
|
||||||
|
Ergebnis `tmp/rechnungen.txt` (die Warnung meldet den Positionstext mit mehr als 30 Zeichen):
|
||||||
|
|
||||||
|
```text
|
||||||
|
DATEI|20261008|000005
|
||||||
|
KOPF|000001|R-1001|20251103|100001|380.50
|
||||||
|
POS|000002|R-1001|001|Beratung|300.00
|
||||||
|
POS|000003|R-1001|002|Reisekosten|80.50
|
||||||
|
KOPF|000004|R-1002|20251105|100002|1200.00
|
||||||
|
POS|000005|R-1002|001|Jahreslizenz Software mit Wartung und Support|1200.00
|
||||||
|
ENDE|000005
|
||||||
|
```
|
||||||
|
|
||||||
|
**Inkrementell und gestreamt** – nur Bewegungen, die seit dem letzten erfolgreichen Lauf dazugekommen sind; jede
|
||||||
|
Zeile wird sofort geschrieben, der Speicherbedarf bleibt auch bei Millionen Zeilen klein.
|
||||||
|
|
||||||
|
<!-- beispiel: examples/bewegungen_inkrementell.rb -->
|
||||||
|
```ruby
|
||||||
|
#
|
||||||
|
# Nur neue Buchungsbewegungen exportieren, Zeile für Zeile (Streaming)
|
||||||
|
# Ein zweiter Lauf ohne neue Daten schreibt nichts.
|
||||||
|
#
|
||||||
|
# bin/etl run examples/bewegungen_inkrementell.rb [--out=tmp]
|
||||||
|
#
|
||||||
|
require 'fileutils'
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
DATA = File.expand_path('data', __dir__)
|
||||||
|
out = DataPipeline.params.fetch(:out, 'tmp')
|
||||||
|
FileUtils.mkdir_p(out)
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
streamed
|
||||||
|
|
||||||
|
# höchste id des letzten erfolgreichen Laufs, gespeichert in tmp/.state.json
|
||||||
|
stand = watermark :bewegungen, column: 'id', file: "#{out}/.state.json", initial: 0
|
||||||
|
|
||||||
|
# bei einer Datenbank gehört der Filter ins SQL: where id > $1, params: [stand.value]
|
||||||
|
read container(:json, file: "#{DATA}/bewegungen.ndjson")
|
||||||
|
filter { |row| row['id'] > stand.value.to_i }
|
||||||
|
map { |row| row.merge('soll_haben' => row['betrag'].negative? ? 'H' : 'S', 'betrag' => row['betrag'].abs) }
|
||||||
|
|
||||||
|
write container(:json,
|
||||||
|
file: "#{out}/bewegungen_#{Time.now.strftime('%Y%m%d_%H%M%S')}.ndjson"
|
||||||
|
)
|
||||||
|
|
||||||
|
on_finish { |stats| puts "neu: #{stats.written}, Stand jetzt: #{stand.current}" }
|
||||||
|
end.run
|
||||||
|
```
|
||||||
|
|
||||||
|
Der erste Lauf schreibt 4 Bewegungen und merkt sich `4`; der zweite schreibt nichts (`neu: 0`) und legt auch keine
|
||||||
|
Datei an. Bricht ein Lauf ab, bleibt der alte Stand stehen.
|
||||||
|
|
||||||
|
**Eigenes Plugin** – eine Quelle, die alle CSV-Dateien eines Verzeichnisses liest. Ein Plugin braucht nur die
|
||||||
|
Methoden, die es anbietet (hier `read`); die Schnittstelle steht in 2.4.
|
||||||
|
|
||||||
|
<!-- beispiel: examples/eigenes_plugin.rb -->
|
||||||
|
```ruby
|
||||||
|
#
|
||||||
|
# Eigenes Plugin: liest alle CSV-Dateien eines Verzeichnisses als eine Quelle
|
||||||
|
#
|
||||||
|
# bin/etl run examples/eigenes_plugin.rb [--out=tmp]
|
||||||
|
#
|
||||||
|
require 'fileutils'
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
DATA = File.expand_path('data', __dir__)
|
||||||
|
out = DataPipeline.params.fetch(:out, 'tmp')
|
||||||
|
FileUtils.mkdir_p(out)
|
||||||
|
|
||||||
|
class VerzeichnisStorage < DataPipeline::StoragePlugin
|
||||||
|
def initialize dir:, pattern: '*.csv', **csv_options
|
||||||
|
@dir = dir
|
||||||
|
@pattern = pattern
|
||||||
|
@csv_options = csv_options
|
||||||
|
end
|
||||||
|
|
||||||
|
# jede Zeile bekommt den Dateinamen dazu; mit Block zeilenweise (für streamed)
|
||||||
|
def read(&block)
|
||||||
|
rows = Dir.glob(File.join(@dir, @pattern)).sort.flat_map do |file|
|
||||||
|
DataPipeline::CsvStorage.new(file:, **@csv_options).read.map { |row| row.merge('datei' => File.basename(file, '.*')) }
|
||||||
|
end
|
||||||
|
return rows unless block
|
||||||
|
|
||||||
|
rows.each(&block)
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
DataPipeline.register :verzeichnis, VerzeichnisStorage
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
read container(:verzeichnis, dir: "#{DATA}/filialen")
|
||||||
|
map { |row| row.merge('menge' => row['menge'].to_i) }
|
||||||
|
write container(:csv, file: "#{out}/alle_filialen.csv", columns: %w[datei artikel menge])
|
||||||
|
end.run
|
||||||
|
```
|
||||||
|
|
||||||
|
Ergebnis `tmp/alle_filialen.csv`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
datei,artikel,menge
|
||||||
|
vaduz,A-100,5
|
||||||
|
vaduz,B-200,2
|
||||||
|
zuerich,A-100,7
|
||||||
|
zuerich,C-300,1
|
||||||
|
```
|
||||||
|
|
||||||
|
Weitere Ausschnitte: REST-API nach Excel in 2.5 (REST ausführlich in 1.4), Benachrichtigung über Slack in 2.7.
|
||||||
|
|
||||||
|
### 1.4 Beispiele mit REST-APIs
|
||||||
|
|
||||||
|
Die drei Beispiele laufen gegen eine kleine Demo-API ([examples/demo_api.rb](examples/demo_api.rb)), die ohne Internet
|
||||||
|
und ohne weitere Gems auskommt. Sie verlangt ein Token, liefert Seiten und Folge-Links, nimmt Daten per POST und PUT an
|
||||||
|
und antwortet einmal absichtlich mit `503`, damit die Wiederholung sichtbar wird.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
ruby examples/demo_api.rb & # http://127.0.0.1:4567, Token demo-token
|
||||||
|
bin/etl run examples/rest_lesen.rb
|
||||||
|
bin/etl run examples/rest_schreiben.rb
|
||||||
|
bin/etl run examples/rest_zu_rest.rb
|
||||||
|
```
|
||||||
|
|
||||||
|
Gegen eine echte API ändern sich nur `--url=…` und das Token (`DEMO_API_TOKEN`, in der Praxis aus der `.env`).
|
||||||
|
|
||||||
|
| Endpunkt der Demo-API | Antwort |
|
||||||
|
| --- | --- |
|
||||||
|
| `GET /api/kunden?page=1&per_page=3` | `{"data": [...], "page": 1}`; Seite 2 antwortet einmal mit `503` |
|
||||||
|
| `GET /api/laender` | `[{"code": "DE", "name": "Deutschland"}, ...]` |
|
||||||
|
| `GET /api/bestellungen?cursor=n` | `{"items": [...], "next": "/api/bestellungen?cursor=n+2"}`, auf der letzten Seite `"next": null` |
|
||||||
|
| `POST /api/import` | ein Objekt oder Array; `422`, wenn `kundennr` oder `name` fehlt |
|
||||||
|
| `PUT /api/archiv/<id>` | legt das Objekt unter der id ab |
|
||||||
|
|
||||||
|
**Lesen: seitenweise, mit Wiederholung und Nachschlagen** – `page_param` holt Seite für Seite, bis eine leer ist;
|
||||||
|
`root` zeigt auf das Array in der Antwort. Ein zweiter Container auf denselben Server dient mit `query` als Lookup.
|
||||||
|
|
||||||
|
<!-- beispiel: examples/rest_lesen.rb -->
|
||||||
|
```ruby
|
||||||
|
#
|
||||||
|
# Kunden seitenweise aus einer REST-API lesen, mit Ländern aus einem zweiten Endpunkt anreichern,
|
||||||
|
# nach CSV und JSON schreiben
|
||||||
|
#
|
||||||
|
# ruby examples/demo_api.rb & # Demo-API starten
|
||||||
|
# bin/etl run examples/rest_lesen.rb [--url=http://127.0.0.1:4567] [--out=tmp]
|
||||||
|
#
|
||||||
|
require 'fileutils'
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
API = DataPipeline.params.fetch(:url, 'http://127.0.0.1:4567')
|
||||||
|
AUTH = "Bearer #{ENV.fetch('DEMO_API_TOKEN', 'demo-token')}"
|
||||||
|
out = DataPipeline.params.fetch(:out, 'tmp')
|
||||||
|
FileUtils.mkdir_p(out)
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
version 'rest_lesen 1.0'
|
||||||
|
|
||||||
|
# GET /api/kunden?page=1&per_page=3, dann page=2, ... bis eine Seite leer ist
|
||||||
|
# die Zeilen stehen in der Antwort unter "data"; 429/5xx und Netzfehler werden bis zu 3-mal wiederholt
|
||||||
|
kunden = container :rest, url: "#{API}/api/kunden", auth: AUTH,
|
||||||
|
root: 'data', page_param: 'page', params: { 'per_page' => 3 },
|
||||||
|
retries: 3, retry_wait: 0.2
|
||||||
|
|
||||||
|
# derselbe Server, anderer Endpunkt: query hängt den Pfad an die url an (GET /api/laender)
|
||||||
|
stamm = container :rest, url: "#{API}/api", auth: AUTH
|
||||||
|
laender = lookup stamm, query: 'laender', key: 'code', unique: true
|
||||||
|
|
||||||
|
read kunden
|
||||||
|
|
||||||
|
on_error :collect, to: container(:csv, file: "#{out}/api_fehler.csv")
|
||||||
|
validate 'id' => { required: true }, 'umsatz' => { type: :decimal }
|
||||||
|
|
||||||
|
map do |kunde|
|
||||||
|
{
|
||||||
|
'id' => kunde['id'],
|
||||||
|
'name' => kunde['name'],
|
||||||
|
'land' => laender.dig(kunde['land'], 'name') || kunde['land'],
|
||||||
|
'umsatz' => kunde['umsatz'].to_f
|
||||||
|
}
|
||||||
|
end
|
||||||
|
|
||||||
|
write container(:csv, file: "#{out}/api_kunden.csv", columns: %w[id name land umsatz])
|
||||||
|
write container(:json, file: "#{out}/api_kunden.json", pretty: true)
|
||||||
|
|
||||||
|
on_finish { |stats| puts "rest_lesen: #{stats}" }
|
||||||
|
end.run
|
||||||
|
```
|
||||||
|
|
||||||
|
```text
|
||||||
|
INFO start rest_lesen 1.0
|
||||||
|
WARN rejected by validate#1: umsatz: not decimal
|
||||||
|
INFO finish read=7 written=6 rejected=1 0.213s
|
||||||
|
```
|
||||||
|
|
||||||
|
Gelesen werden 7 Kunden in 3 Seiten (plus eine leere Seite als Ende); die `503` auf Seite 2 bemerkt man nur an der
|
||||||
|
Laufzeit. Kunde 5 hat den Umsatz `n/a` und steht in `tmp/api_fehler.csv`. Ergebnis `tmp/api_kunden.csv`:
|
||||||
|
|
||||||
|
```text
|
||||||
|
id,name,land,umsatz
|
||||||
|
1,Müller GmbH,Deutschland,12500.5
|
||||||
|
2,Huber & Partner,Österreich,8300.0
|
||||||
|
3,Swiss Trade AG,Schweiz,22000.75
|
||||||
|
4,Rossi Srl,Italien,4100.1
|
||||||
|
6,Nordic AB,SE,990.0
|
||||||
|
7,Vaduz Treuhand,Liechtenstein,15000.0
|
||||||
|
```
|
||||||
|
|
||||||
|
Schweden fehlt in `/api/laender`, deshalb bleibt bei Kunde 6 der Code `SE` stehen.
|
||||||
|
|
||||||
|
**Schreiben: prüfen, dann in Paketen senden** – `batch_size: 2` schickt je 2 Zeilen als JSON-Array. Was die API
|
||||||
|
ablehnen würde, wird vorher geprüft: Lehnt die API ein Paket trotzdem ab (`4xx`), bricht der Lauf ab, weil ein Fehler
|
||||||
|
beim Schreiben nicht einer einzelnen Zeile zugeordnet werden kann.
|
||||||
|
|
||||||
|
<!-- beispiel: examples/rest_schreiben.rb -->
|
||||||
|
```ruby
|
||||||
|
#
|
||||||
|
# Kunden aus CSV prüfen und in Paketen per POST an eine REST-API senden
|
||||||
|
#
|
||||||
|
# ruby examples/demo_api.rb &
|
||||||
|
# bin/etl run examples/rest_schreiben.rb [--url=http://127.0.0.1:4567] [--dry-run]
|
||||||
|
#
|
||||||
|
require 'fileutils'
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
API = DataPipeline.params.fetch(:url, 'http://127.0.0.1:4567')
|
||||||
|
AUTH = "Bearer #{ENV.fetch('DEMO_API_TOKEN', 'demo-token')}"
|
||||||
|
DATA = File.expand_path('data', __dir__)
|
||||||
|
out = DataPipeline.params.fetch(:out, 'tmp')
|
||||||
|
FileUtils.mkdir_p(out)
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
version 'rest_schreiben 1.0'
|
||||||
|
|
||||||
|
read container(:csv, file: "#{DATA}/kunden.csv", csv: { col_sep: ';' })
|
||||||
|
|
||||||
|
# vor dem Senden prüfen: was die API ablehnen würde, landet hier in der Fehlerdatei
|
||||||
|
on_error :collect, to: container(:csv, file: "#{out}/import_fehler.csv")
|
||||||
|
validate 'kundennr' => { required: true, format: /\A\d{6}\z/ }, 'name' => { required: true }
|
||||||
|
|
||||||
|
filter { |row| row['status'] == 'aktiv' }
|
||||||
|
map do |row|
|
||||||
|
{ 'kundennr' => row['kundennr'], 'name' => row['name'].strip, 'land' => row['land'], 'umsatz' => row['umsatz'].to_f }
|
||||||
|
end
|
||||||
|
|
||||||
|
# POST /api/import mit je 2 Kunden als JSON-Array; im Dry-Run wird nichts gesendet
|
||||||
|
write container(:rest, url: "#{API}/api/import", auth: AUTH, batch_size: 2)
|
||||||
|
|
||||||
|
on_finish do |stats|
|
||||||
|
puts stats.dry_run ? 'Dry-Run: nichts gesendet' : "gesendet: #{stats.written} Kunden in #{(stats.written / 2.0).ceil} Paketen"
|
||||||
|
end
|
||||||
|
end.run
|
||||||
|
```
|
||||||
|
|
||||||
|
Ergebnis: 6 aktive Kunden in 3 Anfragen an `POST /api/import`, 2 Kunden in `tmp/import_fehler.csv` (leerer Name,
|
||||||
|
Kundennummer mit 5 Stellen). Mit `--dry-run` wird nichts gesendet; auf dem Bildschirm stehen die Daten, die gesendet
|
||||||
|
würden:
|
||||||
|
|
||||||
|
```text
|
||||||
|
DRY RUN – nothing written: read=9 written=6 rejected=2 0.0s (dry run)
|
||||||
|
{"kundennr":"100001","name":"Müller GmbH","land":"DE","umsatz":12500.5}
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
**Von API zu API: Folge-Links, Streaming, eine URL je Datensatz** – Die Antwort nennt die nächste Seite selbst; das
|
||||||
|
Lambda in `next:` macht daraus eine vollständige URL. `write_url` als Lambda schickt jede Bestellung an ihre eigene
|
||||||
|
Adresse. Mit dem Wasserzeichen archiviert ein zweiter Lauf nur, was neu dazugekommen ist.
|
||||||
|
|
||||||
|
<!-- beispiel: examples/rest_zu_rest.rb -->
|
||||||
|
```ruby
|
||||||
|
#
|
||||||
|
# Von API zu API: neue Bestellungen über Folge-Links lesen und einzeln per PUT archivieren
|
||||||
|
# Ein zweiter Lauf ohne neue Bestellungen sendet nichts.
|
||||||
|
#
|
||||||
|
# ruby examples/demo_api.rb &
|
||||||
|
# bin/etl run examples/rest_zu_rest.rb [--url=http://127.0.0.1:4567] [--out=tmp]
|
||||||
|
#
|
||||||
|
require 'fileutils'
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
API = DataPipeline.params.fetch(:url, 'http://127.0.0.1:4567')
|
||||||
|
AUTH = "Bearer #{ENV.fetch('DEMO_API_TOKEN', 'demo-token')}"
|
||||||
|
out = DataPipeline.params.fetch(:out, 'tmp')
|
||||||
|
FileUtils.mkdir_p(out)
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
streamed
|
||||||
|
|
||||||
|
# höchste Bestellnummer des letzten erfolgreichen Laufs
|
||||||
|
stand = watermark :bestellungen, column: 'id', file: "#{out}/.state.json", initial: 0
|
||||||
|
|
||||||
|
# die Antwort nennt die nächste Seite selbst: {"items": [...], "next": "/api/bestellungen?cursor=2"}
|
||||||
|
read container(:rest, url: "#{API}/api/bestellungen", auth: AUTH, root: 'items',
|
||||||
|
next: ->(body) { body['next'] && "#{API}#{body['next']}" })
|
||||||
|
|
||||||
|
filter { |row| row['id'] > stand.value.to_i }
|
||||||
|
map { |row| row.merge('archiviert' => Date.today.iso8601) }
|
||||||
|
|
||||||
|
# eine url je Bestellung: PUT /api/archiv/<id>
|
||||||
|
write container(:rest, url: "#{API}/api/archiv", auth: AUTH, write_method: :put,
|
||||||
|
write_url: ->(row) { "#{API}/api/archiv/#{row['id']}" })
|
||||||
|
|
||||||
|
on_finish { |stats| puts "archiviert: #{stats.written}, Stand jetzt: #{stand.current}" }
|
||||||
|
end.run
|
||||||
|
```
|
||||||
|
|
||||||
|
```text
|
||||||
|
archiviert: 5, Stand jetzt: 5 # erster Lauf: PUT /api/archiv/1 … /5
|
||||||
|
archiviert: 0, Stand jetzt: 5 # zweiter Lauf: keine Anfrage
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Technische Dokumentation
|
||||||
|
|
||||||
|
### 2.1 Aufbau
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
subgraph Pipeline-Skript
|
||||||
|
DSL[DataPipeline.new do ... end]
|
||||||
|
end
|
||||||
|
DSL -->|container :typ| REG[Registry / autoload<br/><Typ>Storage]
|
||||||
|
REG --> SP[DataPipeline::StoragePlugin<br/>read · write · write_row · query · close]
|
||||||
|
SP --- PL[postgres · sqlite · csv · json<br/>excel · rest · eigene]
|
||||||
|
DSL -->|lookup| LK[(Nachschlage-Hash)]
|
||||||
|
SRC((Quelle)) --> STEPS[Schritte<br/>validate · filter · map · transform]
|
||||||
|
LK -.-> STEPS
|
||||||
|
STEPS --> EDGE[header · trailer<br/>ctx.read · ctx.written]
|
||||||
|
EDGE --> DST1((Ziel 1))
|
||||||
|
EDGE -->|only:| DST2((Ziel 2))
|
||||||
|
STEPS -. on_error .-> REJ((Fehlerdatei))
|
||||||
|
DSL -->|run| ST[Stats · on_finish<br/>Exit-Code über bin/etl]
|
||||||
|
```
|
||||||
|
|
||||||
|
| Datei | Inhalt |
|
||||||
|
| --- | --- |
|
||||||
|
| [lib/datapipeline.rb](lib/datapipeline.rb) | DSL und Ausführung (`run`), `Stats`, `Context` |
|
||||||
|
| [lib/datapipeline/validation.rb](lib/datapipeline/validation.rb) | Regeln für `validate` |
|
||||||
|
| [lib/datapipeline/watermark.rb](lib/datapipeline/watermark.rb) | Wasserzeichen für inkrementelle Läufe |
|
||||||
|
| [lib/datapipeline/layout.rb](lib/datapipeline/layout.rb) | Satzarten-Layouts (Feldliste je Satzart, siehe 2.9) |
|
||||||
|
| [exe/etl](exe/etl) | Kommando `etl` des Gems mit Exit-Codes (siehe 2.7); [bin/etl](bin/etl) startet es im Repository mit dem Bundle |
|
||||||
|
| [datapipeline.gemspec](datapipeline.gemspec), [CHANGELOG.md](CHANGELOG.md) | Gem-Beschreibung, Versionsverlauf |
|
||||||
|
| [examples/](examples/) | Beispiele aus Abschnitt 1 mit Daten in `examples/data/`, Demo-API für die REST-Beispiele |
|
||||||
|
| [lib/datapipeline/storage/storage_plugin.rb](lib/datapipeline/storage/storage_plugin.rb) | Basisklasse der Storage-Plugins |
|
||||||
|
| [lib/datapipeline/storage/postgres.rb](lib/datapipeline/storage/postgres.rb) | Quelle/Ziel PostgreSQL (`pg`) |
|
||||||
|
| [lib/datapipeline/storage/csv.rb](lib/datapipeline/storage/csv.rb) | Quelle/Ziel CSV |
|
||||||
|
| [lib/datapipeline/storage/json.rb](lib/datapipeline/storage/json.rb) | Quelle/Ziel JSON und NDJSON |
|
||||||
|
| [lib/datapipeline/storage/sqlite.rb](lib/datapipeline/storage/sqlite.rb) | Quelle/Ziel SQLite (`sqlite3`) |
|
||||||
|
| [lib/datapipeline/storage/excel.rb](lib/datapipeline/storage/excel.rb) | Quelle/Ziel Excel xlsx (`roo`, `caxlsx`) |
|
||||||
|
| [lib/datapipeline/storage/rest.rb](lib/datapipeline/storage/rest.rb) | Quelle/Ziel REST-API (`http`) |
|
||||||
|
| [lib/datapipeline/notification/slack.rb](lib/datapipeline/notification/slack.rb) | Benachrichtigung, z. B. aus `on_finish` |
|
||||||
|
| [lib/datapipeline/version.rb](lib/datapipeline/version.rb) | `DataPipeline::VERSION` |
|
||||||
|
|
||||||
|
### 2.2 DSL-Referenz
|
||||||
|
|
||||||
|
| Aufruf | Wirkung |
|
||||||
|
| --- | --- |
|
||||||
|
| `DataPipeline.settings(opts)` | Defaults für alle folgenden Pipelines; der Hash wird ersetzt, nicht verändert (laufende Pipelines behalten ihre Kopie); `reset_settings!` setzt zurück |
|
||||||
|
| `DataPipeline.register(:typ, Klasse \| 'Klassenname', path:)` | eigenes Plugin; mit `path:` erst bei der ersten Nutzung geladen |
|
||||||
|
| `DataPipeline.new(**settings) { ... }` | führt den Block per `instance_eval` aus; `settings` gelten nur für diese Pipeline, z. B. `new(limit: 10)` |
|
||||||
|
| `configure(opts)` | Einstellungen dieser Pipeline als Hash: `verbose`, `streamed`, `chained`, `version` |
|
||||||
|
| `version(str)`, `verbose(flag = true)`, `streamed(flag = true)`, `chained(flag = true)` | dasselbe als DSL-Aufruf |
|
||||||
|
| `container(:typ, **opts)` | erzeugt das registrierte Plugin, sonst `<Typ>Storage.new(**opts)` (z. B. `:postgres` → `DataPipeline::PostgresStorage`); fehlt es: „Unsupported storage type“; fehlende Gems, Verbindungs- und Optionsfehler kommen unverändert durch |
|
||||||
|
| `read(storage, opts = {})` | legt die Quelle fest; `opts` gehen an `storage.set` |
|
||||||
|
| `transform(opts = {}) { \|row, index\| ... }` | Transformation; `processing: :prepend \| :normal \| :append`, `condition:` (siehe 2.3) |
|
||||||
|
| `map(opts = {}) { \|row, ctx\| ... }` | Schritt in der Kette: Hash, Array oder `nil` zurückgeben; schaltet Verkettung ein |
|
||||||
|
| `filter(opts = {}) { \|row, ctx\| ... }` | Schritt in der Kette: Zeile bleibt, wenn der Block wahr ist; schaltet Verkettung ein |
|
||||||
|
| `header { \|ctx\| ... }` | Zeile(n) vor den Daten; wird nach den Daten berechnet, kennt also `ctx.written` |
|
||||||
|
| `trailer { \|ctx\| ... }` | Zeile(n) nach den Daten |
|
||||||
|
| `query(storage, query: sql, params: [])` | führt sofort eine Abfrage aus (Subquery in einem Schritt) |
|
||||||
|
| `lookup(storage, key:, unique: false, query:, params:)` | lädt eine Tabelle einmal als Hash: `key => [Zeilen]`, mit `unique: true` `key => Zeile`; `key` als Spaltenname oder Lambda; ohne `query:` wird `storage.read` genutzt |
|
||||||
|
| `write(storage, only: ->(row) { ... })` | Ziel; mehrfach möglich, jedes Ziel bekommt alle Zeilen oder mit `only:` die passenden (auch Kopf- und Schlusssätze) |
|
||||||
|
| `validate(rules \| layout, warn: false, ignore: []) { \|row\| ... }` | Prüfschritt (siehe 2.7, 2.9); `warn: true` nur Warnung, Zeile bleibt; schaltet Verkettung ein |
|
||||||
|
| `on_error(:abort \| :skip \| :collect, to: storage)` | Umgang mit Fehlern in Datenschritten (siehe 2.7) |
|
||||||
|
| `on_finish { \|stats\| ... }` | nach jedem Lauf, auch nach Abbruch |
|
||||||
|
| `watermark(name, column:, file:, initial:)` | letzter verarbeiteter Wert für inkrementelle Läufe (siehe 2.7) |
|
||||||
|
| `dry_run(flag = true)`, `limit(n)` | nichts schreiben bzw. nur die ersten n Quellzeilen |
|
||||||
|
| `dry_run?` | in Schritten mit Nebenwirkungen (API-Aufrufe): `true` im Dry-Run |
|
||||||
|
| `logger(logger)` | eigener Ruby-`Logger`; Default: Warnungen auf `$stderr` |
|
||||||
|
| `params` | Parameter von der Kommandozeile: über `bin/etl` oder direkt aus `ARGV` (`--name=wert`, `--param key=wert`) |
|
||||||
|
| `run` | führt die Pipeline aus und liefert `DataPipeline::Stats` |
|
||||||
|
| jeder andere Aufruf | `NoMethodError` (Tippfehler fallen sofort auf) |
|
||||||
|
|
||||||
|
### 2.3 Ausführungsmodell (`run`)
|
||||||
|
|
||||||
|
1. Batch (Default): `data = source.read` lädt die gesamte Quelle in den Speicher. Streaming: siehe unten.
|
||||||
|
2. Die Schritte werden in drei Gruppen geteilt, innerhalb jeder Gruppe gilt die Reihenfolge der Registrierung:
|
||||||
|
**vorne** (`:prepend`, `header`), **Daten** (`:normal`, `map`, `filter`), **hinten** (`:append`, `trailer`).
|
||||||
|
3. Die Daten-Schritte laufen in einem von zwei Modi:
|
||||||
|
|
||||||
|
| Modus | Wann | Eingabe eines Schritts |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| parallel | Default | immer eine eigene Kopie der Quellzeilen; die Ergebnisse aller Schritte werden hintereinander angehängt |
|
||||||
|
| verkettet | `chained`, oder sobald `map`/`filter` vorkommt, oder ganz ohne Schritte | die Ausgabe des vorherigen Schritts |
|
||||||
|
|
||||||
|
Beispiel mit 2 Quellzeilen und `transform { |r| r.merge('b'=>1) }` + `transform { |r| r.merge('c'=>2) }`:
|
||||||
|
parallel 4 Zeilen (2× mit `b`, dann 2× mit `c`), verkettet 2 Zeilen mit `b` und `c`.
|
||||||
|
4. Rückgabe eines Blocks: ein Hash ergibt eine Zeile, ein Array mehrere Zeilen (1:n, z. B. Kopf + Positionen),
|
||||||
|
`nil` oder `[]` keine Zeile; `nil` innerhalb eines Arrays wird übersprungen.
|
||||||
|
5. `condition:` an `transform` oder `map`: String (per `eval`, Variable `row` sichtbar) oder Lambda `->(row) { ... }`.
|
||||||
|
Zeilen, die die Bedingung nicht erfüllen, fallen in diesem Schritt weg.
|
||||||
|
6. Vorne und hinten:
|
||||||
|
|
||||||
|
| Schritt | Aufruf | Zeitpunkt |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `transform(processing: :prepend)` | `(nil, 0)` | vor den Daten |
|
||||||
|
| `header` | `(ctx)` mit `read`, `written` | nach den Daten, steht aber vorne |
|
||||||
|
| `transform(processing: :append)` | `(nil, n)`: parallel n = Quellzeilen, verkettet n = Datenzeilen | nach den Daten |
|
||||||
|
| `trailer` | `(ctx)` mit `read`, `written` | nach den Daten |
|
||||||
|
|
||||||
|
7. Ohne Schritte wird die Quelle unverändert ins Ziel kopiert.
|
||||||
|
8. Batch: `target.write(ausgabe)` schreibt alles auf einmal.
|
||||||
|
|
||||||
|
Der Kontext `ctx` (`DataPipeline::Context`) enthält `read` (Anzahl Quellzeilen), `written` (Anzahl Datenzeilen nach allen
|
||||||
|
Schritten) und bei `map`/`filter` `index` (Position der Zeile im aktuellen Schritt).
|
||||||
|
|
||||||
|
Mit `verbose` werden im Batch Quelle, Daten nach allen Schritten und Ziel als JSON ausgegeben.
|
||||||
|
|
||||||
|
Nach dem Lauf schliesst `run` Quelle und Ziel (`close`), auch im Fehlerfall.
|
||||||
|
|
||||||
|
**Streaming-Modus** (`streamed`): Jede Zeile geht einzeln von `source.read { |row| ... }` durch die Kette und sofort
|
||||||
|
an `target.write_row`; am Ende `target.finalize`. Der Speicherbedarf bleibt konstant (getestet: 500 000 CSV-Zeilen
|
||||||
|
mit 30 MB statt 614 MB im Batch). Die Datenzeilen sind dieselben wie im Batch, mit diesen Unterschieden:
|
||||||
|
|
||||||
|
| Thema | Batch | Streaming |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `header` | kennt `read` und `written` | wird zuerst geschrieben, `read` und `written` sind `nil` |
|
||||||
|
| `ctx.read` in `map`/`filter` | Anzahl Quellzeilen | `nil` |
|
||||||
|
| mehrere `transform` im Parallelmodus | erlaubt | `ArgumentError`, dafür `chained` setzen |
|
||||||
|
| Postgres als Ziel | eine Transaktion für alles | je Zeile ein `INSERT` ohne gemeinsame Transaktion |
|
||||||
|
|
||||||
|
Postgres als Quelle liest im Streaming über einen serverseitigen Cursor (`fetch_size`, Default 1000). Abfragen und
|
||||||
|
Lookups auf derselben Verbindung bleiben währenddessen möglich.
|
||||||
|
|
||||||
|
### 2.4 Plugin-Schnittstelle
|
||||||
|
|
||||||
|
Ein Storage-Plugin ist eine Klasse `<Name>Storage < DataPipeline::StoragePlugin` mit:
|
||||||
|
|
||||||
|
| Methode | Pflicht | Verwendung |
|
||||||
|
| --- | --- | --- |
|
||||||
|
| `initialize(**opts, &block)` | ja | Verbindung/Datei vorbereiten |
|
||||||
|
| `read` → `Array<Hash<String, …>>` | als Quelle | Batch-Lesen |
|
||||||
|
| `read { \|row\| }` | als Quelle | zeilenweises Lesen |
|
||||||
|
| `write(rows)` | als Ziel | Batch-Schreiben |
|
||||||
|
| `write_row(row)` | als Ziel | zeilenweises Schreiben |
|
||||||
|
| `query(query: sql, params: [])` → `Array<Hash>` | optional | Ad-hoc-Abfragen aus Transformationen |
|
||||||
|
| `set(opts)` | Basisklasse | Optionen nachreichen, wird von `read(storage, opts)` genutzt |
|
||||||
|
| `close` / `finalize` | Basisklasse | Verbindung/Datei freigeben; `run` ruft `close` für Quelle und Ziel am Ende auf |
|
||||||
|
|
||||||
|
Alle Plugins liefern Zeilen mit **String-Schlüsseln**. Die mitgelieferten Plugins liegen im Namensraum `DataPipeline` (`DataPipeline::CsvStorage` usw.) und werden per `autoload` erst bei der ersten Nutzung geladen. Neues Plugin: Klasse `<Name>Storage < DataPipeline::StoragePlugin` in der Anwendung schreiben und mit `DataPipeline.register :name, NameStorage` (oder `'NameStorage', path: 'datei.rb'` für Laden bei Bedarf) bekannt machen – danach ist `container :name` verfügbar. Ohne `register` findet `container :name` eine Klasse `<Name>Storage` auf oberster Ebene.
|
||||||
|
|
||||||
|
### 2.5 Mitgelieferte Plugins
|
||||||
|
|
||||||
|
**DataPipeline::PostgresStorage** – Optionen `host`, `port` (Default 5432), `database`, `username`, `password`, `query`, `params`,
|
||||||
|
`table`, `typed`, `fetch_size`, `truncate`.
|
||||||
|
- `read` / `query` mit Platzhaltern `$1, $2, …` und `params: [...]`; Ergebnis als `Hash<String, String>`
|
||||||
|
(`NULL` = `nil`), mit `typed: true` als Ruby-Typen (Integer, Float, Date, …).
|
||||||
|
- `read { |row| }` liest über einen Cursor in einer Transaktion, je `fetch_size` Zeilen (Default 1000).
|
||||||
|
- `write(rows)` schreibt in `table:` (oder alte Form `write(table, rows)`), alle Zeilen in **einer Transaktion**,
|
||||||
|
Werte als Bind-Parameter, `nil` = `NULL`, Tabellen-/Spaltennamen gequotet (`schema.tabelle` erlaubt).
|
||||||
|
- `truncate: true` leert die Tabelle vor dem ersten Schreiben (bei `write` in derselben Transaktion).
|
||||||
|
|
||||||
|
```ruby
|
||||||
|
pg = container :postgres, host: 'localhost', database: 'crm', username: 'etl', table: 'kunde_import'
|
||||||
|
read pg, query: 'select * from kunde where geaendert >= $1', params: [Date.today - 7]
|
||||||
|
```
|
||||||
|
|
||||||
|
**DataPipeline::CsvStorage** – Optionen `file` (Pflicht), `csv:` (an Rubys `CSV`, gemergt mit den Defaults `headers: true`,
|
||||||
|
`col_sep: ','`), `columns`, `counter`, `verbose`.
|
||||||
|
- `columns: %w[a b c]` legt die Spaltenreihenfolge fest: fehlende Felder bleiben leer, String- und Symbol-Schlüssel
|
||||||
|
werden gefunden, die Kopfzeile kommt aus `columns`. Ohne `columns` gilt die Hash-Reihenfolge (nötig für Dateien mit
|
||||||
|
mehreren Satzarten wie CUDO).
|
||||||
|
- Lesen ohne Kopfzeile (`headers: false`): Schlüssel aus `columns`, sonst `"0"`, `"1"`, …
|
||||||
|
- Leere Daten: ohne `columns` leere Datei, mit `columns` nur die Kopfzeile.
|
||||||
|
- `layout: DataPipeline::Layout` schreibt Dateien mit mehreren Satzarten in der Feldreihenfolge je Satzart, ohne
|
||||||
|
Kopfzeile; ein Feld, das nicht im Layout steht, bricht ab (siehe 2.9).
|
||||||
|
|
||||||
|
Übersicht der übrigen Plugins (alle mit `read`, `read { }`, `write`, `write_row`, `close`):
|
||||||
|
|
||||||
|
| `container` | Klasse | Gem (bei Nutzung geladen) | Wichtige Optionen |
|
||||||
|
| --- | --- | --- | --- |
|
||||||
|
| `:json` | `DataPipeline::JsonStorage` | – | `file`, `format: :array \| :lines` (Default aus Endung: `.ndjson`/`.jsonl` → Zeilen), `root: 'data.items'`, `pretty`; ein einzelnes Objekt wird als eine Zeile gelesen |
|
||||||
|
| `:sqlite`, `:sqlite3` | `DataPipeline::SqliteStorage` | `sqlite3` | `database` (Datei oder `':memory:'`), `query`, `params` (Array für `?`, Hash für `:name`), `table`, `create`, `truncate` |
|
||||||
|
| `:excel`, `:xlsx` | `DataPipeline::ExcelStorage` | `roo` (lesen), `caxlsx` (schreiben) | `file`, `sheet` (Name/Index), `headers`, `columns`, `skip` (Titelzeilen) |
|
||||||
|
| `:rest` | `DataPipeline::RestStorage` | `http` | `url`, `auth`, `headers`, `params`, `root`, Paging `page_param` oder `next: ->(body) { … }`, `retries`, `timeout`, `batch_size`, `write_method`, `write_url` (auch als Lambda je Zeile), `ssl_verify` |
|
||||||
|
|
||||||
|
**DataPipeline::JsonStorage** – NDJSON wird zeilenweise gelesen und geschrieben. Ein JSON-Array wird beim Streaming als
|
||||||
|
`[`…`]` geschrieben und in `close` abgeschlossen, die Datei ist also auch beim Streaming gültiges JSON.
|
||||||
|
|
||||||
|
**DataPipeline::SqliteStorage** – Verhält sich wie Postgres: Bind-Parameter, Schreiben in einer Transaktion, Tabellen- und
|
||||||
|
Spaltennamen gequotet, `nil` = `NULL`, Datum als ISO-Text. Werte kommen mit SQLite-Typen zurück (Integer, Float, String).
|
||||||
|
`create: true` legt die Tabelle beim ersten Schreiben aus der ersten Zeile an (Integer → `integer`, Float → `real`,
|
||||||
|
sonst `text`), `truncate: true` leert sie vorher. Die Datei wird erst bei der ersten Nutzung geöffnet; ein Dry-Run
|
||||||
|
legt keine an.
|
||||||
|
|
||||||
|
**DataPipeline::ExcelStorage** – Lesen von xlsx zeilenweise (`each_row_streaming`), leere Zeilen werden übersprungen. Beim
|
||||||
|
Schreiben bleibt Text immer Text: `'0000000001'` behält die Nullen, `'=1+1'` wird keine Formel. Zahlen und Datumswerte
|
||||||
|
behalten ihren Typ. Die Datei entsteht in `close` (caxlsx baut sie im Speicher auf).
|
||||||
|
|
||||||
|
**DataPipeline::RestStorage** – Liest JSON, der Pfad zum Array kommt aus `root`. Paging über eine Seitennummer (bis eine Seite
|
||||||
|
leer ist) oder über eine Folge-URL aus der Antwort; `max_pages` (Default 1000) stoppt endlose Schleifen. Bei 429, 5xx
|
||||||
|
und Netzwerkfehlern wird `retries`-mal wiederholt (Wartezeit `retry_wait` × Versuch), 4xx bricht sofort mit
|
||||||
|
`DataPipeline::RestStorage::Error` ab. Schreiben je Zeile oder mit `batch_size` als JSON-Array; `write_url: ->(row) { … }` baut
|
||||||
|
je Zeile eine eigene URL (z. B. `PUT /kunden/<id>`, nicht zusammen mit `batch_size`). `query(query: 'pfad', params:)`
|
||||||
|
hängt den Pfad an `url` an und erlaubt `lookup` gegen die API. Ein Fehler beim Schreiben bricht den Lauf ab (siehe 1.4).
|
||||||
|
|
||||||
|
Für APIs im RPC-Stil (Suchen und Anlegen per POST mit Body, z. B. die pds API) gibt es einzelne Aufrufe:
|
||||||
|
`call(:post, 'person/listkunden', json: { … })` liefert das geparste JSON. Mit `retries: 0` wird ein anlegender Aufruf
|
||||||
|
bei `5xx` nicht wiederholt. Ein Fehler wirft `DataPipeline::RestStorage::Error` mit `status` (HTTP-Code, `nil` bei Netzwerkfehlern).
|
||||||
|
`ssl_verify: false` akzeptiert jedes Server-Zertifikat (eigene CA, Zugriff über IP-Adresse).
|
||||||
|
|
||||||
|
```ruby
|
||||||
|
api = container :rest, url: 'https://api.example.com/v1/kunden', auth: "Bearer #{ENV['API_TOKEN']}",
|
||||||
|
root: 'data', page_param: 'page'
|
||||||
|
xlsx = container :excel, file: 'data/kunden.xlsx', sheet: 'Kunden', columns: %w[id name land]
|
||||||
|
read api
|
||||||
|
map { |r| r.slice('id', 'name', 'land') }
|
||||||
|
write xlsx
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2.6 Abhängigkeiten
|
||||||
|
|
||||||
|
Das Gem selbst braucht nur `csv` und `logger` (Ruby `>= 3.4`). Die Gems der Plugins sind optional und gehören ins
|
||||||
|
Gemfile der Anwendung, die das Plugin nutzt:
|
||||||
|
|
||||||
|
| Plugin | Gem |
|
||||||
|
| --- | --- |
|
||||||
|
| `:postgres` | `pg` |
|
||||||
|
| `:sqlite` | `sqlite3` |
|
||||||
|
| `:excel` | `roo` (lesen), `caxlsx` (schreiben) |
|
||||||
|
| `:rest`, `DataPipeline::Slack` | `http` |
|
||||||
|
| `:csv`, `:json` | – |
|
||||||
|
|
||||||
|
Im Repository bindet das `Gemfile` das Gem über `gemspec` ein und ergänzt die Gruppe `plugins` (`pg`, `sqlite3`,
|
||||||
|
`roo`, `caxlsx`, `http`) für die Plugin-Tests und die Gruppe `test` (`minitest`, `rake`). `Gemfile.lock` ist versioniert.
|
||||||
|
Gem bauen: `gem build datapipeline.gemspec`.
|
||||||
|
|
||||||
|
Plugins und ihre Gems werden erst bei der ersten Nutzung geladen (`autoload` und `DataPipeline::StoragePlugin.require_gem`): eine
|
||||||
|
CSV- oder JSON-Pipeline lädt keines dieser Gems. Wer ein Plugin nicht braucht, lässt die Gruppe weg
|
||||||
|
(`bundle config set without plugins`); bei Nutzung meldet das Plugin dann, welches Gem fehlt.
|
||||||
|
|
||||||
|
### 2.7 Betrieb
|
||||||
|
|
||||||
|
**Fehler je Zeile** – `on_error` legt fest, was passiert, wenn ein Datenschritt (`transform`, `map`, `filter`,
|
||||||
|
`validate`) für eine Zeile eine Exception wirft. Fehler in `header`, `trailer`, prepend und append brechen immer ab.
|
||||||
|
|
||||||
|
| Modus | Wirkung |
|
||||||
|
| --- | --- |
|
||||||
|
| `:abort` (Default) | Lauf bricht ab, Exception wird weitergereicht |
|
||||||
|
| `:skip` | Zeile fällt weg, Warnung im Log, Zähler `rejected` |
|
||||||
|
| `:collect, to: storage` | wie `:skip`, zusätzlich die Eingangszeile mit `_step` und `_error` per `write_row` ins Ziel `storage` |
|
||||||
|
|
||||||
|
**Validierung** – `validate` prüft Spalten nach Regeln, nach einem Layout (2.9) und/oder mit einem Block. Eine
|
||||||
|
verletzte Regel löst `DataPipeline::ValidationError` aus und läuft damit über `on_error`. Leere Werte prüft nur
|
||||||
|
`required`. Mit `warn: true` bleibt die Zeile drin, es gibt eine Warnung und `stats.warnings` zählt mit (ins Log
|
||||||
|
kommen höchstens 100); `ignore: %w[FELD]` lässt bekannte Lücken aus.
|
||||||
|
|
||||||
|
```ruby
|
||||||
|
on_error :collect, to: container(:csv, file: 'data/fehler.csv', columns: %w[belnr konto dmbtr _step _error])
|
||||||
|
validate 'konto' => { required: true, format: /\A\d+\z/ },
|
||||||
|
'shkzg' => { in: %w[S H] },
|
||||||
|
'dmbtr' => { type: :decimal },
|
||||||
|
'name1' => { max: 35 }
|
||||||
|
validate { |row| 'Betrag 0' if row['dmbtr'].to_f.zero? }
|
||||||
|
```
|
||||||
|
|
||||||
|
**Ergebnis, Log, Hooks** – `run` liefert `DataPipeline::Stats`:
|
||||||
|
|
||||||
|
| Feld | Inhalt |
|
||||||
|
| --- | --- |
|
||||||
|
| `read` | gelesene Quellzeilen |
|
||||||
|
| `written` | Datenzeilen nach allen Schritten (ohne Kopf- und Schlusssätze) |
|
||||||
|
| `output` | an die Ziele gegebene Zeilen inkl. Kopf- und Schlusssätzen |
|
||||||
|
| `rejected`, `errors` | verworfene Zeilen; die ersten 100 mit Schritt, Meldung und Zeile |
|
||||||
|
| `warnings` | Zeilen mit Warnung aus `validate …, warn: true` |
|
||||||
|
| `duration`, `started_at`, `finished_at` | Laufzeit |
|
||||||
|
| `error` | Exception eines abgebrochenen Laufs |
|
||||||
|
| `ok?` | kein Abbruch und nichts verworfen |
|
||||||
|
| `to_s` | `read=9 written=4 rejected=4 0.097s` |
|
||||||
|
|
||||||
|
Der Logger schreibt `start` und `finish` (Info), jede verworfene Zeile (Warnung) und einen Abbruch (Fehler).
|
||||||
|
`on_finish` läuft nach jedem Lauf, auch nach einem Abbruch; ein Fehler im Hook wird nur geloggt.
|
||||||
|
|
||||||
|
```ruby
|
||||||
|
on_finish { |stats| DataPipeline::Slack.new(channel: '#etl').send_message("Buchungen: #{stats}") unless stats.ok? }
|
||||||
|
```
|
||||||
|
|
||||||
|
**Inkrementelle Läufe** – `watermark` merkt sich je Name den höchsten Wert einer Spalte in einer JSON-Datei. Die
|
||||||
|
Pipeline verfolgt jede Quellzeile und speichert den neuen Wert nur nach einem erfolgreichen Lauf (nicht nach Abbruch,
|
||||||
|
nicht im Dry-Run). Zahlen als Text werden numerisch verglichen. Mit `limit` sollte die Quelle nach der Spalte
|
||||||
|
sortiert sein; verworfene Zeilen schieben das Wasserzeichen mit (sie liegen bei `:collect` in der Fehlerdatei).
|
||||||
|
|
||||||
|
```ruby
|
||||||
|
wm = watermark :rechnungen, column: 'rechnung_id', file: 'data/.state.json', initial: 0
|
||||||
|
read pg, query: 'select * from rechnung where rechnung_id > $1 order by rechnung_id', params: [wm.value]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Dry-Run und Limit** – Im Dry-Run läuft alles, aber nichts wird geschrieben: kein Ziel, keine Fehlerdatei, kein
|
||||||
|
Wasserzeichen; eine CSV-Datei wird nicht angelegt. Am Ende stehen die Zusammenfassung und die ersten Zeilen
|
||||||
|
(`preview`, Default 10) als JSON auf `$stdout`. `limit n` verarbeitet nur die ersten n Quellzeilen, im Streaming
|
||||||
|
wird das Lesen dann beendet.
|
||||||
|
|
||||||
|
**Kommandozeile** – `bin/etl` startet ein Pipeline-Skript; `run` ist optional (`bin/etl skript.rb` genügt):
|
||||||
|
|
||||||
|
```sh
|
||||||
|
bin/etl run export.rb --from=2025-11-01 --till=2025-11-30
|
||||||
|
bin/etl run examples/kunden_import.rb --dry-run --limit=20
|
||||||
|
bin/etl run pipeline.rb --param land=DE # im Skript: params[:land]
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Wirkung |
|
||||||
|
| --- | --- |
|
||||||
|
| `--from`, `--till` | in `params[:from]`/`params[:till]` und unverändert an das Skript weitergereicht (eigener `OptionParser` der Importer) |
|
||||||
|
| `--param KEY=VALUE` | `params[:key]`, mehrfach möglich |
|
||||||
|
| `--dry-run`, `--limit=N` | für alle Pipelines des Skripts |
|
||||||
|
| `--verbose`, `--quiet` | Log-Level Debug bzw. Warnung (Default über `bin/etl`: Info) |
|
||||||
|
| jede andere `--name=wert` | `params[:name]` und unverändert an das Skript, z. B. `--out=data`, `--incremental=1` |
|
||||||
|
|
||||||
|
Exit-Code: `0` ok, `1` Abbruch, `2` fertig, aber mit verworfenen Zeilen. Das `Rakefile` ruft `bin/etl` auf:
|
||||||
|
`rake process:posting[2025-11-01,2025-11-30]`, `rake process:all[2025-11-01,2025-11-30,dry]`. Ein fehlgeschlagener Lauf stoppt Rake.
|
||||||
|
|
||||||
|
### 2.8 Tests
|
||||||
|
|
||||||
|
`rake test` (oder `bundle exec rake`) startet die Minitest-Suite in [test/](test/): rund 165 Fälle für Plugins, Kern,
|
||||||
|
Streaming, Betrieb, Kommandozeile und die Beispiele.
|
||||||
|
|
||||||
|
| Ordner | Inhalt |
|
||||||
|
| --- | --- |
|
||||||
|
| `test/plugins` | je Plugin das gemeinsame Testkit `test/support/storage_contract.rb` (Schreiben und Zurücklesen, zeilenweise, als Quelle und Ziel einer Pipeline) plus eigene Fälle; REST gegen einen lokalen Testserver; Laden der Gems erst bei Nutzung in einem frischen Prozess |
|
||||||
|
| `test/core` | DSL, Einstellungen, Plugin-Registry, Parallel- und Verkettungsmodus, Streaming, Lookups, mehrere Ziele, Betrieb; `compat_test.rb` hält die Ausgabe des `transform`-Modells von v1.2.1 fest |
|
||||||
|
| `test/cli` | `bin/etl` als Prozess: Weitergabe von `--from`/`--till`, Parameter, Dry-Run, Exit-Codes |
|
||||||
|
| `test/examples` | die Beispiele aus Abschnitt 1 (Code in der Doku = Datei, Lauf und Ergebnis; REST gegen die Demo-API) |
|
||||||
|
|
||||||
|
Die `.env` wird in Tests nicht geladen; kein Test erreicht die Produktivdatenbank. Postgres-Tests legen je Test eine
|
||||||
|
Wegwerf-Datenbank auf einem lokalen Server an (`ETL_TEST_PG_HOST`, `_PORT`, `_USER`, `_PASSWORD`; Default
|
||||||
|
`localhost`, `$USER`). Ohne erreichbaren Server werden sie übersprungen (21 Tests). Ein eigenes Plugin lässt sich mit
|
||||||
|
`MemoryStorage` aus `test/test_helper.rb` als Quelle oder Ziel testen; mit `include StorageContract` und einer
|
||||||
|
Methode `contract_storage(env, :write | :read)` bekommt es die gemeinsamen Tests.
|
||||||
|
|
||||||
|
### 2.9 Satzarten-Layouts
|
||||||
|
|
||||||
|
Dateien mit mehreren Satzarten (Kopf, Positionen, Schluss) haben je Satzart eine feste Feldliste. `DataPipeline::Layout`
|
||||||
|
beschreibt sie; die Satzart einer Zeile ist ihr erster Wert (oder `type_field:`), Schlüssel werden ohne Rücksicht auf
|
||||||
|
Gross-/Kleinschreibung zugeordnet.
|
||||||
|
|
||||||
|
```ruby
|
||||||
|
LAYOUT = DataPipeline::Layout.new do
|
||||||
|
record 'DOHD', [['SATYP', 'AN', 4, true], ['LFDNR', 'N', 10, true], ['BKTXT', 'AN', 25, true], ...]
|
||||||
|
record 'LARC', [['SATYP', 'AN', 4, true], ['SUMME', 'N', 10, true]]
|
||||||
|
end
|
||||||
|
|
||||||
|
validate LAYOUT, warn: true # Pflicht, Länge, Format je Feld
|
||||||
|
write container(:csv, file: 'x.ok', layout: LAYOUT, csv: { headers: false, col_sep: '|' })
|
||||||
|
```
|
||||||
|
|
||||||
|
| Typ | Prüfung |
|
||||||
|
| --- | --- |
|
||||||
|
| `AN` | Länge als Text |
|
||||||
|
| `N` | Zahl; Länge `10` = Stellen, `13.2` = 11 Vor- und 2 Nachkommastellen |
|
||||||
|
| `DAT` | `YYYYMMDD`, gültiges Datum |
|
||||||
|
| `TIM` | `HHMMSS` |
|
||||||
|
|
||||||
|
Beim Schreiben stehen die Werte in der Reihenfolge des Layouts, fehlende Felder bleiben leer, ein unbekanntes Feld
|
||||||
|
bricht mit `DataPipeline::LayoutError` ab. Eine falsch benannte SQL-Spalte verschiebt also nicht mehr still alle
|
||||||
|
folgenden Felder. Ein vollständiges Beispiel sind die CUDO-Satzarten der SAP-Schnittstelle im Projekt `etl` (`cudo/layout.rb`),
|
||||||
|
erzeugt aus der Schnittstellenbeschreibung.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. Features und Verbesserungen
|
||||||
|
|
||||||
|
Ziel: aus dem Exportwerkzeug ein generisches, testbares ETL-Framework machen. Reihenfolge = empfohlene Umsetzung.
|
||||||
|
|
||||||
|
### 3.1 Erweiterbarkeit
|
||||||
|
|
||||||
|
1. **Weitere Plugins:** Festbreiten-Format (feste Spaltenbreiten, nutzt die Längen aus dem Layout), SFTP-Ablage.
|
||||||
|
2. **Veröffentlichung:** eigenes Remote-Repository für das Gem, Lizenz festlegen, Versionen taggen.
|
||||||
@@ -0,0 +1,11 @@
|
|||||||
|
require 'rake/testtask'
|
||||||
|
|
||||||
|
# rake test all tests (postgres tests need a local server, see test/test_helper.rb)
|
||||||
|
# rake test TEST=test/core/chained_test.rb
|
||||||
|
Rake::TestTask.new(:test) do |t|
|
||||||
|
t.libs << 'lib' << 'test'
|
||||||
|
t.test_files = FileList['test/**/*_test.rb']
|
||||||
|
t.warning = false
|
||||||
|
end
|
||||||
|
|
||||||
|
task default: :test
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
#!/usr/bin/env ruby
|
||||||
|
#
|
||||||
|
# binstub for this repository: loads the bundle (optional plugin gems) and runs the gem executable
|
||||||
|
# exe/etl. Installed as a gem, the command is just `etl`.
|
||||||
|
#
|
||||||
|
require 'bundler/setup'
|
||||||
|
load File.expand_path('../exe/etl', __dir__)
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
require_relative 'lib/datapipeline/version'
|
||||||
|
|
||||||
|
Gem::Specification.new do |spec|
|
||||||
|
spec.name = 'datapipeline'
|
||||||
|
spec.version = DataPipeline::VERSION
|
||||||
|
spec.authors = ['Karl-Heinz Wild']
|
||||||
|
spec.email = ['kh.wild@wicom.li']
|
||||||
|
spec.summary = 'Schlankes ETL-Framework mit Ruby-DSL: lesen, prüfen, umbauen, schreiben'
|
||||||
|
spec.description = <<~TEXT
|
||||||
|
Pipelines als Ruby-DSL mit austauschbaren Storage-Plugins (PostgreSQL, SQLite, CSV, JSON/NDJSON, Excel, REST),
|
||||||
|
verketteten Schritten (map, filter, validate, header, trailer), Lookups, Streaming, Satzarten-Layouts,
|
||||||
|
Fehlerdateien, Dry-Run, inkrementellen Läufen und dem Kommando etl mit Exit-Codes.
|
||||||
|
TEXT
|
||||||
|
spec.homepage = 'https://git.wicom.li/wkh/datapipeline'
|
||||||
|
spec.license = 'Nonstandard'
|
||||||
|
spec.required_ruby_version = '>= 3.4'
|
||||||
|
|
||||||
|
spec.metadata['changelog_uri'] = "#{spec.homepage}/src/branch/main/CHANGELOG.md"
|
||||||
|
spec.metadata['rubygems_mfa_required'] = 'true'
|
||||||
|
|
||||||
|
spec.files = Dir['lib/**/*.rb', 'exe/*', 'README.md', 'CHANGELOG.md']
|
||||||
|
spec.bindir = 'exe'
|
||||||
|
spec.executables = ['etl']
|
||||||
|
spec.require_paths = ['lib']
|
||||||
|
|
||||||
|
# csv and logger are no default gems any more from ruby 3.4 / 4.0
|
||||||
|
spec.add_dependency 'csv', '~> 3.3'
|
||||||
|
spec.add_dependency 'logger', '~> 1.6'
|
||||||
|
|
||||||
|
# optional, loaded only when the plugin is used (add them to the Gemfile of the application):
|
||||||
|
# pg (postgres), sqlite3 (sqlite), roo + caxlsx (excel), http (rest, slack)
|
||||||
|
end
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
#
|
||||||
|
# Nur neue Buchungsbewegungen exportieren, Zeile für Zeile (Streaming)
|
||||||
|
# Ein zweiter Lauf ohne neue Daten schreibt nichts.
|
||||||
|
#
|
||||||
|
# bin/etl run examples/bewegungen_inkrementell.rb [--out=tmp]
|
||||||
|
#
|
||||||
|
require 'fileutils'
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
DATA = File.expand_path('data', __dir__)
|
||||||
|
out = DataPipeline.params.fetch(:out, 'tmp')
|
||||||
|
FileUtils.mkdir_p(out)
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
streamed
|
||||||
|
|
||||||
|
# höchste id des letzten erfolgreichen Laufs, gespeichert in tmp/.state.json
|
||||||
|
stand = watermark :bewegungen, column: 'id', file: "#{out}/.state.json", initial: 0
|
||||||
|
|
||||||
|
# bei einer Datenbank gehört der Filter ins SQL: where id > $1, params: [stand.value]
|
||||||
|
read container(:json, file: "#{DATA}/bewegungen.ndjson")
|
||||||
|
filter { |row| row['id'] > stand.value.to_i }
|
||||||
|
map { |row| row.merge('soll_haben' => row['betrag'].negative? ? 'H' : 'S', 'betrag' => row['betrag'].abs) }
|
||||||
|
|
||||||
|
write container(:json,
|
||||||
|
file: "#{out}/bewegungen_#{Time.now.strftime('%Y%m%d_%H%M%S')}.ndjson"
|
||||||
|
)
|
||||||
|
|
||||||
|
on_finish { |stats| puts "neu: #{stats.written}, Stand jetzt: #{stand.current}" }
|
||||||
|
end.run
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
{"id":1,"konto":"4000","betrag":120.0}
|
||||||
|
{"id":2,"konto":"4000","betrag":-20.0}
|
||||||
|
{"id":3,"konto":"4400","betrag":75.5}
|
||||||
|
{"id":4,"konto":"4000","betrag":10.0}
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
artikel,menge
|
||||||
|
A-100,5
|
||||||
|
B-200,2
|
||||||
|
@@ -0,0 +1,3 @@
|
|||||||
|
artikel,menge
|
||||||
|
A-100,7
|
||||||
|
C-300,1
|
||||||
|
@@ -0,0 +1,10 @@
|
|||||||
|
kundennr;name;land;status;umsatz
|
||||||
|
100001;Müller GmbH;DE;aktiv;12500.50
|
||||||
|
100002;Huber & Partner;AT;aktiv;8300
|
||||||
|
100003;Swiss Trade AG;CH;aktiv;22000.75
|
||||||
|
100004;Alte Firma KG;DE;inaktiv;0
|
||||||
|
100005; ;DE;aktiv;100
|
||||||
|
10006;Kurz;DE;aktiv;50
|
||||||
|
100007;Nowhere Ltd;XX;aktiv;10
|
||||||
|
100008;Dupont SARL;FR;aktiv;abc
|
||||||
|
100009;Rossi Srl;IT;aktiv;4100.10
|
||||||
|
@@ -0,0 +1,7 @@
|
|||||||
|
[
|
||||||
|
{ "code": "DE", "name": "Deutschland", "eu": true },
|
||||||
|
{ "code": "AT", "name": "Österreich", "eu": true },
|
||||||
|
{ "code": "CH", "name": "Schweiz", "eu": false },
|
||||||
|
{ "code": "FR", "name": "Frankreich", "eu": true },
|
||||||
|
{ "code": "IT", "name": "Italien", "eu": true }
|
||||||
|
]
|
||||||
@@ -0,0 +1,7 @@
|
|||||||
|
[
|
||||||
|
{ "nr": "R-1001", "datum": "2025-11-03", "kunde": "100001",
|
||||||
|
"positionen": [{ "text": "Beratung", "menge": 2, "preis": 150.0 },
|
||||||
|
{ "text": "Reisekosten", "menge": 1, "preis": 80.5 }] },
|
||||||
|
{ "nr": "R-1002", "datum": "2025-11-05", "kunde": "100002",
|
||||||
|
"positionen": [{ "text": "Jahreslizenz Software mit Wartung und Support", "menge": 1, "preis": 1200 }] }
|
||||||
|
]
|
||||||
@@ -0,0 +1,137 @@
|
|||||||
|
#
|
||||||
|
# Kleine Demo-API für die REST-Beispiele, ohne Internet und ohne weitere Gems
|
||||||
|
#
|
||||||
|
# ruby examples/demo_api.rb # http://127.0.0.1:4567, Token: demo-token
|
||||||
|
#
|
||||||
|
# Endpunkte (alle mit Authorization: Bearer demo-token, sonst 401)
|
||||||
|
# GET /api/kunden?page=1&per_page=3 {"data": [...], "page": 1} die 2. Seite antwortet einmal mit 503
|
||||||
|
# GET /api/laender [{"code": "DE", "name": "Deutschland"}, ...]
|
||||||
|
# GET /api/bestellungen[?cursor=n] {"items": [...], "next": "/api/bestellungen?cursor=3"} oder "next": null
|
||||||
|
# POST /api/import ein Objekt oder ein Array; 422 ohne kundennr/name
|
||||||
|
# GET /api/import alles, was angenommen wurde
|
||||||
|
# PUT /api/archiv/<id> ein Objekt ablegen
|
||||||
|
# GET /api/archiv alles im Archiv
|
||||||
|
#
|
||||||
|
require 'json'
|
||||||
|
require 'socket'
|
||||||
|
require 'uri'
|
||||||
|
|
||||||
|
class DemoApi
|
||||||
|
TOKEN = 'demo-token'.freeze
|
||||||
|
|
||||||
|
KUNDEN = [
|
||||||
|
{ 'id' => 1, 'name' => 'Müller GmbH', 'land' => 'DE', 'umsatz' => '12500.50' },
|
||||||
|
{ 'id' => 2, 'name' => 'Huber & Partner', 'land' => 'AT', 'umsatz' => '8300' },
|
||||||
|
{ 'id' => 3, 'name' => 'Swiss Trade AG', 'land' => 'CH', 'umsatz' => '22000.75' },
|
||||||
|
{ 'id' => 4, 'name' => 'Rossi Srl', 'land' => 'IT', 'umsatz' => '4100.10' },
|
||||||
|
{ 'id' => 5, 'name' => 'Dupont SARL', 'land' => 'FR', 'umsatz' => 'n/a' },
|
||||||
|
{ 'id' => 6, 'name' => 'Nordic AB', 'land' => 'SE', 'umsatz' => '990' },
|
||||||
|
{ 'id' => 7, 'name' => 'Vaduz Treuhand', 'land' => 'LI', 'umsatz' => '15000' }
|
||||||
|
].freeze
|
||||||
|
|
||||||
|
LAENDER = [
|
||||||
|
{ 'code' => 'DE', 'name' => 'Deutschland' }, { 'code' => 'AT', 'name' => 'Österreich' },
|
||||||
|
{ 'code' => 'CH', 'name' => 'Schweiz' }, { 'code' => 'IT', 'name' => 'Italien' },
|
||||||
|
{ 'code' => 'FR', 'name' => 'Frankreich' }, { 'code' => 'LI', 'name' => 'Liechtenstein' }
|
||||||
|
].freeze
|
||||||
|
|
||||||
|
BESTELLUNGEN = (1..5).map { |i| { 'id' => i, 'kunde' => (i % 3) + 1, 'betrag' => i * 100.0 } }.freeze
|
||||||
|
|
||||||
|
attr_reader :imported, :archive
|
||||||
|
|
||||||
|
def self.start(port: 4567) = new(port).tap(&:start)
|
||||||
|
|
||||||
|
def initialize port
|
||||||
|
@server = TCPServer.new('127.0.0.1', port)
|
||||||
|
@imported = []
|
||||||
|
@archive = {}
|
||||||
|
@failed_once = false
|
||||||
|
end
|
||||||
|
|
||||||
|
def url = "http://127.0.0.1:#{@server.addr[1]}"
|
||||||
|
|
||||||
|
def start
|
||||||
|
@thread = Thread.new { loop { Thread.new(@server.accept) { |client| serve(client) } } }
|
||||||
|
self
|
||||||
|
end
|
||||||
|
|
||||||
|
def stop
|
||||||
|
@thread&.kill
|
||||||
|
@server.close
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def serve client
|
||||||
|
method, target = client.gets.to_s.split
|
||||||
|
return unless method
|
||||||
|
|
||||||
|
headers = {}
|
||||||
|
while (line = client.gets) && line != "\r\n"
|
||||||
|
key, value = line.split(':', 2)
|
||||||
|
headers[key.downcase] = value.strip
|
||||||
|
end
|
||||||
|
body = headers['content-length'] ? client.read(headers['content-length'].to_i) : ''
|
||||||
|
path, query = target.split('?', 2)
|
||||||
|
params = URI.decode_www_form(query.to_s).to_h
|
||||||
|
|
||||||
|
status, payload = headers['authorization'] == "Bearer #{TOKEN}" ? route(method, path, params, body) : [401, { 'error' => 'token' }]
|
||||||
|
text = JSON.generate(payload)
|
||||||
|
client.write "HTTP/1.1 #{status} X\r\nContent-Type: application/json\r\nContent-Length: #{text.bytesize}\r\n" \
|
||||||
|
"Connection: close\r\n\r\n#{text}"
|
||||||
|
rescue StandardError => e
|
||||||
|
warn "DemoApi: #{e.class}: #{e.message}"
|
||||||
|
ensure
|
||||||
|
client.close
|
||||||
|
end
|
||||||
|
|
||||||
|
def route method, path, params, body
|
||||||
|
case [method, path]
|
||||||
|
in ['GET', '/api/kunden'] then kunden(params)
|
||||||
|
in ['GET', '/api/laender'] then [200, LAENDER]
|
||||||
|
in ['GET', '/api/bestellungen'] then bestellungen(params)
|
||||||
|
in ['POST', '/api/import'] then import(JSON.parse(body))
|
||||||
|
in ['GET', '/api/import'] then [200, @imported]
|
||||||
|
in ['PUT', %r{\A/api/archiv/\w+\z}] then archivieren(path.split('/').last, JSON.parse(body))
|
||||||
|
in ['GET', '/api/archiv'] then [200, @archive.values]
|
||||||
|
else [404, { 'error' => "#{method} #{path}" }]
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def kunden params
|
||||||
|
page = params.fetch('page', '1').to_i
|
||||||
|
per_page = params.fetch('per_page', '3').to_i
|
||||||
|
if page == 2 && !@failed_once
|
||||||
|
@failed_once = true
|
||||||
|
return [503, { 'error' => 'kurz nicht verfügbar' }]
|
||||||
|
end
|
||||||
|
[200, { 'data' => KUNDEN.drop((page - 1) * per_page).first(per_page), 'page' => page }]
|
||||||
|
end
|
||||||
|
|
||||||
|
def bestellungen params
|
||||||
|
cursor = params.fetch('cursor', '0').to_i
|
||||||
|
items = BESTELLUNGEN.drop(cursor).first(2)
|
||||||
|
following = cursor + 2 < BESTELLUNGEN.size ? "/api/bestellungen?cursor=#{cursor + 2}" : nil
|
||||||
|
[200, { 'items' => items, 'next' => following }]
|
||||||
|
end
|
||||||
|
|
||||||
|
def archivieren id, data
|
||||||
|
@archive[id] = data
|
||||||
|
[200, { 'ok' => true }]
|
||||||
|
end
|
||||||
|
|
||||||
|
def import data
|
||||||
|
rows = data.is_a?(Array) ? data : [data]
|
||||||
|
invalid = rows.reject { |r| r['kundennr'] && r['name'] }
|
||||||
|
return [422, { 'error' => 'kundennr und name sind Pflicht', 'zeilen' => invalid }] if invalid.any?
|
||||||
|
|
||||||
|
@imported.concat(rows)
|
||||||
|
[201, { 'angenommen' => rows.size }]
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
if $PROGRAM_NAME == __FILE__
|
||||||
|
api = DemoApi.start(port: ENV.fetch('PORT', 4567).to_i)
|
||||||
|
puts "Demo-API auf #{api.url}, Token #{DemoApi::TOKEN} (Ctrl-C beendet)"
|
||||||
|
sleep
|
||||||
|
end
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
#
|
||||||
|
# Eigenes Plugin: liest alle CSV-Dateien eines Verzeichnisses als eine Quelle
|
||||||
|
#
|
||||||
|
# bin/etl run examples/eigenes_plugin.rb [--out=tmp]
|
||||||
|
#
|
||||||
|
require 'fileutils'
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
DATA = File.expand_path('data', __dir__)
|
||||||
|
out = DataPipeline.params.fetch(:out, 'tmp')
|
||||||
|
FileUtils.mkdir_p(out)
|
||||||
|
|
||||||
|
class VerzeichnisStorage < DataPipeline::StoragePlugin
|
||||||
|
def initialize dir:, pattern: '*.csv', **csv_options
|
||||||
|
@dir = dir
|
||||||
|
@pattern = pattern
|
||||||
|
@csv_options = csv_options
|
||||||
|
end
|
||||||
|
|
||||||
|
# jede Zeile bekommt den Dateinamen dazu; mit Block zeilenweise (für streamed)
|
||||||
|
def read(&block)
|
||||||
|
rows = Dir.glob(File.join(@dir, @pattern)).sort.flat_map do |file|
|
||||||
|
DataPipeline::CsvStorage.new(file:, **@csv_options).read.map { |row| row.merge('datei' => File.basename(file, '.*')) }
|
||||||
|
end
|
||||||
|
return rows unless block
|
||||||
|
|
||||||
|
rows.each(&block)
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
DataPipeline.register :verzeichnis, VerzeichnisStorage
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
read container(:verzeichnis, dir: "#{DATA}/filialen")
|
||||||
|
map { |row| row.merge('menge' => row['menge'].to_i) }
|
||||||
|
write container(:csv, file: "#{out}/alle_filialen.csv", columns: %w[datei artikel menge])
|
||||||
|
end.run
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
#
|
||||||
|
# Kundenliste aus CSV prüfen, mit Ländern anreichern, nach SQLite und Excel schreiben
|
||||||
|
#
|
||||||
|
# bin/etl run examples/kunden_import.rb [--out=tmp] [--dry-run]
|
||||||
|
#
|
||||||
|
require 'fileutils'
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
DATA = File.expand_path('data', __dir__)
|
||||||
|
out = DataPipeline.params.fetch(:out, 'tmp')
|
||||||
|
FileUtils.mkdir_p(out)
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
version 'kunden_import 1.0'
|
||||||
|
|
||||||
|
# Nachschlagetabelle, einmal geladen: { 'DE' => { 'code' => 'DE', 'name' => 'Deutschland', 'eu' => true }, ... }
|
||||||
|
laender = lookup container(:json, file: "#{DATA}/laender.json"), key: 'code', unique: true
|
||||||
|
|
||||||
|
read container(:csv, file: "#{DATA}/kunden.csv", csv: { col_sep: ';' })
|
||||||
|
|
||||||
|
# fehlerhafte Zeilen mit Grund in eine eigene Datei, der Lauf geht weiter
|
||||||
|
on_error :collect, to: container(:csv, file: "#{out}/kunden_fehler.csv")
|
||||||
|
|
||||||
|
# 1. prüfen
|
||||||
|
validate 'kundennr' => { required: true, format: /\A\d{6}\z/ },
|
||||||
|
'name' => { required: true, max: 40 },
|
||||||
|
'umsatz' => { type: :decimal }
|
||||||
|
validate { |row| "Land #{row['land']} unbekannt" unless laender.key?(row['land']) }
|
||||||
|
|
||||||
|
# 2. nur aktive Kunden
|
||||||
|
filter { |row| row['status'] == 'aktiv' }
|
||||||
|
|
||||||
|
# 3. anreichern und umbauen
|
||||||
|
map do |row|
|
||||||
|
land = laender.fetch(row['land'])
|
||||||
|
{
|
||||||
|
'kundennr' => row['kundennr'],
|
||||||
|
'name' => row['name'].strip,
|
||||||
|
'land' => land['name'],
|
||||||
|
'eu' => land['eu'] ? 'ja' : 'nein',
|
||||||
|
'umsatz' => row['umsatz'].to_f.round(2)
|
||||||
|
}
|
||||||
|
end
|
||||||
|
|
||||||
|
# 4. schreiben: alle Kunden nach SQLite, EU-Kunden zusätzlich als Excel-Liste
|
||||||
|
write container(:sqlite, database: "#{out}/kunden.sqlite3", table: 'kunden', create: true, truncate: true)
|
||||||
|
write container(:excel, file: "#{out}/eu_kunden.xlsx", sheet: 'EU-Kunden'), only: ->(row) { row['eu'] == 'ja' }
|
||||||
|
|
||||||
|
on_finish { |stats| puts "kunden_import: #{stats}" }
|
||||||
|
end.run
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
#
|
||||||
|
# Rechnungen aus verschachteltem JSON in eine Datei mit Satzarten (Datei-, Kopf-, Positions-, Schlusssatz)
|
||||||
|
#
|
||||||
|
# bin/etl run examples/rechnungen_satzarten.rb [--out=tmp]
|
||||||
|
#
|
||||||
|
require 'fileutils'
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
DATA = File.expand_path('data', __dir__)
|
||||||
|
out = DataPipeline.params.fetch(:out, 'tmp')
|
||||||
|
FileUtils.mkdir_p(out)
|
||||||
|
|
||||||
|
# Feld: [Name, Typ, Länge, Pflicht]; Typ AN Text, N Zahl (11.2 = 9 Vor-, 2 Nachkommastellen), DAT YYYYMMDD
|
||||||
|
LAYOUT = DataPipeline::Layout.new do
|
||||||
|
record 'DATEI', [['SATYP', 'AN', 5, true], ['ERSTELLT', 'DAT', 8, true], ['ANZAHL', 'N', 6, true]]
|
||||||
|
record 'KOPF', [['SATYP', 'AN', 5, true], ['LFDNR', 'N', 6, true], ['BELNR', 'AN', 10, true],
|
||||||
|
['DATUM', 'DAT', 8, true], ['KUNNR', 'AN', 10, true], ['SUMME', 'N', '11.2', true]]
|
||||||
|
record 'POS', [['SATYP', 'AN', 5, true], ['LFDNR', 'N', 6, true], ['BELNR', 'AN', 10, true],
|
||||||
|
['POSNR', 'N', 3, true], ['TEXT', 'AN', 30], ['BETRAG', 'N', '11.2', true]]
|
||||||
|
record 'ENDE', [['SATYP', 'AN', 5, true], ['ANZAHL', 'N', 6, true]]
|
||||||
|
end
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
read container(:json, file: "#{DATA}/rechnungen.json")
|
||||||
|
|
||||||
|
# 1. eine Rechnung -> Kopfsatz + n Positionssätze
|
||||||
|
map do |rechnung|
|
||||||
|
positionen = rechnung['positionen'].each_with_index.map do |pos, i|
|
||||||
|
{ 'satyp' => 'POS', 'belnr' => rechnung['nr'], 'posnr' => format('%03d', i + 1),
|
||||||
|
'text' => pos['text'], 'betrag' => format('%.2f', pos['menge'] * pos['preis']) }
|
||||||
|
end
|
||||||
|
kopf = { 'satyp' => 'KOPF', 'belnr' => rechnung['nr'], 'datum' => rechnung['datum'].delete('-'),
|
||||||
|
'kunnr' => rechnung['kunde'], 'summe' => format('%.2f', positionen.sum { |p| p['betrag'].to_f }) }
|
||||||
|
[kopf, *positionen]
|
||||||
|
end
|
||||||
|
|
||||||
|
# 2. laufende Nummer über alle Kopf- und Positionssätze
|
||||||
|
map { |row, ctx| row.merge('lfdnr' => format('%06d', ctx.index + 1)) }
|
||||||
|
|
||||||
|
# 3. gegen das Layout prüfen: zu langer Text wird gemeldet, die Zeile bleibt
|
||||||
|
validate LAYOUT, warn: true
|
||||||
|
|
||||||
|
# Datei- und Schlusssatz kennen die Anzahl der Datensätze
|
||||||
|
header { |ctx| { 'satyp' => 'DATEI', 'erstellt' => Date.today.strftime('%Y%m%d'), 'anzahl' => format('%06d', ctx.written) } }
|
||||||
|
trailer { |ctx| { 'satyp' => 'ENDE', 'anzahl' => format('%06d', ctx.written) } }
|
||||||
|
|
||||||
|
# Felder in der Reihenfolge des Layouts; Positionen zusätzlich als NDJSON
|
||||||
|
write container(:csv, file: "#{out}/rechnungen.txt", layout: LAYOUT, csv: { headers: false, col_sep: '|' })
|
||||||
|
write container(:json, file: "#{out}/positionen.ndjson"), only: ->(row) { row['satyp'] == 'POS' }
|
||||||
|
end.run
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
#
|
||||||
|
# Kunden seitenweise aus einer REST-API lesen, mit Ländern aus einem zweiten Endpunkt anreichern,
|
||||||
|
# nach CSV und JSON schreiben
|
||||||
|
#
|
||||||
|
# ruby examples/demo_api.rb & # Demo-API starten
|
||||||
|
# bin/etl run examples/rest_lesen.rb [--url=http://127.0.0.1:4567] [--out=tmp]
|
||||||
|
#
|
||||||
|
require 'fileutils'
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
API = DataPipeline.params.fetch(:url, 'http://127.0.0.1:4567')
|
||||||
|
AUTH = "Bearer #{ENV.fetch('DEMO_API_TOKEN', 'demo-token')}"
|
||||||
|
out = DataPipeline.params.fetch(:out, 'tmp')
|
||||||
|
FileUtils.mkdir_p(out)
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
version 'rest_lesen 1.0'
|
||||||
|
|
||||||
|
# GET /api/kunden?page=1&per_page=3, dann page=2, ... bis eine Seite leer ist
|
||||||
|
# die Zeilen stehen in der Antwort unter "data"; 429/5xx und Netzfehler werden bis zu 3-mal wiederholt
|
||||||
|
kunden = container :rest, url: "#{API}/api/kunden", auth: AUTH,
|
||||||
|
root: 'data', page_param: 'page', params: { 'per_page' => 3 },
|
||||||
|
retries: 3, retry_wait: 0.2
|
||||||
|
|
||||||
|
# derselbe Server, anderer Endpunkt: query hängt den Pfad an die url an (GET /api/laender)
|
||||||
|
stamm = container :rest, url: "#{API}/api", auth: AUTH
|
||||||
|
laender = lookup stamm, query: 'laender', key: 'code', unique: true
|
||||||
|
|
||||||
|
read kunden
|
||||||
|
|
||||||
|
on_error :collect, to: container(:csv, file: "#{out}/api_fehler.csv")
|
||||||
|
validate 'id' => { required: true }, 'umsatz' => { type: :decimal }
|
||||||
|
|
||||||
|
map do |kunde|
|
||||||
|
{
|
||||||
|
'id' => kunde['id'],
|
||||||
|
'name' => kunde['name'],
|
||||||
|
'land' => laender.dig(kunde['land'], 'name') || kunde['land'],
|
||||||
|
'umsatz' => kunde['umsatz'].to_f
|
||||||
|
}
|
||||||
|
end
|
||||||
|
|
||||||
|
write container(:csv, file: "#{out}/api_kunden.csv", columns: %w[id name land umsatz])
|
||||||
|
write container(:json, file: "#{out}/api_kunden.json", pretty: true)
|
||||||
|
|
||||||
|
on_finish { |stats| puts "rest_lesen: #{stats}" }
|
||||||
|
end.run
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
#
|
||||||
|
# Kunden aus CSV prüfen und in Paketen per POST an eine REST-API senden
|
||||||
|
#
|
||||||
|
# ruby examples/demo_api.rb &
|
||||||
|
# bin/etl run examples/rest_schreiben.rb [--url=http://127.0.0.1:4567] [--dry-run]
|
||||||
|
#
|
||||||
|
require 'fileutils'
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
API = DataPipeline.params.fetch(:url, 'http://127.0.0.1:4567')
|
||||||
|
AUTH = "Bearer #{ENV.fetch('DEMO_API_TOKEN', 'demo-token')}"
|
||||||
|
DATA = File.expand_path('data', __dir__)
|
||||||
|
out = DataPipeline.params.fetch(:out, 'tmp')
|
||||||
|
FileUtils.mkdir_p(out)
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
version 'rest_schreiben 1.0'
|
||||||
|
|
||||||
|
read container(:csv, file: "#{DATA}/kunden.csv", csv: { col_sep: ';' })
|
||||||
|
|
||||||
|
# vor dem Senden prüfen: was die API ablehnen würde, landet hier in der Fehlerdatei
|
||||||
|
on_error :collect, to: container(:csv, file: "#{out}/import_fehler.csv")
|
||||||
|
validate 'kundennr' => { required: true, format: /\A\d{6}\z/ }, 'name' => { required: true }
|
||||||
|
|
||||||
|
filter { |row| row['status'] == 'aktiv' }
|
||||||
|
map do |row|
|
||||||
|
{ 'kundennr' => row['kundennr'], 'name' => row['name'].strip, 'land' => row['land'], 'umsatz' => row['umsatz'].to_f }
|
||||||
|
end
|
||||||
|
|
||||||
|
# POST /api/import mit je 2 Kunden als JSON-Array; im Dry-Run wird nichts gesendet
|
||||||
|
write container(:rest, url: "#{API}/api/import", auth: AUTH, batch_size: 2)
|
||||||
|
|
||||||
|
on_finish do |stats|
|
||||||
|
puts stats.dry_run ? 'Dry-Run: nichts gesendet' : "gesendet: #{stats.written} Kunden in #{(stats.written / 2.0).ceil} Paketen"
|
||||||
|
end
|
||||||
|
end.run
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
#
|
||||||
|
# Von API zu API: neue Bestellungen über Folge-Links lesen und einzeln per PUT archivieren
|
||||||
|
# Ein zweiter Lauf ohne neue Bestellungen sendet nichts.
|
||||||
|
#
|
||||||
|
# ruby examples/demo_api.rb &
|
||||||
|
# bin/etl run examples/rest_zu_rest.rb [--url=http://127.0.0.1:4567] [--out=tmp]
|
||||||
|
#
|
||||||
|
require 'fileutils'
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
API = DataPipeline.params.fetch(:url, 'http://127.0.0.1:4567')
|
||||||
|
AUTH = "Bearer #{ENV.fetch('DEMO_API_TOKEN', 'demo-token')}"
|
||||||
|
out = DataPipeline.params.fetch(:out, 'tmp')
|
||||||
|
FileUtils.mkdir_p(out)
|
||||||
|
|
||||||
|
DataPipeline.new do
|
||||||
|
streamed
|
||||||
|
|
||||||
|
# höchste Bestellnummer des letzten erfolgreichen Laufs
|
||||||
|
stand = watermark :bestellungen, column: 'id', file: "#{out}/.state.json", initial: 0
|
||||||
|
|
||||||
|
# die Antwort nennt die nächste Seite selbst: {"items": [...], "next": "/api/bestellungen?cursor=2"}
|
||||||
|
read container(:rest, url: "#{API}/api/bestellungen", auth: AUTH, root: 'items',
|
||||||
|
next: ->(body) { body['next'] && "#{API}#{body['next']}" })
|
||||||
|
|
||||||
|
filter { |row| row['id'] > stand.value.to_i }
|
||||||
|
map { |row| row.merge('archiviert' => Date.today.iso8601) }
|
||||||
|
|
||||||
|
# eine url je Bestellung: PUT /api/archiv/<id>
|
||||||
|
write container(:rest, url: "#{API}/api/archiv", auth: AUTH, write_method: :put,
|
||||||
|
write_url: ->(row) { "#{API}/api/archiv/#{row['id']}" })
|
||||||
|
|
||||||
|
on_finish { |stats| puts "archiviert: #{stats.written}, Stand jetzt: #{stand.current}" }
|
||||||
|
end.run
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
#!/usr/bin/env ruby
|
||||||
|
#
|
||||||
|
# etl [run] <pipeline.rb> [options] ('run' is optional)
|
||||||
|
#
|
||||||
|
# executable of the gem datapipeline; in this repository bin/etl loads the bundle first
|
||||||
|
#
|
||||||
|
# --from=DATE --till=DATE in DataPipeline.params, also passed on to the pipeline script
|
||||||
|
# --param key=value in DataPipeline.params[:key], repeatable
|
||||||
|
# --dry-run run everything, write nothing, print the first rows
|
||||||
|
# --limit=N only the first N source rows
|
||||||
|
# --verbose log level debug, --quiet only warnings (default info)
|
||||||
|
# --name=value any other option: DataPipeline.params[:name] and passed on to the script
|
||||||
|
#
|
||||||
|
# exit code: 0 ok, 1 aborted, 2 finished with rejected rows
|
||||||
|
#
|
||||||
|
require 'optparse'
|
||||||
|
require 'logger'
|
||||||
|
require_relative '../lib/datapipeline'
|
||||||
|
|
||||||
|
options = { settings: { log_level: Logger::INFO }, passthrough: [] }
|
||||||
|
DataPipeline.params = {}
|
||||||
|
parser = OptionParser.new do |opts|
|
||||||
|
opts.banner = "Usage: etl [run] PIPELINE.rb [options]\n\n etl examples/kunden_import.rb --dry-run\n"
|
||||||
|
opts.on('--from=DATE', String) { |v| DataPipeline.params[:from] = v; options[:passthrough] << "--from=#{v}" }
|
||||||
|
opts.on('--till=DATE', String) { |v| DataPipeline.params[:till] = v; options[:passthrough] << "--till=#{v}" }
|
||||||
|
opts.on('--param KEY=VALUE', String) do |v|
|
||||||
|
key, value = v.split('=', 2)
|
||||||
|
DataPipeline.params[key.to_sym] = value
|
||||||
|
end
|
||||||
|
opts.on('--dry-run') { options[:settings][:dry_run] = true }
|
||||||
|
opts.on('--limit=N', Integer) { |v| options[:settings][:limit] = v }
|
||||||
|
opts.on('--verbose') { options[:settings][:log_level] = Logger::DEBUG }
|
||||||
|
opts.on('--quiet') { options[:settings][:log_level] = Logger::WARN }
|
||||||
|
end
|
||||||
|
|
||||||
|
# own options for the parser, everything else (also --name=value of the script) goes to the script
|
||||||
|
own = []
|
||||||
|
args = []
|
||||||
|
argv = ARGV.dup
|
||||||
|
while (arg = argv.shift)
|
||||||
|
if arg.match?(/\A--(from|till|param|dry-run|limit|verbose|quiet)(=|\z)/)
|
||||||
|
own << arg
|
||||||
|
own << argv.shift if %w[--param --limit --from --till].include?(arg)
|
||||||
|
else
|
||||||
|
args << arg
|
||||||
|
end
|
||||||
|
end
|
||||||
|
parser.parse(own)
|
||||||
|
args.shift if args.first == 'run'
|
||||||
|
script = args.shift
|
||||||
|
DataPipeline.params.merge!(DataPipeline.parse_params(args)) { |_key, own_value, _| own_value }
|
||||||
|
abort parser.help if script.nil?
|
||||||
|
abort "etl: #{script} not found\n\n#{parser.help}" unless File.exist?(script)
|
||||||
|
|
||||||
|
DataPipeline.settings(options[:settings])
|
||||||
|
|
||||||
|
# the pipeline script sees only its own arguments
|
||||||
|
ARGV.replace(options[:passthrough] + args)
|
||||||
|
|
||||||
|
begin
|
||||||
|
load File.expand_path(script)
|
||||||
|
rescue StandardError => e
|
||||||
|
warn "#{File.basename(script)}: #{e.class}: #{e.message}"
|
||||||
|
exit 1
|
||||||
|
end
|
||||||
|
|
||||||
|
runs = DataPipeline.runs
|
||||||
|
exit 1 if runs.any?(&:error)
|
||||||
|
exit 2 if runs.any? { |stats| stats.rejected.positive? }
|
||||||
|
exit 0
|
||||||
@@ -0,0 +1,553 @@
|
|||||||
|
require 'json'
|
||||||
|
require 'logger'
|
||||||
|
|
||||||
|
require_relative 'datapipeline/version'
|
||||||
|
require_relative 'datapipeline/storage/storage_plugin'
|
||||||
|
require_relative 'datapipeline/validation'
|
||||||
|
require_relative 'datapipeline/layout'
|
||||||
|
require_relative 'datapipeline/watermark'
|
||||||
|
|
||||||
|
class DataPipeline
|
||||||
|
# plugins are loaded on first use, their gems too: pg (postgres), sqlite3 (sqlite), roo/caxlsx (excel),
|
||||||
|
# http (rest, slack)
|
||||||
|
autoload :CsvStorage, File.expand_path('datapipeline/storage/csv', __dir__)
|
||||||
|
autoload :PostgresStorage, File.expand_path('datapipeline/storage/postgres', __dir__)
|
||||||
|
autoload :JsonStorage, File.expand_path('datapipeline/storage/json', __dir__)
|
||||||
|
autoload :SqliteStorage, File.expand_path('datapipeline/storage/sqlite', __dir__)
|
||||||
|
autoload :ExcelStorage, File.expand_path('datapipeline/storage/excel', __dir__)
|
||||||
|
autoload :RestStorage, File.expand_path('datapipeline/storage/rest', __dir__)
|
||||||
|
autoload :Slack, File.expand_path('datapipeline/notification/slack', __dir__)
|
||||||
|
|
||||||
|
# passed to header, trailer, map and filter
|
||||||
|
# read: number of source rows (nil while streaming, until the trailer)
|
||||||
|
# written: number of body rows after all steps (nil while the body is processed)
|
||||||
|
# index: index of the current row within the current step
|
||||||
|
Context = Struct.new(:read, :written, :index, keyword_init: true)
|
||||||
|
|
||||||
|
# result of a run, returned by run and passed to on_finish
|
||||||
|
# read: source rows, written: body rows, output: rows to the target incl. header/trailer,
|
||||||
|
# rejected: rows dropped by on_error :skip/:collect, errors: first 100 of them,
|
||||||
|
# warnings: rows that failed a validate step with warn: true (kept), error: exception of an aborted run
|
||||||
|
Stats = Struct.new(:version, :read, :written, :output, :rejected, :errors, :warnings, :started_at, :finished_at,
|
||||||
|
:dry_run, :error, keyword_init: true) do
|
||||||
|
def duration = finished_at && started_at ? (finished_at - started_at).round(3) : nil
|
||||||
|
def ok? = error.nil? && rejected.zero?
|
||||||
|
|
||||||
|
def to_s
|
||||||
|
"read=#{read} written=#{written} rejected=#{rejected}#{warnings.to_i.positive? ? " warnings=#{warnings}" : ''} " \
|
||||||
|
"#{duration}s#{dry_run ? ' (dry run)' : ''}" \
|
||||||
|
"#{error ? " ABORTED: #{error.class}: #{error.message}" : ''}"
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# stops reading at the limit, not an error
|
||||||
|
class LimitReached < StandardError; end
|
||||||
|
|
||||||
|
EDGE_BEFORE = %i[prepend header].freeze
|
||||||
|
EDGE_AFTER = %i[append trailer].freeze
|
||||||
|
CHAINING = %i[map filter validate].freeze
|
||||||
|
ON_ERROR = %i[abort skip collect].freeze
|
||||||
|
MAX_ERRORS = 100
|
||||||
|
|
||||||
|
DEFAULT_SETTINGS = {
|
||||||
|
version: nil,
|
||||||
|
streamed: false,
|
||||||
|
chained: false,
|
||||||
|
verbose: false,
|
||||||
|
dry_run: false,
|
||||||
|
limit: nil,
|
||||||
|
preview: 10,
|
||||||
|
on_error: :abort,
|
||||||
|
log_level: Logger::WARN
|
||||||
|
}.freeze
|
||||||
|
|
||||||
|
SETTINGS_LOCK = Mutex.new
|
||||||
|
@defaults = DEFAULT_SETTINGS
|
||||||
|
@plugins = {}
|
||||||
|
|
||||||
|
#
|
||||||
|
# defaults for every following DataPipeline.new, e.g. from bin/etl
|
||||||
|
# the hash is replaced, never changed: a running pipeline keeps its own copy
|
||||||
|
#
|
||||||
|
def self.settings(options = {})
|
||||||
|
SETTINGS_LOCK.synchronize { @defaults = @defaults.merge(options).freeze } if options.any?
|
||||||
|
@defaults
|
||||||
|
end
|
||||||
|
|
||||||
|
def self.reset_settings!
|
||||||
|
SETTINGS_LOCK.synchronize { @defaults = DEFAULT_SETTINGS }
|
||||||
|
end
|
||||||
|
|
||||||
|
#
|
||||||
|
# container :name finds a registered plugin first, then <Name>Storage by convention
|
||||||
|
# (DataPipeline::<Name>Storage of the gem, else a top level <Name>Storage of the application)
|
||||||
|
# DataPipeline.register :mem, MemStorage
|
||||||
|
# DataPipeline.register :sftp, 'SftpStorage', path: '/path/to/sftp_storage' # loaded on first use
|
||||||
|
#
|
||||||
|
def self.register type, klass, path: nil
|
||||||
|
Object.autoload(klass.to_sym, File.expand_path(path)) if path && klass.is_a?(String)
|
||||||
|
SETTINGS_LOCK.synchronize { @plugins = @plugins.merge(type.to_sym => klass).freeze }
|
||||||
|
end
|
||||||
|
|
||||||
|
def self.plugin type
|
||||||
|
klass = @plugins[type.to_sym]
|
||||||
|
klass = Object.const_get(klass) if klass.is_a?(String)
|
||||||
|
return klass if klass
|
||||||
|
|
||||||
|
class_name = type.to_s.capitalize + 'Storage'
|
||||||
|
return const_get(class_name, false) if const_defined?(class_name, false)
|
||||||
|
|
||||||
|
Object.const_get(class_name) if Object.const_defined?(class_name)
|
||||||
|
end
|
||||||
|
|
||||||
|
#
|
||||||
|
# parameters from the command line: bin/etl sets them; a script started with ruby directly
|
||||||
|
# reads them from ARGV on first use: --name=value -> params[:name], --param key=value -> params[:key]
|
||||||
|
#
|
||||||
|
def self.params
|
||||||
|
@params ||= parse_params(ARGV)
|
||||||
|
end
|
||||||
|
|
||||||
|
def self.params= hash
|
||||||
|
@params = hash
|
||||||
|
end
|
||||||
|
|
||||||
|
def self.parse_params args
|
||||||
|
params = {}
|
||||||
|
args.each_with_index do |arg, i|
|
||||||
|
case arg
|
||||||
|
when /\A--param=(\w+)=(.*)\z/ then params[$1.to_sym] = $2
|
||||||
|
when '--param' then key, value = args[i + 1].to_s.split('=', 2); params[key.to_sym] = value if value
|
||||||
|
when /\A--([a-z][\w-]*)=(.*)\z/ then params[$1.tr('-', '_').to_sym] = $2
|
||||||
|
end
|
||||||
|
end
|
||||||
|
params
|
||||||
|
end
|
||||||
|
|
||||||
|
# stats of every run in this process, used by bin/etl for the exit code
|
||||||
|
def self.runs
|
||||||
|
@runs ||= []
|
||||||
|
end
|
||||||
|
|
||||||
|
# DataPipeline.new(verbose: true) { ... } overrides the defaults for this pipeline only
|
||||||
|
def initialize(**settings, &block)
|
||||||
|
@settings = DataPipeline.settings.merge(settings)
|
||||||
|
@transformations = []
|
||||||
|
@targets = []
|
||||||
|
@on_finish = []
|
||||||
|
@logger = Logger.new($stderr, level: @settings[:log_level], progname: 'etl')
|
||||||
|
@logger.formatter = proc { |severity, time, _prog, msg| "#{time.strftime('%F %T')} #{severity} #{msg}\n" }
|
||||||
|
instance_eval(&block)
|
||||||
|
end
|
||||||
|
|
||||||
|
def params
|
||||||
|
self.class.params
|
||||||
|
end
|
||||||
|
|
||||||
|
def configure(options = {})
|
||||||
|
@settings.merge!(options)
|
||||||
|
end
|
||||||
|
|
||||||
|
def version value
|
||||||
|
@settings[:version] = value
|
||||||
|
end
|
||||||
|
|
||||||
|
def verbose flag = true
|
||||||
|
@settings[:verbose] = flag
|
||||||
|
end
|
||||||
|
|
||||||
|
def streamed flag = true
|
||||||
|
@settings[:streamed] = flag
|
||||||
|
end
|
||||||
|
|
||||||
|
# each body step gets the output of the previous step instead of the source rows
|
||||||
|
def chained flag = true
|
||||||
|
@settings[:chained] = flag
|
||||||
|
end
|
||||||
|
|
||||||
|
# run everything, write nothing (target, rejects, watermark); prints the first rows
|
||||||
|
def dry_run flag = true
|
||||||
|
@settings[:dry_run] = flag
|
||||||
|
end
|
||||||
|
|
||||||
|
# for steps with side effects (api calls): true in a dry run
|
||||||
|
def dry_run?
|
||||||
|
@settings[:dry_run] ? true : false
|
||||||
|
end
|
||||||
|
|
||||||
|
# process only the first n source rows
|
||||||
|
def limit count
|
||||||
|
@settings[:limit] = count
|
||||||
|
end
|
||||||
|
|
||||||
|
def logger logger = nil
|
||||||
|
@logger = logger if logger
|
||||||
|
@logger
|
||||||
|
end
|
||||||
|
|
||||||
|
#
|
||||||
|
# what happens when a body step raises (or a validate step fails) for a row
|
||||||
|
# on_error :abort -> stop the run (default)
|
||||||
|
# on_error :skip -> drop the row, log a warning, count it
|
||||||
|
# on_error :collect, to: storage -> like skip, plus the row with _step and _error into storage
|
||||||
|
#
|
||||||
|
def on_error mode, to: nil
|
||||||
|
raise ArgumentError, "on_error: #{mode} unknown, use #{ON_ERROR.join(', ')}" unless ON_ERROR.include?(mode)
|
||||||
|
raise ArgumentError, 'on_error :collect needs to: storage' if mode == :collect && to.nil?
|
||||||
|
|
||||||
|
@settings[:on_error] = mode
|
||||||
|
@reject_target = to
|
||||||
|
end
|
||||||
|
|
||||||
|
# on_finish { |stats| ... } -> after every run, also after an abort (stats.error)
|
||||||
|
def on_finish(&block)
|
||||||
|
@on_finish << block
|
||||||
|
end
|
||||||
|
|
||||||
|
# see DataPipeline::Watermark
|
||||||
|
def watermark name, column:, file:, initial: nil
|
||||||
|
@watermark = Watermark.new(name, column:, file:, initial:)
|
||||||
|
end
|
||||||
|
|
||||||
|
# only a missing plugin is reported as unsupported, errors while loading it
|
||||||
|
# (missing gem) or from the plugin itself (connection, options) are passed through
|
||||||
|
def container type, **options, &block
|
||||||
|
klass = DataPipeline.plugin(type)
|
||||||
|
raise "Unsupported storage type or class missing: #{type.to_s.capitalize}Storage" unless klass
|
||||||
|
|
||||||
|
klass.new(**options, &block)
|
||||||
|
end
|
||||||
|
|
||||||
|
#
|
||||||
|
# transform(options) { |row, index| ... }
|
||||||
|
# processing: :prepend | :normal (default) | :append
|
||||||
|
# condition: ruby expression as string (row is visible) or lambda { |row| ... }
|
||||||
|
#
|
||||||
|
def transform(options = {}, &block)
|
||||||
|
kind = options.fetch(:processing, :normal)
|
||||||
|
kind = :normal unless %i[prepend append].include?(kind)
|
||||||
|
add_step kind, options, &block
|
||||||
|
end
|
||||||
|
|
||||||
|
# map { |row, ctx| ... } -> hash, array of hashes or nil; turns chaining on
|
||||||
|
def map(options = {}, &block)
|
||||||
|
add_step :map, options, &block
|
||||||
|
end
|
||||||
|
|
||||||
|
# filter { |row, ctx| ... } -> keeps the row when the block is truthy; turns chaining on
|
||||||
|
def filter(options = {}, &block)
|
||||||
|
add_step :filter, options, &block
|
||||||
|
end
|
||||||
|
|
||||||
|
# header { |ctx| ... } -> rows in front of the body, called after the body with full counters
|
||||||
|
def header(&block)
|
||||||
|
add_step :header, {}, &block
|
||||||
|
end
|
||||||
|
|
||||||
|
# trailer { |ctx| ... } -> rows after the body
|
||||||
|
def trailer(&block)
|
||||||
|
add_step :trailer, {}, &block
|
||||||
|
end
|
||||||
|
|
||||||
|
#
|
||||||
|
# validate 'kostl' => { required: true, max: 10 }, 'shkzg' => { in: %w[S H] }
|
||||||
|
# validate layout -> fields of a DataPipeline::Layout
|
||||||
|
# validate { |row| 'soll ungleich haben' if ... } -> nil, message or array of messages
|
||||||
|
#
|
||||||
|
# a failing row raises ValidationError, handled by on_error; turns chaining on
|
||||||
|
# warn: true -> only a warning in the log and stats.warnings, the row is kept
|
||||||
|
# ignore: %w[A B] -> messages for these fields are left out (known gaps)
|
||||||
|
#
|
||||||
|
def validate(rules = nil, **options, &block)
|
||||||
|
warn = options.delete(:warn) || false
|
||||||
|
ignore = Array(options.delete(:ignore)).map(&:to_s)
|
||||||
|
rules ||= options # validate 'konto' => { ... } arrives as keywords
|
||||||
|
@transformations << { kind: :validate, block:, options: { rules:, warn:, ignore: } }
|
||||||
|
end
|
||||||
|
|
||||||
|
def read storage, options = {}
|
||||||
|
@source = storage
|
||||||
|
@source.set options if options.any?
|
||||||
|
end
|
||||||
|
|
||||||
|
#
|
||||||
|
# write storage -> all rows
|
||||||
|
# write storage, only: ->(row) { ... } -> only matching rows (also header/trailer rows)
|
||||||
|
#
|
||||||
|
# can be called several times, every target gets its rows
|
||||||
|
#
|
||||||
|
def write storage, only: nil
|
||||||
|
@targets << { storage:, only: }
|
||||||
|
end
|
||||||
|
|
||||||
|
def query storage, options = {}
|
||||||
|
storage.query options
|
||||||
|
end
|
||||||
|
|
||||||
|
#
|
||||||
|
# loads a table once, for lookups in map/transform instead of one query per row
|
||||||
|
#
|
||||||
|
# lookup pg, query: 'select ...', params: [...], key: 'rechnung_dbid' -> { key => [rows] }
|
||||||
|
# lookup pg, query: 'select ...', key: 'id', unique: true -> { key => row }
|
||||||
|
# lookup csv, key: ->(row) { row['land'] } -> reads the whole storage
|
||||||
|
#
|
||||||
|
def lookup storage, key:, unique: false, **options
|
||||||
|
rows = options[:query] ? storage.query(options) : storage.read
|
||||||
|
key_of = key.respond_to?(:call) ? key : ->(row) { row[key] }
|
||||||
|
unique ? rows.to_h { |row| [key_of.call(row), row] } : rows.group_by(&key_of)
|
||||||
|
end
|
||||||
|
|
||||||
|
def run
|
||||||
|
@stats = Stats.new(version: @settings[:version], read: 0, written: 0, output: 0, rejected: 0, errors: [], warnings: 0,
|
||||||
|
started_at: Time.now, dry_run: @settings[:dry_run])
|
||||||
|
@logger.info "start#{" #{@settings[:version]}" if @settings[:version]}#{@settings[:dry_run] ? ' (dry run)' : ''}"
|
||||||
|
@preview = []
|
||||||
|
raise ArgumentError, 'read storage missing' unless @source
|
||||||
|
raise ArgumentError, 'write storage missing' if @targets.empty?
|
||||||
|
|
||||||
|
before = @transformations.select { |t| EDGE_BEFORE.include?(t[:kind]) }
|
||||||
|
after = @transformations.select { |t| EDGE_AFTER.include?(t[:kind]) }
|
||||||
|
body = @transformations - before - after
|
||||||
|
|
||||||
|
@settings[:streamed] ? run_streamed(before, body, after) : run_batch(before, body, after)
|
||||||
|
|
||||||
|
@watermark&.save unless @settings[:dry_run]
|
||||||
|
@stats
|
||||||
|
rescue StandardError => e
|
||||||
|
@stats.error = e
|
||||||
|
@logger.error "abort: #{e.class}: #{e.message}"
|
||||||
|
raise
|
||||||
|
ensure
|
||||||
|
finish
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def add_step kind, options, &block
|
||||||
|
raise ArgumentError, "#{kind}: block missing" unless block
|
||||||
|
|
||||||
|
@transformations << { kind:, block:, options: }
|
||||||
|
end
|
||||||
|
|
||||||
|
# without any transformation the source is copied unchanged
|
||||||
|
def chained?
|
||||||
|
@settings[:chained] || @transformations.empty? ||
|
||||||
|
@transformations.any? { |t| CHAINING.include?(t[:kind]) }
|
||||||
|
end
|
||||||
|
|
||||||
|
def write_row row
|
||||||
|
@targets.each { |t| t[:storage].write_row(row) if t[:only].nil? || t[:only].call(row) }
|
||||||
|
end
|
||||||
|
|
||||||
|
def finish
|
||||||
|
@stats.finished_at = Time.now
|
||||||
|
[@source, *@targets.map { |t| t[:storage] }, @reject_target].compact.uniq.each(&:close)
|
||||||
|
self.class.runs << @stats
|
||||||
|
print_preview if @stats.dry_run && @stats.error.nil?
|
||||||
|
@logger.info "finish #{@stats}"
|
||||||
|
@on_finish.each do |hook|
|
||||||
|
hook.call(@stats)
|
||||||
|
rescue StandardError => e
|
||||||
|
@logger.error "on_finish: #{e.class}: #{e.message}"
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def source_rows
|
||||||
|
data = @source.read
|
||||||
|
data = data.first(@settings[:limit]) if @settings[:limit]
|
||||||
|
data
|
||||||
|
end
|
||||||
|
|
||||||
|
def output rows
|
||||||
|
@stats.output += rows.size
|
||||||
|
@preview.concat(rows.first(@settings[:preview] - @preview.size)) if @preview.size < @settings[:preview]
|
||||||
|
return if @settings[:dry_run]
|
||||||
|
|
||||||
|
yield rows
|
||||||
|
end
|
||||||
|
|
||||||
|
def print_preview
|
||||||
|
puts "DRY RUN – nothing written: #{@stats}"
|
||||||
|
@preview.each { |row| puts JSON.generate(row) }
|
||||||
|
end
|
||||||
|
|
||||||
|
# whole source in memory, one target.write at the end
|
||||||
|
def run_batch before, body, after
|
||||||
|
data = source_rows
|
||||||
|
@stats.read = data.length
|
||||||
|
data.each { |row| @watermark.track(row) } if @watermark
|
||||||
|
dump 'Source#data', data
|
||||||
|
|
||||||
|
# prepend keeps its old timing: called before the body
|
||||||
|
results = {}.compare_by_identity
|
||||||
|
before.select { |t| t[:kind] == :prepend }.each { |t| results[t] = edge(t, nil, 0) }
|
||||||
|
|
||||||
|
rows = []
|
||||||
|
if chained?
|
||||||
|
counters = Array.new(body.size, 0)
|
||||||
|
data.each { |row| push(row, body, counters, data.length) { |out| rows << out } }
|
||||||
|
else
|
||||||
|
body.each { |step| rows.concat apply(step, data) }
|
||||||
|
end
|
||||||
|
dump 'Transformation#data', rows
|
||||||
|
|
||||||
|
ctx = Context.new(read: data.length, written: rows.length)
|
||||||
|
before.select { |t| t[:kind] == :header }.each { |t| results[t] = edge(t, ctx, 0) }
|
||||||
|
|
||||||
|
# parallel mode keeps the old count of source rows for append
|
||||||
|
count = chained? ? rows.length : data.length
|
||||||
|
new_data = []
|
||||||
|
before.each { |t| add_rows new_data, results[t] }
|
||||||
|
new_data.concat rows
|
||||||
|
after.each { |t| add_rows new_data, edge(t, ctx, count) }
|
||||||
|
|
||||||
|
@stats.written = rows.length
|
||||||
|
dump 'Target#data', new_data
|
||||||
|
output(new_data) do |all|
|
||||||
|
@targets.each { |t| t[:storage].write(t[:only] ? all.select(&t[:only]) : all) }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# row by row from source.read { } to target.write_row, header only knows nothing yet
|
||||||
|
def run_streamed before, body, after
|
||||||
|
if !chained? && body.size > 1
|
||||||
|
raise ArgumentError, 'streamed: several transformations need chained (or map/filter)'
|
||||||
|
end
|
||||||
|
|
||||||
|
emit = ->(result) { output(add_rows([], result)) { |rows| rows.each { |row| write_row(row) } } }
|
||||||
|
|
||||||
|
before.each { |t| emit.call edge(t, Context.new, 0) }
|
||||||
|
|
||||||
|
read = written = 0
|
||||||
|
counters = Array.new(body.size, 0)
|
||||||
|
begin
|
||||||
|
@source.read do |row|
|
||||||
|
raise LimitReached if @settings[:limit] && read >= @settings[:limit]
|
||||||
|
|
||||||
|
read += 1
|
||||||
|
@watermark&.track(row)
|
||||||
|
push(row, body, counters, nil) do |out|
|
||||||
|
written += 1
|
||||||
|
emit.call out
|
||||||
|
end
|
||||||
|
end
|
||||||
|
rescue LimitReached
|
||||||
|
@logger.info "limit #{@settings[:limit]} reached"
|
||||||
|
end
|
||||||
|
@stats.read = read
|
||||||
|
@stats.written = written
|
||||||
|
|
||||||
|
ctx = Context.new(read:, written:)
|
||||||
|
after.each { |t| emit.call edge(t, ctx, chained? ? written : read) }
|
||||||
|
@targets.each { |t| t[:storage].finalize }
|
||||||
|
end
|
||||||
|
|
||||||
|
# one row through the chain, depth first; counters hold the index per step
|
||||||
|
def push row, body, counters, read, level = 0, &emit
|
||||||
|
return emit.call(row) if level == body.size
|
||||||
|
|
||||||
|
index = counters[level]
|
||||||
|
counters[level] += 1
|
||||||
|
safe_step_rows(body[level], row, index, read).each do |out|
|
||||||
|
push(out, body, counters, read, level + 1, &emit)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# parallel mode: every step reads its own copy of the source rows
|
||||||
|
def apply step, rows
|
||||||
|
rows.each_with_index.flat_map { |row, index| safe_step_rows(step, row.dup, index, rows.length) }
|
||||||
|
end
|
||||||
|
|
||||||
|
# errors of a step for one row go to on_error
|
||||||
|
def safe_step_rows step, row, index, read
|
||||||
|
step_rows(step, row, index, read)
|
||||||
|
rescue StandardError => e
|
||||||
|
reject(step, row, e)
|
||||||
|
[]
|
||||||
|
end
|
||||||
|
|
||||||
|
def reject step, row, error
|
||||||
|
raise error if @settings[:on_error] == :abort
|
||||||
|
|
||||||
|
name = step_name(step)
|
||||||
|
@stats.rejected += 1
|
||||||
|
@stats.errors << { step: name, error: error.message, row: } if @stats.errors.size < MAX_ERRORS
|
||||||
|
@logger.warn "rejected by #{name}: #{error.message}"
|
||||||
|
return if @settings[:dry_run] || @reject_target.nil?
|
||||||
|
|
||||||
|
@reject_target.write_row(row.merge('_step' => name, '_error' => error.message))
|
||||||
|
end
|
||||||
|
|
||||||
|
def step_name step
|
||||||
|
"#{step[:kind]}##{@transformations.index(step) + 1}"
|
||||||
|
end
|
||||||
|
|
||||||
|
# rows produced by one step for one input row
|
||||||
|
def step_rows step, row, index, read
|
||||||
|
return [] unless condition?(step, row)
|
||||||
|
|
||||||
|
case step[:kind]
|
||||||
|
when :validate
|
||||||
|
rules = step[:options][:rules]
|
||||||
|
messages = rules.is_a?(Layout) ? rules.errors(row) : Validation.errors(row, rules)
|
||||||
|
messages += Array(step[:block].call(row)).compact if step[:block]
|
||||||
|
ignore = step[:options][:ignore]
|
||||||
|
messages = messages.reject { |m| ignore.any? { |field| m.start_with?("#{field}:") } } if ignore.any?
|
||||||
|
return [row] if messages.empty?
|
||||||
|
raise ValidationError, messages.join('; ') unless step[:options][:warn]
|
||||||
|
|
||||||
|
@stats.warnings += 1
|
||||||
|
if @stats.warnings <= MAX_ERRORS
|
||||||
|
@logger.warn "#{step_name(step)}: #{messages.join('; ')}"
|
||||||
|
elsif @stats.warnings == MAX_ERRORS + 1
|
||||||
|
@logger.warn "more than #{MAX_ERRORS} warnings, further ones only counted"
|
||||||
|
end
|
||||||
|
[row]
|
||||||
|
when :filter
|
||||||
|
step[:block].call(row, Context.new(read:, index:)) ? [row] : []
|
||||||
|
when :map
|
||||||
|
add_rows [], step[:block].call(row, Context.new(read:, index:))
|
||||||
|
else
|
||||||
|
add_rows [], step[:block].call(row, index)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def edge step, ctx, count
|
||||||
|
case step[:kind]
|
||||||
|
when :prepend then step[:block].call(nil, 0)
|
||||||
|
when :append then step[:block].call(nil, count)
|
||||||
|
else step[:block].call(ctx)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# string conditions are evaluated with the local variable row
|
||||||
|
def condition? step, row
|
||||||
|
condition = step[:options].fetch(:condition, true)
|
||||||
|
return true if condition == true
|
||||||
|
return condition.call(row) if condition.respond_to?(:call)
|
||||||
|
|
||||||
|
eval(condition)
|
||||||
|
end
|
||||||
|
|
||||||
|
# a hash is one row, an array are several rows, nil is no row
|
||||||
|
def add_rows new_data, result
|
||||||
|
if result.is_a?(Array)
|
||||||
|
new_data.concat(result.compact)
|
||||||
|
elsif result
|
||||||
|
new_data << result
|
||||||
|
end
|
||||||
|
new_data
|
||||||
|
end
|
||||||
|
|
||||||
|
def dump title, data
|
||||||
|
return unless @settings[:verbose]
|
||||||
|
|
||||||
|
puts "#{title}".ljust(28, '-')
|
||||||
|
puts JSON.pretty_generate data
|
||||||
|
puts "----------------------------"
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# short names besides the <Name>Storage convention
|
||||||
|
DataPipeline.register :sqlite3, 'DataPipeline::SqliteStorage'
|
||||||
|
DataPipeline.register :xlsx, 'DataPipeline::ExcelStorage'
|
||||||
@@ -0,0 +1,113 @@
|
|||||||
|
require 'date'
|
||||||
|
|
||||||
|
class DataPipeline
|
||||||
|
class LayoutError < StandardError; end
|
||||||
|
|
||||||
|
#
|
||||||
|
# fixed field list per record type, for files with several record types (e.g. SAP CUDO)
|
||||||
|
#
|
||||||
|
# layout = DataPipeline::Layout.new do
|
||||||
|
# record 'DOHD', [['SATYP', 'AN', 4, true], ['LFDNR', 'N', 10, true], ['BELNR', 'AN', 10], ...]
|
||||||
|
# record 'LARC', [['SATYP', 'AN', 4, true], ['SUMME', 'N', 10, true]]
|
||||||
|
# end
|
||||||
|
#
|
||||||
|
# field: [name, type, length, required]
|
||||||
|
# type AN (text), N (number, length 10 or 13.2 = digits.decimals), DAT (YYYYMMDD), TIM (HHMMSS)
|
||||||
|
#
|
||||||
|
# the record type of a row is the value of type_field (default: the first value of the row).
|
||||||
|
# keys of a row are matched without regard to case, string or symbol.
|
||||||
|
#
|
||||||
|
# write container(:csv, file: ..., layout: layout) -> values in field order, unknown keys raise
|
||||||
|
# validate layout -> required, length, format per field
|
||||||
|
#
|
||||||
|
class Layout
|
||||||
|
Field = Struct.new(:name, :type, :length, :required) do
|
||||||
|
def key = name.downcase
|
||||||
|
|
||||||
|
# 13.2 -> [11, 2] digits before and after the point, 10 -> [10, 0]
|
||||||
|
def digits
|
||||||
|
before, after = length.to_s.split('.').map(&:to_i)
|
||||||
|
after ||= 0
|
||||||
|
[before - after, after]
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
attr_reader :records
|
||||||
|
|
||||||
|
def initialize type_field: nil, strict: true, &block
|
||||||
|
@type_field = type_field&.to_s&.downcase
|
||||||
|
@strict = strict
|
||||||
|
@records = {}
|
||||||
|
instance_eval(&block) if block
|
||||||
|
end
|
||||||
|
|
||||||
|
def record type, fields
|
||||||
|
@records[type.to_s] = fields.map do |name, kind, length, required|
|
||||||
|
Field.new(name.to_s, (kind || 'AN').to_s.upcase, length, required ? true : false)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def fields_for row
|
||||||
|
type = record_type(row)
|
||||||
|
@records.fetch(type.to_s) { raise LayoutError, "record type #{type.inspect} not in layout (#{@records.keys.join(', ')})" }
|
||||||
|
end
|
||||||
|
|
||||||
|
# values in the order of the layout; unknown keys raise (strict), missing keys are nil
|
||||||
|
def values row
|
||||||
|
fields = fields_for(row)
|
||||||
|
normalized = normalize(row)
|
||||||
|
if @strict
|
||||||
|
unknown = normalized.keys - fields.map(&:key)
|
||||||
|
raise LayoutError, "#{record_type(row)}: fields not in layout: #{unknown.join(', ')}" if unknown.any?
|
||||||
|
end
|
||||||
|
fields.map { |field| normalized[field.key] }
|
||||||
|
end
|
||||||
|
|
||||||
|
def errors row
|
||||||
|
fields = fields_for(row)
|
||||||
|
normalized = normalize(row)
|
||||||
|
fields.flat_map { |field| check(field, normalized[field.key]) }
|
||||||
|
rescue LayoutError => e
|
||||||
|
[e.message]
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def record_type row
|
||||||
|
@type_field ? normalize(row)[@type_field] : row.values.first
|
||||||
|
end
|
||||||
|
|
||||||
|
def normalize row
|
||||||
|
row.to_h { |key, value| [key.to_s.downcase, value] }
|
||||||
|
end
|
||||||
|
|
||||||
|
def check field, value
|
||||||
|
text = value.to_s
|
||||||
|
return(field.required ? ["#{field.name}: missing"] : []) if text.strip.empty?
|
||||||
|
|
||||||
|
case field.type
|
||||||
|
when 'N' then check_number(field, text)
|
||||||
|
when 'DAT' then valid_date?(text) ? [] : ["#{field.name}: not a date YYYYMMDD"]
|
||||||
|
when 'TIM' then text.match?(/\A([01]\d|2[0-3])[0-5]\d[0-5]\d\z/) ? [] : ["#{field.name}: not a time HHMMSS"]
|
||||||
|
else
|
||||||
|
field.length && text.length > field.length.to_i ? ["#{field.name}: longer than #{field.length}"] : []
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def check_number field, text
|
||||||
|
return ["#{field.name}: not a number"] unless text.match?(/\A[+-]?\d+(\.\d+)?\z/)
|
||||||
|
return [] unless field.length
|
||||||
|
|
||||||
|
before, after = field.digits
|
||||||
|
int, dec = text.delete('+-').split('.')
|
||||||
|
return ["#{field.name}: more than #{before} digits"] if int.length > before
|
||||||
|
return ["#{field.name}: more than #{after} decimals"] if dec && dec.length > after
|
||||||
|
|
||||||
|
[]
|
||||||
|
end
|
||||||
|
|
||||||
|
def valid_date? text
|
||||||
|
text.match?(/\A\d{8}\z/) && Date.valid_date?(text[0, 4].to_i, text[4, 2].to_i, text[6, 2].to_i)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
require 'http'
|
||||||
|
require 'json'
|
||||||
|
|
||||||
|
class DataPipeline
|
||||||
|
#
|
||||||
|
# Slack.new(channel: '#etl').send_message('posting export done')
|
||||||
|
#
|
||||||
|
# options
|
||||||
|
# token: bot token, default ENV['SLACK_TOKEN']
|
||||||
|
# channel: default '#general'
|
||||||
|
#
|
||||||
|
class Slack
|
||||||
|
API_URL = 'https://slack.com/api/chat.postMessage'.freeze
|
||||||
|
|
||||||
|
DEFAULTS = {
|
||||||
|
token: nil,
|
||||||
|
channel: '#general'
|
||||||
|
}.freeze
|
||||||
|
|
||||||
|
def initialize options = {}
|
||||||
|
@options = DEFAULTS.merge(token: ENV['SLACK_TOKEN']).merge(options)
|
||||||
|
raise ArgumentError, 'Slack: token missing (option token: or ENV SLACK_TOKEN)' unless @options[:token]
|
||||||
|
end
|
||||||
|
|
||||||
|
# returns the parsed slack response, raises when slack reports an error
|
||||||
|
def send_message(content, verbose: false)
|
||||||
|
response = HTTP.auth("Bearer #{@options[:token]}")
|
||||||
|
.post(API_URL, json: { channel: @options[:channel], text: content })
|
||||||
|
body = JSON.parse(response.body.to_s)
|
||||||
|
puts JSON.pretty_generate(body) if verbose
|
||||||
|
raise "Slack#send_message failed: #{body['error']}" unless body['ok']
|
||||||
|
|
||||||
|
body
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,121 @@
|
|||||||
|
require 'csv'
|
||||||
|
require 'json'
|
||||||
|
require_relative 'storage_plugin'
|
||||||
|
|
||||||
|
class DataPipeline
|
||||||
|
#
|
||||||
|
# options
|
||||||
|
# file: path of the csv file
|
||||||
|
# csv: options passed to ruby CSV, merged with DEFAULTS
|
||||||
|
# columns: fixed column order for writing (and keys for reading without headers)
|
||||||
|
# without columns the values are written in hash order
|
||||||
|
# layout: DataPipeline::Layout, field order per record type (several record types in one file),
|
||||||
|
# unknown fields raise; no header line
|
||||||
|
# counter: print a dot per written row
|
||||||
|
# verbose: print settings
|
||||||
|
#
|
||||||
|
class CsvStorage < StoragePlugin
|
||||||
|
DEFAULTS = {
|
||||||
|
verbose: false,
|
||||||
|
counter: false,
|
||||||
|
columns: nil,
|
||||||
|
csv: {
|
||||||
|
headers: true,
|
||||||
|
col_sep: ',',
|
||||||
|
empty_value: ''
|
||||||
|
}
|
||||||
|
}.freeze
|
||||||
|
|
||||||
|
def initialize **options, &block
|
||||||
|
@options = DEFAULTS.merge(options) do |_key, default, value|
|
||||||
|
default.is_a?(Hash) && value.is_a?(Hash) ? default.merge(value) : value
|
||||||
|
end
|
||||||
|
@file_name = @options.fetch(:file) { raise ArgumentError, 'CsvStorage: option file missing' }
|
||||||
|
@verbose = @options[:verbose]
|
||||||
|
|
||||||
|
if @verbose
|
||||||
|
puts "CsvStorage#options--------"
|
||||||
|
puts JSON.pretty_generate @options
|
||||||
|
puts "---------------------------"
|
||||||
|
end
|
||||||
|
|
||||||
|
yield if block_given?
|
||||||
|
end
|
||||||
|
|
||||||
|
def read(&block)
|
||||||
|
if block_given?
|
||||||
|
CSV.foreach(@file_name, **csv_options) { |row| yield to_hash(row) }
|
||||||
|
return
|
||||||
|
end
|
||||||
|
|
||||||
|
CSV.read(@file_name, **csv_options).map { |row| to_hash(row) }
|
||||||
|
end
|
||||||
|
|
||||||
|
def write data
|
||||||
|
puts "> CsvStorage#write" if @verbose
|
||||||
|
open_file
|
||||||
|
data.each { |row| write_row row }
|
||||||
|
ensure
|
||||||
|
close
|
||||||
|
end
|
||||||
|
|
||||||
|
def write_row row
|
||||||
|
open_file
|
||||||
|
if @options[:layout]
|
||||||
|
@file << @options[:layout].values(row)
|
||||||
|
print '.' if @options[:counter]
|
||||||
|
return
|
||||||
|
end
|
||||||
|
|
||||||
|
if !@header_written && csv_options[:headers]
|
||||||
|
header = columns || row.keys
|
||||||
|
@file << header.map(&:to_s) if header
|
||||||
|
end
|
||||||
|
@header_written = true
|
||||||
|
|
||||||
|
@file << (columns ? columns.map { |col| fetch(row, col) } : row.values)
|
||||||
|
print "." if @options[:counter]
|
||||||
|
end
|
||||||
|
|
||||||
|
def close
|
||||||
|
@file&.close
|
||||||
|
@file = nil
|
||||||
|
@header_written = false
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def csv_options
|
||||||
|
@options[:csv]
|
||||||
|
end
|
||||||
|
|
||||||
|
def columns
|
||||||
|
@options[:columns]
|
||||||
|
end
|
||||||
|
|
||||||
|
def open_file
|
||||||
|
return if @file
|
||||||
|
|
||||||
|
@file = CSV.open(@file_name, 'w', **csv_options)
|
||||||
|
@header_written = false
|
||||||
|
return unless columns && csv_options[:headers] && !@options[:layout]
|
||||||
|
|
||||||
|
@file << columns.map(&:to_s)
|
||||||
|
@header_written = true
|
||||||
|
end
|
||||||
|
|
||||||
|
# rows with headers are CSV::Row, without headers plain arrays
|
||||||
|
def to_hash row
|
||||||
|
return row.to_h if row.is_a?(CSV::Row)
|
||||||
|
|
||||||
|
keys = columns&.map(&:to_s) || (0...row.length).map(&:to_s)
|
||||||
|
keys.zip(row).to_h
|
||||||
|
end
|
||||||
|
|
||||||
|
def fetch row, col
|
||||||
|
return row[col] if row.key?(col)
|
||||||
|
|
||||||
|
row.key?(col.to_s) ? row[col.to_s] : row[col.to_s.to_sym]
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,108 @@
|
|||||||
|
require_relative 'storage_plugin'
|
||||||
|
|
||||||
|
class DataPipeline
|
||||||
|
#
|
||||||
|
# options
|
||||||
|
# file: path of the .xlsx file
|
||||||
|
# sheet: read: name or index (0 = first, default), write: name (default 'Daten')
|
||||||
|
# headers: first row holds the column names (default true)
|
||||||
|
# columns: column names when there is no header row, fixed column order for writing
|
||||||
|
# skip: read: rows before the header row (title lines), default 0
|
||||||
|
#
|
||||||
|
# read needs the gem roo, write the gem caxlsx; both are loaded on first use.
|
||||||
|
# text stays text when writing (no number conversion, leading zeros are kept).
|
||||||
|
#
|
||||||
|
class ExcelStorage < StoragePlugin
|
||||||
|
def initialize **options, &block
|
||||||
|
@options = { headers: true, skip: 0 }.merge(options)
|
||||||
|
@file_name = @options.fetch(:file) { raise ArgumentError, 'ExcelStorage: option file missing' }
|
||||||
|
|
||||||
|
yield if block_given?
|
||||||
|
end
|
||||||
|
|
||||||
|
# without block all rows, with block row by row (xlsx is streamed)
|
||||||
|
def read(&block)
|
||||||
|
return enum_for(:each_row).to_a unless block
|
||||||
|
|
||||||
|
each_row(&block)
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
|
||||||
|
def write rows
|
||||||
|
open_sheet
|
||||||
|
rows.each { |row| write_row row }
|
||||||
|
ensure
|
||||||
|
close
|
||||||
|
end
|
||||||
|
|
||||||
|
def write_row row
|
||||||
|
open_sheet
|
||||||
|
if @columns.nil?
|
||||||
|
@columns = (@options[:columns] || row.keys).map(&:to_s)
|
||||||
|
add_row @columns, header: true if @options[:headers]
|
||||||
|
end
|
||||||
|
add_row(@columns.map { |col| fetch(row, col) })
|
||||||
|
end
|
||||||
|
|
||||||
|
def close
|
||||||
|
return unless @package
|
||||||
|
|
||||||
|
@package.serialize(@file_name)
|
||||||
|
@package = @sheet = @columns = nil
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def each_row
|
||||||
|
self.class.require_gem 'roo'
|
||||||
|
book = Roo::Spreadsheet.open(@file_name, extension: File.extname(@file_name).delete('.').to_sym)
|
||||||
|
sheet = book.sheet(@options.fetch(:sheet, 0))
|
||||||
|
keys = @options[:headers] ? nil : @options[:columns]&.map(&:to_s)
|
||||||
|
skip = @options[:skip]
|
||||||
|
|
||||||
|
rows_of(sheet).each do |values|
|
||||||
|
next if (skip -= 1) >= 0
|
||||||
|
next if values.all? { |v| v.nil? || v.to_s.strip.empty? }
|
||||||
|
|
||||||
|
if keys.nil?
|
||||||
|
keys = values.each_with_index.map { |v, i| v.nil? ? i.to_s : v.to_s }
|
||||||
|
next
|
||||||
|
end
|
||||||
|
yield keys.zip(values).to_h
|
||||||
|
end
|
||||||
|
ensure
|
||||||
|
book&.close
|
||||||
|
end
|
||||||
|
|
||||||
|
# xlsx row by row without loading the sheet, other formats over the row api
|
||||||
|
def rows_of sheet
|
||||||
|
if sheet.respond_to?(:each_row_streaming)
|
||||||
|
Enumerator.new do |y|
|
||||||
|
sheet.each_row_streaming(pad_cells: true) { |cells| y << cells.map { |c| c&.value } }
|
||||||
|
end
|
||||||
|
else
|
||||||
|
(sheet.first_row.to_i..sheet.last_row.to_i).lazy.map { |i| sheet.row(i) }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def open_sheet
|
||||||
|
return if @package
|
||||||
|
|
||||||
|
self.class.require_gem 'axlsx', 'caxlsx'
|
||||||
|
@package = Axlsx::Package.new
|
||||||
|
@sheet = @package.workbook.add_worksheet(name: @options.fetch(:sheet, 'Daten').to_s[0, 31])
|
||||||
|
@bold = @package.workbook.styles.add_style(b: true)
|
||||||
|
end
|
||||||
|
|
||||||
|
def add_row values, header: false
|
||||||
|
types = values.map { |v| v.is_a?(String) ? :string : nil }
|
||||||
|
@sheet.add_row(values, types:, style: header ? @bold : nil)
|
||||||
|
end
|
||||||
|
|
||||||
|
def fetch row, col
|
||||||
|
return row[col] if row.key?(col)
|
||||||
|
|
||||||
|
row.key?(col.to_sym) ? row[col.to_sym] : nil
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,79 @@
|
|||||||
|
require 'json'
|
||||||
|
require_relative 'storage_plugin'
|
||||||
|
|
||||||
|
class DataPipeline
|
||||||
|
#
|
||||||
|
# options
|
||||||
|
# file: path of the file
|
||||||
|
# format: :array (one json array) or :lines (one object per line, ndjson)
|
||||||
|
# default from the extension: .ndjson / .jsonl -> :lines, else :array
|
||||||
|
# root: read only: path to the array inside the document, e.g. 'data.items'
|
||||||
|
# a single object (instead of an array) is read as one row
|
||||||
|
# pretty: write :array pretty printed (default false)
|
||||||
|
#
|
||||||
|
class JsonStorage < StoragePlugin
|
||||||
|
LINE_EXTENSIONS = %w[.ndjson .jsonl].freeze
|
||||||
|
|
||||||
|
def initialize **options, &block
|
||||||
|
@options = options
|
||||||
|
@file_name = options.fetch(:file) { raise ArgumentError, 'JsonStorage: option file missing' }
|
||||||
|
@format = options.fetch(:format) { LINE_EXTENSIONS.include?(File.extname(@file_name)) ? :lines : :array }
|
||||||
|
raise ArgumentError, "JsonStorage: format #{@format} unknown, use :array or :lines" unless %i[array lines].include?(@format)
|
||||||
|
|
||||||
|
yield if block_given?
|
||||||
|
end
|
||||||
|
|
||||||
|
# without block all rows, with block row by row (ndjson is read line by line)
|
||||||
|
def read(&block)
|
||||||
|
rows = @format == :lines ? read_lines : read_array
|
||||||
|
return rows.to_a unless block
|
||||||
|
|
||||||
|
rows.each(&block)
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
|
||||||
|
def write rows
|
||||||
|
if @format == :lines
|
||||||
|
File.open(@file_name, 'w') { |f| rows.each { |row| f.puts JSON.generate(row) } }
|
||||||
|
else
|
||||||
|
File.write(@file_name, @options[:pretty] ? JSON.pretty_generate(rows) : JSON.generate(rows))
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# streaming: ndjson line by line, array as "[row,row,...]" closed in close
|
||||||
|
def write_row row
|
||||||
|
@file ||= File.open(@file_name, 'w').tap { |f| f.write('[') if @format == :array }
|
||||||
|
if @format == :array
|
||||||
|
@file.write(',') if @rows_written
|
||||||
|
@file.write(JSON.generate(row))
|
||||||
|
else
|
||||||
|
@file.puts JSON.generate(row)
|
||||||
|
end
|
||||||
|
@rows_written = true
|
||||||
|
end
|
||||||
|
|
||||||
|
def close
|
||||||
|
return unless @file
|
||||||
|
|
||||||
|
@file.write(']') if @format == :array
|
||||||
|
@file.close
|
||||||
|
@file = nil
|
||||||
|
@rows_written = false
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def read_lines
|
||||||
|
File.foreach(@file_name).lazy.map(&:strip).reject(&:empty?).map { |line| JSON.parse(line) }
|
||||||
|
end
|
||||||
|
|
||||||
|
def read_array
|
||||||
|
data = JSON.parse(File.read(@file_name))
|
||||||
|
@options[:root].to_s.split('.').each { |key| data = data.fetch(key) } if @options[:root]
|
||||||
|
data = [data] if data.is_a?(Hash) # a single object is one row
|
||||||
|
raise ArgumentError, "JsonStorage: #{@file_name} does not contain an array" unless data.is_a?(Array)
|
||||||
|
|
||||||
|
data
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,125 @@
|
|||||||
|
require_relative 'storage_plugin'
|
||||||
|
|
||||||
|
class DataPipeline
|
||||||
|
#
|
||||||
|
# options
|
||||||
|
# host, port, database, username, password
|
||||||
|
# query: sql for read, placeholders $1, $2, ...
|
||||||
|
# params: values for the placeholders of query
|
||||||
|
# table: target table for write
|
||||||
|
# truncate: true -> delete all rows of the table before the first write (same transaction)
|
||||||
|
# typed: true -> values come back as ruby types (Integer, Float, Date, ...)
|
||||||
|
# false (default) -> all values as String, NULL as nil
|
||||||
|
# fetch_size: rows per fetch when reading with a block (default 1000)
|
||||||
|
#
|
||||||
|
class PostgresStorage < StoragePlugin
|
||||||
|
def initialize **options, &block
|
||||||
|
self.class.require_gem 'pg'
|
||||||
|
@options = options
|
||||||
|
|
||||||
|
begin
|
||||||
|
@conn = PG.connect(
|
||||||
|
host: options[:host],
|
||||||
|
port: options[:port] || '5432',
|
||||||
|
dbname: options[:database],
|
||||||
|
user: options[:username],
|
||||||
|
password: options[:password]
|
||||||
|
)
|
||||||
|
rescue PG::ConnectionBad => e
|
||||||
|
raise PG::ConnectionBad, "Error connection to database: #{e.message}"
|
||||||
|
end
|
||||||
|
|
||||||
|
@conn.type_map_for_queries = PG::BasicTypeMapForQueries.new(@conn)
|
||||||
|
@conn.type_map_for_results = PG::BasicTypeMapForResults.new(@conn) if options[:typed]
|
||||||
|
|
||||||
|
yield if block_given?
|
||||||
|
end
|
||||||
|
|
||||||
|
# without block all rows at once, with block row by row through a server side cursor
|
||||||
|
def read(&block)
|
||||||
|
raise "Query for PostgresStorage missing" unless @options[:query]
|
||||||
|
|
||||||
|
params = @options.fetch(:params, [])
|
||||||
|
return query(query: @options[:query], params:) unless block_given?
|
||||||
|
|
||||||
|
each_row(@options[:query], params, &block)
|
||||||
|
end
|
||||||
|
|
||||||
|
def query options = {}
|
||||||
|
raise "Query for PostgresStorage:query missing" unless options[:query]
|
||||||
|
|
||||||
|
result = @conn.exec_params(options[:query], options.fetch(:params, []))
|
||||||
|
return result.to_a unless block_given?
|
||||||
|
|
||||||
|
result.each { |row| yield row }
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
|
||||||
|
#
|
||||||
|
# write rows -> table from option table:
|
||||||
|
# write table_name, rows -> explicit table (old signature)
|
||||||
|
#
|
||||||
|
# all rows in one transaction, values as bind parameters, nil -> NULL
|
||||||
|
#
|
||||||
|
def write table_or_data, data = nil
|
||||||
|
table_name, data = data.nil? ? [@options[:table], table_or_data] : [table_or_data, data]
|
||||||
|
raise "Table for PostgresStorage#write missing" unless table_name
|
||||||
|
|
||||||
|
@conn.transaction do
|
||||||
|
truncate_once table_name
|
||||||
|
data.each { |row| insert table_name, row }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def write_row row
|
||||||
|
raise "Table for PostgresStorage#write_row missing" unless @options[:table]
|
||||||
|
|
||||||
|
truncate_once @options[:table]
|
||||||
|
insert @options[:table], row
|
||||||
|
end
|
||||||
|
|
||||||
|
def close
|
||||||
|
@conn.close if @conn && !@conn.finished?
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
# rows are fetched in batches inside a transaction; other queries on the
|
||||||
|
# same connection (lookups, subqueries) are possible while reading
|
||||||
|
def each_row sql, params
|
||||||
|
size = Integer(@options.fetch(:fetch_size, 1000))
|
||||||
|
@conn.transaction do
|
||||||
|
@conn.exec_params("declare etl_read no scroll cursor for #{sql.strip.chomp(';')}", params)
|
||||||
|
loop do
|
||||||
|
result = @conn.exec("fetch #{size} from etl_read")
|
||||||
|
break if result.ntuples.zero?
|
||||||
|
|
||||||
|
result.each { |row| yield row }
|
||||||
|
end
|
||||||
|
@conn.exec('close etl_read')
|
||||||
|
end
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
|
||||||
|
def truncate_once table_name
|
||||||
|
return unless @options[:truncate] && !(@truncated ||= {})[table_name]
|
||||||
|
|
||||||
|
@conn.exec("delete from #{quote_table(table_name)}")
|
||||||
|
@truncated[table_name] = true
|
||||||
|
end
|
||||||
|
|
||||||
|
def insert table_name, row
|
||||||
|
columns = row.keys.map { |col| @conn.quote_ident(col.to_s) }.join(',')
|
||||||
|
placeholders = (1..row.size).map { |i| "$#{i}" }.join(',')
|
||||||
|
@conn.exec_params(
|
||||||
|
"insert into #{quote_table(table_name)} (#{columns}) values (#{placeholders})",
|
||||||
|
row.values
|
||||||
|
)
|
||||||
|
end
|
||||||
|
|
||||||
|
# schema.table -> "schema"."table"
|
||||||
|
def quote_table name
|
||||||
|
name.to_s.split('.').map { |part| @conn.quote_ident(part) }.join('.')
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,180 @@
|
|||||||
|
require 'json'
|
||||||
|
require_relative 'storage_plugin'
|
||||||
|
|
||||||
|
class DataPipeline
|
||||||
|
#
|
||||||
|
# options
|
||||||
|
# url: base url (read and write)
|
||||||
|
# headers: hash of request headers
|
||||||
|
# auth: value of the Authorization header, e.g. "Bearer #{ENV['API_TOKEN']}"
|
||||||
|
# params: query parameters for read
|
||||||
|
# root: path to the array in the response, e.g. 'data.items' (default: the response itself)
|
||||||
|
# timeout: seconds (default 30)
|
||||||
|
# ssl_verify: false -> accept any server certificate (own CA, IP address); default true
|
||||||
|
# retries: repeats on 5xx, 429 and network errors (default 2), wait retry_wait * attempt seconds
|
||||||
|
#
|
||||||
|
# paging for read (one of)
|
||||||
|
# page_param: 'page' -> page_start (default 1), page_start + 1, ... until a page returns no rows
|
||||||
|
# next: ->(body) { body['next'] } -> url of the next page or nil
|
||||||
|
# max_pages: safety stop (default 1000)
|
||||||
|
#
|
||||||
|
# write
|
||||||
|
# write_url: default url; or ->(row) { "#{base}/kunden/#{row['id']}" } for one url per row
|
||||||
|
# write_method: :post (default), :put, :patch
|
||||||
|
# batch_size: nil -> one request per row, n -> json array of n rows per request
|
||||||
|
#
|
||||||
|
# query(query: 'path', params: {...}) -> GET url + path, rows like read (for lookups)
|
||||||
|
#
|
||||||
|
# call(:post, 'person/listkunden', json: {...}) -> one request, parsed json back (for rpc style apis)
|
||||||
|
# retries: 0 for requests that create something, so a 5xx is not sent twice
|
||||||
|
# a failed request raises RestStorage::Error with #status (nil for network errors)
|
||||||
|
#
|
||||||
|
class RestStorage < StoragePlugin
|
||||||
|
class Error < StandardError
|
||||||
|
attr_reader :status
|
||||||
|
|
||||||
|
def initialize message = nil, status: nil
|
||||||
|
super(message)
|
||||||
|
@status = status
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
class RetryableError < Error; end
|
||||||
|
|
||||||
|
RETRY_STATUS = [429, 500, 502, 503, 504].freeze
|
||||||
|
|
||||||
|
def initialize **options, &block
|
||||||
|
self.class.require_gem 'http'
|
||||||
|
@options = { timeout: 30, retries: 2, retry_wait: 1, max_pages: 1000, write_method: :post }.merge(options)
|
||||||
|
@url = @options.fetch(:url) { raise ArgumentError, 'RestStorage: option url missing' }
|
||||||
|
if @options[:batch_size] && @options[:write_url].respond_to?(:call)
|
||||||
|
raise ArgumentError, 'RestStorage: write_url per row does not work with batch_size'
|
||||||
|
end
|
||||||
|
@buffer = []
|
||||||
|
|
||||||
|
yield if block_given?
|
||||||
|
end
|
||||||
|
|
||||||
|
def read(&block)
|
||||||
|
rows = enum_for(:each_page_row, @url, @options.fetch(:params, {}))
|
||||||
|
return rows.to_a unless block
|
||||||
|
|
||||||
|
rows.each(&block)
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
|
||||||
|
def query options = {}
|
||||||
|
path = options[:query].to_s
|
||||||
|
url = path.start_with?('http') ? path : "#{@url.chomp('/')}/#{path.delete_prefix('/')}"
|
||||||
|
rows = enum_for(:each_page_row, url, options.fetch(:params, {}))
|
||||||
|
return rows.to_a unless block_given?
|
||||||
|
|
||||||
|
rows.each { |row| yield row }
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
|
||||||
|
# one request against url + path; json body and/or query params
|
||||||
|
def call method, path, json: nil, params: nil, retries: nil
|
||||||
|
url = path.to_s.start_with?('http') ? path.to_s : "#{@url.chomp('/')}/#{path.to_s.delete_prefix('/')}"
|
||||||
|
args = { json:, params: }.compact
|
||||||
|
request(method, url, retries:, **args)
|
||||||
|
end
|
||||||
|
|
||||||
|
def write rows
|
||||||
|
rows.each { |row| write_row row }
|
||||||
|
ensure
|
||||||
|
close
|
||||||
|
end
|
||||||
|
|
||||||
|
def write_row row
|
||||||
|
if @options[:batch_size]
|
||||||
|
@buffer << row
|
||||||
|
flush if @buffer.size >= @options[:batch_size]
|
||||||
|
else
|
||||||
|
send_rows row
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def close
|
||||||
|
flush
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def flush
|
||||||
|
return if @buffer.empty?
|
||||||
|
|
||||||
|
send_rows @buffer
|
||||||
|
@buffer = []
|
||||||
|
end
|
||||||
|
|
||||||
|
def send_rows payload
|
||||||
|
url = @options.fetch(:write_url, @url)
|
||||||
|
url = url.call(payload) if url.respond_to?(:call)
|
||||||
|
request(@options[:write_method], url, json: payload)
|
||||||
|
end
|
||||||
|
|
||||||
|
def each_page_row url, params
|
||||||
|
page = @options.fetch(:page_start, 1)
|
||||||
|
@options[:max_pages].times do
|
||||||
|
query = @options[:page_param] ? params.merge(@options[:page_param] => page) : params
|
||||||
|
body = request(:get, url, params: query)
|
||||||
|
rows = rows_of(body)
|
||||||
|
rows.each { |row| yield row }
|
||||||
|
|
||||||
|
if @options[:next]
|
||||||
|
url = @options[:next].call(body)
|
||||||
|
params = {}
|
||||||
|
break if url.nil? || url.to_s.empty?
|
||||||
|
elsif @options[:page_param]
|
||||||
|
break if rows.empty?
|
||||||
|
|
||||||
|
page += 1
|
||||||
|
else
|
||||||
|
break
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def rows_of body
|
||||||
|
data = body
|
||||||
|
@options[:root].to_s.split('.').each { |key| data = data.fetch(key) } if @options[:root]
|
||||||
|
data = [] if data.nil?
|
||||||
|
raise Error, "RestStorage: response is not an array (root: #{@options[:root].inspect})" unless data.is_a?(Array)
|
||||||
|
|
||||||
|
data
|
||||||
|
end
|
||||||
|
|
||||||
|
def request method, url, retries: nil, **args
|
||||||
|
retries ||= @options[:retries]
|
||||||
|
args[:ssl_context] = ssl_context if @options[:ssl_verify] == false && url.start_with?('https')
|
||||||
|
attempt = 0
|
||||||
|
begin
|
||||||
|
attempt += 1
|
||||||
|
response = client.request(method, url, **args)
|
||||||
|
status = response.status.code
|
||||||
|
text = response.body.to_s
|
||||||
|
raise RetryableError.new("HTTP #{status}", status:) if RETRY_STATUS.include?(status)
|
||||||
|
raise Error.new("#{method.upcase} #{url}: HTTP #{status} #{text[0, 200]}", status:) unless response.status.success?
|
||||||
|
|
||||||
|
text.strip.empty? ? nil : JSON.parse(text)
|
||||||
|
rescue RetryableError, HTTP::Error, SystemCallError => e
|
||||||
|
raise Error.new("#{method.upcase} #{url}: #{e.message}", status: e.respond_to?(:status) ? e.status : nil) if attempt > retries
|
||||||
|
|
||||||
|
sleep(@options[:retry_wait] * attempt)
|
||||||
|
retry
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def ssl_context
|
||||||
|
require 'openssl'
|
||||||
|
OpenSSL::SSL::SSLContext.new.tap { |ctx| ctx.verify_mode = OpenSSL::SSL::VERIFY_NONE }
|
||||||
|
end
|
||||||
|
|
||||||
|
def client
|
||||||
|
headers = { 'Accept' => 'application/json' }.merge(@options.fetch(:headers, {}))
|
||||||
|
headers['Authorization'] = @options[:auth] if @options[:auth]
|
||||||
|
HTTP.timeout(@options[:timeout]).headers(headers)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,122 @@
|
|||||||
|
require_relative 'storage_plugin'
|
||||||
|
|
||||||
|
class DataPipeline
|
||||||
|
#
|
||||||
|
# options
|
||||||
|
# database: path of the database file or ':memory:'
|
||||||
|
# query: sql for read, placeholders ? or $1 / :name
|
||||||
|
# params: values for the placeholders (array, or hash for :name)
|
||||||
|
# table: target table for write
|
||||||
|
# create: true -> create the table on the first write, columns from the first row
|
||||||
|
# (Integer -> integer, Float -> real, else text)
|
||||||
|
# truncate: true -> delete all rows of the table before the first write
|
||||||
|
#
|
||||||
|
# rows come back as Hash<String, value> with sqlite types (Integer, Float, String, nil).
|
||||||
|
# the database file is opened on first use: a dry run creates no file.
|
||||||
|
#
|
||||||
|
class SqliteStorage < StoragePlugin
|
||||||
|
def initialize **options, &block
|
||||||
|
self.class.require_gem 'sqlite3'
|
||||||
|
@options = options
|
||||||
|
@database = options.fetch(:database) { raise ArgumentError, 'SqliteStorage: option database missing' }.to_s
|
||||||
|
@prepared = {}
|
||||||
|
|
||||||
|
yield if block_given?
|
||||||
|
end
|
||||||
|
|
||||||
|
# without block all rows at once, with block row by row
|
||||||
|
def read(&block)
|
||||||
|
raise 'Query for SqliteStorage missing' unless @options[:query]
|
||||||
|
|
||||||
|
query(query: @options[:query], params: @options.fetch(:params, []), &block)
|
||||||
|
end
|
||||||
|
|
||||||
|
def query options = {}
|
||||||
|
raise 'Query for SqliteStorage:query missing' unless options[:query]
|
||||||
|
|
||||||
|
statement = db.prepare(options[:query])
|
||||||
|
result = statement.execute(*bind_args(options.fetch(:params, [])))
|
||||||
|
return result.map(&:to_h) unless block_given?
|
||||||
|
|
||||||
|
result.each { |row| yield row.to_h }
|
||||||
|
nil
|
||||||
|
ensure
|
||||||
|
statement&.close
|
||||||
|
end
|
||||||
|
|
||||||
|
#
|
||||||
|
# write rows -> table from option table:
|
||||||
|
# write table_name, rows -> explicit table
|
||||||
|
#
|
||||||
|
# all rows in one transaction, values as bind parameters, nil -> NULL
|
||||||
|
#
|
||||||
|
def write table_or_data, data = nil
|
||||||
|
table_name, data = data.nil? ? [@options[:table], table_or_data] : [table_or_data, data]
|
||||||
|
raise 'Table for SqliteStorage#write missing' unless table_name
|
||||||
|
|
||||||
|
return if data.empty?
|
||||||
|
|
||||||
|
db.transaction do
|
||||||
|
prepare_table table_name, data.first
|
||||||
|
data.each { |row| insert table_name, row }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def write_row row
|
||||||
|
raise 'Table for SqliteStorage#write_row missing' unless @options[:table]
|
||||||
|
|
||||||
|
prepare_table @options[:table], row
|
||||||
|
insert @options[:table], row
|
||||||
|
end
|
||||||
|
|
||||||
|
def close
|
||||||
|
@db.close if @db && !@db.closed?
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def db
|
||||||
|
@db ||= SQLite3::Database.new(@database).tap { |d| d.results_as_hash = true }
|
||||||
|
end
|
||||||
|
|
||||||
|
# create / truncate once per table, before its first row
|
||||||
|
def prepare_table table_name, row
|
||||||
|
return if @prepared[table_name]
|
||||||
|
|
||||||
|
if @options[:create]
|
||||||
|
columns = row.map { |key, value| "#{quote(key)} #{column_type(value)}" }.join(', ')
|
||||||
|
db.execute("create table if not exists #{quote_table(table_name)} (#{columns})")
|
||||||
|
end
|
||||||
|
db.execute("delete from #{quote_table(table_name)}") if @options[:truncate]
|
||||||
|
@prepared[table_name] = true
|
||||||
|
end
|
||||||
|
|
||||||
|
def column_type value
|
||||||
|
case value
|
||||||
|
when Integer then 'integer'
|
||||||
|
when Float then 'real'
|
||||||
|
else 'text'
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# a hash binds named parameters, an array positional ones
|
||||||
|
def bind_args params
|
||||||
|
params.is_a?(Hash) ? [params] : Array(params)
|
||||||
|
end
|
||||||
|
|
||||||
|
def insert table_name, row
|
||||||
|
columns = row.keys.map { |col| quote(col) }.join(',')
|
||||||
|
placeholders = (['?'] * row.size).join(',')
|
||||||
|
values = row.values.map { |v| v.is_a?(Date) || v.is_a?(Time) ? v.iso8601 : v }
|
||||||
|
db.execute("insert into #{quote_table(table_name)} (#{columns}) values (#{placeholders})", values)
|
||||||
|
end
|
||||||
|
|
||||||
|
def quote name
|
||||||
|
%("#{name.to_s.gsub('"', '""')}")
|
||||||
|
end
|
||||||
|
|
||||||
|
def quote_table name
|
||||||
|
name.to_s.split('.').map { |part| quote(part) }.join('.')
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
|
||||||
|
|
||||||
|
class DataPipeline
|
||||||
|
#
|
||||||
|
# common interface for all storage plugins
|
||||||
|
#
|
||||||
|
# read -> Array<Hash<String, value>>, with block: yields each row
|
||||||
|
# write rows -> writes Array<Hash>
|
||||||
|
# write_row row -> writes a single row (streaming)
|
||||||
|
# query opts -> Array<Hash>, ad hoc queries from transformations
|
||||||
|
# set opts -> merge options after creation (e.g. read pg, query: ...)
|
||||||
|
# close -> release connections and files, safe to call more than once
|
||||||
|
#
|
||||||
|
class StoragePlugin
|
||||||
|
# loads the gem of a plugin on first use, with a hint when it is missing
|
||||||
|
def self.require_gem feature, gem_name = feature
|
||||||
|
require feature
|
||||||
|
rescue LoadError
|
||||||
|
raise LoadError, "#{name} needs the gem '#{gem_name}': add `gem '#{gem_name}'` to the Gemfile and run bundle install"
|
||||||
|
end
|
||||||
|
|
||||||
|
def initialize **options
|
||||||
|
@options = options
|
||||||
|
end
|
||||||
|
|
||||||
|
def connect
|
||||||
|
raise NotImplementedError, "#{self.class}#connect is missing"
|
||||||
|
end
|
||||||
|
|
||||||
|
def read
|
||||||
|
raise NotImplementedError, "#{self.class}#read is missing"
|
||||||
|
end
|
||||||
|
|
||||||
|
def write data = nil
|
||||||
|
raise NotImplementedError, "#{self.class}#write is missing"
|
||||||
|
end
|
||||||
|
|
||||||
|
def write_row row
|
||||||
|
raise NotImplementedError, "#{self.class}#write_row is missing"
|
||||||
|
end
|
||||||
|
|
||||||
|
def query options = {}
|
||||||
|
raise NotImplementedError, "#{self.class}#query is missing"
|
||||||
|
end
|
||||||
|
|
||||||
|
def set options
|
||||||
|
(@options ||= {}).merge! options
|
||||||
|
self
|
||||||
|
end
|
||||||
|
|
||||||
|
def close
|
||||||
|
end
|
||||||
|
|
||||||
|
def finalize
|
||||||
|
close
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
require 'date'
|
||||||
|
|
||||||
|
class DataPipeline
|
||||||
|
# raised by a validate step, handled like any other row error (on_error)
|
||||||
|
class ValidationError < StandardError; end
|
||||||
|
|
||||||
|
#
|
||||||
|
# rules per column
|
||||||
|
# required: true value present and not empty
|
||||||
|
# max: 35, min: 1 length of the value as text
|
||||||
|
# format: /\A\d+\z/ regex on the value as text
|
||||||
|
# in: %w[S H] allowed values
|
||||||
|
# type: :integer | :decimal | :date
|
||||||
|
#
|
||||||
|
# empty values are only checked by required
|
||||||
|
#
|
||||||
|
module Validation
|
||||||
|
TYPES = {
|
||||||
|
integer: ->(v) { Integer(v.to_s, 10) },
|
||||||
|
decimal: ->(v) { Float(v.to_s) },
|
||||||
|
date: ->(v) { v.is_a?(Date) ? v : Date.iso8601(v.to_s) }
|
||||||
|
}.freeze
|
||||||
|
|
||||||
|
module_function
|
||||||
|
|
||||||
|
def errors row, rules
|
||||||
|
rules.flat_map do |column, rule|
|
||||||
|
value = row.key?(column) ? row[column] : row[column.to_s]
|
||||||
|
check(column, value, rule)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def check column, value, rule
|
||||||
|
empty = value.nil? || value.to_s.strip.empty?
|
||||||
|
return(rule[:required] ? ["#{column}: missing"] : []) if empty
|
||||||
|
|
||||||
|
text = value.to_s
|
||||||
|
messages = []
|
||||||
|
messages << "#{column}: longer than #{rule[:max]}" if rule[:max] && text.length > rule[:max]
|
||||||
|
messages << "#{column}: shorter than #{rule[:min]}" if rule[:min] && text.length < rule[:min]
|
||||||
|
messages << "#{column}: format #{rule[:format].inspect}" if rule[:format] && !text.match?(rule[:format])
|
||||||
|
messages << "#{column}: not in #{rule[:in].inspect}" if rule[:in] && !rule[:in].include?(value)
|
||||||
|
if rule[:type]
|
||||||
|
begin
|
||||||
|
TYPES.fetch(rule[:type]).call(value)
|
||||||
|
rescue ArgumentError, TypeError, Date::Error
|
||||||
|
messages << "#{column}: not #{rule[:type]}"
|
||||||
|
end
|
||||||
|
end
|
||||||
|
messages
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
class DataPipeline
|
||||||
|
VERSION = '2.0.0'.freeze
|
||||||
|
|
||||||
|
# kept for scripts that print the version (rake info:info); the history is in CHANGELOG.md
|
||||||
|
class Version
|
||||||
|
def self.version = "datapipeline #{VERSION}"
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
require 'json'
|
||||||
|
require 'fileutils'
|
||||||
|
|
||||||
|
class DataPipeline
|
||||||
|
#
|
||||||
|
# remembers the highest value of a column between runs (json file, one entry per name)
|
||||||
|
#
|
||||||
|
# wm = watermark :posting, column: 'rechnung_dbid', file: 'data/.state.json', initial: 0
|
||||||
|
# read pg, query: 'select ... where rechnung_dbid > $1', params: [wm.value]
|
||||||
|
#
|
||||||
|
# the pipeline tracks every source row and saves the new value after a successful run
|
||||||
|
# (not after an abort, not in dry run). numeric texts are compared as numbers.
|
||||||
|
#
|
||||||
|
class Watermark
|
||||||
|
attr_reader :name, :column, :file, :value, :current
|
||||||
|
|
||||||
|
def initialize name, column:, file:, initial: nil
|
||||||
|
@name = name.to_s
|
||||||
|
@column = column
|
||||||
|
@file = file
|
||||||
|
@value = load.fetch(@name, initial)
|
||||||
|
@current = @value
|
||||||
|
end
|
||||||
|
|
||||||
|
def track row
|
||||||
|
candidate = row.key?(@column) ? row[@column] : row[@column.to_s]
|
||||||
|
return if candidate.nil?
|
||||||
|
|
||||||
|
@current = candidate if @current.nil? || greater?(candidate, @current)
|
||||||
|
end
|
||||||
|
|
||||||
|
def changed?
|
||||||
|
@current != @value
|
||||||
|
end
|
||||||
|
|
||||||
|
def save
|
||||||
|
return unless changed?
|
||||||
|
|
||||||
|
state = load.merge(@name => @current)
|
||||||
|
FileUtils.mkdir_p(File.dirname(@file))
|
||||||
|
File.write("#{@file}.tmp", JSON.pretty_generate(state))
|
||||||
|
File.rename("#{@file}.tmp", @file)
|
||||||
|
@value = @current
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def load
|
||||||
|
File.exist?(@file) ? JSON.parse(File.read(@file)) : {}
|
||||||
|
end
|
||||||
|
|
||||||
|
def greater? a, b
|
||||||
|
numeric?(a) && numeric?(b) ? a.to_s.to_r > b.to_s.to_r : a.to_s > b.to_s
|
||||||
|
end
|
||||||
|
|
||||||
|
def numeric? value
|
||||||
|
value.is_a?(Numeric) || value.to_s.match?(/\A-?\d+(\.\d+)?\z/)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
require 'open3'
|
||||||
|
|
||||||
|
# bin/etl as a subprocess: arguments, params, dry run, exit codes
|
||||||
|
class EtlCliTest < PipelineTest
|
||||||
|
def etl dir, *args
|
||||||
|
env = { 'BUNDLE_GEMFILE' => File.join(ROOT, 'Gemfile') }
|
||||||
|
Open3.capture3(env, RbConfig.ruby, File.join(ROOT, 'bin/etl'), *args, chdir: dir)
|
||||||
|
end
|
||||||
|
|
||||||
|
def script dir, name, body
|
||||||
|
File.write("#{dir}/in.csv", "a,b\n1,x\n2,y\nz,q\n")
|
||||||
|
File.write("#{dir}/#{name}", "require #{File.join(ROOT, 'lib/datapipeline').inspect}\n#{body}")
|
||||||
|
"#{dir}/#{name}"
|
||||||
|
end
|
||||||
|
|
||||||
|
# like import.*.rb: own OptionParser for --from/--till, transform api
|
||||||
|
def test_legacy_script_gets_from_till_and_limit
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
path = script(dir, 'legacy.rb', <<~RUBY)
|
||||||
|
require 'optparse/date'
|
||||||
|
opts = {}
|
||||||
|
OptionParser.new { |p| p.on('--from=DATE', Date) { |v| opts[:from] = v }; p.on('--till=DATE', Date) { |v| opts[:till] = v } }.parse!
|
||||||
|
File.write('args.txt', "\#{opts[:from]} \#{opts[:till]}")
|
||||||
|
DataPipeline.new do
|
||||||
|
read container(:csv, file: 'in.csv')
|
||||||
|
transform(processing: :prepend) { |_, _| { 'h' => 'HEAD' } }
|
||||||
|
transform { |r, i| r.merge('i' => i) }
|
||||||
|
write container(:csv, file: 'out.csv', csv: { headers: false, col_sep: '|' })
|
||||||
|
end.run
|
||||||
|
RUBY
|
||||||
|
_out, err, status = etl(dir, 'run', path, '--from=2025-11-01', '--till=2025-11-30', '--limit=2')
|
||||||
|
assert_equal 0, status.exitstatus, err
|
||||||
|
assert_equal '2025-11-01 2025-11-30', File.read("#{dir}/args.txt")
|
||||||
|
assert_equal "HEAD\n1|x|0\n2|y|1\n", File.read("#{dir}/out.csv")
|
||||||
|
assert_includes err, 'INFO finish read=2 written=2 rejected=0'
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_params_and_exit_code_2_for_rejected_rows
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
path = script(dir, 'neu.rb', <<~RUBY)
|
||||||
|
DataPipeline.new do
|
||||||
|
on_error :skip
|
||||||
|
read container(:csv, file: 'in.csv')
|
||||||
|
validate 'a' => { format: /\\A\\d+\\z/ }
|
||||||
|
map { |r| r.merge('land' => params[:land]) }
|
||||||
|
write container(:csv, file: 'out.csv')
|
||||||
|
end.run
|
||||||
|
RUBY
|
||||||
|
_out, err, status = etl(dir, 'run', path, '--param', 'land=DE')
|
||||||
|
assert_equal 2, status.exitstatus, err
|
||||||
|
assert_equal "a,b,land\n1,x,DE\n2,y,DE\n", File.read("#{dir}/out.csv")
|
||||||
|
|
||||||
|
File.delete("#{dir}/out.csv")
|
||||||
|
out, _err, status = etl(dir, 'run', path, '--param', 'land=AT', '--dry-run', '--quiet')
|
||||||
|
assert_equal 2, status.exitstatus
|
||||||
|
refute File.exist?("#{dir}/out.csv")
|
||||||
|
assert out.start_with?('DRY RUN')
|
||||||
|
assert_includes out, '"land":"AT"'
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_run_is_optional
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
path = script(dir, 'kopie.rb', "DataPipeline.new { read container(:csv, file: 'in.csv'); write container(:csv, file: 'out.csv') }.run\n")
|
||||||
|
_out, err, status = etl(dir, path, '--quiet')
|
||||||
|
assert_equal 0, status.exitstatus, err
|
||||||
|
assert_equal File.read("#{dir}/in.csv"), File.read("#{dir}/out.csv")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_exit_code_1_on_abort_and_usage
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
path = script(dir, 'kaputt.rb', "DataPipeline.new { read container(:csv, file: 'in.csv'); map { raise 'boom' }; write container(:csv, file: 'k.csv') }.run\n")
|
||||||
|
_out, err, status = etl(dir, 'run', path, '--quiet')
|
||||||
|
assert_equal 1, status.exitstatus
|
||||||
|
assert_includes err, 'RuntimeError: boom'
|
||||||
|
|
||||||
|
_out, err, status = etl(dir, 'gibtsnicht.rb')
|
||||||
|
assert_equal 1, status.exitstatus
|
||||||
|
assert_includes err, 'gibtsnicht.rb not found'
|
||||||
|
|
||||||
|
_out, err, status = etl(dir)
|
||||||
|
assert_equal 1, status.exitstatus
|
||||||
|
assert_includes err, 'Usage: etl [run] PIPELINE.rb'
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,75 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
|
||||||
|
# chained steps: map, filter, header, trailer, context
|
||||||
|
class ChainedTest < PipelineTest
|
||||||
|
ROWS = [{ 'a' => 1 }, { 'a' => 2 }, { 'a' => 3 }].freeze
|
||||||
|
|
||||||
|
def test_chained_transform_gets_previous_output
|
||||||
|
out = output(ROWS) { chained; transform { |r| r.merge('b' => 1) }; transform { |r| r.merge('c' => 2) } }
|
||||||
|
assert_equal ROWS.map { |r| r.merge('b' => 1, 'c' => 2) }, out
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_one_to_many_then_index
|
||||||
|
out = output(ROWS) { chained; transform { |r| [r, r] }; transform { |r, i| r.merge('i' => i) } }
|
||||||
|
assert_equal (0...6).to_a, out.map { |r| r['i'] }
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_map_and_filter
|
||||||
|
out = output(ROWS) do
|
||||||
|
map { |r| r.merge('d' => r['a'] * 2) }
|
||||||
|
filter { |r| r['d'] > 2 }
|
||||||
|
map { |r, ctx| r.merge('i' => ctx.index) }
|
||||||
|
end
|
||||||
|
assert_equal [{ 'a' => 2, 'd' => 4, 'i' => 0 }, { 'a' => 3, 'd' => 6, 'i' => 1 }], out
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_map_turns_chaining_on_for_transform
|
||||||
|
out = output(ROWS) { transform { |r| r.merge('b' => 1) }; map { |r| r.merge('c' => 1) } }
|
||||||
|
assert_equal 3, out.size
|
||||||
|
assert(out.all? { |r| r['b'] && r['c'] })
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_header_and_trailer_with_context
|
||||||
|
out = output(ROWS) do
|
||||||
|
header { |c| { 'H' => c.read, 'W' => c.written } }
|
||||||
|
filter { |r| r['a'] > 1 }
|
||||||
|
map { |r| [r, r] }
|
||||||
|
trailer { |c| { 'T' => c.written } }
|
||||||
|
end
|
||||||
|
assert_equal({ 'H' => 3, 'W' => 4 }, out.first)
|
||||||
|
assert_equal({ 'T' => 4 }, out.last)
|
||||||
|
assert_equal 6, out.size
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_edge_order_follows_registration
|
||||||
|
out = output(ROWS) { header { { 'h' => 1 } }; transform(processing: :prepend) { { 'p' => 1 } }; map { |r| r } }
|
||||||
|
assert_equal [{ 'h' => 1 }, { 'p' => 1 }], out.first(2)
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_header_and_trailer_nil_and_arrays
|
||||||
|
out = output(ROWS) { header { nil }; trailer { [{ 'x' => 1 }, nil, { 'y' => 2 }] }; map { |r| r } }
|
||||||
|
assert_equal ROWS + [{ 'x' => 1 }, { 'y' => 2 }], out
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_condition_on_map
|
||||||
|
assert_equal [{ 'a' => 1 }], output(ROWS) { map(condition: ->(r) { r['a'] == 1 }) { |r| r } }
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_chained_false
|
||||||
|
assert_equal 6, output(ROWS) { chained false; transform { |r| r }; transform { |r| r } }.size
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_csv_end_to_end
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
DataPipeline.new do
|
||||||
|
logger PipelineHelper::QUIET
|
||||||
|
read MemoryStorage.new(rows: ROWS)
|
||||||
|
filter { |r| r['a'].odd? }
|
||||||
|
map { |r| { 'id' => r['a'], 'x' => 'y' } }
|
||||||
|
trailer { |c| { 'id' => 'TOTAL', 'x' => c.written } }
|
||||||
|
write container(:csv, file: "#{dir}/c.csv", columns: %w[id x])
|
||||||
|
end.run
|
||||||
|
assert_equal "id,x\n1,y\n3,y\nTOTAL,2\n", File.read("#{dir}/c.csv")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
|
||||||
|
#
|
||||||
|
# parallel mode (transform without chained/map/filter) must keep the output of v1.2.1,
|
||||||
|
# the version the CUDO importers were built on. expected values were recorded with v1.2.1.
|
||||||
|
#
|
||||||
|
class CompatTest < PipelineTest
|
||||||
|
ROWS = (1..6).map { |i| { 'id' => i, 'land' => i.even? ? 'DE' : 'AT' } }
|
||||||
|
|
||||||
|
def assert_output expected_json, &block
|
||||||
|
assert_equal expected_json, JSON.generate(output(ROWS, &block))
|
||||||
|
end
|
||||||
|
|
||||||
|
# pattern of import.customer.rb
|
||||||
|
def test_customer_pattern
|
||||||
|
assert_output '[{"SATYP":"FIRC","LFDNR":0},{"id":1,"land":"AT","lfdnr":"0000000001"},' \
|
||||||
|
'{"id":2,"land":"DE","lfdnr":"0000000002"},{"id":3,"land":"AT","lfdnr":"0000000003"},' \
|
||||||
|
'{"id":4,"land":"DE","lfdnr":"0000000004"},{"id":5,"land":"AT","lfdnr":"0000000005"},' \
|
||||||
|
'{"id":6,"land":"DE","lfdnr":"0000000006"},{"topic":"LARC","counter":"0000000006"}]' do
|
||||||
|
transform(processing: :prepend) { |_row, n| { 'SATYP': 'FIRC', 'LFDNR': n } }
|
||||||
|
transform(processing: :append) { |_row, n| { 'topic': 'LARC', 'counter': format('%010d', n) } }
|
||||||
|
transform { |row, _i| row.merge('lfdnr' => format('%010d', row['id'])) }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# pattern of import.posting.rb: closure counter, 1:n
|
||||||
|
def test_posting_pattern
|
||||||
|
expected = [{ SATYP: 'FIRC', LFDNR: 0 }]
|
||||||
|
lfdnr = 0
|
||||||
|
ROWS.each do |row|
|
||||||
|
expected << row.merge('lfdnr' => lfdnr += 1)
|
||||||
|
row['id'].times { |k| expected << { 'pos' => k, 'lfdnr' => lfdnr += 1 } }
|
||||||
|
end
|
||||||
|
expected << { SATYPE: 'LARC', SUMME: '0000000027' }
|
||||||
|
|
||||||
|
assert_output JSON.generate(expected) do
|
||||||
|
cnt = 1
|
||||||
|
transform(processing: :prepend) { |_row, _n| { 'SATYP': 'FIRC', 'LFDNR': 0 } }
|
||||||
|
transform do |row|
|
||||||
|
out = [row.merge('lfdnr' => cnt)]
|
||||||
|
cnt += 1
|
||||||
|
row['id'].times do |k|
|
||||||
|
out << { 'pos' => k, 'lfdnr' => cnt }
|
||||||
|
cnt += 1
|
||||||
|
end
|
||||||
|
out
|
||||||
|
end
|
||||||
|
transform(processing: :append) { |_row| { 'SATYPE': 'LARC', 'SUMME': format('%010d', cnt - 1) } }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_append_registered_before_prepend
|
||||||
|
assert_output '[{"h":0},{"id":1,"land":"AT"},{"id":2,"land":"DE"},{"id":3,"land":"AT"},' \
|
||||||
|
'{"id":4,"land":"DE"},{"id":5,"land":"AT"},{"id":6,"land":"DE"},{"t":6}]' do
|
||||||
|
transform(processing: :append) { |_, n| { 't' => n } }
|
||||||
|
transform { |r| r }
|
||||||
|
transform(processing: :prepend) { |_, n| { 'h' => n } }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_condition_string
|
||||||
|
assert_output '[{"id":2,"land":"DE","i":1},{"id":4,"land":"DE","i":3},{"id":6,"land":"DE","i":5}]' do
|
||||||
|
transform(condition: "row['land'] == 'DE'") { |r, i| r.merge('i' => i) }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_two_parallel_transforms
|
||||||
|
expected = ROWS.map { |r| r.merge('a' => 1) } + ROWS.map { |r| r.merge('b' => 1) }
|
||||||
|
assert_output JSON.generate(expected) do
|
||||||
|
transform { |r| r.merge('a' => 1) }
|
||||||
|
transform { |r| [r.merge('b' => 1), nil] }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_nil_and_empty_array
|
||||||
|
assert_output '[{"id":1,"land":"AT"}]' do
|
||||||
|
transform { |r| r['id'] > 3 ? nil : [] }
|
||||||
|
transform { |r| r['id'] == 1 ? r : nil }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,74 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
|
||||||
|
class LayoutTest < PipelineTest
|
||||||
|
LAYOUT = DataPipeline::Layout.new do
|
||||||
|
record 'HEAD', [['SATYP', 'AN', 4, true], ['DATUM', 'DAT', 8, true], ['ZEIT', 'TIM', 6]]
|
||||||
|
record 'POS', [['SATYP', 'AN', 4, true], ['NR', 'N', 3, true], ['TEXT', 'AN', 5], ['BETRAG', 'N', '13.2']]
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_values_in_field_order_any_key_case
|
||||||
|
assert_equal ['POS', '1', 'abc', nil], LAYOUT.values({ SATYP: 'POS', betrag: nil, 'text' => 'abc', 'NR' => '1' })
|
||||||
|
assert_equal ['HEAD', '20251103', nil], LAYOUT.values({ 'satyp' => 'HEAD', 'datum' => '20251103' })
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_unknown_field_and_record_type
|
||||||
|
assert_includes assert_raises(DataPipeline::LayoutError) { LAYOUT.values({ 'satyp' => 'POS', 'extra' => 1 }) }.message, 'extra'
|
||||||
|
assert_includes assert_raises(DataPipeline::LayoutError) { LAYOUT.values({ 'satyp' => 'XXXX' }) }.message, 'XXXX'
|
||||||
|
lenient = DataPipeline::Layout.new(strict: false) { record 'A', [['SATYP']] }
|
||||||
|
assert_equal ['A'], lenient.values({ 'satyp' => 'A', 'extra' => 1 })
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_type_field
|
||||||
|
layout = DataPipeline::Layout.new(type_field: 'art') { record 'X', [['ID'], ['ART']] }
|
||||||
|
assert_equal %w[7 X], layout.values({ 'id' => '7', 'art' => 'X' })
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_errors
|
||||||
|
assert_empty LAYOUT.errors({ 'satyp' => 'POS', 'nr' => '001', 'text' => 'abcde', 'betrag' => '-123.45' })
|
||||||
|
assert_equal ['NR: missing', 'TEXT: longer than 5', 'BETRAG: more than 2 decimals'],
|
||||||
|
LAYOUT.errors({ 'satyp' => 'POS', 'text' => 'abcdef', 'betrag' => '1.234' })
|
||||||
|
assert_equal ['NR: more than 3 digits', 'BETRAG: not a number'], LAYOUT.errors({ 'satyp' => 'POS', 'nr' => '1000', 'betrag' => '1,5' })
|
||||||
|
assert_equal ['DATUM: not a date YYYYMMDD', 'ZEIT: not a time HHMMSS'], LAYOUT.errors({ 'satyp' => 'HEAD', 'datum' => '20251331', 'zeit' => '250000' })
|
||||||
|
assert_equal ['record type "?" not in layout (HEAD, POS)'], LAYOUT.errors({ 'satyp' => '?' })
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_csv_with_layout
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
rows = [{ SATYP: 'HEAD', DATUM: '20251103' }, { 'satyp' => 'POS', 'betrag' => '1.50', 'nr' => '1' }]
|
||||||
|
DataPipeline::CsvStorage.new(file: "#{dir}/l.ok", layout: LAYOUT, csv: { col_sep: '|', quote_char: '' }).write(rows)
|
||||||
|
assert_equal "HEAD|20251103|\nPOS|1||1.50\n", File.read("#{dir}/l.ok")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_validate_layout_abort_warn_and_ignore
|
||||||
|
rows = [{ 'satyp' => 'POS', 'nr' => '1' }, { 'satyp' => 'POS', 'text' => 'zu lang' }]
|
||||||
|
assert_raises(DataPipeline::ValidationError) { run_pipeline(rows) { validate LAYOUT } }
|
||||||
|
|
||||||
|
target, stats = run_pipeline(rows) { validate LAYOUT, warn: true }
|
||||||
|
assert_equal 2, target.output.size
|
||||||
|
assert_equal 1, stats.warnings
|
||||||
|
assert_match(/warnings=1/, stats.to_s)
|
||||||
|
|
||||||
|
_target, stats = run_pipeline(rows) { validate LAYOUT, warn: true, ignore: %w[NR TEXT] }
|
||||||
|
assert_equal 0, stats.warnings
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_warning_log_is_capped
|
||||||
|
log = StringIO.new
|
||||||
|
stats = DataPipeline.new do
|
||||||
|
logger Logger.new(log)
|
||||||
|
read MemoryStorage.new(rows: Array.new(150) { { 'satyp' => 'POS' } })
|
||||||
|
write MemoryStorage.new
|
||||||
|
validate LAYOUT, warn: true
|
||||||
|
end.run
|
||||||
|
assert_equal 150, stats.warnings
|
||||||
|
assert_equal 101, log.string.lines.count { |l| l.include?('WARN') }
|
||||||
|
assert_includes log.string, 'more than 100 warnings'
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_validate_hash_without_braces
|
||||||
|
target, stats = run_pipeline([{ 'a' => 'x' }, { 'a' => '1' }]) { on_error :skip; validate 'a' => { format: /\A\d\z/ } }
|
||||||
|
assert_equal [{ 'a' => '1' }], target.output
|
||||||
|
assert_equal 1, stats.rejected
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,178 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
|
||||||
|
# on_error, validate, stats, hooks, watermark, dry run, limit
|
||||||
|
class OperationsTest < PipelineTest
|
||||||
|
ROWS = (1..5).map { |i| { 'id' => i.to_s, 'shkzg' => i == 3 ? 'X' : 'S', 'betrag' => i == 4 ? 'abc' : "#{i}.50" } }.freeze
|
||||||
|
RULES = { 'id' => { required: true, format: /\A\d+\z/ }, 'shkzg' => { in: %w[S H] }, 'betrag' => { type: :decimal } }.freeze
|
||||||
|
|
||||||
|
def failing_map = proc { map { |r| r['id'] == '2' ? raise('kaputt') : r } }
|
||||||
|
|
||||||
|
def test_abort_is_default
|
||||||
|
error = assert_raises(RuntimeError) { run_pipeline(ROWS, &failing_map) }
|
||||||
|
assert_equal 'kaputt', error.message
|
||||||
|
assert_equal 'kaputt', DataPipeline.runs.last.error.message
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_skip
|
||||||
|
map = failing_map
|
||||||
|
target, stats = run_pipeline(ROWS) { on_error :skip; instance_eval(&map) }
|
||||||
|
assert_equal 4, target.output.size
|
||||||
|
assert_equal 1, stats.rejected
|
||||||
|
assert_equal 'map#1', stats.errors.first[:step]
|
||||||
|
refute stats.ok?
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_collect_batch_and_stream
|
||||||
|
[false, true].each do |streamed|
|
||||||
|
rejects = MemoryStorage.new
|
||||||
|
map = failing_map
|
||||||
|
target, = run_pipeline(ROWS, streamed:) { on_error :collect, to: rejects; instance_eval(&map) }
|
||||||
|
assert_equal [ROWS[1].merge('_step' => 'map#1', '_error' => 'kaputt')], rejects.streamed_rows
|
||||||
|
assert_equal 4, target.output.size
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_on_error_arguments
|
||||||
|
assert_raises(ArgumentError) { DataPipeline.new { on_error :collect } }
|
||||||
|
assert_raises(ArgumentError) { DataPipeline.new { on_error :ignore } }
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_edge_errors_always_abort
|
||||||
|
assert_raises(RuntimeError) { run_pipeline(ROWS) { on_error :skip; header { raise 'h' }; map { |r| r } } }
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_skip_in_parallel_mode
|
||||||
|
target, stats = run_pipeline(ROWS) { on_error :skip; transform { |r| r['id'] == '1' ? raise('x') : r } }
|
||||||
|
assert_equal 4, target.output.size
|
||||||
|
assert_equal 1, stats.rejected
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_validate_rules
|
||||||
|
assert_raises(DataPipeline::ValidationError) { run_pipeline(ROWS) { validate RULES } }
|
||||||
|
target, stats = run_pipeline(ROWS) { on_error :skip; validate RULES }
|
||||||
|
assert_equal %w[1 2 5], target.output.map { |r| r['id'] }
|
||||||
|
assert_equal 2, stats.rejected
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_validate_block
|
||||||
|
target, stats = run_pipeline(ROWS) { on_error :skip; validate { |r| 'zu gross' if r['id'].to_i > 3 } }
|
||||||
|
assert_equal 3, target.output.size
|
||||||
|
assert_equal 'zu gross', stats.errors.first[:error]
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_validation_messages
|
||||||
|
errors = DataPipeline::Validation.errors({ 'a' => '12345', 'd' => '2025-13-01' },
|
||||||
|
'a' => { max: 3, format: /x/ }, 'd' => { type: :date }, 'b' => { required: true })
|
||||||
|
assert_equal ['a: longer than 3', 'a: format /x/', 'd: not date', 'b: missing'], errors
|
||||||
|
assert_empty DataPipeline::Validation.errors({ 'a' => '' }, 'a' => { max: 1, type: :integer })
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_validate_turns_chaining_on
|
||||||
|
target, = run_pipeline(ROWS) { on_error :skip; transform { |r| r.merge('x' => 1) }; validate RULES }
|
||||||
|
assert_equal 3, target.output.size
|
||||||
|
assert(target.output.all? { |r| r['x'] })
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_stats
|
||||||
|
_target, stats = run_pipeline(ROWS) { map { |r| [r, r] } }
|
||||||
|
assert_equal [5, 10, 10], [stats.read, stats.written, stats.output]
|
||||||
|
assert stats.ok?
|
||||||
|
assert_kind_of Float, stats.duration
|
||||||
|
map = failing_map
|
||||||
|
_target, stats = run_pipeline(ROWS) { on_error :skip; instance_eval(&map) }
|
||||||
|
assert_match(/\Aread=5 written=4 rejected=1/, stats.to_s)
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_on_finish_after_success_and_abort
|
||||||
|
seen = []
|
||||||
|
run_pipeline(ROWS) { on_finish { |s| seen << s.error } }
|
||||||
|
assert_raises(RuntimeError) { run_pipeline(ROWS) { on_finish { |s| seen << s.error.class }; map { raise 'x' } } }
|
||||||
|
assert_equal [nil, RuntimeError], seen
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_hook_error_is_only_logged
|
||||||
|
log = StringIO.new
|
||||||
|
stats = DataPipeline.new do
|
||||||
|
logger Logger.new(log)
|
||||||
|
read MemoryStorage.new(rows: ROWS)
|
||||||
|
write MemoryStorage.new
|
||||||
|
on_finish { raise 'hook' }
|
||||||
|
end.run
|
||||||
|
assert stats.ok?
|
||||||
|
assert_includes log.string, 'on_finish: RuntimeError: hook'
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_logger_messages
|
||||||
|
log = StringIO.new
|
||||||
|
DataPipeline.new do
|
||||||
|
logger Logger.new(log)
|
||||||
|
on_error :skip
|
||||||
|
read MemoryStorage.new(rows: ROWS)
|
||||||
|
write MemoryStorage.new
|
||||||
|
map { |r| r['id'] == '1' ? raise('weg') : r }
|
||||||
|
end.run
|
||||||
|
assert_includes log.string, 'rejected by map#1: weg'
|
||||||
|
assert_includes log.string, 'finish read=5 written=4 rejected=1'
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_watermark
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
file = "#{dir}/state.json"
|
||||||
|
wm = nil
|
||||||
|
run_pipeline(ROWS) { wm = watermark :t, column: 'id', file:, initial: 0 }
|
||||||
|
assert_equal '5', wm.value
|
||||||
|
|
||||||
|
run_pipeline([{ 'id' => '9' }, { 'id' => '10' }]) { watermark :t, column: 'id', file: }
|
||||||
|
assert_equal '10', JSON.parse(File.read(file))['t'], 'numeric texts compared as numbers'
|
||||||
|
|
||||||
|
assert_raises(RuntimeError) { run_pipeline([{ 'id' => '99' }]) { watermark :t, column: 'id', file:; map { raise 'x' } } }
|
||||||
|
assert_equal '10', JSON.parse(File.read(file))['t'], 'not saved after abort'
|
||||||
|
|
||||||
|
run_pipeline(ROWS, streamed: true) { watermark :s, column: 'id', file: }
|
||||||
|
assert_equal({ 't' => '10', 's' => '5' }, JSON.parse(File.read(file)))
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_dry_run
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
rejects = MemoryStorage.new
|
||||||
|
target = nil
|
||||||
|
text = capture_stdout do
|
||||||
|
target, = run_pipeline(ROWS, dry_run: true, preview: 2) do
|
||||||
|
watermark :d, column: 'id', file: "#{dir}/state.json"
|
||||||
|
on_error :collect, to: rejects
|
||||||
|
map { |r| r['id'] == '1' ? raise('x') : r }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
assert_nil target.written
|
||||||
|
assert_empty target.streamed_rows
|
||||||
|
assert_empty rejects.streamed_rows
|
||||||
|
refute File.exist?("#{dir}/state.json")
|
||||||
|
assert text.start_with?('DRY RUN')
|
||||||
|
assert_equal 3, text.lines.size
|
||||||
|
assert DataPipeline.runs.last.dry_run
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_dry_run_creates_no_csv_file
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
capture_stdout do
|
||||||
|
DataPipeline.new(dry_run: true) do
|
||||||
|
logger PipelineHelper::QUIET
|
||||||
|
read MemoryStorage.new(rows: ROWS)
|
||||||
|
write container(:csv, file: "#{dir}/dry.csv")
|
||||||
|
end.run
|
||||||
|
end
|
||||||
|
refute File.exist?("#{dir}/dry.csv")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_limit
|
||||||
|
target, stats = run_pipeline(ROWS, limit: 2) { map { |r| r } }
|
||||||
|
assert_equal [2, 2], [target.output.size, stats.read]
|
||||||
|
|
||||||
|
target, stats = run_pipeline(ROWS, streamed: true, limit: 3) { trailer { |c| { 'n' => c.read } } }
|
||||||
|
assert_equal({ 'n' => 3 }, target.output.last)
|
||||||
|
assert_equal 3, stats.read
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,131 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
|
||||||
|
# dsl, settings, plugin lookup, parallel mode
|
||||||
|
class PipelineCoreTest < PipelineTest
|
||||||
|
ROWS = [{ 'a' => 1 }, { 'a' => 2 }, { 'a' => 3 }].freeze
|
||||||
|
|
||||||
|
def settings_of(pipeline) = pipeline.instance_variable_get(:@settings)
|
||||||
|
|
||||||
|
def test_nil_filters_rows
|
||||||
|
assert_equal [{ 'a' => 1 }], output(ROWS) { transform { |r| r['a'] == 1 ? r : nil } }
|
||||||
|
assert_equal ROWS, output(ROWS) { transform { |r| [r, nil] } }
|
||||||
|
assert_equal ROWS, output(ROWS) { transform(processing: :prepend) { nil }; transform { |r| r }; transform(processing: :append) { nil } }
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_dsl_switches
|
||||||
|
assert_equal false, settings_of(DataPipeline.new { verbose false })[:verbose]
|
||||||
|
assert_equal true, settings_of(DataPipeline.new { verbose })[:verbose]
|
||||||
|
assert_equal false, settings_of(DataPipeline.new { streamed false })[:streamed]
|
||||||
|
assert_equal '1.0', settings_of(DataPipeline.new { version '1.0' })[:version]
|
||||||
|
assert_equal true, settings_of(DataPipeline.new { configure verbose: true })[:verbose]
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_typo_raises
|
||||||
|
assert_raises(NoMethodError) { DataPipeline.new { verbsoe true } }
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_unknown_plugin
|
||||||
|
error = assert_raises(RuntimeError) { DataPipeline.new { container :gibtsnicht } }
|
||||||
|
assert_includes error.message, 'GibtsnichtStorage'
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_plugin_errors_pass_through
|
||||||
|
assert_raises(ArgumentError) { DataPipeline.new { container :csv } }
|
||||||
|
broken = Class.new(DataPipeline::StoragePlugin) { def initialize(**) = undefined_thing }
|
||||||
|
DataPipeline.register :broken, broken
|
||||||
|
assert_includes assert_raises(NameError) { DataPipeline.new { container :broken } }.message, 'undefined_thing'
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_register_lazy_path
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
File.write("#{dir}/lazy_storage.rb", "class LazyStorage < MemoryStorage; end\n")
|
||||||
|
DataPipeline.register :lazy, 'LazyStorage', path: "#{dir}/lazy_storage.rb"
|
||||||
|
assert_empty $LOADED_FEATURES.grep(/lazy_storage/)
|
||||||
|
storage = nil
|
||||||
|
DataPipeline.new { storage = container :lazy }
|
||||||
|
assert_equal 'LazyStorage', storage.class.name
|
||||||
|
refute_empty $LOADED_FEATURES.grep(/lazy_storage/)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_csv_plugin_by_convention
|
||||||
|
assert_equal DataPipeline::CsvStorage, DataPipeline.plugin(:csv)
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_settings_are_frozen_and_replaced
|
||||||
|
assert DataPipeline.settings.frozen?
|
||||||
|
pipeline = DataPipeline.new { logger PipelineHelper::QUIET }
|
||||||
|
DataPipeline.settings(verbose: true)
|
||||||
|
assert_equal false, settings_of(pipeline)[:verbose]
|
||||||
|
assert_equal true, DataPipeline.settings[:verbose]
|
||||||
|
DataPipeline.reset_settings!
|
||||||
|
assert_equal DataPipeline::DEFAULT_SETTINGS, DataPipeline.settings
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_settings_per_pipeline
|
||||||
|
_target, stats = run_pipeline(ROWS, limit: 1)
|
||||||
|
assert_equal 1, stats.read
|
||||||
|
assert_nil DataPipeline.settings[:limit]
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_parallel_threads
|
||||||
|
reads = Array.new(8) { |i| Thread.new { run_pipeline(ROWS, limit: (i % 3) + 1).last.read } }.map(&:value)
|
||||||
|
assert_equal Array.new(8) { |i| (i % 3) + 1 }, reads
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_parallel_mode_is_side_by_side
|
||||||
|
out = output(ROWS) { transform { |r| r.merge('b' => 1) }; transform { |r| r.merge('c' => 2) } }
|
||||||
|
assert_equal 6, out.size
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_parallel_mode_copies_rows_per_step
|
||||||
|
out = output(ROWS) { transform { |r| r['x'] = 1; r }; transform { |r| r.key?('x') ? nil : r } }
|
||||||
|
assert_equal 6, out.size
|
||||||
|
assert_equal [{ 'a' => 10 }, { 'a' => 20 }, { 'a' => 30 }], output(ROWS) { transform { |r| [r.transform_values! { |v| v * 10 }] } }
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_stable_order_and_all_prepends_appends
|
||||||
|
out = output(ROWS) { 20.times { |k| transform { |r| r['a'] == 1 ? { 'k' => k } : nil } } }
|
||||||
|
assert_equal (0...20).to_a, out.map { |r| r['k'] }
|
||||||
|
|
||||||
|
out = output(ROWS) do
|
||||||
|
transform(processing: :prepend) { { 'h' => 1 } }
|
||||||
|
transform(processing: :prepend) { { 'h' => 2 } }
|
||||||
|
transform { |r| r }
|
||||||
|
transform(processing: :append) { { 't' => 1 } }
|
||||||
|
transform(processing: :append) { { 't' => 2 } }
|
||||||
|
end
|
||||||
|
assert_equal [{ 'h' => 1 }, { 'h' => 2 }, *ROWS, { 't' => 1 }, { 't' => 2 }], out
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_append_count_parallel_vs_chained
|
||||||
|
assert_equal({ 'n' => 3 }, output(ROWS) { transform { |r| [r, r] }; transform(processing: :append) { |_, n| { 'n' => n } } }.last)
|
||||||
|
assert_equal({ 'n' => 6 }, output(ROWS) { chained; transform { |r| [r, r] }; transform(processing: :append) { |_, n| { 'n' => n } } }.last)
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_conditions
|
||||||
|
assert_equal [{ 'a' => 1 }, { 'a' => 3 }], output(ROWS) { transform(condition: ->(r) { r['a'] != 2 }) { |r| r } }
|
||||||
|
assert_equal [{ 'a' => 2 }], output(ROWS) { transform(condition: "row['a'] == 2") { |r| r } }
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_copy_without_steps
|
||||||
|
assert_equal ROWS, output(ROWS)
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_closes_source_and_target
|
||||||
|
source = MemoryStorage.new(rows: ROWS)
|
||||||
|
target = MemoryStorage.new
|
||||||
|
DataPipeline.new { logger PipelineHelper::QUIET; read source; write target }.run
|
||||||
|
assert source.closed?
|
||||||
|
assert target.closed?
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_read_and_write_required
|
||||||
|
assert_includes assert_raises(ArgumentError) { DataPipeline.new { logger PipelineHelper::QUIET; read MemoryStorage.new }.run }.message, 'write'
|
||||||
|
assert_includes assert_raises(ArgumentError) { DataPipeline.new { logger PipelineHelper::QUIET; write MemoryStorage.new }.run }.message, 'read'
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_block_required
|
||||||
|
assert_raises(ArgumentError) { DataPipeline.new { map } }
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,153 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
|
||||||
|
# streamed mode, lookups, several targets
|
||||||
|
class StreamingTest < PipelineTest
|
||||||
|
ROWS = [{ 'a' => 1 }, { 'a' => 2 }, { 'a' => 3 }].freeze
|
||||||
|
|
||||||
|
CHAIN = proc do
|
||||||
|
header { |c| { 'H' => c.written } }
|
||||||
|
filter { |r| r['a'] > 1 }
|
||||||
|
map { |r, c| [r, r.merge('k' => c.index)] }
|
||||||
|
map { |r, c| r.merge('i' => c.index) }
|
||||||
|
trailer { |c| { 'T' => c.written, 'R' => c.read } }
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_batch_and_stream_same_body
|
||||||
|
batch = output(ROWS, &CHAIN)
|
||||||
|
stream = output(ROWS, streamed: true, &CHAIN)
|
||||||
|
assert_equal batch[1..-2], stream[1..-2]
|
||||||
|
assert_equal({ 'H' => 4 }, batch.first)
|
||||||
|
assert_equal({ 'H' => nil }, stream.first)
|
||||||
|
assert_equal({ 'T' => 4, 'R' => 3 }, stream.last)
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_stream_prepend_append
|
||||||
|
out = output(ROWS, streamed: true) do
|
||||||
|
chained
|
||||||
|
transform(processing: :prepend) { |_, n| { 'p' => n } }
|
||||||
|
transform { |r| [r, r] }
|
||||||
|
transform(processing: :append) { |_, n| { 'n' => n } }
|
||||||
|
end
|
||||||
|
assert_equal({ 'p' => 0 }, out.first)
|
||||||
|
assert_equal({ 'n' => 6 }, out.last)
|
||||||
|
assert_equal 8, out.size
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_stream_parallel_single_step_and_error_for_several
|
||||||
|
out = output(ROWS, streamed: true) { transform { |r| [r, r] }; transform(processing: :append) { |_, n| { 'n' => n } } }
|
||||||
|
assert_equal({ 'n' => 3 }, out.last)
|
||||||
|
error = assert_raises(ArgumentError) { output(ROWS, streamed: true) { transform { |r| r }; transform { |r| r } } }
|
||||||
|
assert_includes error.message, 'chained'
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_stream_condition_and_copy
|
||||||
|
assert_equal [{ 'a' => 1 }, { 'a' => 3 }], output(ROWS, streamed: true) { map(condition: ->(r) { r['a'] != 2 }) { |r| r } }
|
||||||
|
assert_equal ROWS, output(ROWS, streamed: true)
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_stream_csv_to_csv
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
File.write("#{dir}/in.csv", "a,b\n1,x\n2,y\n3,z\n")
|
||||||
|
DataPipeline.new do
|
||||||
|
logger PipelineHelper::QUIET
|
||||||
|
streamed
|
||||||
|
read container(:csv, file: "#{dir}/in.csv")
|
||||||
|
filter { |r| r['a'] != '2' }
|
||||||
|
trailer { |c| { 'a' => 'T', 'b' => c.written } }
|
||||||
|
write container(:csv, file: "#{dir}/out.csv")
|
||||||
|
end.run
|
||||||
|
assert_equal "a,b\n1,x\n3,z\nT,2\n", File.read("#{dir}/out.csv")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_stream_postgres_with_query_on_same_connection
|
||||||
|
with_pg_database do |db|
|
||||||
|
pg_exec db, "create table k as select i as id, case when i % 2 = 0 then 'DE' else 'AT' end as land from generate_series(1, 5) i;
|
||||||
|
create table l (land text, name text); insert into l values ('DE','Deutschland'),('AT','Oesterreich')"
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
DataPipeline.new do
|
||||||
|
logger PipelineHelper::QUIET
|
||||||
|
streamed
|
||||||
|
pg = container :postgres, **db
|
||||||
|
read pg, query: 'select id, land from k where id <= $1 order by id', params: [5], fetch_size: 2
|
||||||
|
map { |r| r.merge('name' => query(pg, query: 'select name from l where land = $1', params: [r['land']]).first['name']) }
|
||||||
|
write container(:csv, file: "#{dir}/o.csv", columns: %w[id name])
|
||||||
|
end.run
|
||||||
|
assert_equal 6, File.readlines("#{dir}/o.csv").size
|
||||||
|
assert_includes File.read("#{dir}/o.csv"), '2,Deutschland'
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_lookup_grouped_unique_and_lambda
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
File.write("#{dir}/lk.csv", "id,grp\n1,a\n2,b\n3,a\n")
|
||||||
|
grouped = unique = by_lambda = nil
|
||||||
|
DataPipeline.new do
|
||||||
|
csv = container(:csv, file: "#{dir}/lk.csv")
|
||||||
|
grouped = lookup csv, key: 'grp'
|
||||||
|
unique = lookup csv, key: 'id', unique: true
|
||||||
|
by_lambda = lookup csv, key: ->(r) { r['grp'].upcase }
|
||||||
|
end
|
||||||
|
assert_equal 2, grouped['a'].size
|
||||||
|
assert_equal 'b', unique['2']['grp']
|
||||||
|
assert_equal %w[A B], by_lambda.keys.sort
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_lookup_postgres_join_without_n_plus_one
|
||||||
|
with_pg_database do |db|
|
||||||
|
pg_exec db, "create table k as select i as id, case when i % 2 = 0 then 'DE' else 'AT' end as land from generate_series(1, 3) i;
|
||||||
|
create table l (land text, name text); insert into l values ('DE','Deutschland'),('AT','Oesterreich')"
|
||||||
|
out = nil
|
||||||
|
DataPipeline.new do
|
||||||
|
logger PipelineHelper::QUIET
|
||||||
|
pg = container :postgres, **db
|
||||||
|
names = lookup pg, query: 'select * from l where land like $1', params: ['%'], key: 'land', unique: true
|
||||||
|
read pg, query: 'select id, land from k order by id'
|
||||||
|
map { |r| r.merge('name' => names.dig(r['land'], 'name')) }
|
||||||
|
write(out = MemoryStorage.new)
|
||||||
|
end.run
|
||||||
|
assert_equal %w[Oesterreich Deutschland Oesterreich], out.output.map { |r| r['name'] }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_several_targets_with_only
|
||||||
|
rows = [{ 'satyp' => 'DOHD', 'n' => 1 }, { 'satyp' => 'DOPS', 'n' => 2 }, { 'satyp' => 'DOPS', 'n' => 3 }]
|
||||||
|
[false, true].each do |streamed|
|
||||||
|
all = MemoryStorage.new
|
||||||
|
heads = MemoryStorage.new
|
||||||
|
positions = MemoryStorage.new
|
||||||
|
trailers = MemoryStorage.new
|
||||||
|
DataPipeline.new(streamed:) do
|
||||||
|
logger PipelineHelper::QUIET
|
||||||
|
read MemoryStorage.new(rows:)
|
||||||
|
header { { 'satyp' => 'FIRC' } }
|
||||||
|
map { |r| r }
|
||||||
|
trailer { |c| { 'satyp' => 'LARC', 'n' => c.written } }
|
||||||
|
write all
|
||||||
|
write heads, only: ->(r) { r['satyp'] == 'DOHD' }
|
||||||
|
write positions, only: ->(r) { r['satyp'] == 'DOPS' }
|
||||||
|
write trailers, only: ->(r) { r['satyp'] == 'LARC' }
|
||||||
|
end.run
|
||||||
|
assert_equal 5, all.output.size, "streamed=#{streamed}"
|
||||||
|
assert_equal [1], heads.output.map { |r| r['n'] }
|
||||||
|
assert_equal 2, positions.output.size
|
||||||
|
assert_equal [{ 'satyp' => 'LARC', 'n' => 3 }], trailers.output
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_memory_stays_flat_when_streaming
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
File.open("#{dir}/big.csv", 'w') { |f| f.puts 'id,text'; 50_000.times { |i| f.puts "#{i},#{'x' * 80}" } }
|
||||||
|
stats = DataPipeline.new(streamed: true) do
|
||||||
|
logger PipelineHelper::QUIET
|
||||||
|
read container(:csv, file: "#{dir}/big.csv")
|
||||||
|
map { |r| r.merge('len' => r['text'].size) }
|
||||||
|
write container(:csv, file: "#{dir}/out.csv", columns: %w[id len])
|
||||||
|
end.run
|
||||||
|
assert_equal 50_000, stats.written
|
||||||
|
assert_equal 50_001, File.foreach("#{dir}/out.csv").count
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,155 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
require_relative '../../examples/demo_api'
|
||||||
|
require 'open3'
|
||||||
|
require 'fileutils'
|
||||||
|
|
||||||
|
#
|
||||||
|
# the examples of README.md section 1: the code in the documentation is the file,
|
||||||
|
# and every example runs and produces the documented result
|
||||||
|
#
|
||||||
|
class DocExamplesTest < PipelineTest
|
||||||
|
DOC = File.read(File.join(ROOT, 'README.md'))
|
||||||
|
EXAMPLES = DOC.scan(/<!-- beispiel: (\S+) -->\n```ruby\n(.*?)\n```/m)
|
||||||
|
|
||||||
|
def run_example dir, name, *args
|
||||||
|
env = { 'BUNDLE_GEMFILE' => File.join(ROOT, 'Gemfile') }
|
||||||
|
Open3.capture3(env, RbConfig.ruby, File.join(ROOT, 'bin/etl'), 'run', File.join(ROOT, name), "--out=#{dir}", *args,
|
||||||
|
chdir: dir)
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_documented_code_equals_files
|
||||||
|
assert_equal %w[examples/kunden_import.rb examples/rechnungen_satzarten.rb examples/bewegungen_inkrementell.rb
|
||||||
|
examples/eigenes_plugin.rb examples/rest_lesen.rb examples/rest_schreiben.rb
|
||||||
|
examples/rest_zu_rest.rb], EXAMPLES.map(&:first)
|
||||||
|
EXAMPLES.each do |file, code|
|
||||||
|
assert_equal File.read(File.join(ROOT, file)).rstrip, code.rstrip, "#{file} differs from README.md"
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_kunden_import
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
out, err, status = run_example(dir, 'examples/kunden_import.rb')
|
||||||
|
assert_equal 2, status.exitstatus, err
|
||||||
|
assert_includes out, 'kunden_import: read=9 written=4 rejected=4'
|
||||||
|
|
||||||
|
db = DataPipeline::SqliteStorage.new(database: "#{dir}/kunden.sqlite3", query: 'select * from kunden order by kundennr')
|
||||||
|
rows = db.read
|
||||||
|
db.close
|
||||||
|
assert_equal %w[100001 100002 100003 100009], rows.map { |r| r['kundennr'] }
|
||||||
|
assert_equal({ 'kundennr' => '100002', 'name' => 'Huber & Partner', 'land' => 'Österreich', 'eu' => 'ja', 'umsatz' => 8300.0 }, rows[1])
|
||||||
|
|
||||||
|
excel = DataPipeline::ExcelStorage.new(file: "#{dir}/eu_kunden.xlsx", sheet: 'EU-Kunden').read
|
||||||
|
assert_equal %w[100001 100002 100009], excel.map { |r| r['kundennr'] }
|
||||||
|
|
||||||
|
errors = DataPipeline::CsvStorage.new(file: "#{dir}/kunden_fehler.csv").read
|
||||||
|
assert_equal ['name: missing', 'kundennr: format /\A\d{6}\z/', 'Land XX unbekannt', 'umsatz: not decimal'],
|
||||||
|
errors.map { |r| r['_error'] }
|
||||||
|
|
||||||
|
# a second run replaces the table content (truncate)
|
||||||
|
run_example(dir, 'examples/kunden_import.rb')
|
||||||
|
db = DataPipeline::SqliteStorage.new(database: "#{dir}/kunden.sqlite3", query: 'select count(*) as n from kunden')
|
||||||
|
assert_equal 4, db.read.first['n']
|
||||||
|
db.close
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_kunden_import_dry_run_creates_no_files
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
out, _err, status = run_example(dir, 'examples/kunden_import.rb', '--dry-run')
|
||||||
|
assert_equal 2, status.exitstatus
|
||||||
|
assert_includes out, 'DRY RUN'
|
||||||
|
assert_empty Dir.children(dir)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_rechnungen_satzarten
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
_out, err, status = run_example(dir, 'examples/rechnungen_satzarten.rb')
|
||||||
|
assert status.success?, err
|
||||||
|
assert_includes err, 'TEXT: longer than 30'
|
||||||
|
documented = DOC[/Ergebnis `tmp\/rechnungen.txt`.*?```text\n(.*?)```/m, 1]
|
||||||
|
expected = documented.sub(/\ADATEI\|\d{8}/, "DATEI|#{Date.today.strftime('%Y%m%d')}")
|
||||||
|
assert_equal expected, File.read("#{dir}/rechnungen.txt")
|
||||||
|
assert_equal 3, File.readlines("#{dir}/positionen.ndjson").size
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_bewegungen_inkrementell
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
out, err, status = run_example(dir, 'examples/bewegungen_inkrementell.rb')
|
||||||
|
assert status.success?, err
|
||||||
|
assert_includes out, 'neu: 4, Stand jetzt: 4'
|
||||||
|
file = Dir["#{dir}/bewegungen_*.ndjson"].first
|
||||||
|
assert_equal({ 'id' => 2, 'konto' => '4000', 'betrag' => 20.0, 'soll_haben' => 'H' }, JSON.parse(File.readlines(file)[1]))
|
||||||
|
|
||||||
|
FileUtils.rm(file)
|
||||||
|
out, = run_example(dir, 'examples/bewegungen_inkrementell.rb')
|
||||||
|
assert_includes out, 'neu: 0, Stand jetzt: 4'
|
||||||
|
assert_empty Dir["#{dir}/bewegungen_*.ndjson"], 'no file without new rows'
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_eigenes_plugin
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
_out, err, status = run_example(dir, 'examples/eigenes_plugin.rb')
|
||||||
|
assert status.success?, err
|
||||||
|
documented = DOC[/Ergebnis `tmp\/alle_filialen.csv`:\n\n```text\n(.*?)```/m, 1]
|
||||||
|
assert_equal documented, File.read("#{dir}/alle_filialen.csv")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def with_demo_api
|
||||||
|
api = DemoApi.start(port: 0)
|
||||||
|
yield api
|
||||||
|
ensure
|
||||||
|
api&.stop
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_rest_lesen
|
||||||
|
with_demo_api do |api|
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
out, err, status = run_example(dir, 'examples/rest_lesen.rb', "--url=#{api.url}")
|
||||||
|
assert_equal 2, status.exitstatus, err
|
||||||
|
assert_includes out, 'rest_lesen: read=7 written=6 rejected=1'
|
||||||
|
documented = DOC[/Ergebnis `tmp\/api_kunden.csv`:\n\n```text\n(.*?)```/m, 1]
|
||||||
|
assert_equal documented, File.read("#{dir}/api_kunden.csv")
|
||||||
|
assert_equal 6, JSON.parse(File.read("#{dir}/api_kunden.json")).size
|
||||||
|
assert_equal ['umsatz: not decimal'], DataPipeline::CsvStorage.new(file: "#{dir}/api_fehler.csv").read.map { |r| r['_error'] }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_rest_schreiben_and_dry_run
|
||||||
|
with_demo_api do |api|
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
out, _err, status = run_example(dir, 'examples/rest_schreiben.rb', "--url=#{api.url}", '--dry-run')
|
||||||
|
assert_equal 2, status.exitstatus
|
||||||
|
assert_includes out, 'DRY RUN'
|
||||||
|
assert_empty api.imported, 'dry run sends nothing'
|
||||||
|
|
||||||
|
out, err, status = run_example(dir, 'examples/rest_schreiben.rb', "--url=#{api.url}")
|
||||||
|
assert_equal 2, status.exitstatus, err
|
||||||
|
assert_includes out, 'gesendet: 6 Kunden in 3 Paketen'
|
||||||
|
assert_equal %w[100001 100002 100003 100007 100008 100009], api.imported.map { |r| r['kundennr'] }
|
||||||
|
assert_equal 2, DataPipeline::CsvStorage.new(file: "#{dir}/import_fehler.csv").read.size
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_rest_zu_rest
|
||||||
|
with_demo_api do |api|
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
out, err, status = run_example(dir, 'examples/rest_zu_rest.rb', "--url=#{api.url}")
|
||||||
|
assert status.success?, err
|
||||||
|
assert_includes out, 'archiviert: 5, Stand jetzt: 5'
|
||||||
|
assert_equal %w[1 2 3 4 5], api.archive.keys
|
||||||
|
assert_equal Date.today.iso8601, api.archive['3']['archiviert']
|
||||||
|
|
||||||
|
api.archive.clear
|
||||||
|
out, = run_example(dir, 'examples/rest_zu_rest.rb', "--url=#{api.url}")
|
||||||
|
assert_includes out, 'archiviert: 0, Stand jetzt: 5'
|
||||||
|
assert_empty api.archive
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,89 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
require_relative '../support/storage_contract'
|
||||||
|
|
||||||
|
class CsvStorageTest < PipelineTest
|
||||||
|
include StorageContract
|
||||||
|
|
||||||
|
def contract_storage(dir, _purpose) = DataPipeline::CsvStorage.new(file: "#{dir}/data.csv")
|
||||||
|
|
||||||
|
def test_read_with_defaults
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
File.write("#{dir}/in.csv", "a,b\n1,x\n2,\"\"\n")
|
||||||
|
assert_equal [{ 'a' => '1', 'b' => 'x' }, { 'a' => '2', 'b' => '' }], DataPipeline::CsvStorage.new(file: "#{dir}/in.csv").read
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_read_without_header_uses_columns_or_index
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
File.write("#{dir}/in.csv", "1|x\n2|y\n")
|
||||||
|
csv = { headers: false, col_sep: '|' }
|
||||||
|
assert_equal [{ 'id' => '1', 'txt' => 'x' }, { 'id' => '2', 'txt' => 'y' }],
|
||||||
|
DataPipeline::CsvStorage.new(file: "#{dir}/in.csv", columns: %w[id txt], csv:).read
|
||||||
|
assert_equal({ '0' => '1', '1' => 'x' }, DataPipeline::CsvStorage.new(file: "#{dir}/in.csv", csv:).read.first)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_read_streaming_with_block
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
File.write("#{dir}/in.csv", "a\n1\n2\n")
|
||||||
|
rows = []
|
||||||
|
DataPipeline::CsvStorage.new(file: "#{dir}/in.csv").read { |row| rows << row }
|
||||||
|
assert_equal %w[1 2], rows.map { |r| r['a'] }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_defaults_are_merged
|
||||||
|
options = DataPipeline::CsvStorage.new(file: 'x', csv: { col_sep: ';' }).instance_variable_get(:@options)[:csv]
|
||||||
|
assert_equal ';', options[:col_sep]
|
||||||
|
assert options[:headers]
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_file_is_required
|
||||||
|
error = assert_raises(ArgumentError) { DataPipeline::CsvStorage.new }
|
||||||
|
assert_includes error.message, 'file'
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_write_without_csv_options
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
DataPipeline::CsvStorage.new(file: "#{dir}/o.csv").write([{ 'a' => 1, 'b' => 2 }])
|
||||||
|
assert_equal "a,b\n1,2\n", File.read("#{dir}/o.csv")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_columns_fix_order_missing_fields_and_key_types
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
DataPipeline::CsvStorage.new(file: "#{dir}/o.csv", columns: %w[a b c]).write([{ 'b' => 2, 'a' => 1 }, { c: 3, a: 4 }])
|
||||||
|
assert_equal "a,b,c\n1,2,\n4,,3\n", File.read("#{dir}/o.csv")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_empty_data
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
DataPipeline::CsvStorage.new(file: "#{dir}/a.csv").write([])
|
||||||
|
DataPipeline::CsvStorage.new(file: "#{dir}/b.csv", columns: %w[a b]).write([])
|
||||||
|
assert_equal '', File.read("#{dir}/a.csv")
|
||||||
|
assert_equal "a,b\n", File.read("#{dir}/b.csv")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_write_row_and_close
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
csv = DataPipeline::CsvStorage.new(file: "#{dir}/o.csv")
|
||||||
|
csv.write_row({ 'a' => 1 })
|
||||||
|
csv.write_row({ 'a' => 2 })
|
||||||
|
csv.close
|
||||||
|
assert_equal "a\n1\n2\n", File.read("#{dir}/o.csv")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# options of the CUDO exports: no header, pipe, no quotes, mixed record types and key types
|
||||||
|
def test_cudo_format_mixed_record_types
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
rows = [{ SATYP: 'FIRC', LFDNR: 0 }, { 'satyp' => 'DOHD', 'belnr' => 'AR1', 'wwert' => nil },
|
||||||
|
{ 'satyp' => 'DOPS', 'dmbtr' => '1.50' }, { 'SATYPE' => 'LARC', 'SUMME' => '0000000002' }]
|
||||||
|
DataPipeline::CsvStorage.new(file: "#{dir}/c.ok", csv: { headers: false, col_sep: '|', quote_char: '', encoding: 'utf-8' })
|
||||||
|
.write(rows)
|
||||||
|
assert_equal "FIRC|0\nDOHD|AR1|\nDOPS|1.50\nLARC|0000000002\n", File.read("#{dir}/c.ok")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,57 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
require_relative '../support/storage_contract'
|
||||||
|
|
||||||
|
class ExcelStorageTest < PipelineTest
|
||||||
|
include StorageContract
|
||||||
|
|
||||||
|
def contract_storage(dir, _purpose) = DataPipeline::ExcelStorage.new(file: "#{dir}/data.xlsx")
|
||||||
|
|
||||||
|
# an empty text is an empty cell in excel
|
||||||
|
def contract_rows(rows) = rows.map { |row| row.transform_values { |v| v == '' ? nil : v } }
|
||||||
|
|
||||||
|
def test_text_stays_text_numbers_and_dates_keep_type
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
rows = [{ 'lfdnr' => '0000000001', 'betrag' => 1.5, 'menge' => 3, 'datum' => Date.new(2025, 11, 3), 'text' => '=1+1' }]
|
||||||
|
DataPipeline::ExcelStorage.new(file: "#{dir}/t.xlsx").write(rows)
|
||||||
|
row = DataPipeline::ExcelStorage.new(file: "#{dir}/t.xlsx").read.first
|
||||||
|
assert_equal '0000000001', row['lfdnr']
|
||||||
|
assert_in_delta 1.5, row['betrag']
|
||||||
|
assert_equal 3, row['menge'].to_i
|
||||||
|
assert_equal Date.new(2025, 11, 3), row['datum'].to_date
|
||||||
|
assert_equal '=1+1', row['text']
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_columns_sheet_name_and_symbol_keys
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
DataPipeline::ExcelStorage.new(file: "#{dir}/c.xlsx", sheet: 'Buchungen', columns: %w[b a])
|
||||||
|
.write([{ a: '1', b: '2' }, { 'b' => '4' }])
|
||||||
|
assert_equal [{ 'b' => '2', 'a' => '1' }, { 'b' => '4', 'a' => nil }],
|
||||||
|
DataPipeline::ExcelStorage.new(file: "#{dir}/c.xlsx", sheet: 'Buchungen').read
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_without_header_and_skip_title_rows
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
DataPipeline::ExcelStorage.new(file: "#{dir}/h.xlsx", headers: false).write([{ 'x' => 'Titel' }, { 'x' => 'Kopf' }, { 'x' => 'Wert' }])
|
||||||
|
assert_equal [{ 'name' => 'Titel' }, { 'name' => 'Kopf' }, { 'name' => 'Wert' }],
|
||||||
|
DataPipeline::ExcelStorage.new(file: "#{dir}/h.xlsx", headers: false, columns: %w[name]).read
|
||||||
|
assert_equal [{ 'Kopf' => 'Wert' }], DataPipeline::ExcelStorage.new(file: "#{dir}/h.xlsx", skip: 1).read
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_empty_rows_are_skipped_and_short_name
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
DataPipeline.new do
|
||||||
|
logger PipelineHelper::QUIET
|
||||||
|
read MemoryStorage.new(rows: [{ 'a' => '1' }, { 'a' => nil }, { 'a' => '3' }])
|
||||||
|
write container(:xlsx, file: "#{dir}/e.xlsx")
|
||||||
|
end.run
|
||||||
|
assert_equal [{ 'a' => '1' }, { 'a' => '3' }], DataPipeline::ExcelStorage.new(file: "#{dir}/e.xlsx").read
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_file_required
|
||||||
|
assert_raises(ArgumentError) { DataPipeline::ExcelStorage.new }
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,56 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
require_relative '../support/storage_contract'
|
||||||
|
|
||||||
|
class JsonArrayStorageTest < PipelineTest
|
||||||
|
include StorageContract
|
||||||
|
|
||||||
|
def contract_storage(dir, _purpose) = DataPipeline::JsonStorage.new(file: "#{dir}/data.json")
|
||||||
|
|
||||||
|
def test_streamed_array_is_valid_json
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
json = DataPipeline::JsonStorage.new(file: "#{dir}/a.json")
|
||||||
|
json.write_row({ 'a' => 1 })
|
||||||
|
json.write_row({ 'a' => 2 })
|
||||||
|
json.close
|
||||||
|
assert_equal [{ 'a' => 1 }, { 'a' => 2 }], JSON.parse(File.read("#{dir}/a.json"))
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_root_path_and_pretty
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
File.write("#{dir}/api.json", JSON.generate('data' => { 'items' => [{ 'id' => 1 }] }))
|
||||||
|
assert_equal [{ 'id' => 1 }], DataPipeline::JsonStorage.new(file: "#{dir}/api.json", root: 'data.items').read
|
||||||
|
DataPipeline::JsonStorage.new(file: "#{dir}/p.json", pretty: true).write([{ 'a' => 1 }])
|
||||||
|
assert_includes File.read("#{dir}/p.json"), "\n"
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_single_object_is_one_row_and_scalar_raises
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
File.write("#{dir}/o.json", '{"a":1}')
|
||||||
|
assert_equal [{ 'a' => 1 }], DataPipeline::JsonStorage.new(file: "#{dir}/o.json").read
|
||||||
|
File.write("#{dir}/s.json", '42')
|
||||||
|
assert_raises(ArgumentError) { DataPipeline::JsonStorage.new(file: "#{dir}/s.json").read }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_options
|
||||||
|
assert_raises(ArgumentError) { DataPipeline::JsonStorage.new }
|
||||||
|
assert_raises(ArgumentError) { DataPipeline::JsonStorage.new(file: 'x.json', format: :xml) }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
class JsonLinesStorageTest < PipelineTest
|
||||||
|
include StorageContract
|
||||||
|
|
||||||
|
def contract_storage(dir, _purpose) = DataPipeline::JsonStorage.new(file: "#{dir}/data.ndjson")
|
||||||
|
|
||||||
|
def test_format_from_extension_and_blank_lines
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
File.write("#{dir}/d.jsonl", "{\"a\":1}\n\n{\"a\":2}\n")
|
||||||
|
assert_equal [{ 'a' => 1 }, { 'a' => 2 }], DataPipeline::JsonStorage.new(file: "#{dir}/d.jsonl").read
|
||||||
|
DataPipeline::JsonStorage.new(file: "#{dir}/x.txt", format: :lines).write([{ 'a' => 1 }, { 'a' => 2 }])
|
||||||
|
assert_equal "{\"a\":1}\n{\"a\":2}\n", File.read("#{dir}/x.txt")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,45 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
require 'open3'
|
||||||
|
|
||||||
|
# in a fresh process: plugins and their gems are loaded only when used
|
||||||
|
class LazyLoadingTest < PipelineTest
|
||||||
|
def loaded_after code
|
||||||
|
script = <<~RUBY
|
||||||
|
require 'bundler/setup'
|
||||||
|
require #{File.join(ROOT, 'lib/datapipeline').inspect}
|
||||||
|
#{code}
|
||||||
|
features = $LOADED_FEATURES.join(' ')
|
||||||
|
puts %w[pg sqlite3 roo caxlsx http].select { |gem| features.match?(%r{/gems/\#{gem}-\\d}) }.join(',')
|
||||||
|
RUBY
|
||||||
|
out, err, status = Open3.capture3({ 'BUNDLE_GEMFILE' => File.join(ROOT, 'Gemfile') }, RbConfig.ruby, '-e', script)
|
||||||
|
assert status.success?, err
|
||||||
|
out.strip.split(',')
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_nothing_loaded_at_start
|
||||||
|
assert_empty loaded_after('')
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_json_and_csv_need_no_gem
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
code = "DataPipeline.new { logger Logger.new(IO::NULL); read container(:json, file: '#{dir}/x.json').tap { File.write('#{dir}/x.json', '[]') }; write container(:csv, file: '#{dir}/y.csv') }.run"
|
||||||
|
assert_empty loaded_after(code)
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_each_plugin_loads_only_its_gem
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
assert_equal %w[sqlite3], loaded_after("DataPipeline::SqliteStorage.new(database: ':memory:').close")
|
||||||
|
assert_equal %w[caxlsx], loaded_after("DataPipeline::ExcelStorage.new(file: '#{dir}/a.xlsx').write([{ 'a' => 1 }])")
|
||||||
|
assert_equal %w[roo caxlsx].sort, loaded_after("DataPipeline::ExcelStorage.new(file: '#{dir}/b.xlsx').write([{ 'a' => 1 }]); DataPipeline::ExcelStorage.new(file: '#{dir}/b.xlsx').read").sort
|
||||||
|
assert_equal %w[http], loaded_after("DataPipeline::RestStorage.new(url: 'http://localhost')")
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_missing_gem_message
|
||||||
|
plugin = Class.new(DataPipeline::StoragePlugin) { def initialize(**) = self.class.require_gem('gibtsnicht_gem') }
|
||||||
|
Object.const_set(:GibtsNichtStorage, plugin) unless defined?(GibtsNichtStorage)
|
||||||
|
error = assert_raises(LoadError) { GibtsNichtStorage.new }
|
||||||
|
assert_includes error.message, "gem 'gibtsnicht_gem'"
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,133 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
require_relative '../support/storage_contract'
|
||||||
|
|
||||||
|
class PostgresStorageTest < PipelineTest
|
||||||
|
include StorageContract
|
||||||
|
|
||||||
|
def with_contract_env
|
||||||
|
with_pg_database do |db|
|
||||||
|
pg_exec db, 'create table rows (id text, code text, name text, note text)'
|
||||||
|
yield db
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def contract_storage db, purpose
|
||||||
|
purpose == :write ? DataPipeline::PostgresStorage.new(**db, table: 'rows') : DataPipeline::PostgresStorage.new(**db, query: 'select * from rows order by id')
|
||||||
|
end
|
||||||
|
|
||||||
|
def with_table
|
||||||
|
with_pg_database do |db|
|
||||||
|
pg_exec db, 'create table t (id int, name text, note text); create schema s; create table s."Mixed Case" ("Col A" text)'
|
||||||
|
storage = DataPipeline::PostgresStorage.new(**db)
|
||||||
|
yield db, storage
|
||||||
|
ensure
|
||||||
|
storage&.close
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_write_rows_with_table_option_quotes_and_null
|
||||||
|
with_table do |db, pg|
|
||||||
|
DataPipeline::PostgresStorage.new(**db, table: 't').tap do |w|
|
||||||
|
w.write([{ 'id' => 1, 'name' => "O'Brien", 'note' => nil }, { id: 2, name: "x'); drop table t; --" }])
|
||||||
|
w.close
|
||||||
|
end
|
||||||
|
assert_equal [{ 'id' => '1', 'name' => "O'Brien", 'note' => nil },
|
||||||
|
{ 'id' => '2', 'name' => "x'); drop table t; --", 'note' => nil }],
|
||||||
|
pg.query(query: 'select id, name, note from t order by id')
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_write_old_signature_and_schema_quoting
|
||||||
|
with_table do |_db, pg|
|
||||||
|
pg.write('t', [{ 'id' => 3 }])
|
||||||
|
pg.write('s.Mixed Case', [{ 'Col A' => 'v' }])
|
||||||
|
assert_equal '1', pg.query(query: 'select count(*) from t').first['count']
|
||||||
|
assert_equal [{ 'Col A' => 'v' }], pg.query(query: 'select * from s."Mixed Case"')
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_truncate_before_first_write
|
||||||
|
with_table do |db, pg|
|
||||||
|
pg.write('t', [{ 'id' => 1 }, { 'id' => 2 }])
|
||||||
|
writer = DataPipeline::PostgresStorage.new(**db, table: 't', truncate: true)
|
||||||
|
writer.write([{ 'id' => 3 }])
|
||||||
|
writer.write_row({ 'id' => 4 })
|
||||||
|
writer.close
|
||||||
|
assert_equal %w[3 4], pg.query(query: 'select id from t order by id').map { |r| r['id'] }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_write_is_one_transaction
|
||||||
|
with_table do |_db, pg|
|
||||||
|
assert_raises(PG::Error) { pg.write('t', [{ 'id' => 10 }, { 'id' => 'kein int' }]) }
|
||||||
|
assert_equal '0', pg.query(query: 'select count(*) from t').first['count']
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_write_without_table
|
||||||
|
with_table do |_db, pg|
|
||||||
|
assert_includes assert_raises(RuntimeError) { pg.write([{ 'id' => 1 }]) }.message, 'Table'
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_query_and_read_with_params
|
||||||
|
with_table do |db, pg|
|
||||||
|
pg.write('t', [{ 'id' => 1, 'name' => 'a' }, { 'id' => 2, 'name' => 'b' }, { 'id' => 3 }])
|
||||||
|
assert_equal [{ 'name' => 'a' }], pg.query(query: 'select name from t where id = $1', params: [1])
|
||||||
|
|
||||||
|
reader = DataPipeline::PostgresStorage.new(**db)
|
||||||
|
reader.set(query: 'select id from t where id > $1 order by id', params: [1])
|
||||||
|
assert_equal [{ 'id' => '2' }, { 'id' => '3' }], reader.read
|
||||||
|
reader.close
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_typed_and_default_string_values
|
||||||
|
with_table do |db, pg|
|
||||||
|
typed = DataPipeline::PostgresStorage.new(**db, typed: true)
|
||||||
|
row = typed.query(query: 'select 1::int as i, 1.5::float8 as f, current_date as d').first
|
||||||
|
typed.close
|
||||||
|
assert_equal 1, row['i']
|
||||||
|
assert_in_delta 1.5, row['f']
|
||||||
|
assert_kind_of Date, row['d']
|
||||||
|
assert_equal '1', pg.query(query: 'select 1::int as i').first['i']
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_cursor_read_in_batches
|
||||||
|
with_pg_database do |db|
|
||||||
|
pg_exec db, 'create table k as select i as id from generate_series(1, 2500) i'
|
||||||
|
reader = DataPipeline::PostgresStorage.new(**db, query: "select id from k where id > $1 order by id;\n", params: [500],
|
||||||
|
fetch_size: 100)
|
||||||
|
count = 0
|
||||||
|
reader.read { count += 1 }
|
||||||
|
reader.close
|
||||||
|
assert_equal 2000, count
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_cursor_rollback_on_error_keeps_connection_usable
|
||||||
|
with_pg_database do |db|
|
||||||
|
pg_exec db, 'create table k as select 1 as id'
|
||||||
|
pg = DataPipeline::PostgresStorage.new(**db, query: 'select id from k')
|
||||||
|
assert_raises(RuntimeError) { pg.read { raise 'x' } }
|
||||||
|
assert_equal [{ 'x' => '1' }], pg.query(query: 'select 1 as x')
|
||||||
|
pg.close
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_close_twice
|
||||||
|
with_pg_database do |db|
|
||||||
|
pg = DataPipeline::PostgresStorage.new(**db)
|
||||||
|
pg.close
|
||||||
|
pg.close
|
||||||
|
assert pg.instance_variable_get(:@conn).finished?
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_connection_error_keeps_cause
|
||||||
|
skip 'no local postgres' unless PgHelper.available?
|
||||||
|
error = assert_raises(PG::ConnectionBad) { DataPipeline::PostgresStorage.new(host: 'localhost', port: 1, database: 'x') }
|
||||||
|
assert_includes error.message, 'Error connection to database:'
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,151 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
require_relative '../support/storage_contract'
|
||||||
|
require_relative '../support/test_http_server'
|
||||||
|
|
||||||
|
class RestStorageTest < PipelineTest
|
||||||
|
include StorageContract
|
||||||
|
|
||||||
|
# a tiny api: POST /items stores one row or an array, GET /items returns all
|
||||||
|
def with_contract_env
|
||||||
|
store = []
|
||||||
|
server = TestHttpServer.new do |req|
|
||||||
|
if req[:method] == 'POST'
|
||||||
|
Array(JSON.parse(req[:body]).then { |b| b.is_a?(Array) ? b : [b] }).each { |row| store << row }
|
||||||
|
[201, { 'ok' => true }]
|
||||||
|
else
|
||||||
|
[200, { 'data' => { 'items' => store } }]
|
||||||
|
end
|
||||||
|
end
|
||||||
|
yield server
|
||||||
|
ensure
|
||||||
|
server&.stop
|
||||||
|
end
|
||||||
|
|
||||||
|
def contract_storage(server, _purpose) = DataPipeline::RestStorage.new(url: "#{server.url}/items", root: 'data.items')
|
||||||
|
|
||||||
|
# handler: ->(request) { [status, body] }
|
||||||
|
def server handler
|
||||||
|
s = TestHttpServer.new(&handler)
|
||||||
|
yield s
|
||||||
|
ensure
|
||||||
|
s&.stop
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_page_param_paging
|
||||||
|
pages = { '1' => [{ 'id' => 1 }, { 'id' => 2 }], '2' => [{ 'id' => 3 }] }
|
||||||
|
server(->(req) { [200, pages.fetch(req[:query]['page'], [])] }) do |s|
|
||||||
|
rows = DataPipeline::RestStorage.new(url: "#{s.url}/kunden", page_param: 'page', params: { 'land' => 'DE' }).read
|
||||||
|
assert_equal [1, 2, 3], rows.map { |r| r['id'] }
|
||||||
|
assert_equal %w[1 2 3], s.requests.map { |r| r[:query]['page'] }
|
||||||
|
assert(s.requests.all? { |r| r[:query]['land'] == 'DE' })
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_next_url_paging
|
||||||
|
base = nil
|
||||||
|
handler = lambda do |req|
|
||||||
|
if req[:path] == '/a'
|
||||||
|
[200, { 'rows' => [{ 'n' => 1 }], 'next' => "#{base}/b" }]
|
||||||
|
else
|
||||||
|
[200, { 'rows' => [{ 'n' => 2 }], 'next' => nil }]
|
||||||
|
end
|
||||||
|
end
|
||||||
|
server(handler) do |s|
|
||||||
|
base = s.url
|
||||||
|
rows = DataPipeline::RestStorage.new(url: "#{s.url}/a", root: 'rows', next: ->(body) { body['next'] }).read
|
||||||
|
assert_equal [1, 2], rows.map { |r| r['n'] }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_auth_and_headers
|
||||||
|
server(->(_req) { [200, []] }) do |s|
|
||||||
|
DataPipeline::RestStorage.new(url: s.url, auth: 'Bearer geheim', headers: { 'X-Mandant' => '001' }).read
|
||||||
|
assert_equal 'Bearer geheim', s.requests.first[:headers]['authorization']
|
||||||
|
assert_equal '001', s.requests.first[:headers]['x-mandant']
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_retry_on_503_then_success
|
||||||
|
calls = 0
|
||||||
|
server(->(_req) { (calls += 1) < 3 ? [503, 'busy'] : [200, [{ 'ok' => 1 }]] }) do |s|
|
||||||
|
assert_equal [{ 'ok' => 1 }], DataPipeline::RestStorage.new(url: s.url, retries: 2, retry_wait: 0).read
|
||||||
|
assert_equal 3, calls
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_no_retry_on_404_and_error_message
|
||||||
|
server(->(_req) { [404, { 'error' => 'nicht da' }] }) do |s|
|
||||||
|
error = assert_raises(DataPipeline::RestStorage::Error) { DataPipeline::RestStorage.new(url: "#{s.url}/x", retry_wait: 0).read }
|
||||||
|
assert_includes error.message, 'HTTP 404'
|
||||||
|
assert_includes error.message, 'nicht da'
|
||||||
|
assert_equal 1, s.requests.size
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_retries_exhausted_and_connection_refused
|
||||||
|
server(->(_req) { [500, 'kaputt'] }) do |s|
|
||||||
|
assert_includes assert_raises(DataPipeline::RestStorage::Error) { DataPipeline::RestStorage.new(url: s.url, retries: 1, retry_wait: 0).read }.message, 'HTTP 500'
|
||||||
|
assert_equal 2, s.requests.size
|
||||||
|
end
|
||||||
|
closed = TCPServer.new('127.0.0.1', 0).then { |t| port = t.addr[1]; t.close; port }
|
||||||
|
assert_raises(DataPipeline::RestStorage::Error) { DataPipeline::RestStorage.new(url: "http://127.0.0.1:#{closed}", retries: 1, retry_wait: 0).read }
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_batch_write_and_write_method
|
||||||
|
server(->(_req) { [200, ''] }) do |s|
|
||||||
|
rest = DataPipeline::RestStorage.new(url: "#{s.url}/import", batch_size: 2, write_method: :put)
|
||||||
|
rest.write([{ 'a' => 1 }, { 'a' => 2 }, { 'a' => 3 }])
|
||||||
|
assert_equal [[{ 'a' => 1 }, { 'a' => 2 }], [{ 'a' => 3 }]], s.requests.map { |r| JSON.parse(r[:body]) }
|
||||||
|
assert(s.requests.all? { |r| r[:method] == 'PUT' })
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_write_url_per_row
|
||||||
|
server(->(_req) { [200, ''] }) do |s|
|
||||||
|
DataPipeline::RestStorage.new(url: s.url, write_method: :put, write_url: ->(row) { "#{s.url}/kunden/#{row['id']}" })
|
||||||
|
.write([{ 'id' => 7 }, { 'id' => 8 }])
|
||||||
|
assert_equal %w[/kunden/7 /kunden/8], s.requests.map { |r| r[:path] }
|
||||||
|
assert_equal({ 'id' => 8 }, JSON.parse(s.requests.last[:body]))
|
||||||
|
end
|
||||||
|
assert_raises(ArgumentError) { DataPipeline::RestStorage.new(url: 'http://x', batch_size: 2, write_url: ->(_row) { 'http://x' }) }
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_call_json_status_and_retries
|
||||||
|
calls = 0
|
||||||
|
server(->(req) { calls += 1; req[:path] == '/api/fail' ? [503, ''] : [201, { 'echo' => JSON.parse(req[:body]) }] }) do |s|
|
||||||
|
rest = DataPipeline::RestStorage.new(url: "#{s.url}/api/", retries: 2, retry_wait: 0)
|
||||||
|
assert_equal({ 'echo' => { 'a' => 1 } }, rest.call(:post, 'person/create', json: { 'a' => 1 }))
|
||||||
|
assert_equal '/api/person/create', s.requests.last[:path]
|
||||||
|
|
||||||
|
calls = 0
|
||||||
|
error = assert_raises(DataPipeline::RestStorage::Error) { rest.call(:post, 'fail', json: {}, retries: 0) }
|
||||||
|
assert_equal 503, error.status
|
||||||
|
assert_equal 1, calls, 'retries: 0 sends once'
|
||||||
|
end
|
||||||
|
server(->(_req) { [404, { 'message' => 'weg' }] }) do |s|
|
||||||
|
assert_equal 404, assert_raises(DataPipeline::RestStorage::Error) { DataPipeline::RestStorage.new(url: s.url).call(:get, 'x') }.status
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_ssl_verify_false_context
|
||||||
|
context = DataPipeline::RestStorage.new(url: 'https://192.168.178.251:50443/pds/rest/api', ssl_verify: false).send(:ssl_context)
|
||||||
|
assert_equal OpenSSL::SSL::VERIFY_NONE, context.verify_mode
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_query_for_lookup
|
||||||
|
server(->(req) { [200, [{ 'land' => req[:query]['land'], 'pfad' => req[:path] }]] }) do |s|
|
||||||
|
names = nil
|
||||||
|
DataPipeline.new do
|
||||||
|
api = container :rest, url: "#{s.url}/api/"
|
||||||
|
names = lookup api, query: '/laender', params: { 'land' => 'DE' }, key: 'land', unique: true
|
||||||
|
end
|
||||||
|
assert_equal '/api/laender', names['DE']['pfad']
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_not_an_array
|
||||||
|
server(->(_req) { [200, { 'a' => 1 }] }) do |s|
|
||||||
|
assert_raises(DataPipeline::RestStorage::Error) { DataPipeline::RestStorage.new(url: s.url).read }
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
|
||||||
|
# no network: only construction and options
|
||||||
|
class SlackTest < PipelineTest
|
||||||
|
def test_token_required
|
||||||
|
token = ENV.delete('SLACK_TOKEN')
|
||||||
|
assert_raises(ArgumentError) { DataPipeline::Slack.new }
|
||||||
|
ensure
|
||||||
|
ENV['SLACK_TOKEN'] = token if token
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_options_per_instance
|
||||||
|
a = DataPipeline::Slack.new(token: 'x', channel: '#a')
|
||||||
|
b = DataPipeline::Slack.new(token: 'y')
|
||||||
|
assert_equal '#a', a.instance_variable_get(:@options)[:channel]
|
||||||
|
assert_equal '#general', b.instance_variable_get(:@options)[:channel]
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
require_relative '../test_helper'
|
||||||
|
require_relative '../support/storage_contract'
|
||||||
|
|
||||||
|
class SqliteStorageTest < PipelineTest
|
||||||
|
include StorageContract
|
||||||
|
|
||||||
|
def contract_storage dir, purpose
|
||||||
|
db = "#{dir}/test.sqlite3"
|
||||||
|
unless File.exist?(db)
|
||||||
|
DataPipeline::SqliteStorage.new(database: db).tap do |s|
|
||||||
|
s.query(query: 'create table rows (id text, code text, name text, note text)')
|
||||||
|
s.close
|
||||||
|
end
|
||||||
|
end
|
||||||
|
purpose == :write ? DataPipeline::SqliteStorage.new(database: db, table: 'rows') : DataPipeline::SqliteStorage.new(database: db, query: 'select * from rows order by id')
|
||||||
|
end
|
||||||
|
|
||||||
|
def memory_db
|
||||||
|
DataPipeline::SqliteStorage.new(database: ':memory:').tap do |s|
|
||||||
|
s.query(query: 'create table t (id integer, name text, betrag real, datum text)')
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_types_null_and_quotes
|
||||||
|
db = memory_db
|
||||||
|
db.write('t', [{ 'id' => 1, 'name' => "O'Brien", 'betrag' => 1.5, 'datum' => Date.new(2025, 11, 3) }, { id: 2, name: nil }])
|
||||||
|
assert_equal [{ 'id' => 1, 'name' => "O'Brien", 'betrag' => 1.5, 'datum' => '2025-11-03' },
|
||||||
|
{ 'id' => 2, 'name' => nil, 'betrag' => nil, 'datum' => nil }],
|
||||||
|
db.query(query: 'select * from t order by id')
|
||||||
|
db.close
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_positional_and_named_params
|
||||||
|
db = memory_db
|
||||||
|
db.write('t', [{ 'id' => 1, 'name' => 'a' }, { 'id' => 2, 'name' => 'b' }])
|
||||||
|
assert_equal [{ 'name' => 'b' }], db.query(query: 'select name from t where id = ?', params: [2])
|
||||||
|
assert_equal [{ 'name' => 'a' }], db.query(query: 'select name from t where id = :id', params: { id: 1 })
|
||||||
|
db.close
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_write_is_one_transaction
|
||||||
|
db = memory_db
|
||||||
|
db.query(query: 'create table u (id integer not null)')
|
||||||
|
assert_raises(SQLite3::Exception) { db.write('u', [{ 'id' => 1 }, { 'id' => nil }]) }
|
||||||
|
assert_equal 0, db.query(query: 'select count(*) as n from u').first['n']
|
||||||
|
db.close
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_sql_injection_in_values_and_identifiers
|
||||||
|
db = memory_db
|
||||||
|
db.query(query: 'create table "Mixed ""Name""" ("Col A" text)')
|
||||||
|
db.write('Mixed "Name"', [{ 'Col A' => "x'); drop table t; --" }])
|
||||||
|
assert_equal [{ 'Col A' => "x'); drop table t; --" }], db.query(query: 'select * from "Mixed ""Name"""')
|
||||||
|
assert_equal 0, db.query(query: 'select count(*) as n from t').first['n']
|
||||||
|
db.close
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_create_truncate_and_lazy_open
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
path = "#{dir}/neu.sqlite3"
|
||||||
|
storage = DataPipeline::SqliteStorage.new(database: path, table: 'k', create: true, truncate: true)
|
||||||
|
refute File.exist?(path), 'opened only on first use'
|
||||||
|
storage.write([{ 'id' => 1, 'betrag' => 1.5, 'name' => 'a' }, { 'id' => 2, 'betrag' => 2.0, 'name' => 'b' }])
|
||||||
|
storage.close
|
||||||
|
|
||||||
|
again = DataPipeline::SqliteStorage.new(database: path, table: 'k', create: true, truncate: true)
|
||||||
|
again.write_row({ 'id' => 3, 'betrag' => 3.0, 'name' => 'c' })
|
||||||
|
assert_equal [{ 'id' => 3, 'betrag' => 3.0, 'name' => 'c' }], again.query(query: 'select * from k')
|
||||||
|
assert_equal %w[integer real text], again.query(query: 'pragma table_info(k)').map { |c| c['type'].downcase }
|
||||||
|
again.close
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_missing_options
|
||||||
|
assert_raises(ArgumentError) { DataPipeline::SqliteStorage.new }
|
||||||
|
db = memory_db
|
||||||
|
assert_raises(RuntimeError) { db.write([{ 'id' => 1 }]) }
|
||||||
|
assert_raises(RuntimeError) { db.read }
|
||||||
|
db.close
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_short_name_and_lookup
|
||||||
|
with_tmpdir do |dir|
|
||||||
|
path = "#{dir}/l.sqlite3"
|
||||||
|
names = nil
|
||||||
|
DataPipeline.new do
|
||||||
|
db = container :sqlite3, database: path
|
||||||
|
db.query(query: 'create table land (code text, name text)')
|
||||||
|
db.write('land', [{ 'code' => 'DE', 'name' => 'Deutschland' }])
|
||||||
|
names = lookup db, query: 'select * from land where code like ?', params: ['%'], key: 'code', unique: true
|
||||||
|
end
|
||||||
|
assert_equal 'Deutschland', names['DE']['name']
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,73 @@
|
|||||||
|
#
|
||||||
|
# shared tests for every storage plugin that can write and read back
|
||||||
|
#
|
||||||
|
# a test class includes StorageContract and implements
|
||||||
|
# contract_storage(env, purpose) -> a new storage; purpose is :write or :read
|
||||||
|
# optionally
|
||||||
|
# with_contract_env { |env| ... } -> environment per test (default: a tmp dir)
|
||||||
|
# contract_rows(rows) -> rows as the plugin reads them back (default: unchanged)
|
||||||
|
#
|
||||||
|
module StorageContract
|
||||||
|
ROWS = [
|
||||||
|
{ 'id' => '1', 'code' => '007', 'name' => "O'Brien", 'note' => nil },
|
||||||
|
{ 'id' => '2', 'code' => '010', 'name' => 'Zürich | "ä", ß', 'note' => 'zwei' },
|
||||||
|
{ 'id' => '3', 'code' => '100', 'name' => 'drei', 'note' => '' }
|
||||||
|
].freeze
|
||||||
|
|
||||||
|
def with_contract_env(&block) = with_tmpdir(&block)
|
||||||
|
def contract_rows(rows) = rows
|
||||||
|
|
||||||
|
def test_contract_write_and_read_back
|
||||||
|
with_contract_env do |env|
|
||||||
|
contract_storage(env, :write).tap { |s| s.write(ROWS) }.close
|
||||||
|
reader = contract_storage(env, :read)
|
||||||
|
assert_equal contract_rows(ROWS), reader.read
|
||||||
|
reader.close
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_contract_write_row_and_read_with_block
|
||||||
|
with_contract_env do |env|
|
||||||
|
writer = contract_storage(env, :write)
|
||||||
|
ROWS.each { |row| writer.write_row(row) }
|
||||||
|
writer.close
|
||||||
|
rows = []
|
||||||
|
reader = contract_storage(env, :read)
|
||||||
|
assert_nil reader.read { |row| rows << row }
|
||||||
|
reader.close
|
||||||
|
assert_equal contract_rows(ROWS), rows
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_contract_as_pipeline_source_and_target
|
||||||
|
[false, true].each do |streamed|
|
||||||
|
with_contract_env do |env|
|
||||||
|
target = contract_storage(env, :write)
|
||||||
|
stats = DataPipeline.new(streamed:) do
|
||||||
|
logger PipelineHelper::QUIET
|
||||||
|
read MemoryStorage.new(rows: ROWS)
|
||||||
|
write target
|
||||||
|
end.run
|
||||||
|
assert_equal 3, stats.written
|
||||||
|
|
||||||
|
source = contract_storage(env, :read)
|
||||||
|
copy = MemoryStorage.new
|
||||||
|
DataPipeline.new(streamed:) do
|
||||||
|
logger PipelineHelper::QUIET
|
||||||
|
read source
|
||||||
|
write copy
|
||||||
|
end.run
|
||||||
|
assert_equal contract_rows(ROWS), copy.output, "streamed=#{streamed}"
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
def test_contract_close_twice
|
||||||
|
with_contract_env do |env|
|
||||||
|
storage = contract_storage(env, :write)
|
||||||
|
storage.close
|
||||||
|
storage.close
|
||||||
|
pass
|
||||||
|
end
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
require 'socket'
|
||||||
|
require 'uri'
|
||||||
|
require 'json'
|
||||||
|
|
||||||
|
#
|
||||||
|
# minimal http server for DataPipeline::RestStorage tests, one request per connection
|
||||||
|
# server = TestHttpServer.new { |req| [200, { 'items' => [] }] }
|
||||||
|
# req: { method:, path:, query:, headers:, body: } -> [status, hash/array or string]
|
||||||
|
#
|
||||||
|
class TestHttpServer
|
||||||
|
attr_reader :requests
|
||||||
|
|
||||||
|
def initialize(&handler)
|
||||||
|
@handler = handler
|
||||||
|
@requests = []
|
||||||
|
@server = TCPServer.new('127.0.0.1', 0)
|
||||||
|
@thread = Thread.new { loop { serve(@server.accept) } }
|
||||||
|
end
|
||||||
|
|
||||||
|
def url = "http://127.0.0.1:#{@server.addr[1]}"
|
||||||
|
|
||||||
|
def stop
|
||||||
|
@thread.kill
|
||||||
|
@server.close
|
||||||
|
end
|
||||||
|
|
||||||
|
private
|
||||||
|
|
||||||
|
def serve client
|
||||||
|
method, target = client.gets.to_s.split
|
||||||
|
return unless method
|
||||||
|
|
||||||
|
headers = {}
|
||||||
|
while (line = client.gets) && line != "\r\n"
|
||||||
|
key, value = line.split(':', 2)
|
||||||
|
headers[key.downcase] = value.strip
|
||||||
|
end
|
||||||
|
body = headers['content-length'] ? client.read(headers['content-length'].to_i) : ''
|
||||||
|
path, query = target.split('?', 2)
|
||||||
|
request = { method:, path:, query: URI.decode_www_form(query.to_s).to_h, headers:, body: }
|
||||||
|
@requests << request
|
||||||
|
|
||||||
|
status, payload = @handler.call(request)
|
||||||
|
text = payload.is_a?(String) ? payload : JSON.generate(payload)
|
||||||
|
client.write "HTTP/1.1 #{status} X\r\nContent-Type: application/json\r\n" \
|
||||||
|
"Content-Length: #{text.bytesize}\r\nConnection: close\r\n\r\n#{text}"
|
||||||
|
rescue StandardError => e
|
||||||
|
warn "TestHttpServer: #{e.class}: #{e.message}"
|
||||||
|
ensure
|
||||||
|
client.close
|
||||||
|
end
|
||||||
|
end
|
||||||
@@ -0,0 +1,148 @@
|
|||||||
|
#
|
||||||
|
# rake test or ruby -Itest test/core/chained_test.rb
|
||||||
|
#
|
||||||
|
# .env is NOT loaded: no test may reach the production database.
|
||||||
|
# postgres tests use a local server and a throw-away database per test:
|
||||||
|
# ETL_TEST_PG_HOST (default localhost), ETL_TEST_PG_PORT, ETL_TEST_PG_USER (default $USER), ETL_TEST_PG_PASSWORD
|
||||||
|
# without a reachable server they are skipped.
|
||||||
|
#
|
||||||
|
require 'bundler/setup'
|
||||||
|
require 'minitest/autorun'
|
||||||
|
require 'json'
|
||||||
|
require 'logger'
|
||||||
|
require 'stringio'
|
||||||
|
require 'tmpdir'
|
||||||
|
|
||||||
|
require 'datapipeline'
|
||||||
|
|
||||||
|
ROOT = File.expand_path('..', __dir__)
|
||||||
|
|
||||||
|
# source and target in memory
|
||||||
|
class MemoryStorage < DataPipeline::StoragePlugin
|
||||||
|
attr_reader :rows, :written, :streamed_rows
|
||||||
|
|
||||||
|
def initialize rows: [], **options
|
||||||
|
@options = options
|
||||||
|
@rows = rows
|
||||||
|
@written = nil
|
||||||
|
@streamed_rows = []
|
||||||
|
@closed = false
|
||||||
|
end
|
||||||
|
|
||||||
|
def read(&block)
|
||||||
|
return @rows.map(&:dup) unless block
|
||||||
|
|
||||||
|
@rows.map(&:dup).each(&block)
|
||||||
|
nil
|
||||||
|
end
|
||||||
|
|
||||||
|
def write(rows) = (@written = rows)
|
||||||
|
def write_row(row) = (@streamed_rows << row)
|
||||||
|
def close = (@closed = true)
|
||||||
|
def closed? = @closed
|
||||||
|
|
||||||
|
# rows of the last batch write or of the streamed rows
|
||||||
|
def output = @written || @streamed_rows
|
||||||
|
end
|
||||||
|
|
||||||
|
DataPipeline.register :memory, MemoryStorage
|
||||||
|
|
||||||
|
module PipelineHelper
|
||||||
|
QUIET = Logger.new(IO::NULL)
|
||||||
|
|
||||||
|
def setup
|
||||||
|
super
|
||||||
|
DataPipeline.reset_settings!
|
||||||
|
DataPipeline.runs.clear
|
||||||
|
DataPipeline.params.clear
|
||||||
|
end
|
||||||
|
|
||||||
|
# runs a pipeline from rows into a MemoryStorage, returns [target, stats]
|
||||||
|
def run_pipeline rows = [], **settings, &block
|
||||||
|
source = MemoryStorage.new(rows:)
|
||||||
|
target = MemoryStorage.new
|
||||||
|
stats = DataPipeline.new(**settings) do
|
||||||
|
logger QUIET
|
||||||
|
read source
|
||||||
|
write target
|
||||||
|
instance_eval(&block) if block
|
||||||
|
end.run
|
||||||
|
[target, stats]
|
||||||
|
end
|
||||||
|
|
||||||
|
def output rows = [], **settings, &block
|
||||||
|
run_pipeline(rows, **settings, &block).first.output
|
||||||
|
end
|
||||||
|
|
||||||
|
def with_tmpdir(&block)
|
||||||
|
Dir.mktmpdir('etl-test', &block)
|
||||||
|
end
|
||||||
|
|
||||||
|
def capture_stdout
|
||||||
|
old = $stdout
|
||||||
|
$stdout = StringIO.new
|
||||||
|
yield
|
||||||
|
$stdout.string
|
||||||
|
ensure
|
||||||
|
$stdout = old
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
module PgHelper
|
||||||
|
def self.config
|
||||||
|
{
|
||||||
|
host: ENV.fetch('ETL_TEST_PG_HOST', 'localhost'),
|
||||||
|
port: ENV.fetch('ETL_TEST_PG_PORT', '5432'),
|
||||||
|
username: ENV.fetch('ETL_TEST_PG_USER', ENV['USER']),
|
||||||
|
password: ENV.fetch('ETL_TEST_PG_PASSWORD', nil)
|
||||||
|
}
|
||||||
|
end
|
||||||
|
|
||||||
|
def self.admin
|
||||||
|
require 'pg'
|
||||||
|
c = config
|
||||||
|
PG.connect(host: c[:host], port: c[:port], user: c[:username], password: c[:password], dbname: 'postgres',
|
||||||
|
connect_timeout: 3)
|
||||||
|
end
|
||||||
|
|
||||||
|
def self.available?
|
||||||
|
return @available unless @available.nil?
|
||||||
|
|
||||||
|
@available = begin
|
||||||
|
admin.close
|
||||||
|
true
|
||||||
|
rescue StandardError, LoadError
|
||||||
|
false
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# yields connection options for a fresh database, dropped afterwards
|
||||||
|
def with_pg_database
|
||||||
|
skip 'no local postgres (ETL_TEST_PG_HOST)' unless PgHelper.available?
|
||||||
|
|
||||||
|
name = "etl_test_#{Process.pid}_#{rand(1_000_000)}"
|
||||||
|
admin = PgHelper.admin
|
||||||
|
admin.exec("create database #{name}")
|
||||||
|
yield PgHelper.config.merge(database: name)
|
||||||
|
ensure
|
||||||
|
if admin
|
||||||
|
admin.exec("select pg_terminate_backend(pid) from pg_stat_activity where datname = '#{name}' and pid <> pg_backend_pid()")
|
||||||
|
admin.exec("drop database if exists #{name}")
|
||||||
|
admin.close
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
# plain sql, several statements allowed (setup of test tables)
|
||||||
|
def pg_exec options, sql
|
||||||
|
conn = PG.connect(host: options[:host], port: options[:port], user: options[:username],
|
||||||
|
password: options[:password], dbname: options[:database])
|
||||||
|
conn.exec(sql).to_a
|
||||||
|
ensure
|
||||||
|
conn&.close
|
||||||
|
end
|
||||||
|
end
|
||||||
|
|
||||||
|
class PipelineTest < Minitest::Test
|
||||||
|
include PipelineHelper
|
||||||
|
include PgHelper
|
||||||
|
end
|
||||||
Reference in New Issue
Block a user