user@devops:~$ cat README.md
Apache Beam — Production ML Pipelines
# Description
Production Machine Learning pipeline built with Apache Beam: the same DAG definition runs on DirectRunner (local, threads) or on a distributed runner (Dataflow) without changing a single line. It generates 24,000 synthetic telecom churn rows with known logistic ground truth and ~4% dirty rows (nulls/impossible values). The Beam training pipeline (ReadFromText → ParseRow → CleanRow → FeatureEngineer → Partition 80/20) removes 2.8% of invalid rows and produces JSONL artifacts. On those features it trains a 400-tree RandomForest (Accuracy 0.72, F1 0.73, ROC-AUC 0.72) and runs batch inference at scale over 300,000 pre-featurized rows. Key findings: predicting row-by-row inside a DoFn costs ~80ms per element due to data-plane serialization, while a single 2D numpy call per bundle is 32x faster (12 → 374 rows/s); vectorized scoring releases the GIL and genuinely scales with DirectRunner workers. Kubeflow-style orchestration with a 7-component DAG, dependencies and artifacts. 7 visualizations: pipeline DAG, EDA, feature importance, confusion matrix + ROC, batching patterns, worker scaling, and production summary table.
# Key features
$ End-to-end Beam pipeline: ReadFromText → ParseRow → CleanRow → FeatureEngineer → Partition(80/20) with raw/clean row counters
$ 24,000 synthetic churn rows with logistic ground truth and ~4% dirty rows (2.8% removed by the pipeline)
$ 400-tree RandomForest on Beam features: Accuracy 0.72 / F1 0.73 / ROC-AUC 0.72
$ Batch inference at scale: 300,000 pre-featurized rows at ~50,000 rows/s (best config)
$ Vectorized batching 32x faster than row-by-row prediction inside a DoFn (12 → 374 rows/s)
$ Same DAG on DirectRunner (local) and Dataflow (distributed): only PipelineOptions change
$ Kubeflow-style orchestration: 7-component DAG with dependencies and artifacts
$ 7 visualizations: DAG, EDA, feature importance, confusion + ROC, batching, scaling, summary
# Gallery
# Technologies used
