0
0
Large language models often excel at generating quick Python scripts for simple tasks like parsing a single JSON file. However, they frequently struggle when asked to construct complex, production-grade data pipelines involving thousands of documents, text chunking, quality scoring, and noise filtering for Retrieval-Augmented Generation systems.
Researchers from Peking University, Zhongguancun Academy, and Shanghai’s Institute for Advanced Algorithms Research developed DataFlow-Harness to guide AI agents toward creating structured, visual workflows instead of disposable free-form code. The framework converts natural language requests into persistent directed acyclic graphs that integrate with existing MLOps platforms.
This approach addresses the NL2Pipeline gap, where agents produce scripts that cannot be audited or maintained through standard workflow tools. By enforcing typed mutations on a live operator registry, the system ensures every stage aligns with registered datasets and model services.
DataFlow-Harness organizes synthesis through four coordinated layers: a backend storing the authoritative DAG, a web interface for human-AI collaboration, an MCP tools layer exposing platform state, and markdown-based skills that encode domain rules for operator selection and schema matching.
In benchmarks covering twelve industrial tasks, the framework reached a 93.3 percent end-to-end pass rate. It cut API costs by 72.5 percent and latency by 49.9 percent compared with unconstrained coding agents, while staying within one percentage point of agents given full codebase context.
One added benefit for enterprises is reduced technical debt. Instead of accumulating unmaintainable scripts, teams receive editable artifacts that support visual inspection and version control. This matters as organizations scale RAG deployments, where pipeline failures can cascade across retrieval and generation stages.
Another relevant context is the growing pressure on data engineering teams to support synthetic data generation and math-focused cleaning pipelines. Experiments showed models trained on DataFlow-Harness outputs achieved higher accuracy on AIME24 and AIME25 benchmarks than those using vanilla agent outputs.
Implementation requires maintaining an operator registry and encoding recurring procedures as skills. The framework is not a plug-in for Airflow or Spark; adapters must be built for other execution backends. Teams should reserve it for recurring, high-value workflows rather than one-off transformations.
The open-source release under Apache 2.0 allows developers to inspect the DAG editor and MCP integration directly. As model context windows expand, the division of labor between agents handling repetitive construction and engineers overseeing policy decisions is expected to sharpen further.
