Q: What is a Data Flow?
A Data Flow is a new module within Data Engine that enables you to create a sequence of interconnected nodes to clean, transform, and process your data in a visual, drag-and-drop interface.
Q: Why should I start using Data Flows?
Data Flows simplify data processing by offering a centralized, visual interface. They improve efficiency, transparency, and collaboration while enabling automated, scheduled executions.
Q: Where do I find Data Flows module?
You can access Data Flows from the ‘Data’ tab in your Data Engine account.
Q: How do I use Data Flows?
Simply log into your Data Engine account, create a new Data Flow by selecting input tables, setting up transformations, and defining output tables. You can schedule or execute your Data Flow as needed. Detailed instructions are found in the Data Flow - User Guide.
Q: What are the advantages of using Data Flows?
Data Flows provide enhanced visibility, improved data processing, faster decision-making, greater flexibility, and increased performance compared to traditional data processing methods.
Q: Are all functionalities available as I see them for Data Flows?
Although visible in the node selector, the nodes from AI & Processing group are not yet available for use. These are likely to be enabled in the future when the pricing and enablement plan has been finalised.
Q: Can I use existing Views, Merges and Fusions into Data Flows?
No. Existing objects created in Data Engine as Views, Merges and Fusions cannot be Input tables in a Data Flow. They can, however, be recreated easily within the Data Flow through transformations.
Q: What are Fusions?
Table fusions are a type of object that were previously available to customers on the Professional Edition of Workspace, which allowed two tables to be fused together (an equivalent of a Union join in SQL). This feature is now available as a node in Data Flows and therefore will be available to all customers.
Q: How can I integrate existing data to use Data Flows?
All Data Flows start with an Input table. Any physical table can be used as an input table (i.e. Data Clones, Custom Tables, Snapshot Tables , etc). Virtual tables (Views, Merges, Fusion), regardless if it’s cached, cannot be used.
Q: What resources are Data Flows consuming?
One of the standout features of Data Flows is the ability to control when and how your data gets processed. Unlike traditional Views and Merges that refresh automatically, Data Flows give you the flexibility to schedule or trigger executions as needed, improving efficiency and performance. Each output table that gets refreshed consumes a Data Refresh as part of the allowance.
Q: What does this mean for Views, Merges and Fusions?
Views, Merges and Fusions will continue to be available to users and supported. However, new features will not be developed on these. Users are encouraged to use Data Flows as a go forward solution.
Q: How can I migrate existing Views, Merges and Fusions over to Data Flows?
All the features of Views, Merges and Fusions are available as nodes within a Data Flow, and therefore can be replicated manually. In future there is likely to be tooling available to help automatically re-create Views, Merges and Fusions as a Data Flow to help with the migration.
Q: What sort of control I have over Data Flows?
You have complete control over when and how Data Flows execute, allowing you to schedule or trigger transformations as needed.
Q: What supporting materials are available about Data Flows?
There is available a complete Data Flow – User Guide, we have the launching article about Data Flows and the current list of Questions and Answers. In case you cannot find the information you need, you can always rely on Support.
Q: Where can I get support from if I experience issues with Data Flows?
You can access support through our Customer Support Center, where your dedicated team will assist you, and redirect when needed, with any issues related to Data Flows.