Skip to content

Data Processing

Jump to bottom
Erwin de Kreuk edited this page Apr 21, 2026 · 3 revisions

Configure and load bulk data

This article describes how to configure and load demo data into the Fabric Metadata-Driven Framework (FMD) using the PL_TOOLING_POST_ASQL_TO_FMD pipeline.

Overview

The PL_TOOLING_POST_ASQL_TO_FMD pipeline assists in inserting metadata from one of your SQL connections into the metadata-driven database. Before running the pipeline, ensure that a SQL connection is created and registered in the framework.

PL_TOOLING_POST_ASQL_TO_FMD

Required parameters

Set the following parameters when running the pipeline:

Parameter name Description Example
ConnectionGuid GUID of the SQL connection cf673e6a-13f6-4ebb-9cbb-4ba4ab390818
Data_WorkspaceGuid Workspace GUID where the Lakehouse is deployed pf7647b15-7df1-4db5-99cb-90eee61737a4
DatasourceName Name of the database WideWorldImporters
DatasourceNamespace Prefix for tables or files wwi
DatasourceType Type depending on your connection ASQL_01
Tables List of schema.tables to include (comma-separated) (''Warehouse.StockItems'',''Warehouse.PackageTypes'',''Purchasing.PurchaseOrders'' ,''Purchasing.PurchaseOrderLines'' ,''Sales.vCustomers'',''Sales.CustomerCategories'',''Sales.Orders'' ,''Sales.OrderLines'' ,''Sales.Invoices'' ,''Sales.InvoiceLines'' ,''Sales.BuyingGroups'' ,''Sales.CustomerCategories'')

Tip

Ensure all parameter values match your environment and connection details.

Load Demo data csv file

When load_demo_data = True, you have to upload a csv file (which is available in this repo). With this you can easily test if every pipeline and the full process is working

  1. Upload customer.csv from demodata folder in this repo to the file section of LH_DATA_LANDINGZONE in the development environment.
  2. Create table: Generate a table named in_customer from the uploaded file. If you use schema-enabled lakehouse, create schema in and use in.customer.
  3. Run process: Execute the process to validate deployment, by start the PL_LOAD_ALL pipeline in the FMD_FRAMEWORK_CODE (D) workspace. Example of loading the file and creating the table:

Load File to table

Example of loading the file and creating the table with Lakehouse schema enabled:

Load File to table

Next steps

  • After configuring the parameters, run the PL_TOOLING_POST_ASQL_TO_FMD pipeline to load bulk data and metadata into the FMD Framework.
  • For more information on configuring connections and workspaces, see FMD Framework Deployment Guide.

Clone this wiki locally