Quickstart using Web App
Use GlassFlow's UI interface to create a pipeline to process incoming data, transform it, and make it available for consumption in real-time.
There is also a video tutorial available.
Create a GlassFlow account
Visit app.glassflow.dev to create your account. Click on "Sign Up" to initiate the sign-up process.
Choose a signup option using your Google or GitHub account and press continue.
After successfully creating an account, you can access the GlassFlow dashboard.
Creating Your First Pipeline
Create a Transformation Function
Create a new file transform.py
and write a short Python code that transforms incoming data. The function you write will parse the incoming JSON data, convert product quantities based on unit measurements, and categorize product items.
Note that the handler function is mandatory to implement in your code. Without it, the running transformation function will not be successful.
You can also import
other Python dependencies (packages) in the transformation function. See supported libraries with GlassFlow.
Create Your Pipeline
Visit the GlassFlow Web App, locate the "Pipelines" section, and click on "Create new Pipeline".
Enter a name for your pipeline (e.g., "inventorymanagement").
Upload the
transform.py
Python script file you implemented previously.Click "Create Pipeline" to create the pipeline.
Important: Now the transformation function is deployed and running on GlassFlow’s Serverless Execution Engine in the cloud, and you do not need to configure or maintain any infrastructure on your side.
Get Your Pipeline Credentials
Once the pipeline is created and configured, locate its credentials. Choose the pipeline you created, click, and open the pipeline view.
Pipeline ID is the Identifier value under the Details section on the pipeline view.
Access Token can be found in the Access Tokens list section:
Space ID can be found by navigating to the dropdown Actions and clicking on Copy Space ID:
Note down the Pipeline ID, Space ID, and Access Token. You'll set them as environment variables in the upcoming section.
Sending Data to Your Pipeline
Prerequisites
To complete this part you'll need the following:
Python is installed on your machine.
Download and Install Pip to manage project packages.
Create a new folder
Start by creating a dedicated project folder. Create a directory for this project named glassflow-playground
, where you will place all the necessary files.
Create an environment configuration file
Create a .env
file in your project directory with the following content, replacing the placeholders with your actual GlassFlow pipeline credentials:
Install GlassFlow SDK
Install the GlassFlow Python SDK and virtual environment package python-dotenv
using pip
.
Optional: Create a virtual environment before installing Python dependencies. Run the following command: python -m venv .venv && source .venv/bin/activate
Now you can start sending data to the pipeline. GlassFlow will automatically run your transformation function on each event entering the pipeline and make your transformed data available in milliseconds.
Create a producer
Create a new Python file in your project root directory called producer.py
. Write a script to read auto-generated inventory data and send it to the GlassFlow pipeline. Copy and paste the following code to producer.py
file:
Run the above Python script with the following command in your terminal:
When you run the above script, on every event that enters the pipeline, GlassFlow will invoke the transformation function that you have defined in the inventorymanagement
. In this case, it will run the transform.py
file and transform every raw data you send to the pipeline.
Consuming Data from Your Pipeline
Create a new Python file called consumer.py
. The consumer is responsible for pulling transformed data from the pipeline. It continuously checks for new data, processes it as needed, and acts upon the transformed information. Copy and paste the following code to consumer.py
the file:
Run the above Python script consumer.py
in a separate terminal window to see the output side-by-side:
Now you activated the consumer side of the pipeline. This consumer.py
script retrieves the processed data from the pipeline in real-time and you will see the results of data transformation written to a new local file called (supplier_inventory.txt
).
You can watch the new transformations by running the following command on a different terminal window:
Congratulations! You've set up a real-time inventory management system pipeline using GlassFlow.
Conclusion
In this Quickstart you've learned the following:
How to install GlassFlow and set up a new project.
How to create a data pipeline using the GlassFlow Web App.
How to implement transform function and process data in real-time.
How to publish data into the pipeline.
How to consume data from the pipeline.
Last updated