Real-time log data anomaly detection

A practical example of creating a pipeline for real-time logs data anomaly detection using AI.

This example data transformation pipeline demonstrates data anomaly detection with GlassFlow and AI to monitor server logs to detect unusual patterns or suspicious activities and send notifications to Slack.

Real-time data anomalies detection pipeline with GlassFlow

Link to the GitHub project repository

We'll use the GlassFlow CLI to create a new space and configure the data pipeline.

Prerequisites

Make sure that you have the following before proceeding with the installation:

• Docker is installed on your machine

• You created a GlassFlow account.

• You installed GlassFlow CLI and logged into your account via the CLI.

• You have an OpenAI API account.

• Slack account: If don't have a Slack account, sign up for a new free one here and go to the Slack Get Started page.

• Slack workspace: You need access to a Slack workspace where you're an admin. If you are creating just a new workspace, follow this guide.

• You created an incoming webhook for your Slack workspace.

Installation

1. Clone the glassflow-examples repository to your local machine:

   git clone https://github.com/glassflow/glassflow-examples.git

2. Navigate to the project directory:

   cd use-cases/real-time-data-anomaly-detection

Steps to run the GlassFlow pipeline

1. Get a new OpenAI API Key

Create an API key to use OpenAI API endpoints.

2. Set OpenAI API Key

Open transform.py file and replace {REPLACE_WITH_YOUR_OPENAI_API_KEY} with your actual API key.

3. Create a Space via CLI

Open a terminal and create a new space called examples to organize multiple pipelines:

glassflow space create examples

After the space is created successfully, you will get a SpaceID in the terminal.

4. Create a Pipeline via CLI

Use the GlassFlow CLI to create a new data pipeline inside the space.

glassflow pipeline create anomalies-detection --space-id={space_id} --function=transform.py --requirements=openai

With --requirements=openai parameter to the CLI command, we specify which external libraries to use in the transformation function (Similarly how you define Python project dependencies in the requirements.txt file). You can also import other Python dependencies in the transformation function. See supported libraries with GlassFlow.

This command initializes the pipeline with a name anomalies-detection in the examples space and specifies the transformation function transform.py. After running the command, it returns a new Pipeline ID with its Access Token.

5. Create an environment configuration file

Add a .env file in the project directory and add the following configuration variables:

SLACK_WEBHOOK_URL=your_slack_workspace_webhook_url
PIPELINE_ID=your_pipeline_id
SPACE_ID=your_space_id
PIPELINE_ACCESS_TOKEN=your_pipeline_access_token

Replace your_pipeline_id, your_space_id, and your_pipeline_access_token with appropriate values obtained from your GlassFlow account.

6. Run the project with Docker Compose:

docker compose up