Kafka Outbound Events / Signal - Capture

Outbound Events (Signals) push real-time notifications when nodes or relationships are captured in your IKG.

This configuration sets up event streaming to Kafka (Confluent) for data capture events:

1. Define Confluent as the event provider.

2. Route capture events matching specific criteria to a Kafka topic.

3. Filter events by node label and property values (e.g., Car nodes where manufacturer = "pontiac").

Use case

Scenario: You need to react when specific data is captured in your IKG.

Each time a node matching your criteria is created, updated, or deleted, an event is sent to your Kafka topic. This enables:

- Real-time data synchronization with external systems.

- Triggering workflows when specific node types are modified.

- Selective event filtering to reduce noise and processing overhead.

Note: Only one Outbound Events configuration can be active per project.

Requirements

- ServiceAccount credentials created in the IndyKite Hub for your organization.

- Application Agent credentials for capturing data via the Capture API.

- A Confluent Cloud environment with a valid API Key.

- A Kafka topic created in your Confluent environment.

Steps

1. Create a topic in your Confluent environment to receive events.

2. Apply the Terraform configuration with indykite.audit.capture.* eventType and filter criteria.

3. Capture nodes with the Car label and manufacturer property set to "pontiac".

4. Add more nodes matching the same criteria.

5. Verify that event messages appear in your Kafka topic for matching nodes.

6. Confirm that nodes with different properties do not trigger events.

Step 2

main.tf


terraform {
  required_providers {
    indykite = {
      source  = "indykite/indykite"
      version = 1.26. # or latest version
    }
  }
}

# indykite provider integrates IndyKite platform with Terraform scripting.
# Provider for now does not support any parameters and all is set within service account credential file.
provider "indykite" {}
resource "time_static" "example" {}

resource "indykite_event_sink" "outbound_event" {
  name         = "outbound-event"
  display_name = "Outbound Event"
  location     = "project_id"
  providers {
    provider_name = "provider-name"
    kafka {
      brokers  = ["broker"]
      topic    = "topic_signal"
      username = "api_key"
      password = "api_secret"
    }
  }
  routes {
    provider_id     = "provider-name"
    stop_processing = true
     keys_values_filter {
      key_value_pairs {
        key   = "manufacturer"
        value = "pontiac"
      }
       key_value_pairs {
        key   = "captureLabel"
        value = "Car"
      }
      event_type = "indykite.audit.capture.upsert.node"
    }
    route_display_name = "Configuration Audit Events"
    route_id           = "config-audit-log"
  }
}

Step 3

Capture the nodes needed for this use case.

POST https://eu.api.indykite.com/capture/v1/nodes/

{
  "nodes": [
    {
      "external_id": "alice",
      "is_identity": true,
      "type": "Person",
      "properties": [
        {
          "type": "email",
          "value": "alice@email.com"
        },
        {
          "type": "given_name",
          "value": "Alice"
        },
        {
          "type": "last_name",
          "value": "Smith"
        }
      ]
    },
    {
      "external_id": "knightrider",
      "type": "Person",
      "is_identity": true,
      "properties": [
        {
          "type": "email",
          "value": "knightrider@demo.com"
        },
        {
          "type": "name",
          "value": "Michael Knight"
        }
      ]
    },
    {
      "external_id": "satchmo",
      "type": "Person",
      "is_identity": true,
      "properties": [
        {
          "type": "email",
          "value": "satchmo@demo.com"
        },
        {
          "type": "name",
          "value": "Louis Armstrong"
        }
      ]
    },
    {
      "external_id": "karel",
      "type": "Person",
      "is_identity": true,
      "properties": [
        {
          "type": "email",
          "value": "karel@demo.com"
        },
        {
          "type": "name",
          "value": "Karel Plihal"
        }
      ]
    },
    {
      "external_id": "kitt",
      "type": "Car",
      "is_identity": false,
      "properties": [
        {
          "type": "manufacturer",
          "value": "pontiac"
        },
        {
          "type": "model",
          "value": "Firebird"
        }
      ]
    },
    {
      "external_id": "cadillacv16",
      "type": "Car",
      "is_identity": false,
      "properties": [
        {
          "type": "manufacturer",
          "value": "Cadillac"
        },
        {
          "type": "model",
          "value": "V-16"
        }
      ]
    },
    {
      "external_id": "harmonika",
      "type": "Bus",
      "is_identity": false,
      "properties": [
        {
          "type": "manufacturer",
          "value": "Ikarus"
        },
        {
          "type": "model",
          "value": "280"
        }
      ]
    },
    {
      "external_id": "listek",
      "type": "Ticket",
      "is_identity": false
    },
    {
      "external_id": "airbook-xyz",
      "type": "Laptop",
      "is_identity": false
    }
  ]
}

Step 4

Capture additional nodes to trigger more events.

POST https://eu.api.indykite.com/capture/v1/nodes/

{
  "nodes": [
    {
      "external_id": "kitten",
      "type": "Car",
      "is_identity": false,
      "properties": [
        {
          "type": "manufacturer",
          "value": "pontiac"
        },
        {
          "type": "model",
          "value": "Bonneville"
        }
      ]
    },
    {
      "external_id": "kitty",
      "type": "Car",
      "is_identity": false,
      "properties": [
        {
          "type": "manufacturer",
          "value": "pontiac"
        },
        {
          "type": "model",
          "value": "Catalina"
        }
      ]
    }
  ]
}