Building Data Products from Well-Defined Event Schemas on AWS

Most teams start their data journey by pointing analytics tools directly at their operational database. It works — until it doesn’t. Queries slow down production. Schema changes break dashboards. And every consumer ends up with its own interpretation of what a “customer” or “transaction” means.

There’s a better approach: anchor your data products in well-defined business events, not in database tables. Let the domain tell the story, and let each consumer shape that story for its own needs.

This article walks through how to implement this on AWS — with EventBridge, S3, and Glue Data Catalog — at a scale that works for small teams shipping real products.

The Problem with Database-Driven Analytics

When your analytics pipeline reads directly from DynamoDB (or any operational store), you get tight coupling:

graph LR
    A[React App] --> B[API Gateway + Lambda]
    B --> C[DynamoDB]
    C --> D[Analytics Query]
    C --> E[Dashboard]
    C --> F[ML Pipeline]
    style C fill:#ef4444,stroke:#991b1b,color:#fff
    style D fill:#fbbf24,stroke:#92400e
    style E fill:#fbbf24,stroke:#92400e
    style F fill:#fbbf24,stroke:#92400e

Every consumer reads the same operational schema. If you rename a column, add a GSI, or change how you store a field — every downstream system breaks. The database becomes a shared contract it was never designed to be.

The Event-Driven Data Product

Instead of coupling consumers to your database, you publish well-defined business events directly from your application logic. These events flow through an event bus into a data lake, where each consumer transforms them into exactly the shape it needs.

graph TD
    subgraph Operational Layer
        A[React App] --> B[API Gateway + Lambda]
        B --> C[DynamoDB]
    end

    subgraph Event Layer
        B -->|"Well-defined<br/>business events"| F[EventBridge]
    end

    subgraph Data Product Layer
        F --> G[Firehose → S3 Data Lake]
        G --> H[Glue Data Catalog]
    end

    subgraph Consumers
        H --> I[Athena - Analytics]
        H --> J[QuickSight - Dashboards]
        H --> K[Lambda - ML Pipeline]
    end

    style F fill:#7c3aed,stroke:#5b21b6,color:#fff
    style H fill:#2563eb,stroke:#1e40af,color:#fff

The key insight: the event schema is the contract, not the database schema. Your business Lambda emits a well-defined event as part of each operation, completely decoupled from how the data is stored.

What Makes an Event Schema “Well-Defined”?

A good event schema is not just a data dump of your database item. It carries business meaning and follows a recognised standard:

{
  "Source": "myapp.orders",
  "DetailType": "order.completed",
  "Detail": {
    "specversion": "1.0",
    "id": "evt-a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "source": "/myapp/orders",
    "type": "com.myapp.orders.order.completed",
    "time": "2026-06-17T08:30:00Z",
    "datacontenttype": "application/json",
    "subject": "order-12345",
    "data": {
      "orderId": "order-12345",
      "customerId": "cust-67890",
      "totalAmount": 149.99,
      "currency": "GBP",
      "itemCount": 3,
      "completedAt": "2026-06-17T08:30:00Z"
    }
  }
}

Notice what this event does not include: DynamoDB keys (PK, SK), internal TTLs, GSI projections, or any storage implementation detail. It speaks the language of the business: an order was completed, by this customer, for this amount, at this time. The outer envelope (Source, DetailType) gives EventBridge what it needs for routing; the inner payload follows the CloudEvents specification for portability.

The Event Catalogue

For each event type, you need to document:

Field	Purpose
Name	`order.completed`, `order.shipped`, `user.plan-changed`
Description	What happened in business terms
Schema	Exact JSON shape with types and constraints
Owner	Which team/service owns this event
Version	Schema version for evolution tracking
Consumers	Who subscribes and why

This catalogue is your data contract. It’s what makes the event a product, not just a side-effect.

Aligning with CloudEvents

CloudEvents is a CNCF specification (graduated January 2024) for describing events in a common way. It has been adopted by AWS, Azure, Google Cloud, and over 40 platforms and frameworks. Using CloudEvents gives your events a portable, well-understood structure that any team or system can consume without custom parsing logic.

Required and Recommended Attributes

The specification defines a small set of required context attributes, plus several recommended ones:

Attribute	Required	Purpose
`specversion`	Yes	CloudEvents spec version (always `"1.0"`)
`id`	Yes	Unique identifier for this event
`source`	Yes	Identifies the context in which the event happened
`type`	Yes	Type of event (reverse-DNS convention)
`time`	Recommended	Timestamp of when the event occurred
`subject`	Recommended	Subject of the event in context of the source
`datacontenttype`	Recommended	Content type of `data` (e.g. `application/json`)
`data`	Recommended	The event payload

CloudEvents on EventBridge

On AWS EventBridge, the best practice is to use the EventBridge envelope for routing and embed a CloudEvents-compliant payload inside the Detail field. This gives you the best of both worlds: EventBridge’s native filtering and routing via Source and DetailType, plus full CloudEvents portability inside Detail.

{
  "Source": "myapp.orders",
  "DetailType": "order.completed",
  "EventBusName": "my-event-bus",
  "Detail": {
    "specversion": "1.0",
    "id": "evt-a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "source": "/myapp/orders",
    "type": "com.myapp.orders.order.completed",
    "time": "2026-06-17T08:30:00Z",
    "datacontenttype": "application/json",
    "subject": "order-12345",
    "data": {
      "orderId": "order-12345",
      "customerId": "cust-67890",
      "totalAmount": 149.99,
      "currency": "GBP",
      "itemCount": 3,
      "completedAt": "2026-06-17T08:30:00Z"
    }
  }
}

When to Use Which Format

Use Case	Recommended Format	Why
Publishing to EventBridge	EventBridge envelope + CloudEvents in Detail	Best routing + filtering
Publishing to Kafka	Pure top-level CloudEvents	Native support
Publishing via HTTP	Pure top-level CloudEvents	Standard wire format
Sending to external/third-party	Pure CloudEvents (via API Destinations)	Interoperability
Maximum portability	Pure CloudEvents	Vendor-neutral

For further reading, see the CloudEvents specification and the CloudEvents visual guide.

Implementation on AWS

Here’s how each piece fits together, using services that work well at small scale and grow with you.

Step 1: Emit Business Events from Your Lambda

Rather than capturing database changes after the fact, emit well-defined events directly from your business logic. This gives you full control over what your events contain and when they are published.

# Inside your business Lambda — emit a well-defined event
import json
import os
import uuid
from datetime import datetime, timezone
import boto3

eventbridge = boto3.client('events')

def publish_event(event_type, domain, subject, data):
    """Publish a CloudEvents-compliant event to EventBridge."""
    eventbridge.put_events(Entries=[{
        'Source': f'myapp.{domain}',
        'DetailType': event_type,
        'EventBusName': os.environ['EVENT_BUS_NAME'],
        'Detail': json.dumps({
            'specversion': '1.0',
            'id': str(uuid.uuid4()),
            'source': f'/myapp/{domain}',
            'type': f'com.myapp.{domain}.{event_type}',
            'time': datetime.now(timezone.utc).isoformat(),
            'datacontenttype': 'application/json',
            'subject': subject,
            'data': data,
        })
    }])

# Usage in your handler
def create_order(event, context):
    order = save_order(event)  # your business logic

    publish_event(
        event_type='order.created',
        domain='orders',
        subject=f'order-{order["id"]}',
        data={
            'orderId': order['id'],
            'customerId': order['customerId'],
            'totalAmount': order['total'],
            'itemCount': len(order['items']),
            'createdAt': order['createdAt'],
        }
    )
    return {'statusCode': 201, 'body': json.dumps(order)}

This approach has a clear advantage: the event is part of your business logic, not a side-effect of a database write. You choose exactly which fields to include, you control the naming, and you can emit events for operations that don’t even touch a database (e.g. sending a notification, calling a third-party API).

Step 2: Route Events to the Data Lake

EventBridge rules route business events to Amazon Data Firehose, which batches and delivers them to S3 in Parquet format:

# SAM template excerpt
EventRule:
  Type: AWS::Events::Rule
  Properties:
    EventBusName: my-event-bus
    EventPattern:
      source:
        - myapp.orders
        - myapp.users
        - myapp.payments
    Targets:
      - Arn: !GetAtt DeliveryStream.Arn
        Id: DataLakeDelivery
        RoleArn: !GetAtt EventBridgeDeliveryRole.Arn

DeliveryStream:
  Type: AWS::KinesisFirehose::DeliveryStream
  Properties:
    DeliveryStreamType: DirectPut
    ExtendedS3DestinationConfiguration:
      BucketARN: !GetAtt DataLakeBucket.Arn
      Prefix: "events/source=!{partitionKeyFromQuery:source}/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/"
      ErrorOutputPrefix: "errors/"
      BufferingHints:
        IntervalInSeconds: 300
        SizeInMBs: 5
      DataFormatConversionConfiguration:
        Enabled: true
        InputFormatConfiguration:
          Deserializer:
            OpenXJsonSerDe: {}
        OutputFormatConfiguration:
          Serializer:
            ParquetSerDe: {}
        SchemaConfiguration:
          DatabaseName: !Ref GlueDatabase
          TableName: !Ref GlueEventsTable
          RoleARN: !GetAtt FirehoseGlueRole.Arn

Step 3: Register Schemas in Glue Data Catalog

The AWS Glue Data Catalog serves as your living registry of event metadata. Every event schema is a table definition that any consumer can discover and query.

GlueDatabase:
  Type: AWS::Glue::Database
  Properties:
    CatalogId: !Ref AWS::AccountId
    DatabaseInput:
      Name: my_events
      Description: "Business events — the single source of truth for analytics."

GlueEventsTable:
  Type: AWS::Glue::Table
  Properties:
    CatalogId: !Ref AWS::AccountId
    DatabaseName: !Ref GlueDatabase
    TableInput:
      Name: order_events
      Description: "Order lifecycle events: created, completed, shipped, cancelled."
      TableType: EXTERNAL_TABLE
      Parameters:
        classification: parquet
      StorageDescriptor:
        Location: !Sub "s3://${DataLakeBucket}/events/source=myapp.orders/"
        InputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetInputFormat
        OutputFormat: org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat
        SerdeInfo:
          SerializationLibrary: org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe
        Columns:
          - Name: orderId
            Type: string
            Comment: "Unique order identifier"
          - Name: customerId
            Type: string
            Comment: "Customer who placed the order"
          - Name: totalAmount
            Type: double
            Comment: "Order total in the specified currency"
          - Name: currency
            Type: string
            Comment: "ISO 4217 currency code"
          - Name: itemCount
            Type: int
            Comment: "Number of items in the order"
          - Name: completedAt
            Type: timestamp
            Comment: "When the order was completed"
          - Name: schemaVersion
            Type: string
            Comment: "Event schema version for evolution tracking"
      PartitionKeys:
        - Name: year
          Type: string
        - Name: month
          Type: string
        - Name: day
          Type: string

Now anyone on the team can open the Glue Data Catalog and discover exactly what events exist, what fields they carry, and what each field means. For richer governance, layer AWS DataZone on top (more on that below).

Step 4: Query with Athena

With events in S3 and schemas in Glue, you can query your data product instantly with Athena — no infrastructure to manage:

-- Orders completed per customer this month
SELECT customerId,
       COUNT(*) as orders_completed,
       SUM(totalAmount) as total_spent
FROM my_events.order_events
WHERE year = '2026' AND month = '06'
GROUP BY customerId
ORDER BY total_spent DESC;

The Full Architecture

Here’s the complete flow from user action to queryable data product:

graph TD
    subgraph "1 — Operational System"
        U[User Action] --> API[API Gateway]
        API --> L1[Business Lambda]
        L1 --> DDB[DynamoDB]
    end

    subgraph "2 — Event Production"
        L1 -->|"Well-defined<br/>business events"| EB[EventBridge]
    end

    subgraph "3 — Event Delivery"
        EB --> FH[Data Firehose]
        FH -->|Parquet| S3[S3 Data Lake]
    end

    subgraph "4 — Data Catalogue & Governance"
        S3 --> GC[Glue Data Catalog]
        GC --> DZ[DataZone Portal]
    end

    subgraph "5 — Data Consumers"
        GC --> ATH[Athena Queries]
        GC --> QS[QuickSight Dashboards]
        GC --> ML[ML Pipelines]
        GC --> EXP[Data Export / API]
    end

    style EB fill:#7c3aed,stroke:#5b21b6,color:#fff
    style GC fill:#2563eb,stroke:#1e40af,color:#fff
    style S3 fill:#16a34a,stroke:#166534,color:#fff

Glue Data Catalog, DataZone, and DataHub

When building a data catalogue, you have three main options on AWS. The good news is that two of them are free and complement each other.

Glue Data Catalog — The Foundation

Glue Data Catalog is the schema registry and metastore that Athena, Firehose, and every other AWS analytics service reads from. It is always part of the architecture.

AWS DataZone — Governance from Day One

DataZone adds a governance and discovery layer on top of Glue Data Catalog. It is worth enabling from the start because it has no base cost — you only pay for the Glue and Athena queries you would be running anyway (DataZone charges per metadata API call, which is negligible).

What DataZone provides:

Searchable data portal — business users find datasets without knowing S3 paths or Glue database names
Business glossary — define terms like “active customer” or “completed order” once, and link them to the actual columns in your tables
Subscription-based access — data producers publish assets; consumers request access through a workflow, creating an audit trail
In-portal SQL — DataZone includes an Athena query environment, so analysts can discover and query data in a single interface
Cross-account sharing — when you grow to multiple AWS accounts, DataZone manages access across them

For a small team, DataZone gives you a professional data portal with governed access at zero marginal cost. As you grow, the same portal scales to multi-team and cross-account data mesh without re-architecture.

DataHub (Open Source)

LinkedIn’s DataHub is a powerful open-source metadata platform, but it requires self-hosting — Kubernetes, Kafka, Elasticsearch. That operational overhead is hard to justify when DataZone provides comparable discovery and governance as a managed service with no base cost.

Feature	Glue Data Catalog	AWS DataZone	DataHub (OSS)
What it is	Schema registry + metastore	Data governance portal	Self-hosted metadata platform
Cost	Free (included with Glue)	Free (per metadata API call, negligible)	Self-hosted (K8s, Kafka, ES)
Discovery	API + console	Search portal with business glossary	Search portal with lineage
Access control	IAM + Lake Formation	Subscription model with audit trail	Policy-based
Best for	Always required (foundation)	Any team wanting governed discovery	Teams already running Kubernetes

Schema Evolution: The Hard Part

Events will change over time. New fields get added, old fields get deprecated. The key rule: never break existing consumers.

graph LR
    subgraph "v1.0 — Launch"
        A1["order.completed<br/>orderId, customerId, totalAmount"]
    end

    subgraph "v1.1 — Additive"
        A2["order.completed<br/>orderId, customerId, totalAmount,<br/><b>+ currency, itemCount</b>"]
    end

    subgraph "v2.0 — Breaking"
        A3["order.completed<br/>orderId, <b>accountId</b>, totalAmount,<br/>currency, itemCount"]
    end

    A1 -->|"Safe: new optional fields"| A2
    A2 -->|"Requires migration"| A3

    style A1 fill:#d1fae5,stroke:#065f46
    style A2 fill:#d1fae5,stroke:#065f46
    style A3 fill:#fecaca,stroke:#991b1b

Safe changes (backward-compatible):

Adding new optional fields
Adding new event types
Widening a type (int → long)

Breaking changes (require coordination):

Removing or renaming fields
Changing field types
Changing the event structure

For breaking changes, version the event (schemaVersion: "2.0"), publish both versions during a transition period, and give consumers time to migrate.

What This Costs at Small Scale

This architecture is cheap to run when you’re starting out:

Service	Monthly cost (low volume)
EventBridge	< £1 (£1 per million events)
Data Firehose	~£5-10 (per GB ingested)
S3	< £1 (pennies per GB stored)
Glue Data Catalog	Free (first million objects)
DataZone	Free (per metadata API call, negligible)
Athena	£4 per TB scanned

For a small product with thousands of events per day, you’re looking at under £20/month for a fully governed data product pipeline with a professional discovery portal.

When Not to Use This Pattern

This is not the right approach when:

You just need a dashboard. If one QuickSight dashboard on one DynamoDB table solves the problem, do that. Don’t build a data pipeline for a single query.
Your events have no business meaning. If you’re just dumping raw database changes to S3, you haven’t built a data product — you’ve built a backup. The value comes from the semantic transformation.
You have one consumer. The event contract pattern pays off when multiple teams or systems consume the same events. With one consumer, a direct integration is simpler.

Beyond Raw Events: The Enrichment Layer

In practice, raw events are useful for auditing but painful for analytics. Every query needs to join user context (email, name, plan tier) from DynamoDB — expensive and slow at scale. The solution is a two-stream architecture:

graph TD
    subgraph "Event Sources"
        L1[Business Lambdas] -->|"emit_event()"| EB[EventBridge]
    end

    subgraph "Raw Pipeline"
        EB -->|"EventCaptureRule"| FH1[Firehose]
        FH1 --> S3R["S3 raw/ (Parquet)"]
    end

    subgraph "Enrichment Pipeline"
        EB -->|"EnrichmentRule"| EL[Enrichment Lambda]
        EL -->|"joins user profile<br/>+ entitlements"| EB2[EventBridge .enriched]
        EB2 --> FH2[Firehose]
        FH2 --> S3E["S3 enriched/ (Parquet)"]
    end

    subgraph "Query Layer"
        S3R --> GC1[Glue Catalog]
        S3E --> GC1
        GC1 --> ATH[Athena Views]
    end

    style EL fill:#7c3aed,stroke:#5b21b6,color:#fff
    style GC1 fill:#2563eb,stroke:#1e40af,color:#fff

The enrichment Lambda fires on every pweb.* event, looks up the user’s profile and plan from DynamoDB, and re-emits the event with an .enriched suffix. A second Firehose stream captures these enriched events into a separate S3 prefix. Glue crawlers catalog both — giving you two databases: raw (audit trail) and enriched (analytics-ready).

Now your Athena views can reference email, userName, and userPlan directly without joining operational tables:

SELECT
    json_extract_scalar(detail, '$.data.enrichment.userEmail') AS email,
    json_extract_scalar(detail, '$.data.enrichment.userPlan') AS plan,
    COUNT(*) AS events
FROM pweb_enriched.all_events
WHERE year = '2026' AND month = '06'
GROUP BY 1, 2;

Business Flow Views: From Counters to Funnels

Daily counters (events per day, active users per day) are a good start, but they don’t answer the questions that actually drive product decisions:

What percentage of signups generate their first document within 7 days?
Which users are going silent after being active?
How long does it take from signup to paid conversion?

These require cross-time views that track individual users across their lifecycle. Here are the patterns we found most useful:

User Journey Funnel

Track per-user milestones and classify lifecycle stage:

SELECT
    user_id,
    MIN(CASE WHEN event_type = 'user.signup' THEN time END) AS signed_up_at,
    MIN(CASE WHEN event_type = 'streaming.session.started' THEN time END) AS first_session_at,
    MIN(CASE WHEN event_type = 'document.generated' THEN time END) AS first_doc_at,
    MIN(CASE WHEN event_type = 'billing.checkout.completed' THEN time END) AS first_paid_at,
    CASE
        WHEN first_paid_at IS NOT NULL THEN 'converted'
        WHEN first_doc_at IS NOT NULL THEN 'activated'
        WHEN first_session_at IS NOT NULL THEN 'tried'
        ELSE 'signed_up_only'
    END AS lifecycle_stage
FROM enriched_events
GROUP BY user_id;

Activation Cohorts

Weekly signup cohorts with conversion rates at 1, 7, 14, and 30 days — the standard SaaS growth metric:

SELECT
    DATE_TRUNC('week', signup_ts) AS cohort_week,
    COUNT(DISTINCT user_id) AS cohort_size,
    ROUND(100.0 * COUNT(DISTINCT CASE
        WHEN first_session_ts <= signup_ts + INTERVAL '7' DAY
        THEN user_id END) / COUNT(DISTINCT user_id), 1) AS pct_session_7d,
    ROUND(100.0 * COUNT(DISTINCT CASE
        WHEN first_doc_ts <= signup_ts + INTERVAL '7' DAY
        THEN user_id END) / COUNT(DISTINCT user_id), 1) AS pct_doc_7d
FROM cohort_milestones
GROUP BY cohort_week;

Churn Signals

Flag users who were active but have gone silent — ranked by risk:

SELECT user_id, email, plan,
    DATE_DIFF('day', last_seen_at, NOW()) AS days_since_last_activity,
    CASE
        WHEN days_inactive >= 30 THEN 'dormant'
        WHEN days_inactive >= 14 THEN 'at_risk'
        WHEN days_inactive >= 7 THEN 'cooling'
        ELSE 'active'
    END AS risk_level
FROM user_activity
WHERE total_events >= 3
ORDER BY days_since_last_activity DESC;

These views use the enriched database — user context is already joined at write time, so the queries are fast and cheap.

Data Lineage with OpenLineage and DataZone

One question that gets harder as your event pipeline grows: where does this data come from, what transforms it, and who consumes it?

AWS DataZone supports OpenLineage — a CNCF standard for data lineage metadata. You can post lineage events programmatically, and DataZone renders them as a visual graph in its portal.

Runtime Lineage from Your Enrichment Lambda

Add lineage emission to any Lambda that transforms data:

import json, uuid, boto3
from datetime import datetime, timezone

dz = boto3.client('datazone')

def post_lineage(domain_id, job_name, inputs, outputs):
    run_event = {
        "eventType": "COMPLETE",
        "eventTime": datetime.now(timezone.utc).isoformat(),
        "producer": "https://github.com/your-org/your-repo",
        "schemaURL": "https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent",
        "run": {"runId": str(uuid.uuid4())},
        "job": {"namespace": "my-app-prod", "name": job_name},
        "inputs": inputs,
        "outputs": outputs,
    }
    dz.post_lineage_event(
        domainIdentifier=domain_id,
        event=json.dumps(run_event).encode('utf-8'),
    )

Static Lineage for Infrastructure Pipelines

For infrastructure components (Firehose, Glue crawlers) that don’t run your code, seed lineage events once at deploy time. Write the OpenLineage JSON to a temp file and use the AWS CLI:

# Write event to temp file — CLI expects a blob, not a JSON string
tmpfile=$(mktemp)
cat > "$tmpfile" <<EOF
{"eventType":"COMPLETE","eventTime":"$(date -u +%Y-%m-%dT%H:%M:%S.000000Z)",
 "producer":"https://github.com/your-org/your-repo",
 "schemaURL":"https://openlineage.io/spec/2-0-2/OpenLineage.json#/\$defs/RunEvent",
 "run":{"runId":"$(uuidgen)"},
 "job":{"namespace":"my-app-prod","name":"RawFirehoseDelivery"},
 "inputs":[{"namespace":"aws:eventbridge","name":"my-event-bus"}],
 "outputs":[{"namespace":"arn:aws:glue:eu-west-2:123456789:","name":"table/my_db/events"}]}
EOF

aws datazone post-lineage-event \
  --domain-identifier $DOMAIN_ID \
  --event "fileb://$tmpfile"
rm -f "$tmpfile"

Important: Use fileb:// (binary file) — not a raw JSON string. The CLI --event parameter expects a blob, and passing raw JSON produces Invalid base64 errors.

Viewing Lineage in DataZone

Once lineage events are posted, the visual graph appears in the DataZone portal:

Navigate to the Catalog view (not the Data browser)
Search for your asset (e.g. all_events)
Click the Lineage tab on the asset details page

The graph shows upstream sources, transformation jobs, and downstream consumers — all connected automatically from the OpenLineage events you posted.

Key requirement: The lineage graph only renders when at least one node matches an existing DataZone asset. The OpenLineage dataset namespace and name must match the asset’s sourceIdentifier. For Glue tables, use the ARN format: arn:aws:glue:<region>:<account>:table/<database>/<table>.

Business Glossary in DataZone

Define your business terms in a YAML file and sync them to DataZone at deploy time:

glossary:
  name: My Platform
  description: Business terms for the data portal

terms:
  customer:
    description: A person with an account. Identified by Cognito sub.
    source_table: users-table
    related: [subscription, order]

  subscription:
    description: A Stripe subscription linking a customer to a plan.
    source: Stripe
    values: [free, pro, enterprise]
    related: [customer, plan]

Use the DataZone API to create the glossary and terms programmatically. This makes your data portal self-documenting — analysts can look up what “customer” or “subscription” means without asking the engineering team.

Note: Creating glossaries requires a CREATE_GLOSSARY policy grant on the DataZone domain unit. This is DataZone’s internal authorization — separate from IAM. Grant it via add-policy-grant with the project principal type.

Declarative Data Lineage Manifest

For documentation and code review, maintain a YAML manifest alongside your infrastructure that maps every data flow:

sources:
  cognito:
    type: identity-provider
    produces:
      - event: user.signup
        domain: auth

  stripe:
    type: payment-provider
    produces:
      - event: billing.checkout.completed
        domain: billing

pipelines:
  event-to-analytics:
    steps:
      - source: EventBridge
        target: Firehose → S3 raw/
      - source: EventBridge
        transform: EnrichmentFunction
        target: Firehose → S3 enriched/
      - source: S3
        transform: Glue Crawler
        target: Glue Catalog → Athena views

This costs nothing at runtime — it’s version-controlled metadata that keeps your team aligned on where data comes from and where it goes.

Lessons Learned in Production

After implementing this pattern across multiple services, here are the practical lessons:

Set EVENT_BUS_NAME globally. We had five Lambda functions calling emit_event() that silently failed because the environment variable wasn’t set. Configure it in the SAM Globals section, not per-function.
Enrichment must be idempotent. The enrichment Lambda re-emits to the same EventBridge bus with a .enriched suffix. Without a filter rule excluding .enriched events, you get an infinite loop. Use anything-but: {suffix: ".enriched"} in the EventBridge pattern.
DataZone has dual authorization. IAM permissions alone aren’t enough — DataZone has its own internal authorization layer. Project membership, policy grants, and blueprint configuration are all separate from IAM. Budget time for this.
The Lineage tab requires asset linkage. Posted OpenLineage events create lineage nodes, but the visual graph only appears on an asset’s Lineage tab when the node’s sourceIdentifier matches the asset’s sourceIdentifier. For Glue tables, use the full ARN format.
Use fileb:// for the CLI. The aws datazone post-lineage-event --event parameter expects a binary blob. Passing raw JSON produces cryptic Invalid base64 errors. Write to a temp file and use fileb://.
Business flow views belong on the enriched database. Daily counters can use the raw database, but funnel analysis, cohort tracking, and churn detection need the enriched stream — otherwise every query re-joins user context from DynamoDB.

Key Takeaways

Anchor in events, not tables. Your data product’s contract is the event schema, not the database schema. This decouples analytics from operations.
Adopt CloudEvents early. The CNCF CloudEvents standard gives your events portability across AWS, Kafka, HTTP, and third-party systems. On EventBridge, use the native envelope for routing and embed CloudEvents in the Detail field for interoperability.
Build a two-stream architecture. Raw events for audit, enriched events for analytics. The enrichment Lambda joins user context at write time so every downstream query is fast and cheap.
Go beyond daily counters. User journey funnels, activation cohorts, and churn signals are the views that actually drive product decisions. They require cross-time analysis on enriched data.
Wire up lineage from day one. OpenLineage events are cheap to post and make the DataZone portal genuinely useful. Seed static lineage for infrastructure, post runtime lineage from transforms.
Use Glue Data Catalog as your foundation and DataZone for governance. Document every event type, field, and version in Glue. Layer DataZone on top for a searchable portal, business glossary, lineage graphs, and subscription-based access — it costs nothing to add.
Start small. Direct event emission from your Lambdas, plus EventBridge, Firehose, S3, Glue, and DataZone costs under £20/month at low volume and gives you a fully governed, discoverable data product from day one.

The goal is a data product that tells your business’s story — not a mirror of your database, but a curated, well-documented stream of events that every consumer can trust.

Building event-driven data products on AWS? See our Architecture & Cloud Engineering services or get in touch for a free architecture review.